Skip to contentSkip to conversation

utf-8 issues on b4 master

PATCH In review Michael S. Tsirkin 3 participants 8 messages Jul 17, 2021 – Aug 3, 2021

lore
Share & tools
lore
#0 lore raw

Passing message id [email protected] to b4 gives this backtrace:

Traceback (most recent call last):
File "/scm/b4/b4/command.py", line 263, in <module>

    cmd()

File "/scm/b4/b4/command.py", line 246, in cmd
cmdargs.func(cmdargs)
File "/scm/b4/b4/command.py", line 41, in cmd_mbox
b4.mbox.main(cmdargs)
File "/scm/b4/b4/mbox.py", line 581, in main
msgid, msgs = get_msgs(cmdargs)
File "/scm/b4/b4/mbox.py", line 523, in get_msgs
msgid = b4.get_msgid(cmdargs)
File "/scm/b4/b4/__init__.py", line 2080, in get_msgid
msgid = get_msgid_from_stdin()
File "/scm/b4/b4/__init__.py", line 2072, in get_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 5886: invalid continuation byte

mutt does not seem to have trouble decoding this ... weird.

Signature · 1 line
MST
#1 lore raw
Quoted 23 lines · Michael S. Tsirkin

Passing message id [email protected] to b4 gives this backtrace:

Traceback (most recent call last):
File "/scm/b4/b4/command.py", line 263, in <module>

    cmd()

File "/scm/b4/b4/command.py", line 246, in cmd
cmdargs.func(cmdargs)
File "/scm/b4/b4/command.py", line 41, in cmd_mbox
b4.mbox.main(cmdargs)
File "/scm/b4/b4/mbox.py", line 581, in main
msgid, msgs = get_msgs(cmdargs)
File "/scm/b4/b4/mbox.py", line 523, in get_msgs
msgid = b4.get_msgid(cmdargs)
File "/scm/b4/b4/__init__.py", line 2080, in get_msgid
msgid = get_msgid_from_stdin()
File "/scm/b4/b4/__init__.py", line 2072, in get_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 5886: invalid continuation byte

Quoted 1 line

mutt does not seem to have trouble decoding this ... weird.

I'm confused by that backtrace. I think get_msgid_from_stdin() should be called only when a message is fed on stdin. You say you're passing a message ID. That's as a positional argument, right?

Fwiw I wasn't able to trigger the issue on my end.

  $ b4 am [email protected]
  Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com
  Analyzing 5 messages in the thread
  ---
    [PATCH] virtio-balloon: Use virtio_find_vqs() helper
      + Reviewed-by: David Hildenbrand <[email protected]>
  ---
  Total patches: 1
  ---
   Link: https://lore.kernel.org/r/[email protected]
   Base: not specified
         git am ./20210713_xianting_tian_virtio_balloon_use_virtio_find_vqs_helper.mbx
  $ b4 mbox [email protected]
  Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com
  5 messages in the thread
  Saved ./[email protected]

That's with

Python 3.7.3
b4 v0.7.0-32-g45ef591
patatt v0.4.6

#2 lore raw
Quoted 25 lines · Kyle Meyer
Quoted 23 lines · Michael S. Tsirkin

Passing message id [email protected] to b4 gives this backtrace:

Traceback (most recent call last):
File "/scm/b4/b4/command.py", line 263, in <module>

    cmd()

File "/scm/b4/b4/command.py", line 246, in cmd
cmdargs.func(cmdargs)
File "/scm/b4/b4/command.py", line 41, in cmd_mbox
b4.mbox.main(cmdargs)
File "/scm/b4/b4/mbox.py", line 581, in main
msgid, msgs = get_msgs(cmdargs)
File "/scm/b4/b4/mbox.py", line 523, in get_msgs
msgid = b4.get_msgid(cmdargs)
File "/scm/b4/b4/__init__.py", line 2080, in get_msgid
msgid = get_msgid_from_stdin()
File "/scm/b4/b4/__init__.py", line 2072, in get_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 5886: invalid continuation byte

Quoted 5 lines
Quoted 1 line

mutt does not seem to have trouble decoding this ... weird.

I'm confused by that backtrace. I think get_msgid_from_stdin() should be called only when a message is fed on stdin. You say you're passing a message ID. That's as a positional argument, right?

Sorry. I passed the message on the stdin. I supplied the message ID so you can get the original from the list archives.

To reproduce:

wget -O - https://lore.kernel.org/lkml/[email protected]/raw | b4 mbox

Quoted 20 lines

Fwiw I wasn't able to trigger the issue on my end.

  $ b4 am [email protected]
  Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com
  Analyzing 5 messages in the thread
  ---
    [PATCH] virtio-balloon: Use virtio_find_vqs() helper
      + Reviewed-by: David Hildenbrand <[email protected]>
  ---
  Total patches: 1
  ---
   Link: https://lore.kernel.org/r/[email protected]
   Base: not specified
         git am ./20210713_xianting_tian_virtio_balloon_use_virtio_find_vqs_helper.mbx
  $ b4 mbox [email protected]
  Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com
  5 messages in the thread
  Saved ./[email protected]
Quoted 5 lines

That's with

Python 3.7.3
b4 v0.7.0-32-g45ef591
patatt v0.4.6

b4 v0.7.0-32-g45ef591

python3-3.9.5-2.fc33.x86_64

I don't know about patatt.

Signature · 1 line
MST
#3 lore raw

cover Avoid decoding errors when extracting message ID from stdin

Quoted 21 lines · Michael S. Tsirkin
Quoted 14 lines · Kyle Meyer
Quoted 12 lines · Michael S. Tsirkin

Passing message id [email protected] to b4 gives this backtrace:

Traceback (most recent call last):
[....]
File "/scm/b4/b4/__init__.py", line 2072, in get_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 5886: invalid continuation byte

Quoted 5 lines
Quoted 1 line

mutt does not seem to have trouble decoding this ... weird.

I'm confused by that backtrace. I think get_msgid_from_stdin() should be called only when a message is fed on stdin. You say you're passing a message ID. That's as a positional argument, right?

Quoted 6 lines

Sorry. I passed the message on the stdin. I supplied the message ID so you can get the original from the list archives.

To reproduce:

wget -O - https://lore.kernel.org/lkml/[email protected]/raw | b4 mbox

Thanks. I can trigger that on my end too. Here's a possible fix. The first patch is the actual fix. The second patch makes this code path do a little less work but isn't necessary.

[1/2] Avoid decoding errors when extracting message ID from stdin [2/2] Parse just headers when extracting message ID from stdin mbox

 b4/__init__.py | 4 +++-
 b4/pr.py       | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

base-commit: 06cc7c8820aea85d1329911b785d7bf4ecaacb1f

Signature · 1 line
2.32.0
#5 lore raw

PATCH 2/2 Parse just headers when extracting message ID from stdin mbox

When the pr, mbox, and am subcommands grab a message ID from the mbox on stdin, they call message_from_bytes(), which in turn calls BytesParser().parsebytes(s).

parsebytes() has a headersonly parameter that can be used to tell it to stop parsing after reading the headers. The headers are all that's needed here, so use BytesParser directly and set headersonly.

---

Note: I've tested only b4 am/mbox' with the reproducer message mentioned in upthread; I haven't tested b4 pr'.

 b4/__init__.py | 4 +++-
 b4/pr.py       | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)
b4/__init__.py +3 -1
index 5b32fb4..4722826 100644
@@ -1948,7 +1948,9 @@ def get_requests_session():
 
 def get_msgid_from_stdin():
     if not sys.stdin.isatty():
-        message = email.message_from_bytes(sys.stdin.buffer.read())
+        from email.parser import BytesParser
+        message = BytesParser().parsebytes(
+            sys.stdin.buffer.read(), headersonly=True)
         return message.get('Message-ID', None)
     return None
 
b4/pr.py +4 -1
index fbb2a71..e52c2ab 100644
@@ -433,7 +433,10 @@ def main(cmdargs):
 
     if not sys.stdin.isatty():
         logger.debug('Getting PR message from stdin')
-        msg = email.message_from_bytes(sys.stdin.buffer.read())
+        from email.parser import BytesHeaderParser
+        from email.parser import BytesHeaderParser
+        msg = BytesParser().parsebytes(
+            sys.stdin.buffer.read(), headersonly=True)
         msgid = b4.LoreMessage.get_clean_msgid(msg)
         lmsg = parse_pr_data(msg)
     else:
Signature · 1 line
2.32.0
#4 lore raw

PATCH 1/2 Avoid decoding errors when extracting message ID from stdin

In reply to Kyle Meyer

The mbox, am, and pr subcommands accept an mbox on stdin and extract the message ID. When stdin.read() is called, Python assumes the encoding is locale.getpreferredencoding(False). This may not match the content encoding, leading to a decoding error.

Instead feed the stdin bytes to message_from_bytes(), which leads to a decode('ASCII', errors='surrogateescape') underneath. That's sufficient to get the message ID from the ASCII headers.

---

Note: I've tested only b4 am/mbox' with the reproducer message mentioned in upthread; I haven't tested b4 pr'.

 b4/__init__.py | 2 +-
 b4/pr.py       | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
b4/__init__.py +1 -1
index 0e007be..5b32fb4 100644
@@ -1948,7 +1948,7 @@ def get_requests_session():
 
 def get_msgid_from_stdin():
     if not sys.stdin.isatty():
-        message = email.message_from_string(sys.stdin.read())
+        message = email.message_from_bytes(sys.stdin.buffer.read())
         return message.get('Message-ID', None)
     return None
 
b4/pr.py +1 -1
index d8ff7f4..fbb2a71 100644
@@ -433,7 +433,7 @@ def main(cmdargs):
 
     if not sys.stdin.isatty():
         logger.debug('Getting PR message from stdin')
-        msg = email.message_from_string(sys.stdin.read())
+        msg = email.message_from_bytes(sys.stdin.buffer.read())
         msgid = b4.LoreMessage.get_clean_msgid(msg)
         lmsg = parse_pr_data(msg)
     else:
Signature · 1 line
2.32.0
#6 lore raw

Re: [PATCH b4 2/2] Parse just headers when extracting message ID from stdin mbox

In reply to Kyle Meyer

b4/pr.py def main(cmdargs): · line 437

Quoted 11 lines · Kyle Meyer
b4/pr.py +2 -1
index fbb2a71..e52c2ab 100644
@@ -433,7 +433,10 @@ def main(cmdargs):
 
     if not sys.stdin.isatty():
         logger.debug('Getting PR message from stdin')
-        msg = email.message_from_bytes(sys.stdin.buffer.read())
+        from email.parser import BytesHeaderParser
+        from email.parser import BytesHeaderParser

Doh, there's a repeated import here :/

#7 lore raw

Re: [PATCH b4 0/2] Avoid decoding errors when extracting message ID from stdin

In reply to Kyle Meyer
Quoted 6 lines · Kyle Meyer

Thanks. I can trigger that on my end too. Here's a possible fix. The first patch is the actual fix. The second patch makes this code path do a little less work but isn't necessary.

[1/2] Avoid decoding errors when extracting message ID from stdin [2/2] Parse just headers when extracting message ID from stdin mbox

I've applied 1/2 and partial 2/2. In the case of b4/pr.py, we actually want to parse the entire message, not just headers.

Thanks!

-K