devpyshell fixes
cover devpyshell fixes
Hi,
This patchset fixes several devpyshell bugs caused by moving to Python 3.
Please, note that it contains fix for double unbuffering, which was sent in a separate patchset targeting jethro, krogoth and master. It's included here to keep it together with the rest of devpyshell fixes for master.
The following changes since commit 5c11e365e19357f721c49d076971567e7b64b61b:
lib/oeqa: add Galculator to SDK and runtime tests (2016-07-01 16:22:48 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/oe-core/devpyshell-python3
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/devpyshell-python3
Ed Bartosh (4):
devshell.bbclass: fix double unbuffering
bitbake-worker: don't reassign sys.stdout
devpyshell: python3: flush stdout explicitly
oepydevshell-internal: python3: encode/decode pty content
bitbake/bin/bitbake-worker | 3 +-- meta/classes/devshell.bbclass | 5 ++--- scripts/oepydevshell-internal.py | 12 +++++------- 3 files changed, 8 insertions(+), 12 deletions(-)
--
Regards,
Ed
PATCH 1/4 devshell.bbclass: fix double unbuffering
stdout is already unbuffered in bitbake code. Attempt to do it again in devshell.bbclass causes this crash when running devpyshell:
File "scripts/oepydevshell-internal.py", line 29, in <module>
pty = open(sys.argv[1], "w+b", 0)
IOError: [Errno 13] Permission denied: '/dev/pts/6'
--- meta/classes/devshell.bbclass | 1 - 1 file changed, 1 deletion(-)
meta/classes/devshell.bbclass +0 -1
@@ -65,7 +65,6 @@ def devpyshell(d): os.dup2(m, sys.stdout.fileno()) os.dup2(m, sys.stderr.fileno()) - sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0) bb.utils.nonblockingfd(sys.stdout)
Signature · 1 line
2.1.4
PATCH 2/4 bitbake-worker: don't reassign sys.stdout
In reply to Ed BartoshWorker needs input stream in binary mode as it reads binary content from it. Current code does it by detaching a buffer from sys.stdin and assigning it back to sys.stdin.
Detached buffer is io.BufferedReader in binary mode. This operation is implicit as its purpose is not easily understandable from the code. Replacing it with fdopen(sys.stdin.fileno(), 'rb') should make the code more understandable.
Assigning the buffer to sys.stdin is not needed as worker doesn't use sys.stdin. Moreover, it leads to difficult to debug issues down the stack. For example, devpyshell doesn't work without reopening sys.stdin in text mode. This is not needed anymore after this fix as sys.stdin is not changed in worker code and remains in text mode.
--- bitbake/bin/bitbake-worker | 3 +-- meta/classes/devshell.bbclass | 2 -- 2 files changed, 1 insertion(+), 4 deletions(-)
bitbake/bin/bitbake-worker +1 -2
@@ -436,8 +436,7 @@ class BitbakeWorker(object): self.build_pipes[pipe].read() try: - sys.stdin = sys.stdin.detach() - worker = BitbakeWorker(sys.stdin) + worker = BitbakeWorker(os.fdopen(sys.stdin.fileno(), 'rb')) if not profiling: worker.serve() else:
meta/classes/devshell.bbclass +0 -2
@@ -65,8 +65,6 @@ def devpyshell(d): os.dup2(m, sys.stdout.fileno()) os.dup2(m, sys.stderr.fileno()) - sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0) - bb.utils.nonblockingfd(sys.stdout) bb.utils.nonblockingfd(sys.stderr) bb.utils.nonblockingfd(sys.stdin)
Signature · 1 line
2.1.4
PATCH 3/4 devpyshell: python3: flush stdout explicitly
In reply to Ed BartoshOpening text sream in unbuffered mode raises the following exception In Python 3:
ValueError: can't have unbuffered text I/O
Fixed by leaving std* streams in text mode and flushing stdout explicitly.
--- meta/classes/devshell.bbclass | 2 ++ scripts/oepydevshell-internal.py | 4 +--- 2 files changed, 3 insertions(+), 3 deletions(-)
meta/classes/devshell.bbclass +2 -0
@@ -90,6 +90,7 @@ def devpyshell(d): else: prompt = ps1 sys.stdout.write(prompt) + sys.stdout.flush() # Restore Ctrl+C since bitbake masks this def signal_handler(signal, frame):
@@ -111,6 +112,7 @@ def devpyshell(d): continue except EOFError as e: sys.stdout.write("\n") + sys.stdout.flush() except (OSError, IOError) as e: if e.errno == 11: continue
scripts/oepydevshell-internal.py +1 -3
@@ -29,9 +29,6 @@ if len(sys.argv) != 3: pty = open(sys.argv[1], "w+b", 0) parent = int(sys.argv[2]) -# Don't buffer output by line endings -sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) -sys.stdin = os.fdopen(sys.stdin.fileno(), 'r', 0) nonblockingfd(pty) nonblockingfd(sys.stdin)
@@ -64,6 +61,7 @@ try: # Write a page at a time to avoid overflowing output # d.keys() is a good way to do that sys.stdout.write(i[:4096]) + sys.stdout.flush() i = i[4096:] if sys.stdin in ready: echonocbreak(sys.stdin.fileno())
Signature · 1 line
2.1.4
PATCH 4/4 oepydevshell-internal: python3: encode/decode pty content
In reply to Ed BartoshAs /dev/pty opened in binary mode its content has to be decoded when reading from it and encoded when writing to it.
--- scripts/oepydevshell-internal.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/oepydevshell-internal.py +4 -4
@@ -47,7 +47,7 @@ try: # Need cbreak/noecho whilst in select so we trigger on any keypress cbreaknoecho(sys.stdin.fileno()) # Send our PID to the other end so they can kill us. - pty.write(str(os.getpid()) + "\n") + pty.write(str(os.getpid()).encode('utf-8') + b"\n") while True: try: writers = []
@@ -56,7 +56,7 @@ try: (ready, _, _) = select.select([pty, sys.stdin], writers , [], 0) try: if pty in ready: - i = i + pty.read() + i = i + pty.read().decode('utf-8') if i: # Write a page at a time to avoid overflowing output # d.keys() is a good way to do that
@@ -65,9 +65,9 @@ try: i = i[4096:] if sys.stdin in ready: echonocbreak(sys.stdin.fileno()) - o = input() + o = input().encode('utf-8') cbreaknoecho(sys.stdin.fileno()) - pty.write(o + "\n") + pty.write(o + b"\n") except (IOError, OSError) as e: if e.errno == 11: continue
Signature · 1 line
2.1.4