Skip to contentSkip to conversation

StGIT cannot import properly from stdin

DISCUSSION Pierre Ossman 2 participants 9 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

The import command of stgit does not handle stdin in a very sane way. The logic expects to be able to read the patch several times which isn't possible with stdin. So for it to work you have to give it the patch twice.

A simple solution would be to store it to disk and then handle it the same way as if a file had been given on the command line.

Rgds
Pierre

#1 lore raw
Quoted 3 lines · Pierre Ossman

The import command of stgit does not handle stdin in a very sane way. The logic expects to be able to read the patch several times which isn't possible with stdin. So for it to work you have to give it the patch twice.

It works for me with the latest snapshot. What might happen in your case is a missing "---" line after the patch description. The import command reads the sys.stdin file descriptor until the first "---" line. After that, the git.apply() function will read the rest of the lines and pass them to git-apply. You can put some prints in the git.apply() function to check where it starts reading from.

Patches without description cannot be read from stdin at the moment. Maybe I could add another option to prevent it from reading the description.

Signature · 1 line
Catalin
#2 lore raw
Quoted 10 lines · Catalin Marinas

It works for me with the latest snapshot. What might happen in your case is a missing "---" line after the patch description. The import command reads the sys.stdin file descriptor until the first "---" line. After that, the git.apply() function will read the rest of the lines and pass them to git-apply. You can put some prints in the git.apply() function to check where it starts reading from.

Hmm... it seems you're kind of right. It doesn't need the patch twice. But it needs EOF more than once. The for-loop in __parse_patch() doesn't start until the EOF is recieved. And the second for-loop in _input() requires a new EOF.

Quoted 6 lines

Patches without description cannot be read from stdin at the moment. Maybe I could add another option to prevent it from reading the description.

If you would cache the input to a file (or memory) you would remove both of these problems.

Rgds
Pierre

#3 lore raw
Quoted 11 lines · Pierre Ossman
Quoted 6 lines

It works for me with the latest snapshot. What might happen in your case is a missing "---" line after the patch description. The import command reads the sys.stdin file descriptor until the first "---" line. After that, the git.apply() function will read the rest of the lines and pass them to git-apply. You can put some prints in the git.apply() function to check where it starts reading from.

Hmm... it seems you're kind of right. It doesn't need the patch twice. But it needs EOF more than once. The for-loop in __parse_patch() doesn't start until the EOF is recieved. And the second for-loop in _input() requires a new EOF.

The line below in __parse_patch() ensures that the 'for' loop does not consume all the input lines:

        if re.match('---\s*$', line) or re.match('diff -', line):
            break

Unfortunately, if the '---' line is missing, it consumes all the input lines. Is this the case with your patch?

Quoted 2 lines

If you would cache the input to a file (or memory) you would remove both of these problems.

True, but it requires more work than simply writing the patch to a file and reading it twice.

Signature · 1 line
Catalin
#4 lore raw
Quoted 6 lines · Catalin Marinas

Unfortunately, if the '---' line is missing, it consumes all the input lines. Is this the case with your patch?

Nope. It consumes the correct ammount of lines. The problem is that I have to give it a kick (^D) to start things twice. The first for __parse_patch() and the second for _input(). I put some debug-prints in there and the for-loops will not start executing on the first complete line, only EOF.

Quoted 10 lines
Quoted 4 lines

If you would cache the input to a file (or memory) you would remove both of these problems.

True, but it requires more work than simply writing the patch to a file and reading it twice.

If it supports everything when given a filename then shouldn't adding a preprocessing step to dump stdin to disk give the same code path?

Rgds
Pierre

#5 lore raw
Quoted 8 lines · Pierre Ossman
Quoted 2 lines

Unfortunately, if the '---' line is missing, it consumes all the input lines. Is this the case with your patch?

Nope. It consumes the correct ammount of lines. The problem is that I have to give it a kick (^D) to start things twice. The first for __parse_patch() and the second for _input(). I put some debug-prints in there and the for-loops will not start executing on the first complete line, only EOF.

I tried it with reading from the terminal directly and you are right.

Using 'stg import -n patch < patch-file' is fine since the stdin is not re-opened and the _input() loop starts automatically.

'cat | stg import -n patch' also works fine since cat exits when stdin receives EOF.

It looks like a problem with Python since it re-opens sys.stdin once it receives EOF.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

If you would cache the input to a file (or memory) you would remove both of these problems.

True, but it requires more work than simply writing the patch to a file and reading it twice.

If it supports everything when given a filename then shouldn't adding a preprocessing step to dump stdin to disk give the same code path?

I was referring to the 'cache the input to memory' variant, which is a bit more complicated. Caching to a file is simple but do you need to read patches from the terminal directly?

Signature · 1 line
Catalin
#6 lore raw
Quoted 5 lines · Catalin Marinas

It looks like a problem with Python since it re-opens sys.stdin once it receives EOF.

Bummer. I checked with the local Python guru here and he suggested changing the for loop to an infinite while and f.readline().

Quoted 6 lines

I was referring to the 'cache the input to memory' variant, which is a bit more complicated. Caching to a file is simple but do you need to read patches from the terminal directly?

I don't explicitly need it, but it keeps things simple for me. I can copy and paste the patch from my mailer without having to bother with saving it to a temporary file.

#7 lore raw
Quoted 2 lines · Pierre Ossman

Bummer. I checked with the local Python guru here and he suggested changing the for loop to an infinite while and f.readline().

It works, thanks for this. I attached a patch if you want to try.

--
Catalin

#8 lore raw
Quoted 1 line · Catalin Marinas

It works, thanks for this. I attached a patch if you want to try.

Oops, gmail attached it as a binary. It is inserted below (as text/plain):

======================================================
Fix importing from stdin

The current stdin patch importing expects two EOFs since the 'for' loop doesn't start before one EOF is received. As suggested, this patch changes the 'for' loop with a 'while True' loop.

---

 stgit/commands/imprt.py |    6 +++++-
 stgit/git.py            |    5 ++++-
 2 files changed, 9 insertions(+), 2 deletions(-)
stgit/commands/imprt.py +5 -1
@@ -134,7 +134,11 @@ def __parse_patch(filename = None):
     authname = authemail = authdate = None
 
     descr = ''
-    for line in f:
+    while True:
+        line = f.readline()
+        if not line:
+            break
+
         # the first 'Signed-of-by:' is the author
         if not authname and re.match('signed-off-by:\s+', line, re.I):
             auth = re.findall('^.*?:\s+(.*)$', line)[0]
stgit/git.py +4 -1
@@ -113,7 +113,10 @@ def get_conflicts():
 
 def _input(cmd, file_desc):
     p = popen2.Popen3(cmd)
-    for line in file_desc:
+    while True:
+        line = file_desc.readline()
+        if not line:
+            break
         p.tochild.write(line)
     p.tochild.close()
     if p.wait():