git-p4 tests, filetypes, shell metacharacters
v2 cover git-p4 tests, filetypes, shell metacharacters
This series of six patches has been reviewed on the mailing list and should be ready for inclusion.
1 - Clean up git-p4 tests, including Junio review comments
2..4 - Handle p4 filetypes better, including Chris's suggested
utf16 fix
5 - Fix p4 keyword parsing
6 - From Luke: avoid using the shell, so filenames with metacharacters
are not errantly expanded
Luke Diamand (1):
git-p4: handle files with shell metacharacters
Pete Wyckoff (5):
git-p4 tests: refactor and cleanup
git-p4: handle utf16 filetype properly
git-p4: recognize all p4 filetypes
git-p4: stop ignoring apple filetype
git-p4: keyword flattening fixes
contrib/fast-import/git-p4 | 287 +++++++++++++------- t/lib-git-p4.sh | 74 +++++ t/t9800-git-p4.sh | 574 ++++++++++++++++++---------------------- t/t9801-git-p4-branch.sh | 233 ++++++++++++++++ t/t9802-git-p4-filetype.sh | 108 ++++++++ t/t9803-git-shell-metachars.sh | 64 +++++ 6 files changed, 918 insertions(+), 422 deletions(-) create mode 100644 t/lib-git-p4.sh create mode 100755 t/t9801-git-p4-branch.sh create mode 100755 t/t9802-git-p4-filetype.sh create mode 100755 t/t9803-git-shell-metachars.sh
PATCH v2 1/6 git-p4 tests: refactor and cleanup
Introduce a library for functions that are common to multiple git-p4 test files.
Be a bit more clever about starting and stopping p4d. Specify a unique port number for each test, so that tests can run in parallel. Start p4d not in daemon mode, and save the pid, to be able to kill it cleanly later. Never kill p4d at startup; always shutdown cleanly.
Handle directory changes better. Always chdir inside a subshell, and remove any post-test directory changes.
Clean up whitespace, and use test_cmp and test_must_fail more consistently.
Separate the tests related to detecting p4 branches into their own file, and add a few more.
--- t/lib-git-p4.sh | 74 ++++++ t/t9800-git-p4.sh | 574 ++++++++++++++++++++-------------------------- t/t9801-git-p4-branch.sh | 233 +++++++++++++++++++ 3 files changed, 558 insertions(+), 323 deletions(-) create mode 100644 t/lib-git-p4.sh create mode 100755 t/t9801-git-p4-branch.sh
t/lib-git-p4.sh new file +74 -0
@@ -0,0 +1,74 @@ +# +# Library code for git-p4 tests +# + +. ./test-lib.sh + +if ! test_have_prereq PYTHON; then + skip_all='skipping git-p4 tests; python not available' + test_done +fi +( p4 -h && p4d -h ) >/dev/null 2>&1 || { + skip_all='skipping git-p4 tests; no p4 or p4d' + test_done +} + +GITP4="$GIT_BUILD_DIR/contrib/fast-import/git-p4" + +# Try to pick a unique port: guess a large number, then hope +# no more than one of each test is running. +# +# This does not handle the case where somebody else is running the +# same tests and has chosen the same ports. +testid=${this_test#t} +git_p4_test_start=9800 +P4DPORT=$((10669 + (testid - git_p4_test_start))) + +export P4PORT=localhost:$P4DPORT +export P4CLIENT=client + +db="$TRASH_DIRECTORY/db" +cli="$TRASH_DIRECTORY/cli" +git="$TRASH_DIRECTORY/git" +pidfile="$TRASH_DIRECTORY/p4d.pid" + +start_p4d() { + mkdir -p "$db" "$cli" "$git" && + ( + p4d -q -r "$db" -p $P4DPORT & + echo $! >"$pidfile" + ) && + for i in 1 2 3 4 5 ; do + p4 info >/dev/null 2>&1 && break || true && + echo waiting for p4d to start && + sleep 1 + done && + # complain if it never started + p4 info >/dev/null && + ( + cd "$cli" && + p4 client -i <<-EOF + Client: client + Description: client + Root: $cli + View: //depot/... //client/... + EOF + ) +} + +kill_p4d() { + pid=$(cat "$pidfile") + # it had better exist for the first kill + kill $pid && + for i in 1 2 3 4 5 ; do + kill $pid >/dev/null 2>&1 || break + sleep 1 + done && + # complain if it would not die + test_must_fail kill $pid >/dev/null 2>&1 && + rm -rf "$db" "$cli" "$pidfile" +} + +cleanup_git() { + rm -rf "$git" +}
t/t9800-git-p4.sh +251 -323
678 lines not on this page · raw message
t/t9801-git-p4-branch.sh new file +233 -0
233 lines not on this page · raw message
Signature · 1 line
1.7.6.3
Re: [PATCH 1/6] git-p4 tests: refactor and cleanup
Hi Pete, hope you don't mind a couple of minor nits from an outsider ...
t/lib-git-p4.sh line 26
Quoted 9 lines
t/lib-git-p4.sh new file +9
-0+# Try to pick a unique port: guess a large number, then hope +# no more than one of each test is running. +# +# This does not handle the case where somebody else is running the +# same tests and has chosen the same ports. +testid=${this_test#t} +git_p4_test_start=9800 +P4DPORT=$((10669 + (testid - git_p4_test_start))) +
This won't work with older versions of the Almquist shell (without prepending testid' and git_p4_test_start' with a `$', that is):
$ ash-0.5.2 -c 'a=4; b=2; echo $(( 1 + (a - b) ))' ash-0.5.2: arith: syntax error: " 1 + (a - b) "
Still, it works with all the other POSIX-ish shells that I've tried, including dash 0.5.5.1 on GNU/Linux, NetBSD 5.1 /bin/sh and /bin/ksh, and Solaris 10 /bin/ksh and /usr/xpg4/bin/sh, so it might not be worth worrying about.
Same goes for other similar usages in the rest of the patch, such as "git config git-p4.detectCopies $((level + 2))"
Regards,
Stefano
Re: [PATCH 1/6] git-p4 tests: refactor and cleanup
OK, I have to partially correct myself here ...
Quoted 7 lines · Stefano Lattarini
This won't work with older versions of the Almquist shell (without prepending
testid' andgit_p4_test_start' with a `$', that is):$ ash-0.5.2 -c 'a=4; b=2; echo $(( 1 + (a - b) ))' ash-0.5.2: arith: syntax error: " 1 + (a - b) "
OTOH, this behaviour seems in contrast with the POSIX standard, which says that:
``If the shell variable x contains a value that forms a valid integer
constant, then the arithmetic expansions "$((x))" and "$(($x))" shall return the same value.''
Reference:
<http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_04>
So, since the git testsuite explicitly requires a POSIX shell, there is probably nothing to fix here.
Sorry for the noise,
Stefano
Re: [PATCH 1/6] git-p4 tests: refactor and cleanup
[email protected] wrote on Sun, 16 Oct 2011 11:54 +0200:
Quoted 17 lines
OK, I have to partially correct myself here ...
Quoted 7 lines · Stefano Lattarini
This won't work with older versions of the Almquist shell (without prepending
testid' andgit_p4_test_start' with a `$', that is):$ ash-0.5.2 -c 'a=4; b=2; echo $(( 1 + (a - b) ))' ash-0.5.2: arith: syntax error: " 1 + (a - b) "OTOH, this behaviour seems in contrast with the POSIX standard, which says that:
``If the shell variable x contains a value that forms a valid integer
constant, then the arithmetic expansions "$((x))" and "$(($x))" shall return the same value.''
Quoted 8 lines
Reference:
<http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_04>So, since the git testsuite explicitly requires a POSIX shell, there is probably nothing to fix here.
Sorry for the noise,
Stefano
No, this is a good catch. The use of $x inside arithmetic expansions is indeed called out in the CodingGuidlines. Thanks for the careful read.
I'm going to resend the series, since it's not in next yet. Here's the patch for this bit, fyi.
-- Pete
-------------------8<-------------------
From 53d5ad0761cda36da97bf1498ab1dc1a80a52772 Mon Sep 17 00:00:00 2001
From: Pete Wyckoff <[email protected]>
Date: Sun, 16 Oct 2011 09:59:05 -0400
Subject: [PATCH] git-p4 tests: use explicit $ for variables in $(( .. ))
--- t/lib-git-p4.sh | 2 +- t/t9800-git-p4.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-)
t/lib-git-p4.sh +1 -1
@@ -22,7 +22,7 @@ GITP4="$GIT_BUILD_DIR/contrib/fast-import/git-p4" # same tests and has chosen the same ports. testid=${this_test#t} git_p4_test_start=9800 -P4DPORT=$((10669 + (testid - git_p4_test_start))) +P4DPORT=$((10669 + ($testid - $git_p4_test_start))) export P4PORT=localhost:$P4DPORT export P4CLIENT=client
t/t9800-git-p4.sh +4 -4
@@ -293,7 +293,7 @@ test_expect_success 'detect renames' ' git diff-tree -r -M HEAD && level=$(git diff-tree -r -M HEAD | sed 1d | cut -f1 | cut -d" " -f5 | sed "s/R0*//") && test -n "$level" && test "$level" -gt 0 && test "$level" -lt 98 && - git config git-p4.detectRenames $((level + 2)) && + git config git-p4.detectRenames $(($level + 2)) && "$GITP4" submit && p4 filelog //depot/file6 && p4 filelog //depot/file6 | test_must_fail grep -q "branch from" &&
@@ -305,7 +305,7 @@ test_expect_success 'detect renames' ' git diff-tree -r -M HEAD && level=$(git diff-tree -r -M HEAD | sed 1d | cut -f1 | cut -d" " -f5 | sed "s/R0*//") && test -n "$level" && test "$level" -gt 2 && test "$level" -lt 100 && - git config git-p4.detectRenames $((level - 2)) && + git config git-p4.detectRenames $(($level - 2)) && "$GITP4" submit && p4 filelog //depot/file7 && p4 filelog //depot/file7 | grep -q "branch from //depot/file6"
@@ -376,7 +376,7 @@ test_expect_success 'detect copies' ' test -n "$level" && test "$level" -gt 0 && test "$level" -lt 98 && src=$(git diff-tree -r -C --find-copies-harder HEAD | sed 1d | cut -f2) && test "$src" = file10 && - git config git-p4.detectCopies $((level + 2)) && + git config git-p4.detectCopies $(($level + 2)) && "$GITP4" submit && p4 filelog //depot/file12 && p4 filelog //depot/file12 | test_must_fail grep -q "branch from" &&
@@ -390,7 +390,7 @@ test_expect_success 'detect copies' ' test -n "$level" && test "$level" -gt 2 && test "$level" -lt 100 && src=$(git diff-tree -r -C --find-copies-harder HEAD | sed 1d | cut -f2) && test "$src" = file10 && - git config git-p4.detectCopies $((level - 2)) && + git config git-p4.detectCopies $(($level - 2)) && "$GITP4" submit && p4 filelog //depot/file13 && p4 filelog //depot/file13 | grep -q "branch from //depot/file"
Signature · 1 line
1.7.7
PATCH v2 2/6 git-p4: handle utf16 filetype properly
In reply to Pete WyckoffOne of the filetypes that p4 supports is utf16. Its behavior is odd in this case. The data delivered through "p4 -G print" is not encoded in utf16, although "p4 print -o" will produce the proper utf16-encoded file.
When dealing with this filetype, discard the data from -G, and instead read the contents directly.
An alternate approach would be to try to encode the data in python. That worked for true utf16 files, but for other files marked as utf16, p4 delivers mangled text in no recognizable encoding.
Add a test case to check utf16 handling, and +k and +ko handling.
--- contrib/fast-import/git-p4 | 11 +++++ t/t9802-git-p4-filetype.sh | 108 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+), 0 deletions(-) create mode 100755 t/t9802-git-p4-filetype.sh
contrib/fast-import/git-p4 +11 -0
@@ -1238,6 +1238,15 @@ class P4Sync(Command, P4UserMap): data = ''.join(contents) contents = [data[:-1]] + if file['type'].startswith("utf16"): + # p4 delivers different text in the python output to -G + # than it does when using "print -o", or normal p4 client + # operations. utf16 is converted to ascii or utf8, perhaps. + # But ascii text saved as -t utf16 is completely mangled. + # Invoke print -o to get the real contents. + text = p4_read_pipe('print -q -o - "%s"' % file['depotFile']) + contents = [ text ] + if self.isWindows and file["type"].endswith("text"): mangled = [] for data in contents:
@@ -1245,6 +1254,8 @@ class P4Sync(Command, P4UserMap): mangled.append(data) contents = mangled + # Note that we do not try to de-mangle keywords on utf16 files, + # even though in theory somebody may want that. if file['type'] in ('text+ko', 'unicode+ko', 'binary+ko'): contents = map(lambda text: re.sub(r'(?i)\$(Id|Header):[^$]*\$',r'$\1$', text), contents) elif file['type'] in ('text+k', 'ktext', 'kxtext', 'unicode+k', 'binary+k'):
t/t9802-git-p4-filetype.sh new file +108 -0
108 lines not on this page · raw message
Signature · 1 line
1.7.6.3
Re: [PATCH 2/6] git-p4: handle utf16 filetype properly
Hello Pete.
One more "outsider nit" here ...
Quoted 20 lines · Pete Wyckoff
[SNIP]
Add a test case to check utf16 handling, and +k and +ko handling.
[SNIP]
t/t9802-git-p4-filetype.sh new file +6
-0@@ -0,0 +1,108 @@ +#!/bin/sh + +test_description='git-p4 p4 filetype tests' + +. ./lib-git-p4.sh +
t/t9802-git-p4-filetype.sh line 16
Quoted 10 lines
t/t9802-git-p4-filetype.sh new file +10
-0+test_expect_success 'start p4d' ' + start_p4d +' + +test_expect_success 'utf-16 file create' ' + ( + cd "$cli" && + + # p4 saves this verbatim + echo -e "three\nline\ntext" >f-ascii &&
Not portable to (at least) solaris /usr/xpg4/bin/sh and /bin/ksh:
$ /bin/ksh -c 'echo -e "three\nline\ntext"' -e three line text
In fact, use of options and/or escape sequences in "echo" arguments is highly unportable (see the entry for `echo' in the "Limitations of Shell Builtins" section of the autoconf manual); your best bet is to use printf, which is more portable and well-behaved (at least on systems that are not musuem pieces).
t/t9802-git-p4-filetype.sh line 76
Quoted 10 lines
[SNIP]
t/t9802-git-p4-filetype.sh new file +6
-0+ +build_smush() { + cat >k_smush.py <<-EOF && + import re, sys + sys.stdout.write(re.sub(r'(?i)\\\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$]*\\\$', r'$\1$', sys.stdin.read())) + EOF
This is a basically a stylistic nit, so fell free to disregard it completely, but ... wouldn't it be simpler to quote the "EOF" at the here-doc beginning, so that you don't have to escape all the backslashes in the here-doc itself? Similarly for other later usages.
Regards,
Stefano
Re: [PATCH 2/6] git-p4: handle utf16 filetype properly
[email protected] wrote on Sun, 16 Oct 2011 11:52 +0200:
Quoted 42 lines
Hello Pete.
One more "outsider nit" here ...
Quoted 20 lines · Pete Wyckoff
[SNIP]
Add a test case to check utf16 handling, and +k and +ko handling.
[SNIP]
t/t9802-git-p4-filetype.sh new file +6
-0@@ -0,0 +1,108 @@ +#!/bin/sh + +test_description='git-p4 p4 filetype tests' + +. ./lib-git-p4.sh +t/t9802-git-p4-filetype.sh line 16
Quoted 10 lines
t/t9802-git-p4-filetype.sh new file +10
-0+test_expect_success 'start p4d' ' + start_p4d +' + +test_expect_success 'utf-16 file create' ' + ( + cd "$cli" && + + # p4 saves this verbatim + echo -e "three\nline\ntext" >f-ascii &&Not portable to (at least) solaris
/usr/xpg4/bin/shand/bin/ksh:$ /bin/ksh -c 'echo -e "three\nline\ntext"' -e three line text
Quoted 5 lines
In fact, use of options and/or escape sequences in "echo" arguments is highly unportable (see the entry for `echo' in the "Limitations of Shell Builtins" section of the autoconf manual); your best bet is to use printf, which is more portable and well-behaved (at least on systems that are not musuem pieces).
Indeed. The t/ source consistently uses printf where \n is needed.
Quoted 14 lines
t/t9802-git-p4-filetype.sh line 76
Quoted 10 lines
[SNIP]
t/t9802-git-p4-filetype.sh new file +6
-0+ +build_smush() { + cat >k_smush.py <<-EOF && + import re, sys + sys.stdout.write(re.sub(r'(?i)\\\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$]*\\\$', r'$\1$', sys.stdin.read())) + EOFThis is a basically a stylistic nit, so fell free to disregard it completely, but ... wouldn't it be simpler to quote the "EOF" at the here-doc beginning, so that you don't have to escape all the backslashes in the here-doc itself? Similarly for other later usages.
Switched to using \EOF and got rid of the extranneous \.
Thanks,
-- Pete
PATCH v2 3/6 git-p4: recognize all p4 filetypes
In reply to Pete WyckoffThe previous code was approximate in the filetypes it recognized. Put in the canonical list and be more careful about matching elements of the file type.
This might change behavior in some cases, hopefully for the better. Windows newline mangling will now happen on all text files. Previously some like "text+ko" were oddly exempt.
Files with multiple combinations of modifiers, like "text+klx", are now recognized for keyword expansion. I expect these to be seen only rarely.
--- contrib/fast-import/git-p4 | 71 ++++++++++++++++++++++++++++++++------------ 1 files changed, 52 insertions(+), 19 deletions(-)
contrib/fast-import/git-p4 +52 -19
@@ -118,13 +118,41 @@ def p4_system(cmd): real_cmd = p4_build_cmd(cmd) return system(real_cmd) -def isP4Exec(kind): - """Determine if a Perforce 'kind' should have execute permission +# +# Canonicalize the p4 type and return a tuple of the +# base type, plus any modifiers. See "p4 help filetypes" +# for a list and explanation. +# +def split_p4_type(p4type): + + p4_filetypes_historical = { + "ctempobj": "binary+Sw", + "ctext": "text+C", + "cxtext": "text+Cx", + "ktext": "text+k", + "kxtext": "text+kx", + "ltext": "text+F", + "tempobj": "binary+FSw", + "ubinary": "binary+F", + "uresource": "resource+F", + "uxbinary": "binary+Fx", + "xbinary": "binary+x", + "xltext": "text+Fx", + "xtempobj": "binary+Swx", + "xtext": "text+x", + "xunicode": "unicode+x", + "xutf16": "utf16+x", + } + if p4type in p4_filetypes_historical: + p4type = p4_filetypes_historical[p4type] + mods = "" + s = p4type.split("+") + base = s[0] + mods = "" + if len(s) > 1: + mods = s[1] + return (base, mods) - 'p4 help filetypes' gives a list of the types. If it starts with 'x', - or x follows one of a few letters. Otherwise, if there is an 'x' after - a plus sign, it is also executable""" - return (re.search(r"(^[cku]?x)|\+.*x", kind) != None) def setP4ExecBit(file, mode): # Reopens an already open file and changes the execute bit to match
@@ -1229,16 +1257,18 @@ class P4Sync(Command, P4UserMap): if verbose: sys.stderr.write("%s\n" % relPath) - mode = "644" - if isP4Exec(file["type"]): - mode = "755" - elif file["type"] == "symlink": - mode = "120000" - # p4 print on a symlink contains "target\n", so strip it off + (type_base, type_mods) = split_p4_type(file["type"]) + + git_mode = "100644" + if "x" in type_mods: + git_mode = "100755" + if type_base == "symlink": + git_mode = "120000" + # p4 print on a symlink contains "target\n"; remove the newline data = ''.join(contents) contents = [data[:-1]] - if file['type'].startswith("utf16"): + if type_base == "utf16": # p4 delivers different text in the python output to -G # than it does when using "print -o", or normal p4 client # operations. utf16 is converted to ascii or utf8, perhaps.
@@ -1247,7 +1277,9 @@ class P4Sync(Command, P4UserMap): text = p4_read_pipe('print -q -o - "%s"' % file['depotFile']) contents = [ text ] - if self.isWindows and file["type"].endswith("text"): + # Perhaps windows wants unicode, utf16 newlines translated too; + # but this is not doing it. + if self.isWindows and type_base == "text": mangled = [] for data in contents: data = data.replace("\r\n", "\n")
@@ -1256,12 +1288,13 @@ class P4Sync(Command, P4UserMap): # Note that we do not try to de-mangle keywords on utf16 files, # even though in theory somebody may want that. - if file['type'] in ('text+ko', 'unicode+ko', 'binary+ko'): - contents = map(lambda text: re.sub(r'(?i)\$(Id|Header):[^$]*\$',r'$\1$', text), contents) - elif file['type'] in ('text+k', 'ktext', 'kxtext', 'unicode+k', 'binary+k'): - contents = map(lambda text: re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$\n]*\$',r'$\1$', text), contents) + if type_base in ("text", "unicode", "binary"): + if "ko" in type_mods: + contents = map(lambda text: re.sub(r'(?i)\$(Id|Header):[^$]*\$', r'$\1$', text), contents) + elif "k" in type_mods: + contents = map(lambda text: re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$\n]*\$', r'$\1$', text), contents) - self.gitStream.write("M %s inline %s\n" % (mode, relPath)) + self.gitStream.write("M %s inline %s\n" % (git_mode, relPath)) # total length... length = 0
Signature · 1 line
1.7.6.3
PATCH v2 4/6 git-p4: stop ignoring apple filetype
In reply to Pete WyckoffCurrently "apple" filetype is ignored explicitly, and the file is not even included in the git repository. This seems wrong. Remove this, letting it be treated like a "binary" filetype.
--- contrib/fast-import/git-p4 | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-)
contrib/fast-import/git-p4 +0 -5
@@ -1247,11 +1247,6 @@ class P4Sync(Command, P4UserMap): # - helper for streamP4Files def streamOneP4File(self, file, contents): - if file["type"] == "apple": - print "\nfile %s is a strange apple file that forks. Ignoring" % \ - file['depotFile'] - return - relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes) relPath = self.wildcard_decode(relPath) if verbose:
Signature · 1 line
1.7.6.3
PATCH v2 5/6 git-p4: keyword flattening fixes
In reply to Pete WyckoffJoin the text before looking for keywords. There is nothing to prevent the p4 output marshaller from splitting in the middle of a keyword, although it has never been known to happen.
Also remove the (?i) regexp modifier; perforce keywords are documented as case-sensitive.
Remove the "\n" end-character match. I don't know why that is in there, and every keyword in a fairly large production p4 repository always ends with a $.
--- contrib/fast-import/git-p4 | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
contrib/fast-import/git-p4 +6 -2
@@ -1285,9 +1285,13 @@ class P4Sync(Command, P4UserMap): # even though in theory somebody may want that. if type_base in ("text", "unicode", "binary"): if "ko" in type_mods: - contents = map(lambda text: re.sub(r'(?i)\$(Id|Header):[^$]*\$', r'$\1$', text), contents) + text = ''.join(contents) + text = re.sub(r'\$(Id|Header):[^$]*\$', r'$\1$', text) + contents = [ text ] elif "k" in type_mods: - contents = map(lambda text: re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$\n]*\$', r'$\1$', text), contents) + text = ''.join(contents) + text = re.sub(r'\$(Id|Header|Author|Date|DateTime|Change|File|Revision):[^$]*\$', r'$\1$', text) + contents = [ text ] self.gitStream.write("M %s inline %s\n" % (git_mode, relPath))
Signature · 1 line
1.7.6.3
PATCH v2 6/6 git-p4: handle files with shell metacharacters
In reply to Pete WyckoffFrom: Luke Diamand <[email protected]>
git-p4 used to simply pass strings into system() and popen(), and relied on the shell doing the necessary expansion. This though meant that shell metacharacters in file names would be corrupted - for example files with $ or space in them.
Switch to using subprocess.Popen() and friends, and pass in explicit arrays in the places where it matters. This then avoids needing shell expansion.
Add trivial helper functions for some common perforce operations. Add test case.
[pw: test cleanup]
--- contrib/fast-import/git-p4 | 200 ++++++++++++++++++++++++--------------- t/t9803-git-shell-metachars.sh | 64 +++++++++++++ 2 files changed, 187 insertions(+), 77 deletions(-) create mode 100755 t/t9803-git-shell-metachars.sh
contrib/fast-import/git-p4 +123 -77
@@ -22,36 +22,39 @@ def p4_build_cmd(cmd): location. It means that hooking into the environment, or other configuration can be done more easily. """ - real_cmd = "%s " % "p4" + real_cmd = ["p4"] user = gitConfig("git-p4.user") if len(user) > 0: - real_cmd += "-u %s " % user + real_cmd += ["-u",user] password = gitConfig("git-p4.password") if len(password) > 0: - real_cmd += "-P %s " % password + real_cmd += ["-P", password] port = gitConfig("git-p4.port") if len(port) > 0: - real_cmd += "-p %s " % port + real_cmd += ["-p", port] host = gitConfig("git-p4.host") if len(host) > 0: - real_cmd += "-h %s " % host + real_cmd += ["-h", host] client = gitConfig("git-p4.client") if len(client) > 0: - real_cmd += "-c %s " % client + real_cmd += ["-c", client] - real_cmd += "%s" % (cmd) - if verbose: - print real_cmd + + if isinstance(cmd,basestring): + real_cmd = ' '.join(real_cmd) + ' ' + cmd + else: + real_cmd += cmd return real_cmd def chdir(dir): - if os.name == 'nt': - os.environ['PWD']=dir + # P4 uses the PWD environment variable rather than getcwd(). Since we're + # not using the shell, we have to set it ourselves. + os.environ['PWD']=dir os.chdir(dir) def die(msg):
@@ -61,29 +64,34 @@ def die(msg): sys.stderr.write(msg + "\n") sys.exit(1) -def write_pipe(c, str): +def write_pipe(c, stdin): if verbose: - sys.stderr.write('Writing pipe: %s\n' % c) + sys.stderr.write('Writing pipe: %s\n' % str(c)) - pipe = os.popen(c, 'w') - val = pipe.write(str) - if pipe.close(): - die('Command failed: %s' % c) + expand = isinstance(c,basestring) + p = subprocess.Popen(c, stdin=subprocess.PIPE, shell=expand) + pipe = p.stdin + val = pipe.write(stdin) + pipe.close() + if p.wait(): + die('Command failed: %s' % str(c)) return val -def p4_write_pipe(c, str): +def p4_write_pipe(c, stdin): real_cmd = p4_build_cmd(c) - return write_pipe(real_cmd, str) + return write_pipe(real_cmd, stdin) def read_pipe(c, ignore_error=False): if verbose: - sys.stderr.write('Reading pipe: %s\n' % c) + sys.stderr.write('Reading pipe: %s\n' % str(c)) - pipe = os.popen(c, 'rb') + expand = isinstance(c,basestring) + p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand) + pipe = p.stdout val = pipe.read() - if pipe.close() and not ignore_error: - die('Command failed: %s' % c) + if p.wait() and not ignore_error: + die('Command failed: %s' % str(c)) return val
@@ -93,12 +101,14 @@ def p4_read_pipe(c, ignore_error=False): def read_pipe_lines(c): if verbose: - sys.stderr.write('Reading pipe: %s\n' % c) - ## todo: check return status - pipe = os.popen(c, 'rb') + sys.stderr.write('Reading pipe: %s\n' % str(c)) + + expand = isinstance(c, basestring) + p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand) + pipe = p.stdout val = pipe.readlines() - if pipe.close(): - die('Command failed: %s' % c) + if pipe.close() or p.wait(): + die('Command failed: %s' % str(c)) return val
@@ -108,15 +118,37 @@ def p4_read_pipe_lines(c): return read_pipe_lines(real_cmd) def system(cmd): + expand = isinstance(cmd,basestring) if verbose: - sys.stderr.write("executing %s\n" % cmd) - if os.system(cmd) != 0: - die("command failed: %s" % cmd) + sys.stderr.write("executing %s\n" % str(cmd)) + subprocess.check_call(cmd, shell=expand) def p4_system(cmd): """Specifically invoke p4 as the system command. """ real_cmd = p4_build_cmd(cmd) - return system(real_cmd) + expand = isinstance(real_cmd, basestring) + subprocess.check_call(real_cmd, shell=expand) + +def p4_integrate(src, dest): + p4_system(["integrate", "-Dt", src, dest]) + +def p4_sync(path): + p4_system(["sync", path]) + +def p4_add(f): + p4_system(["add", f]) + +def p4_delete(f): + p4_system(["delete", f]) + +def p4_edit(f): + p4_system(["edit", f]) + +def p4_revert(f): + p4_system(["revert", f]) + +def p4_reopen(type, file): + p4_system(["reopen", "-t", type, file]) # # Canonicalize the p4 type and return a tuple of the
@@ -167,12 +199,12 @@ def setP4ExecBit(file, mode): if p4Type[-1] == "+": p4Type = p4Type[0:-1] - p4_system("reopen -t %s %s" % (p4Type, file)) + p4_reopen(p4Type, file) def getP4OpenedType(file): # Returns the perforce file type for the given file. - result = p4_read_pipe("opened %s" % file) + result = p4_read_pipe(["opened", file]) match = re.match(".*\((.+)\)\r?$", result) if match: return match.group(1)
@@ -228,9 +260,17 @@ def isModeExecChanged(src_mode, dst_mode): return isModeExec(src_mode) != isModeExec(dst_mode) def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None): - cmd = p4_build_cmd("-G %s" % (cmd)) + + if isinstance(cmd,basestring): + cmd = "-G " + cmd + expand = True + else: + cmd = ["-G"] + cmd + expand = False + + cmd = p4_build_cmd(cmd) if verbose: - sys.stderr.write("Opening pipe: %s\n" % cmd) + sys.stderr.write("Opening pipe: %s\n" % str(cmd)) # Use a temporary file to avoid deadlocks without # subprocess.communicate(), which would put another copy
@@ -238,11 +278,16 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None): stdin_file = None if stdin is not None: stdin_file = tempfile.TemporaryFile(prefix='p4-stdin', mode=stdin_mode) - stdin_file.write(stdin) + if isinstance(stdin,basestring): + stdin_file.write(stdin) + else: + for i in stdin: + stdin_file.write(i + '\n') stdin_file.flush() stdin_file.seek(0) - p4 = subprocess.Popen(cmd, shell=True, + p4 = subprocess.Popen(cmd, + shell=expand, stdin=stdin_file, stdout=subprocess.PIPE)
@@ -275,7 +320,7 @@ def p4Where(depotPath): if not depotPath.endswith("/"): depotPath += "/" depotPath = depotPath + "..." - outputList = p4CmdList("where %s" % depotPath) + outputList = p4CmdList(["where", depotPath]) output = None for entry in outputList: if "depotFile" in entry:
@@ -477,8 +522,10 @@ def originP4BranchesExist(): def p4ChangesForPaths(depotPaths, changeRange): assert depotPaths - output = p4_read_pipe_lines("changes " + ' '.join (["%s...%s" % (p, changeRange) - for p in depotPaths])) + cmd = ['changes'] + for p in depotPaths: + cmd += ["%s...%s" % (p, changeRange)] + output = p4_read_pipe_lines(cmd) changes = {} for line in output:
@@ -561,7 +608,7 @@ class P4Debug(Command): def run(self, args): j = 0 - for output in p4CmdList(" ".join(args)): + for output in p4CmdList(args): print 'Element: %d' % j j += 1 print output
@@ -715,7 +762,7 @@ class P4Submit(Command, P4UserMap): break if not client: die("could not get client spec") - results = p4CmdList("changes -c %s -m 1" % client) + results = p4CmdList(["changes", "-c", client, "-m", "1"]) for r in results: if r.has_key('change'): return r['change']
@@ -778,7 +825,7 @@ class P4Submit(Command, P4UserMap): # remove lines in the Files section that show changes to files outside the depot path we're committing into template = "" inFilesSection = False - for line in p4_read_pipe_lines("change -o"): + for line in p4_read_pipe_lines(['change', '-o']): if line.endswith("\r\n"): line = line[:-2] + "\n" if inFilesSection:
@@ -835,7 +882,7 @@ class P4Submit(Command, P4UserMap): modifier = diff['status'] path = diff['src'] if modifier == "M": - p4_system("edit \"%s\"" % path) + p4_edit(path) if isModeExecChanged(diff['src_mode'], diff['dst_mode']): filesToChangeExecBit[path] = diff['dst_mode'] editedFiles.add(path)
@@ -850,21 +897,21 @@ class P4Submit(Command, P4UserMap): filesToAdd.remove(path) elif modifier == "C": src, dest = diff['src'], diff['dst'] - p4_system("integrate -Dt \"%s\" \"%s\"" % (src, dest)) + p4_integrate(src, dest) if diff['src_sha1'] != diff['dst_sha1']: - p4_system("edit \"%s\"" % (dest)) + p4_edit(dest) if isModeExecChanged(diff['src_mode'], diff['dst_mode']): - p4_system("edit \"%s\"" % (dest)) + p4_edit(dest) filesToChangeExecBit[dest] = diff['dst_mode'] os.unlink(dest) editedFiles.add(dest) elif modifier == "R": src, dest = diff['src'], diff['dst'] - p4_system("integrate -Dt \"%s\" \"%s\"" % (src, dest)) + p4_integrate(src, dest) if diff['src_sha1'] != diff['dst_sha1']: - p4_system("edit \"%s\"" % (dest)) + p4_edit(dest) if isModeExecChanged(diff['src_mode'], diff['dst_mode']): - p4_system("edit \"%s\"" % (dest)) + p4_edit(dest) filesToChangeExecBit[dest] = diff['dst_mode'] os.unlink(dest) editedFiles.add(dest)
@@ -887,9 +934,9 @@ class P4Submit(Command, P4UserMap): if response == "s": print "Skipping! Good luck with the next patches..." for f in editedFiles: - p4_system("revert \"%s\"" % f); + p4_revert(f) for f in filesToAdd: - system("rm %s" %f) + os.remove(f) return elif response == "a": os.system(applyPatchCmd)
@@ -910,10 +957,10 @@ class P4Submit(Command, P4UserMap): system(applyPatchCmd) for f in filesToAdd: - p4_system("add \"%s\"" % f) + p4_add(f) for f in filesToDelete: - p4_system("revert \"%s\"" % f) - p4_system("delete \"%s\"" % f) + p4_revert(f) + p4_delete(f) # Set/clear executable bits for f in filesToChangeExecBit.keys():
@@ -935,7 +982,7 @@ class P4Submit(Command, P4UserMap): del(os.environ["P4DIFF"]) diff = "" for editedFile in editedFiles: - diff += p4_read_pipe("diff -du %r" % editedFile) + diff += p4_read_pipe(['diff', '-du', editedFile]) newdiff = "" for newFile in filesToAdd:
@@ -987,7 +1034,7 @@ class P4Submit(Command, P4UserMap): submitTemplate = message[:message.index(separatorLine)] if self.isWindows: submitTemplate = submitTemplate.replace("\r\n", "\n") - p4_write_pipe("submit -i", submitTemplate) + p4_write_pipe(['submit', '-i'], submitTemplate) if self.preserveUser: if p4User:
@@ -998,10 +1045,10 @@ class P4Submit(Command, P4UserMap): else: for f in editedFiles: - p4_system("revert \"%s\"" % f); + p4_revert(f) for f in filesToAdd: - p4_system("revert \"%s\"" % f); - system("rm %s" %f) + p4_revert(f) + os.remove(f) os.remove(fileName) else:
@@ -1054,8 +1101,7 @@ class P4Submit(Command, P4UserMap): chdir(self.clientPath) print "Synchronizing p4 checkout..." - p4_system("sync ...") - + p4_sync("...") self.check() commits = []
@@ -1269,7 +1315,7 @@ class P4Sync(Command, P4UserMap): # operations. utf16 is converted to ascii or utf8, perhaps. # But ascii text saved as -t utf16 is completely mangled. # Invoke print -o to get the real contents. - text = p4_read_pipe('print -q -o - "%s"' % file['depotFile']) + text = p4_read_pipe(['print', '-q', '-o', '-', file['depotFile']]) contents = [ text ] # Perhaps windows wants unicode, utf16 newlines translated too;
@@ -1365,10 +1411,11 @@ class P4Sync(Command, P4UserMap): def streamP4FilesCbSelf(entry): self.streamP4FilesCb(entry) - p4CmdList("-x - print", - '\n'.join(['%s#%s' % (f['path'], f['rev']) - for f in filesToRead]), - cb=streamP4FilesCbSelf) + fileArgs = ['%s#%s' % (f['path'], f['rev']) for f in filesToRead] + + p4CmdList(["-x", "-", "print"], + stdin=fileArgs, + cb=streamP4FilesCbSelf) # do the last chunk if self.stream_file.has_key('depotFile'):
@@ -1429,8 +1476,8 @@ class P4Sync(Command, P4UserMap): if self.verbose: print "Change %s is labelled %s" % (change, labelDetails) - files = p4CmdList("files " + ' '.join (["%s...@%s" % (p, change) - for p in branchPrefixes])) + files = p4CmdList(["files"] + ["%s...@%s" % (p, change) + for p in branchPrefixes]) if len(files) == len(labelRevisions):
@@ -1478,9 +1525,9 @@ class P4Sync(Command, P4UserMap): newestChange = 0 if self.verbose: print "Querying files for label %s" % label - for file in p4CmdList("files " - + ' '.join (["%s...@%s" % (p, label) - for p in self.depotPaths])): + for file in p4CmdList(["files"] + + ["%s...@%s" % (p, label) + for p in self.depotPaths]): revisions[file["depotFile"]] = file["rev"] change = int(file["change"]) if change > newestChange:
@@ -1735,10 +1782,9 @@ class P4Sync(Command, P4UserMap): newestRevision = 0 fileCnt = 0 - for info in p4CmdList("files " - + ' '.join(["%s...%s" - % (p, revision) - for p in self.depotPaths])): + fileArgs = ["%s...%s" % (p,revision) for p in self.depotPaths] + + for info in p4CmdList(["files"] + fileArgs): if 'code' in info and info['code'] == 'error': sys.stderr.write("p4 returned an error: %s\n"
t/t9803-git-shell-metachars.sh new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
1.7.6.3
Quoted 5 lines · Pete Wyckoff
This series of six patches has been reviewed on the mailing list and should be ready for inclusion.
1 - Clean up git-p4 tests, including Junio review comments
Quoted 7 lines
2..4 - Handle p4 filetypes better, including Chris's suggested
utf16 fix5 - Fix p4 keyword parsing
6 - From Luke: avoid using the shell, so filenames with metacharacters
are not errantly expanded
I've just tried running the tests in parallel (master branch) and it works a treat.
You've also tidied up my earlier test cases very nicely.
There doesn't seem to be any way to cope with multiple instances of the same test running at the same time. If p4d could write its pid somewhere then something better could be done, but it doesn't.
Ack.
Thanks
Luke
Quoted 22 lines
Luke Diamand (1):
git-p4: handle files with shell metacharactersPete Wyckoff (5):
git-p4 tests: refactor and cleanup
git-p4: handle utf16 filetype properly
git-p4: recognize all p4 filetypes
git-p4: stop ignoring apple filetype
git-p4: keyword flattening fixescontrib/fast-import/git-p4 | 287 +++++++++++++------- t/lib-git-p4.sh | 74 +++++ t/t9800-git-p4.sh | 574 ++++++++++++++++++---------------------- t/t9801-git-p4-branch.sh | 233 ++++++++++++++++ t/t9802-git-p4-filetype.sh | 108 ++++++++ t/t9803-git-shell-metachars.sh | 64 +++++ 6 files changed, 918 insertions(+), 422 deletions(-)create mode 100644
t/lib-git-p4.sh
create mode 100755t/t9801-git-p4-branch.shcreate mode 100755t/t9802-git-p4-filetype.shcreate mode 100755t/t9803-git-shell-metachars.sh