add option to import series directly from a tar archive
PATCH 1/1 add option to import series directly from a tar archive
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Catalin/Karl,
Attached is my first cut at adding the ability to import a patch series by specifying the tarball. For example, the following command:
$ stg import --tarfile patch-2.6.26.3-rt6.bz2
will apply the latest -rt patch series to your current kernel tree.
No Karl, I haven't developed a test for it (yet). I wanted to see what you guys thought first :)
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iEYEARECAAYFAkjDTscACgkQqA4JVb61b9dNRgCZAW+tOCgz5Y+A0IdomcOA4X7v u8MAnRvFWMXRJ0Kxv1rAnBRnheq6Iidi
=W7Dl
-----END PGP SIGNATURE-----
patch to allow importing a series from a tar archive
From: Clark Williams <[email protected]>
--- stgit/commands/imprt.py | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 45 insertions(+), 1 deletions(-)
stgit/commands/imprt.py +45 -1
@@ -87,7 +87,10 @@ options = [make_option('-m', '--mail', make_option('--commname', help = 'use COMMNAME as the committer name'), make_option('--commemail', - help = 'use COMMEMAIL as the committer e-mail') + help = 'use COMMEMAIL as the committer e-mail'), + make_option('--tarfile', + help = 'import a series from a tar archive', + action = "store_true"), ] + make_sign_options()
@@ -287,6 +290,45 @@ def __import_url(url, options): urllib.urlretrieve(url, filename) __import_file(filename, options) +def __import_tarfile(tar, options): + """Import patch series from a tar archive + """ + import tarfile + import tempfile + + if not tarfile.is_tarfile(tar): + raise CmdException, "%s is not a tarfile!" % tar + + + t = tarfile.open(tar, 'r') + names = t.getnames() + + # verify paths in the tarfile are safe + for n in names: + if n.startswith('/'): + raise CmdException, "Absolute path found in %s" % tar + if n.startswith("../"): + raise CmdException, "Relative path found in %s" % tar + + # find the series file + seriesfile = ''; + for m in names: + if m.endswith('/series') or m == 'series': + seriesfile = m + break + if seriesfile == '': + raise CmdException, "no series file found in %s" % tar + + # unpack into a tmp dir + tmpdir = tempfile.mkdtemp('.stg') + t.extractall(tmpdir) + + # apply the series + __import_series(os.path.join(tmpdir, seriesfile), options) + + # cleanup the tmpdir + os.system('rm -rf %s' % tmpdir) + def func(parser, options, args): """Import a GNU diff file as a new patch """
@@ -308,6 +350,8 @@ def func(parser, options, args): __import_mbox(filename, options) elif options.url: __import_url(filename, options) + elif options.tarfile: + __import_tarfile(filename, options) else: __import_file(filename, options)
Quoted 2 lines · Clark Williams
Attached is my first cut at adding the ability to import a patch series by specifying the tarball.
Thanks!
Quoted 2 lines
No Karl, I haven't developed a test for it (yet). I wanted to see what you guys thought first :)
I don't see a problem with it, and if you took the time to code it there is obviously at least one user (I have no idea how common patch series tarballs are). I do have some comments below, but nothing that would prevent you from writing a test or two right away. ;-)
stgit/commands/imprt.py options = [make_option('-m', '--mail', · line 93
Quoted 3 lines
stgit/commands/imprt.py +3
-0+ make_option('--tarfile', + help = 'import a series from a tar archive', + action = "store_true"),
As I hint below, you might want to autodetect tarballs with --series instead, since a tarball is just a tarred series directory.
stgit/commands/imprt.py def __import_url(url, options): · line 311
Quoted 2 lines
stgit/commands/imprt.py +2
-0+ if n.startswith("../"): + raise CmdException, "Relative path found in %s" % tar
I guess any occurrence of /../ in the middle of n should be caught as well? Or can't that happen?
By the way, is the separator always '/' in tarfile? Or should you use os.sep? (There is also os.pardir which you could use instead of '..', but that might be overdoing it a little ...)
Quoted 1 line
+ raise CmdException, "no series file found in %s" % tar
Perhaps "no 'series' file ...", to make it clear what the name should be?
stgit/commands/imprt.py def __import_url(url, options): · line 327
Quoted 6 lines
stgit/commands/imprt.py +6
-0+ # unpack into a tmp dir + tmpdir = tempfile.mkdtemp('.stg') + t.extractall(tmpdir) + + # apply the series + __import_series(os.path.join(tmpdir, seriesfile), options)
Hmm. It seems like such a waste to go via the file system here, when tarfile has such nice file extraction methods.
What you could do is something like this:
1. Make two small classes with the same interface, one backed by a tarfile and one backed by a directory, that have two methods:
get_series() and get_file(filename). Both methods return file-like objects (created by either open() or tarfile.extractfile()).
2. Change __import_series() to use objects of this class rather than a directory directly -- starting with creating an instance of one or the other depending on tarfile.is_tarfile(). This will involve teaching __import_file to accept a file-like object instead of just a file name, but that's a one-liner.
3. Drop the --tarfile flag, since you've just taught the --series flag to handle tarballs!
That said, if you don't feel like doing it the hard way, I won't insist. The way you coded it is in no way bad (in particular, you chose the right function to create a temp dir).
stgit/commands/imprt.py def __import_url(url, options): · line 330
Quoted 2 lines
stgit/commands/imprt.py +2
-0+ # cleanup the tmpdir + os.system('rm -rf %s' % tmpdir)
Aaah! My eyes! My _eyes_!!!!!
Seriously, though, you'd want to use something like shutil.rmtree here.
Signature · 2 lines
Karl Hasselström, [email protected] www.treskal.com/kalle
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 7 lines · Karl Hasselström
Quoted 2 lines · Clark Williams
Attached is my first cut at adding the ability to import a patch series by specifying the tarball.
Thanks!
Quoted 8 lines
Quoted 2 lines
No Karl, I haven't developed a test for it (yet). I wanted to see what you guys thought first :)
I don't see a problem with it, and if you took the time to code it there is obviously at least one user (I have no idea how common patch series tarballs are). I do have some comments below, but nothing that would prevent you from writing a test or two right away. ;-)
Patch series tarballs are quite common from people who use quilt (e.g. many of the kernel -rt series developers). My biggest problem (now that I can directly import them) is to see if I can ease StGit's patch import rules a bit, since quilt accepts pretty much anything as long as there's a diff in there somewhere. I bomb out regularly importing the -rt series using StGit, because some people don't put complete email addresses in their patches.
As to the test, I'll get right on that...:)
Quoted 6 lines
stgit/commands/imprt.py options = [make_option('-m', '--mail', · line 93
Quoted 3 lines
stgit/commands/imprt.py +3
-0+ make_option('--tarfile', + help = 'import a series from a tar archive', + action = "store_true"),As I hint below, you might want to autodetect tarballs with --series instead, since a tarball is just a tarred series directory.
Yeah I thought about that, as well as auto-detecting it in the file case. I'll look into that a bit more.
Quoted 7 lines
stgit/commands/imprt.py def __import_url(url, options): · line 311
Quoted 3 lines
stgit/commands/imprt.py +2
-0+ if n.startswith("../"): + raise CmdException, "Relative path found in %s" % tarI guess any occurrence of /../ in the middle of n should be caught as well? Or can't that happen?
Hence the "would you guys look at this?". Yeah, I need to detect sneaky stuff like that too.
Quoted 3 lines
By the way, is the separator always '/' in tarfile? Or should you use os.sep? (There is also os.pardir which you could use instead of '..', but that might be overdoing it a little ...)
I doubt there are many Windows-generated tarballs out there (except for the Cygwin case; I believe they use '/'), but I shouldn't be so Unix-centric. I'll work on cleaning it up.
I did consider adding Zipfile support as well, but didn't get a very good match-up between tar functionality and zip functionality. Maybe later...
Quoted 6 lines
Quoted 2 lines
+ raise CmdException, "no series file found in %s" % tar
Perhaps "no 'series' file ...", to make it clear what the name should be?
Yeah, that makes sense.
Quoted 24 lines
stgit/commands/imprt.py def __import_url(url, options): · line 327
Quoted 6 lines
stgit/commands/imprt.py +6
-0+ # unpack into a tmp dir + tmpdir = tempfile.mkdtemp('.stg') + t.extractall(tmpdir) + + # apply the series + __import_series(os.path.join(tmpdir, seriesfile), options)Hmm. It seems like such a waste to go via the file system here, when tarfile has such nice file extraction methods.
What you could do is something like this:
1. Make two small classes with the same interface, one backed by a tarfile and one backed by a directory, that have two methods:
get_series()andget_file(filename). Both methods return file-like objects (created by eitheropen()ortarfile.extractfile()).2. Change
__import_series()to use objects of this class rather than a directory directly -- starting with creating an instance of one or the other depending ontarfile.is_tarfile(). This will involve teaching__import_fileto accept a file-like object instead of just a file name, but that's a one-liner.
Quoted 6 lines
3. Drop the --tarfile flag, since you've just taught the --series flag to handle tarballs!
That said, if you don't feel like doing it the hard way, I won't insist. The way you coded it is in no way bad (in particular, you chose the right function to create a temp dir).
I did consider pulling directly from the tarball. I'll look into it.
Quoted 9 lines
stgit/commands/imprt.py def __import_url(url, options): · line 330
Quoted 3 lines
stgit/commands/imprt.py +2
-0+ # cleanup the tmpdir + os.system('rm -rf %s' % tmpdir)Aaah! My eyes! My _eyes_!!!!!
Seriously, though, you'd want to use something like shutil.rmtree here.
Man, I could not for the life of me remember which module had that in it. To be fair I wasn't up at work with my Python Essential Reference, which would have pointed me directly at it, but I would have thought I could have gotten there through the Python docs. Sigh...
You can dock my StGit pay for the visit to the eye doctor :)
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iEYEARECAAYFAkjFatgACgkQqA4JVb61b9fMRQCeLfK0zhPNEq3t5M4HW+vbRtaG VhgAn0rtszqVLbd1bz12MS0b/3r0OkT2
=gkf1
-----END PGP SIGNATURE-----
Quoted 2 lines · Clark Williams
Patch series tarballs are quite common from people who use quilt (e.g. many of the kernel -rt series developers).
Ah.
Quoted 5 lines
My biggest problem (now that I can directly import them) is to see if I can ease StGit's patch import rules a bit, since quilt accepts pretty much anything as long as there's a diff in there somewhere. I bomb out regularly importing the -rt series using StGit, because some people don't put complete email addresses in their patches.
Yes, that would be a welcome addition.
Quoted 1 line
As to the test, I'll get right on that...:)
Swell!
Quoted 9 lines
Quoted 3 lines · Karl Hasselström
By the way, is the separator always '/' in tarfile? Or should you use os.sep? (There is also os.pardir which you could use instead of '..', but that might be overdoing it a little ...)
I doubt there are many Windows-generated tarballs out there (except for the Cygwin case; I believe they use '/'), but I shouldn't be so Unix-centric. I'll work on cleaning it up.
Well, it's no big deal, really. Just thought I'd mention it.
Quoted 3 lines
I did consider adding Zipfile support as well, but didn't get a very good match-up between tar functionality and zip functionality. Maybe later...
I had a quick look at the zipfile module, and it looks like it too could easily be wrapped in a small class like I suggested in point (1).
Quoted 1 line
I did consider pulling directly from the tarball. I'll look into it.
Just don't let my suggestions take all the fun out of contributing ... only do it my way if you really think it's better.
Quoted 10 lines
Quoted 9 lines · Clark Williams
stgit/commands/imprt.py def __import_url(url, options): · line 330
Quoted 3 lines
stgit/commands/imprt.py +2
-0+ # cleanup the tmpdir + os.system('rm -rf %s' % tmpdir)Aaah! My eyes! My _eyes_!!!!!
Seriously, though, you'd want to use something like shutil.rmtree here.
Quoted 7 lines
Man, I could not for the life of me remember which module had that in it. To be fair I wasn't up at work with my Python Essential Reference, which would have pointed me directly at it, but I would have thought I could have gotten there through the Python docs. Sigh...
You can dock my StGit pay for the visit to the eye doctor :)
:-)
Signature · 2 lines
Karl Hasselström, [email protected] www.treskal.com/kalle
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Clark" == Clark Williams <[email protected]> writes:
Clark> [...] is to see if I can ease StGit's patch
Clark> import rules a bit, since quilt accepts pretty much anything as Clark> long as there's a diff in there somewhere. I bomb out regularly Clark> importing the -rt series using StGit, because some people don't Clark> put complete email addresses in their patches.
Two things that would be great would be:
- to be able to import patches with "-p0" (people not using git often sends such patches)
- to be able to find where the patch should be applied; I sometimes receive patches for GCC directory "gcc/ada/", diffed from there, and if StGit could see that the patch only makes sense there and not at the top-level it would be great as well
Sam
Signature · 1 line
Samuel Tardieu -- [email protected] -- http://www.rfc1149.net/
Quoted 4 lines · Samuel Tardieu
Two things that would be great would be:
- to be able to import patches with "-p0" (people not using git often sends such patches)
This should be trivial to implement, since git-apply (pardon the dash) has a -p flag with precisely this meaning.
Quoted 4 lines
- to be able to find where the patch should be applied; I sometimes receive patches for GCC directory "gcc/ada/", diffed from there, and if StGit could see that the patch only makes sense there and not at the top-level it would be great as well
I don't believe git-apply can do this (please correct me if I'm wrong), and the right way to teach StGit to do it would arguably be to teach it to git-apply and then make StGit use it. It'd be _possible_ to do it directly in StGit, but it wouldn't be quite the right level, and git users wouldn't benefit.
Signature · 2 lines
Karl Hasselström, [email protected] www.treskal.com/kalle
Quoted 9 lines · Karl Hasselström
Quoted 4 lines · Samuel Tardieu
- to be able to find where the patch should be applied; I sometimes receive patches for GCC directory "gcc/ada/", diffed from there, and if StGit could see that the patch only makes sense there and not at the top-level it would be great as well
I don't believe git-apply can do this (please correct me if I'm wrong)
It does have a --directory flag, but that requires the user to specify the path manually.
Signature · 2 lines
Karl Hasselström, [email protected] www.treskal.com/kalle
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 8 lines · Samuel Tardieu
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Clark" == Clark Williams <[email protected]> writes:
Clark> [...] is to see if I can ease StGit's patch
Clark> import rules a bit, since quilt accepts pretty much anything as Clark> long as there's a diff in there somewhere. I bomb out regularly Clark> importing the -rt series using StGit, because some people don't Clark> put complete email addresses in their patches.
Quoted 4 lines
Two things that would be great would be:
- to be able to import patches with "-p0" (people not using git often sends such patches)
I'm not sure how easy this is going to be. It looks like the patch is applied with 'git --apply' from the file stgit/git.py:apply_patch(). The default '-p' value is 1, so we'd have to figure out how to pass the 0 along and then get it into the apply_patch() function.
Quoted 6 lines
- to be able to find where the patch should be applied; I sometimes receive patches for GCC directory "gcc/ada/", diffed from there, and if StGit could see that the patch only makes sense there and not at the top-level it would be great as well
Zowie, I thought I only had to worry about folks sending patches with incomplete information. So you get patches to the ada compiler that are rooted in gcc/ada (e.g. patch in tarball says "./ChangeLog", instead of gcc/ada/ChangeLog) rather than at a top level? Only way I could see to deal with that would be to try and pass in the appropriate prefix from the command line.
My current plans are to clean up the first cut at the tarfile logic, then write a test to keep Karl happy, then try to come up with a way to deal with importing patches that don't have complete email addresses, no descriptions, etc. Once I get through that, I'll see if we can deal with weirdly rooted patch series.
Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iEYEARECAAYFAkjKZ0sACgkQqA4JVb61b9eg2ACffDv+FXsL1NifMvxr1tbO2c3s Hc4AoJPb/RZJrpqT6QybeZrj8rNFJg1y
=ccj/
-----END PGP SIGNATURE-----
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Clark" == Clark Williams <[email protected]> writes:
Clark> Zowie, I thought I only had to worry about folks sending Clark> patches with incomplete information. So you get patches to the Clark> ada compiler that are rooted in gcc/ada (e.g. patch in tarball Clark> says "./ChangeLog", instead of gcc/ada/ChangeLog) rather than Clark> at a top level? Only way I could see to deal with that would be Clark> to try and pass in the appropriate prefix from the command Clark> line.
Yes, passing the prefix and strip levels would be fine.
Sam
Signature · 1 line
Samuel Tardieu -- [email protected] -- http://www.rfc1149.net/
Quoted 1 line · Clark Williams
then write a test to keep Karl happy,
Technically, you write the test to make sure that your new feature works as intended and won't break in the future. But since that's rather a mouthful, I guess "Karl" will do as an acronym. ;-)
Quoted 4 lines
then try to come up with a way to deal with importing patches that don't have complete email addresses, no descriptions, etc. Once I get through that, I'll see if we can deal with weirdly rooted patch series.
Nice.
Signature · 2 lines
Karl Hasselström, [email protected] www.treskal.com/kalle