git-mv is not able to handle big directories

When moving directory with large number of files git-mv says:

Can't exec "git-update-index": Argument list too long at /usr/local/bin/git-mv line 193.
git-update-index failed to add new names with code -1

This patch fixes this by building list of files with limited len (currently 5000) and executing git-update-index few times until all files will be processed. I don't know how to determinate limit of command line but 5000 seems safe enougth to me.

@@ -185,13 +185,36 @@
 }
 	
 my $rc;
-if (scalar @changedfiles >0) {
-	$rc = system("git-update-index","--",@changedfiles);
+while (scalar @changedfiles >0) {
+	my @toHandle = ();
+	my $len = 0;
+	while ($len < 5000 && scalar(@changedfiles) >0) {
+		my $f = pop(@changedfiles);
+		$len += length($f) + 1;
+		push(@toHandle, $f);
+	}
+	$rc = system("git-update-index","--",@toHandle);
 	die "git-update-index failed to update changed files with code $?\n" if $rc;
 }
-if (scalar @addedfiles >0) {
-	$rc = system("git-update-index","--add","--",@addedfiles);
+while (scalar @addedfiles >0) {
+	my @toHandle = ();
+	my $len = 0;
+	while ($len < 5000 && scalar(@addedfiles) >0) {
+		my $f = pop(@addedfiles);
+		$len += length($f) + 1;
+		push(@toHandle, $f);
+	}
+	$rc = system("git-update-index","--add","--",@toHandle);
 	die "git-update-index failed to add new names with code $?\n" if $rc;
 }
-$rc = system("git-update-index","--remove","--",@deletedfiles);
-die "git-update-index failed to remove old names with code $?\n" if $rc;
+while (scalar @deletedfiles > 0) {
+	my @toHandle = ();
+	my $len = 0;
+	while ($len < 5000 && scalar(@deletedfiles) >0) {
+		my $f = pop(@deletedfiles);
+		$len += length($f) + 1;
+		push(@toHandle, $f);
+	}
+	$rc = system("git-update-index","--remove","--",@toHandle);
+	die "git-update-index failed to remove old names with code $?\n" if $rc;
+}

Two comments.

(1) the argument limit is enforced by the operating system in bytes (including environment size unfortunately) so we might want to count bytes not number of paths. I heard GNU xargs uses 131072 as the default limit.

(2) I wonder if we can detect this particular failure case and

    then fall back on splitting the arguments dynamically, maybe
    something like this:

	sub xargs_system {
        	my ($cmd, @args) = @_;
                my $rc = system(@$cmd, @args);
                if ($rc == 'argument list too long error') {
                	my (@args0) = splice(@args, 0, @args/2);
			$rc = xargs_system($cmd, @args0);
                        return $c if ($rc);
                        return xargs_system($cmd, @args);
		}
		return $rc;
	}

    and:

	$rc = xargs_system([qw(git-update-index --)], @changedfiles);
	$rc = xargs_system([qw(git-update-index --add --)], @addedfiles);
	...

I have found one error during directory movig: If I move directory with one file somewhere in it this script will try to add target directory instead of file. Commenting lines starting from 190 solve this error. But I don't understand what is the logic behind this case ? Why do target directory checked instead of target file ? Should we replace $dst my $destfiles[0] ?

at line 190 in git-mv:

    if (scalar @srcfiles == 1) {
	if ($overwritten{$dst} ==1) {
	    push @changedfiles, $dst;
	} else {
	    push @addedfiles, $dst;
	}
    }
    else {
	push @addedfiles, @dstfiles;
    }

$src/$dst pairs match the arguments given on the command line of git-mv, And git-mv (as was git-rename) is able to move directory trees around.

It is simplier to keep the granularity at directories, as multiple moves of directories can not overlap each other. If you do it at file level, you have to remote and create directories yourself.

Of course, for git it is done at file level (that are the @srcfiles/@dstfiles arrays), but this is quite trivial as git does not work with directories, but only with files.

Yes, that was part of my patch.

Josef

Are you saying this setup
foodir/somefile.c <--file
newdir/ <--directory

with this command
git-mv foodir/ newdir

tries to create
newdir/foodir/somefile.c <-- directory

or does it create
newdir/somefile.c <-- file

?

It should create
newdir/foodir/somefile.c <-- file

Otherwise it's misbehaving.

Try running it with the -v switch to make it shout out loud what it's trying to do, and then paste the output here.

This is broken. It only checks if there's just one source-file regardless of whether or not it resided in a subdirectory. I'm not exactly fluent in perl so I can't submit a patch, but the src option needs to be directory aware, traverse all source directories and then move the files axing everything but the bottom-most dirname to the destination directory.

Any takers?

Andreas Ericsson                   [email protected]
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

It is calling git-update-index --add newdir but actiual file structire is correct:
newdir/somefile.c - is a file

I still does not understand what this part should do. I know perl enought to fix it but I don't understand the logic.

Yes.
For git-update-index we have to use the file inside the directory.

I just sent a patch for this.

Josef

(3) Even better, 'git-update-index -z --stdin'

	if (@changedfiles) {
            open my $oh, qw(|- git-update-index -z --stdin)
                    or die "oops";
            for (@changedfiles) {
                print $oh "$_\0";
	    }
	    close $oh;
	}



	JC "added too many features that myself cannot remember" Hamano

Junio> open my $oh, qw(|- git-update-index -z --stdin)
Junio> or die "oops";

This is Perl 5.6 or later. Breaks on Perl 5.5, which is still in use in some places.

To be compatible with 5.5, you have to create a handle explicitly:

        require IO::Handle;
        my $oh = IO::Handle->new;
        open $oh, qw(...) ...;

That works all the way back to 5.4, which is the earliest Perl supported by the core team.

Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[email protected]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

I should have known better than posting Perl code to the list where a real guru is lurking and making afool of myself ;-).

Thanks for the advice. How much do we care about 5.5? IOW, is "in some places" wide enough to matter?

git-cvsimport, git-svnimport and git-shortlog share the same problem (svnimport declares that it wants 5.8). perl58delta.pod does say list form of open for pipes is new in that version. So what I wrote above requires 5.8 or better, perhaps? cvsimport uses that same structure. That is doubly Ouch.

No, you're not using the list form for pipes. I use that in git-graft-ripple:

	open(P,"-|","git-rev-list","--parents","--merge-order",$range)

For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's really only the commercial Unixes where requiring 5.8 would be annoying.

Randal, is my guess even remotely accurate?

Ryan Anderson
  sometimes Pug Majere

Oh, I didn't?

Sorry, I read after waking up and totally didn't see the "qw" in there.
You did, my bad.

Ryan> For the kernel, requiring 5.8 shouldn't be a big issue. I suspect it's Ryan> really only the commercial Unixes where requiring 5.8 would be annoying.

Ryan> Randal, is my guess even remotely accurate?

I'd say that 50% of the Perl-using population is at 5.6, with 25% each at 5.5 and 5.8. Those on 5.5 are generally unable to upgrade Perl for corporate reasons.

Targetting Perl 5.6 would assist broad acceptance of git for the typical commercial end user. Targetting 5.5 where possible would ensure practical success for everyone.

However, I have not seen the "target market" of git discussed yet (I came late to the party), so if support for 5.6 (or 5.5) is not chosen, it merely limits the market.

If you'd like, I can review all the Perl code with a tool that determines the minimum Perl version, and provide patches to bring the code to 5.5 level. But if it's likely that someone will say "this is not important to us", I'd rather not waste my time. :)

Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[email protected]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

There are a lot of Perl modules we use, so limiting it to 5.5 is probably a showstopper.

I'm very surprised you say that 5.6 is more prevalent than 5.8.

-hpa

You are overestimating the update-eagerness of the world.

SunOS 5.8 seems to come with 5.005_03
SunOS 5.9 seems to come with v5.6.1.

("Seems" because I am only 99% sure no-one around here has mucked with the perl in /usr/bin/.)

M.

H> There are a lot of Perl modules we use, so limiting it to 5.5 is H> probably a showstopper.

Hmm. I should look at that then. Most Perl Modules are 5.5 compatible, unless they've been written by naive people recently. :)

H> I'm very surprised you say that 5.6 is more prevalent than 5.8.

For individual early adopters, 5.8 is nearly universal. But a lot of my corporate clients upgrade *very* slowly, and are on 5.6 for now.

Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[email protected]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Well, I think the general target market[1] for Linux probably applies here, as well. On the other hand, the users limited to Perl 5.5 for corporate reasons are probably also prevented from installing new tools for general use, so the difference may not matter very much.

[1] - "World domination", of course.

Ryan Anderson
  sometimes Pug Majere

Servers with software installation restrictions are rarely used for development either and the server side part of git is all C. So long as that's true I don't think we need to bother very much with staying backwards compatible. It's nice if we manage it, but we shouldn't, imho, bend over backwards for the sake of perl 5.5 (or even 5.6).

Just my two öre. :)

Andreas Ericsson                   [email protected]
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

I think the target market is people who use whatever POSIXy systems for their own work. To them, git and its dependencies are merely tools to get their job done, and while they might be capable and even willing to match the version requirements if it is absolutely necessary, they'd rather spend their time on what they do best (i.e. write software or documentation or whatever, whose changes are to be managed by git), than installing or upgrading base software just to satisfy git. And I'd like to see their time spent on what they do best, instead of caring and feeding of git, too.

So in that sense, excluding 75% of people with "5.8 only" is not acceptable at all, and covering 75% with "5.6 or better" might be good enough, at least as a starter.

If it is not too much trouble, I'd appreciate it, at least the first half that can be done without taking too much of your time. Is it a mechanical "lint" like thing, that says "line 47 you have list form of pipe open -- that's 5.8 and better"?

Junio> I think the target market is people who use whatever POSIXy Junio> systems for their own work. To them, git and its dependencies Junio> are merely tools to get their job done, and while they might be Junio> capable and even willing to match the version requirements if it Junio> is absolutely necessary, they'd rather spend their time on what Junio> they do best (i.e. write software or documentation or whatever, Junio> whose changes are to be managed by git), than installing or Junio> upgrading base software just to satisfy git. And I'd like to Junio> see their time spent on what they do best, instead of caring and Junio> feeding of git, too.

Yes, this is why I keep pointing out GNU tools being used instead of POSIXy versions. :) Mostly because they break on either my Darwin laptop or my OpenBSD host server.

Junio> So in that sense, excluding 75% of people with "5.8 only" is not Junio> acceptable at all, and covering 75% with "5.6 or better" might Junio> be good enough, at least as a starter.

I think we're in agreement.

Junio> If it is not too much trouble, I'd appreciate it, at least the Junio> first half that can be done without taking too much of your time. Junio> Is it a mechanical "lint" like thing, that says "line 47 you Junio> have list form of pipe open -- that's 5.8 and better"?

The "perlver" tool installed with the "Perl::MinimumVersion" module from the CPAN will do about 80% of it. The other 20% are things that I can use my brain for.

Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[email protected]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Alexander> I have made this change. I also belive it will work on earlier perl but I Alexander> can't test this.

This patch looks good back to 5.5. I didn't execute it either, but the Perl version installed in my head gave it a thumbs-up. :)

Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[email protected]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!