git-send-email: add ~/.authinfo parsing

From: Michal Nazarewicz <[email protected]>

Make git-send-email read password from a ~/.authinfo file instead of requiring it to be stored in git configuration, passed as command line argument or typed in.

There are various other applications that use this file for authentication information so letting users use it for git-send-email is convinient. Furthermore, some users store their ~/.gitconfig file in a public repository and having to store password there makes it easy to publish the password.

Not to introduce any new dependencies, ~/.authinfo file is parsed only if Text::CSV Perl module is installed. If it's not, a notification is printed and the file is ignored.

---
 Documentation/git-send-email.txt | 15 +++++++--
 git-send-email.perl              | 69 +++++++++++++++++++++++++++++++++-------
 2 files changed, 70 insertions(+), 14 deletions(-)

index eeb561c..b83576e 100644
@@ -158,14 +158,25 @@ Sending
 --smtp-pass[=<password>]::
 	Password for SMTP-AUTH. The argument is optional: If no
 	argument is specified, then the empty string is used as
-	the password. Default is the value of 'sendemail.smtppass',
-	however '--smtp-pass' always overrides this value.
+	the password. Default is the value of 'sendemail.smtppass'
+	or value read from '~/.authinfo' file, however '--smtp-pass'
+	always overrides this value.
 +
 Furthermore, passwords need not be specified in configuration files
 or on the command line. If a username has been specified (with
 '--smtp-user' or a 'sendemail.smtpuser'), but no password has been
 specified (with '--smtp-pass' or 'sendemail.smtppass'), then the
 user is prompted for a password while the input is masked for privacy.
++
+The '~/.authinfo' file is read if Text::CSV Perl module is installed
+on the system; if it's missing, a notification message will be printed
+and the file ignored altogether.  The file should contain a line with
+the following format:
++
+  machine <domain> port <port> login <user> password <pass>
++
+Contrary to other tools, 'git-send-email' does not support symbolic
+port names like 'imap' thus `<port>` must be a number.
 
 --smtp-server=<host>::
 	If set, specifies the outgoing SMTP server to use (e.g.

1.8.1

Makes one wonder why .authinfo and not .netrc;

http://www.gnu.org/software/emacs/manual/html_node/auth/Help-for-users.html

phrases it amusingly:

“Netrc” files are usually called .authinfo or .netr nowadays .authinfo seems to be more popular and the auth-source library encourages this confusion by accepting both

Either way it still encourages a plaintext password to be on disk, which may not be what we want, even though it may be slight if not really much of an improvement. Again the Help-for-users has this amusing bit:

You could just say (but we don't recommend it, we're just showing that it's possible)

password mypassword

to use the same password everywhere. Again, DO NOT DO THIS or you will be pwned as the kids say.

From: Michal Nazarewicz <[email protected]>

Make git-send-email read password from a ~/.authinfo or a ~/.netrc file instead of requiring it to be stored in git configuration, passed as command line argument or typed in.

There are various other applications that use this file for authentication information so letting users use it for git-send-email is convinient. Furthermore, some users store their ~/.gitconfig file in a public repository and having to store password there makes it easy to publish the password.

---
 Documentation/git-send-email.txt |  34 +++++++++--
 git-send-email.perl              | 124 +++++++++++++++++++++++++++++++++++----
 2 files changed, 140 insertions(+), 18 deletions(-)

Fine… Let's parse both. ;)

Well… Users store passwords on disks in a lot of places. I wager that most have mail clients configured not to ask for password but instead store it on hard drive. I don't see that changing any time soon, so at least we can try and minimise number of places where a password is stored.

I was worried about spaces in password. CVS should handle such case nicely, whereas simple split won't. Nonetheless, I guess that in the end this is not likely enough to add the dependency.

Parsing /etc/services added.

index eeb561c..ee20714 100644
@@ -158,14 +158,36 @@ Sending
 --smtp-pass[=<password>]::
 	Password for SMTP-AUTH. The argument is optional: If no
 	argument is specified, then the empty string is used as
-	the password. Default is the value of 'sendemail.smtppass',
-	however '--smtp-pass' always overrides this value.
+	the password. Default is the value of 'sendemail.smtppass'
+	or value read from ~/.authinfo file, however '--smtp-pass'
+	always overrides this value.
 +
-Furthermore, passwords need not be specified in configuration files
-or on the command line. If a username has been specified (with
+Furthermore, passwords need not be specified in configuration files or
+on the command line. If a username has been specified (with
 '--smtp-user' or a 'sendemail.smtpuser'), but no password has been
-specified (with '--smtp-pass' or 'sendemail.smtppass'), then the
-user is prompted for a password while the input is masked for privacy.
+specified (with '--smtp-pass', 'sendemail.smtppass' or via
+~/.authinfo file), then the user is prompted for a password while
+the input is masked for privacy.
++
+The ~/.authinfo file should contain a line with the following
+format:
++
+  machine <domain> port <port> login <user> password <pass>
++
+Each pair (expect for `password <pass>`) can be omitted which will
+skip matching of the given value.  Lines are interpreted in order and
+password from the first line that matches will be used.  `<port>` can
+be either an integer or a symbolic name.  In the latter case, it is
+looked up in `/etc/services` file (if it exists).  For instance, you
+can put
++
+  machine example.com login testuser port ssmtp password smtppassword
+  machine example.com login testuser            password testpassword
++
+if you want to use `smtppassword` for authenticating to a service at
+port 465 (SSMTP) and `testpassword` for all other services.  As shown
+in the example, `<port>` can use   If ~/.authinfo file is
+missing, 'git-send-email' will also try ~/.netrc file.
 
 --smtp-server=<host>::
 	If set, specifies the outgoing SMTP server to use (e.g.

1.8.1

But .netrc/.authinfo format separates its entries with SP, HT, or LF. An entry begins with "machine <remote-hostname>" token pair.

split(/\s+/) will not work for an entry that span multiple lines but CSV will not help, either.

Is it bad to use Net::Netrc instead? This looks like exactly the use case that module was written for, no?

Hmph. I would have expected to see getservbyname.

From: Michal Nazarewicz <[email protected]>

Make git-send-email read password from a ~/.authinfo or ~/.netrc file instead of requiring it to be stored in git configuration, passed as command line argument or typed in.

There are various other applications that use this file for authentication information so letting users use it for git-send-email is convinient. Furthermore, some users store their ~/.gitconfig file in a public repository and having to store password there makes it easy to publish the password.

---
 Documentation/git-send-email.txt | 47 +++++++++++++++++---
 git-send-email.perl              | 93 ++++++++++++++++++++++++++++++++++------
 2 files changed, 122 insertions(+), 18 deletions(-)

I don't think that's the case. For one, Net::Netrc does not seem to process port number.

There is a Text::Authinfo module but it just uses Text::CSV.

I can change the code to use Net::Netrc, but I dunno if that's really the best option, since I feel people would expect parsing to be somehow compatible with
<http://www.gnu.org/software/emacs/manual/html_node/gnus/NNTP.html>
rather than the original .netrc file format.

Ha! Even better. :]

index eeb561c..ac020d1 100644
@@ -158,14 +158,49 @@ Sending
 --smtp-pass[=<password>]::
 	Password for SMTP-AUTH. The argument is optional: If no
 	argument is specified, then the empty string is used as
-	the password. Default is the value of 'sendemail.smtppass',
-	however '--smtp-pass' always overrides this value.
+	the password. Default is the value of 'sendemail.smtppass'
+	or value read from ~/.authinfo file, however '--smtp-pass'
+	always overrides this value.
 +
-Furthermore, passwords need not be specified in configuration files
-or on the command line. If a username has been specified (with
+Furthermore, passwords need not be specified in configuration files or
+on the command line. If a username has been specified (with
 '--smtp-user' or a 'sendemail.smtpuser'), but no password has been
-specified (with '--smtp-pass' or 'sendemail.smtppass'), then the
-user is prompted for a password while the input is masked for privacy.
+specified (with '--smtp-pass', 'sendemail.smtppass' or via
+~/.authinfo file), then the user is prompted for a password while
+the input is masked for privacy.
++
+The ~/.authinfo file should contain a line with the following
+format:
++
+  machine <domain> port <port> login <user> password <pass>
++
+Instead of `machine <domain>` pair a `default` token can be used
+instead in which case all domains will match.  Similarly, `port
+<port>` and `login <user>` pairs can be omitted in which case matching
+of the given value will be skipped.  `<port>` can be either an integer
+or a symbolic name.  Lines are interpreted in order and password from
+the first line that matches will be used.  For instance, one may end
+up with:
++
+  machine example.com login jane port ssmtp password smtppassword
+  machine example.com login jane            password janepassword
+  default             login janedoe         password doepassword
++
+if she wants to use `smtppassword` for authenticating as `jane` to
+a service at example.com:465 (SSMTP), `janepassword` for all other
+services at example.com; and `doepassword` when authonticating as
+`janedoe` to any service.  If ~/.authinfo file is missing,
+'git-send-email' will also try ~/.netrc file (even though parsing is
+not fully compatible with ftp's .netrc file format).
++
+Note that you should never make ~/.authinfo file world-readable.  To
+help guarantee that, you might want to create the file with the
+following command:
++
+  ( umask 077; cat >~/.authinfo <<EOF
+  ... file contents ...
+  EOF
+  )
 
 --smtp-server=<host>::
 	If set, specifies the outgoing SMTP server to use (e.g.

1.8.1

Thanks for pushing back (I wish more contributors did so when I suggest nonsense ;-)); you are right that both canned modules are lacking.

Will queue. Thanks.

I do not mind a .netrc or .authinfo parser, because while those formats do have security problems, they are standard files that may already be in use. So as long as we are not encouraging their use, I do not see a problem in supporting them (and we already do the same with curl's netrc support).

But it would probably make sense for send-email to support the existing git-credential subsystem, so that it can take advantage of secure system-specific storage. And that is where we should be pointing new users. I think contrib/mw-to-git even has credential support written in perl, so it would just need to be factored out to Git.pm.

-Peff

Yeah, that sounds like a neat idea.

On Wed, 30 Jan 2013 07:57:29 -0800 Junio C Hamano <[email protected]> wrote:

JCH> Yeah, that sounds like a neat idea.

Jeff, is there a way for git-credential to currently support authinfo/netrc parsing? I assume that's the right way, instead of using Michal's proposal to parse internally?

I'd like to add that, plus support for the 'string' and "string" formats, and authinfo.gpg decoding through GPG. I'd write it in Perl, if there's a choice.

Ted

Yes, you could write a credential helper that understands netrc and friends; git talks to the helpers over a socket, so there is no problem with writing it in Perl. See Documentation/technical/api-credentials.txt for an overview, or the sample implementation in credential-store.c for a simple example.

-Peff

On Thu, 31 Jan 2013 14:38:45 -0500 Jeff King <[email protected]> wrote:

JK> Yes, you could write a credential helper that understands netrc and JK> friends; git talks to the helpers over a socket, so there is no problem JK> with writing it in Perl. See Documentation/technical/api-credentials.txt JK> for an overview, or the sample implementation in credential-store.c for a JK> simple example.

I wrote a Perl credential helper for netrc parsing which is pretty robust, has built-in docs with -h, and doesn't depend on external modules. The netrc parser regex was stolen from Net::Netrc.

It will by default use ~/.authinfo.gpg, ~/.netrc.gpg, ~/.authinfo, and ~/.netrc (whichever is found first) and this can be overridden with -f.

If the file name ends with ".gpg", it will run "gpg --decrypt FILE" and use the output. So non-interactively, that could hang if GPG was waiting for input. Does Git handle that, or should I check for a TTY?

Take a look at the proposed patch and let me know if it's usable, if you need a formal copyright assignment, etc.

Thanks
Ted

commit 3d28bc2a610ebcc988eba5443d82d0ded92c24bc

Author: Ted Zlatanov <[email protected]>
Date:   Sat Feb 2 06:42:13 2013 -0500

Add contrib/credentials/netrc with GPG support

new file mode 100755
index 0000000..92fc306
@@ -0,0 +1,242 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use Data::Dumper;
+
+use Getopt::Long;
+use File::Basename;
+
+my $VERSION = "0.1";
+
+my %options = (
+               help => 0,
+               debug => 0,
+
+               # identical token maps, e.g. host -> host, will be inserted later
+               tmap => {
+                        port => 'protocol',
+                        machine => 'host',
+                        path => 'path',
+                        login => 'username',
+                        user => 'username',
+                        password => 'password',
+                       }
+              );
+
+foreach my $v (values %{$options{tmap}})
+{
+ $options{tmap}->{$v} = $v;
+}
+
+foreach my $suffix ('.gpg', '')
+{
+ foreach my $base (qw/authinfo netrc/)
+ {
+  my $file = glob("~/.$base$suffix");
+  next unless (defined $file && -f $file);
+  $options{file} = $file ;
+ }
+}
+
+Getopt::Long::Configure("bundling");
+
+# TODO: maybe allow the token map $options{tmap} to be configurable.
+GetOptions(\%options,
+           "help|h",
+           "debug|d",
+           "file|f=s",
+          );
+
+if ($options{help})
+{
+ my $shortname = basename($0);
+ $shortname =~ s/git-credential-//;
+
+ print <<EOHIPPUS;
+
+$0 [-f AUTHFILE] [-d] get
+
+Version $VERSION by tzz\@lifelogs.com.  License: any use is OK.
+
+Options:
+  -f AUTHFILE: specify a netrc-style file
+  -d: turn on debugging
+
+To enable (note that Git will prepend "git-credential-" to the helper
+name and look for it in the path):
+
+  git config credential.helper '$shortname -f AUTHFILE'
+
+And if you want lots of debugging info:
+
+  git config credential.helper '$shortname -f AUTHFILE -d'
+
+Only "get" mode is supported by this credential helper.  It opens
+AUTHFILE and looks for entries that match the requested search
+criteria:
+
+ 'port|protocol':
+   The protocol that will be used (e.g., https). (protocol=X)
+
+ 'machine|host':
+   The remote hostname for a network credential. (host=X)
+
+ 'path':
+   The path with which the credential will be used. (path=X)
+
+ 'login|user|username':
+   The credential’s username, if we already have one. (username=X)
+
+Thus, when we get "protocol=https\nusername=tzz", this credential
+helper will look for lines in AUTHFILE that match
+
+port https login tzz
+
+OR
+
+protocol https login tzz
+
+OR... etc. acceptable tokens as listed above.  Any unknown tokens are
+simply ignored.
+
+Then, the helper will print out whatever tokens it got from the line,
+including "password" tokens, mapping e.g. "port" back to "protocol".
+
+The first matching line is used.  Tokens can be quoted as 'STRING' or
+"STRING".
+
+No caching is performed by this credential helper.
+
+EOHIPPUS
+
+ exit;
+}
+
+my $mode = shift @ARGV;
+
+# credentials may get 'get', 'store', or 'erase' as parameters but
+# only acknowledge 'get'
+die "Syntax: $0 [-f AUTHFILE] [-d] get" unless defined $mode;
+
+# only support 'get' mode
+exit unless $mode eq 'get';
+
+my $debug = $options{debug};
+my $file = $options{file};
+
+die "Sorry, you need to specify an existing netrc file (with or without a .gpg extension) with -f AUTHFILE"
+ unless defined $file;
+
+die "Sorry, the specified netrc $file is not accessible"
+ unless -f $file;
+
+if ($file =~ m/\.gpg$/)
+{
+ $file = "gpg --decrypt $file|";
+}
+
+my @data = load($file);
+chomp @data;
+
+die "Sorry, we could not load data from [$file]"
+ unless (scalar @data);
+
+# the query
+my %q;
+
+foreach my $v (values %{$options{tmap}})
+{
+ undef $q{$v};
+}
+
+while (<STDIN>)
+{
+ next unless m/([a-z]+)=(.+)/;
+
+ my ($token, $value) = ($1, $2);
+ die "Unknown search token $1" unless exists $q{$token};
+ $q{$token} = $value;
+}
+
+# build reverse token map
+my %rmap;
+foreach my $k (keys %{$options{tmap}})
+{
+ push @{$rmap{$options{tmap}->{$k}}}, $k;
+}
+
+# there are CPAN modules to do this better, but we want to avoid
+# dependencies and generally, complex netrc-style files are rare
+
+if ($debug)
+{
+ foreach (sort keys %q)
+ {
+  printf STDERR "searching for %s = %s\n",
+   $_, $q{$_} || '(any value)';
+ }
+}
+
+LINE: foreach my $line (@data)
+{
+
+ print STDERR "line [$line]\n" if $debug;
+ my @tok;
+ # gratefully stolen from Net::Netrc
+ while (length $line &&
+        $line =~ s/^("((?:[^"]+|\\.)*)"|((?:[^\\\s]+|\\.)*))\s*//)
+ {
+  (my $tok = $+) =~ s/\\(.)/$1/g;
+  push(@tok, $tok);
+ }
+
+ my %tokens;
+ while (@tok)
+ {
+  my ($k, $v) = (shift @tok, shift @tok);
+  next unless defined $v;
+  next unless exists $options{tmap}->{$k};
+  $tokens{$options{tmap}->{$k}} = $v;
+ }
+
+ foreach my $check (sort keys %q)
+ {
+  if (exists $tokens{$check} && defined $q{$check})
+  {
+   print STDERR "comparing [$tokens{$check}] to [$q{$check}] in line [$line]\n" if $debug;
+   next LINE unless $tokens{$check} eq $q{$check};
+  }
+  else
+  {
+   print STDERR "we could not find [$check] but it's OK\n" if $debug;
+  }
+ }
+
+ print STDERR "line has passed all the search checks\n" if $debug;
+ foreach my $token (sort keys %rmap)
+ {
+  print STDERR "looking for useful token $token\n" if $debug;
+  next unless exists $tokens{$token}; # did we match?
+
+  foreach my $rctoken (@{$rmap{$token}})
+  {
+   next if defined $q{$rctoken};           # don't re-print given tokens
+  }
+
+  print STDERR "FOUND: $token=$tokens{$token}\n" if $debug;
+  printf "%s=%s\n", $token, $tokens{$token};
+ }
+
+ last;
+}
+
+sub load
+{
+ my $file = shift;
+ # this supports pipes too
+ my $io = new IO::File($file) or die "Could not open $file: $!\n";
+
+ return <$io>;                          # whole file
+}

Cool, thanks for working on this.

No, git does not do anything special with respect to credential helpers and ttys (nor should it, since one use of helpers is to get credentials when there is no tty). I think it is GPG's problem to deal with, though. We will invoke it, and it is up to it to decide whether it can acquire the passphrase or not (either through the tty, or possibly from gpg-agent). So it would be wrong to do the tty check yourself.

I haven't tested GPG, but I assume it properly tries to read from /dev/tty and not stdin. Your helper's stdio is connected to git and speaking the credential-helper protocol, so GPG reading from stdin would either steal your input (if run before you read it), or just get EOF (if you have read all of the pipe content already). If GPG isn't well behaved, it may be worth redirecting its stdin from /dev/null as a safety measure.

Overall looks sane to me, though my knowledge of .netrc is not especially good. Usually we try to send patches inline in the email (i.e., as generated by git-format-patch), and include a "Signed-off-by" line indicating that content is released to the project; see Documentation/SubmittingPatches.

I don't see it used here. Leftover from debugging?

Cute, I haven't seen that one before.

             load('-|', qw(gpg --decrypt), $file) :
             load('<', $file);

Probably not that interesting a corner case, but this means we die on an empty .netrc, whereas it might be more sensible for it to behave as "no match".

For the same reason, it might be worth silently exiting when we don't find a .netrc (or any of its variants). That lets people who share their dot-files across machines configure git globally, even if they don't necessarily have a netrc on every machine.

  my %q = map { $_ => undef } values(%{$options{tmap}});

The rest of it looks fine to me. I don't think any of my comments are show-stoppers. Tests would be nice, but integrating contrib/ stuff with the test harness is kind of a pain.

-Peff

---
 contrib/credential/netrc/git-credential-netrc |  242 +++++++++++++++++++++++++
 1 files changed, 242 insertions(+), 0 deletions(-)
 create mode 100755 contrib/credential/netrc/git-credential-netrc

new file mode 100755
index 0000000..92fc306
@@ -0,0 +1,242 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+
+use Data::Dumper;
+
+use Getopt::Long;
+use File::Basename;
+
+my $VERSION = "0.1";
+
+my %options = (
+               help => 0,
+               debug => 0,
+
+               # identical token maps, e.g. host -> host, will be inserted later
+               tmap => {
+                        port => 'protocol',
+                        machine => 'host',
+                        path => 'path',
+                        login => 'username',
+                        user => 'username',
+                        password => 'password',
+                       }
+              );
+
+foreach my $v (values %{$options{tmap}})
+{
+ $options{tmap}->{$v} = $v;
+}
+
+foreach my $suffix ('.gpg', '')
+{
+ foreach my $base (qw/authinfo netrc/)
+ {
+  my $file = glob("~/.$base$suffix");
+  next unless (defined $file && -f $file);
+  $options{file} = $file ;
+ }
+}
+
+Getopt::Long::Configure("bundling");
+
+# TODO: maybe allow the token map $options{tmap} to be configurable.
+GetOptions(\%options,
+           "help|h",
+           "debug|d",
+           "file|f=s",
+          );
+
+if ($options{help})
+{
+ my $shortname = basename($0);
+ $shortname =~ s/git-credential-//;
+
+ print <<EOHIPPUS;
+
+$0 [-f AUTHFILE] [-d] get
+
+Version $VERSION by tzz\@lifelogs.com.  License: any use is OK.
+
+Options:
+  -f AUTHFILE: specify a netrc-style file
+  -d: turn on debugging
+
+To enable (note that Git will prepend "git-credential-" to the helper
+name and look for it in the path):
+
+  git config credential.helper '$shortname -f AUTHFILE'
+
+And if you want lots of debugging info:
+
+  git config credential.helper '$shortname -f AUTHFILE -d'
+
+Only "get" mode is supported by this credential helper.  It opens
+AUTHFILE and looks for entries that match the requested search
+criteria:
+
+ 'port|protocol':
+   The protocol that will be used (e.g., https). (protocol=X)
+
+ 'machine|host':
+   The remote hostname for a network credential. (host=X)
+
+ 'path':
+   The path with which the credential will be used. (path=X)
+
+ 'login|user|username':
+   The credential’s username, if we already have one. (username=X)
+
+Thus, when we get "protocol=https\nusername=tzz", this credential
+helper will look for lines in AUTHFILE that match
+
+port https login tzz
+
+OR
+
+protocol https login tzz
+
+OR... etc. acceptable tokens as listed above.  Any unknown tokens are
+simply ignored.
+
+Then, the helper will print out whatever tokens it got from the line,
+including "password" tokens, mapping e.g. "port" back to "protocol".
+
+The first matching line is used.  Tokens can be quoted as 'STRING' or
+"STRING".
+
+No caching is performed by this credential helper.
+
+EOHIPPUS
+
+ exit;
+}
+
+my $mode = shift @ARGV;
+
+# credentials may get 'get', 'store', or 'erase' as parameters but
+# only acknowledge 'get'
+die "Syntax: $0 [-f AUTHFILE] [-d] get" unless defined $mode;
+
+# only support 'get' mode
+exit unless $mode eq 'get';
+
+my $debug = $options{debug};
+my $file = $options{file};
+
+die "Sorry, you need to specify an existing netrc file (with or without a .gpg extension) with -f AUTHFILE"
+ unless defined $file;
+
+die "Sorry, the specified netrc $file is not accessible"
+ unless -f $file;
+
+if ($file =~ m/\.gpg$/)
+{
+ $file = "gpg --decrypt $file|";
+}
+
+my @data = load($file);
+chomp @data;
+
+die "Sorry, we could not load data from [$file]"
+ unless (scalar @data);
+
+# the query
+my %q;
+
+foreach my $v (values %{$options{tmap}})
+{
+ undef $q{$v};
+}
+
+while (<STDIN>)
+{
+ next unless m/([a-z]+)=(.+)/;
+
+ my ($token, $value) = ($1, $2);
+ die "Unknown search token $1" unless exists $q{$token};
+ $q{$token} = $value;
+}
+
+# build reverse token map
+my %rmap;
+foreach my $k (keys %{$options{tmap}})
+{
+ push @{$rmap{$options{tmap}->{$k}}}, $k;
+}
+
+# there are CPAN modules to do this better, but we want to avoid
+# dependencies and generally, complex netrc-style files are rare
+
+if ($debug)
+{
+ foreach (sort keys %q)
+ {
+  printf STDERR "searching for %s = %s\n",
+   $_, $q{$_} || '(any value)';
+ }
+}
+
+LINE: foreach my $line (@data)
+{
+
+ print STDERR "line [$line]\n" if $debug;
+ my @tok;
+ # gratefully stolen from Net::Netrc
+ while (length $line &&
+        $line =~ s/^("((?:[^"]+|\\.)*)"|((?:[^\\\s]+|\\.)*))\s*//)
+ {
+  (my $tok = $+) =~ s/\\(.)/$1/g;
+  push(@tok, $tok);
+ }
+
+ my %tokens;
+ while (@tok)
+ {
+  my ($k, $v) = (shift @tok, shift @tok);
+  next unless defined $v;
+  next unless exists $options{tmap}->{$k};
+  $tokens{$options{tmap}->{$k}} = $v;
+ }
+
+ foreach my $check (sort keys %q)
+ {
+  if (exists $tokens{$check} && defined $q{$check})
+  {
+   print STDERR "comparing [$tokens{$check}] to [$q{$check}] in line [$line]\n" if $debug;
+   next LINE unless $tokens{$check} eq $q{$check};
+  }
+  else
+  {
+   print STDERR "we could not find [$check] but it's OK\n" if $debug;
+  }
+ }
+
+ print STDERR "line has passed all the search checks\n" if $debug;
+ foreach my $token (sort keys %rmap)
+ {
+  print STDERR "looking for useful token $token\n" if $debug;
+  next unless exists $tokens{$token}; # did we match?
+
+  foreach my $rctoken (@{$rmap{$token}})
+  {
+   next if defined $q{$rctoken};           # don't re-print given tokens
+  }
+
+  print STDERR "FOUND: $token=$tokens{$token}\n" if $debug;
+  printf "%s=%s\n", $token, $tokens{$token};
+ }
+
+ last;
+}
+
+sub load
+{
+ my $file = shift;
+ # this supports pipes too
+ my $io = new IO::File($file) or die "Could not open $file: $!\n";
+
+ return <$io>;                          # whole file
+}

1.7.9.rc2

[administrivia: I would really wish you didn't put "Mail-copies-to:
never" above].

Do we really need to refer readers to Wikipedia or something to learn about extinct equid ungulates ;-)?

On Mon, 04 Feb 2013 09:24:03 -0800 Junio C Hamano <[email protected]> wrote:

JCH> [administrivia: I would really wish you didn't put "Mail-copies-to:
JCH> never" above].

I normally post through GMane and don't need the extra CC on any list I read. I'll make an effort to remove that header here, and apologize for the inconvenience.

JCH> Please follow the styles of existing Perl scripts, e.g. indent with JCH> tab, etc. Style requests are not optional; it is a prerequisite to JCH> make the patch readable and reviewable.

Sorry, I didn't realize contrib/ stuff was under the same rules. I will attempt to make my contributions fit the project's requirements.

It would help if the requirements were codified as the fairly standard Emacs file-local variables, so I can just put them in the Perl code or in .dir-locals.el in the source tree. At least for Perl I'd like that, and it could be nice for the Emacs users who write C too.

Would you like me to propose that as a patch?

Either way, I guessed that these settings are what you want as far as tabs and indentation (I use cperl-mode but perl-mode is the same):

# -*- mode: cperl; tab-width: 8; cperl-indent-level: 4; indent-tabs-mode: t; -*-

...plus hanging braces and avoiding one-line blocks. I hope that's right.

JCH> Do we really need to refer readers to Wikipedia or something to JCH> learn about extinct equid ungulates ;-)?

I think the marker's name is irrelevant, and hope you are OK with leaving it.

Since the change is a pretty big reformatting, should I squash my 3 commits plus the reformatting commit into one patch, or keep them as a series?

I am appending the script in its current form so you can review it and tell me if there's anything else I should add or change in the formatting.

Thanks
Ted

#!/usr/bin/perl
# -*- mode: cperl; tab-width: 8; cperl-indent-level: 4; indent-tabs-mode: t; -*-

use strict;
use warnings;

use Getopt::Long;
use File::Basename;

my $VERSION = "0.1";

my %options = (

               help => 0,
               debug => 0,

               # identical token maps, e.g. host -> host, will be inserted later
               tmap => {
                        port => 'protocol',
                        machine => 'host',
                        path => 'path',
                        login => 'username',
                        user => 'username',
                        password => 'password',
                       }
              );

# map each credential protocol token to itself on the netrc side
$options{tmap}->{$_} = $_ foreach my $v (values %{$options{tmap}});

foreach my $suffix ('.gpg', '') {
    foreach my $base (qw/authinfo netrc/) {
	my $file = glob("~/.$base$suffix");
	next unless (defined $file && -f $file);
	$options{file} = $file ;
    }
}

Getopt::Long::Configure("bundling");

# TODO: maybe allow the token map $options{tmap} to be configurable. GetOptions(\%options,

           "help|h",
           "debug|d",
           "file|f=s",
          );

if ($options{help}) {
    my $shortname = basename($0);
    $shortname =~ s/git-credential-//;

    print <<EOHIPPUS;

$0 [-f AUTHFILE] [-d] get

Version $VERSION by tzz\@lifelogs.com. License: BSD.

Options:
-f AUTHFILE: specify a netrc-style file
-d: turn on debugging

To enable (note that Git will prepend "git-credential-" to the helper name and look for it in the path):

git config credential.helper '$shortname -f AUTHFILE'

And if you want lots of debugging info:

git config credential.helper '$shortname -f AUTHFILE -d'

Only "get" mode is supported by this credential helper. It opens AUTHFILE and looks for entries that match the requested search criteria:

'port|protocol':
The protocol that will be used (e.g., https). (protocol=X)

'machine|host':
The remote hostname for a network credential. (host=X)

'path':
The path with which the credential will be used. (path=X)

'login|user|username':
The credential’s username, if we already have one. (username=X)

Thus, when we get "protocol=https\nusername=tzz", this credential helper will look for lines in AUTHFILE that match

port https login tzz

OR

protocol https login tzz

OR... etc. acceptable tokens as listed above. Any unknown tokens are simply ignored.

Then, the helper will print out whatever tokens it got from the line, including "password" tokens, mapping e.g. "port" back to "protocol".

The first matching line is used. Tokens can be quoted as 'STRING' or "STRING".

No caching is performed by this credential helper.

EOHIPPUS

    exit;
}

my $mode = shift @ARGV;

# credentials may get 'get', 'store', or 'erase' as parameters but
# only acknowledge 'get'
die "Syntax: $0 [-f AUTHFILE] [-d] get" unless defined $mode;

# only support 'get' mode
exit unless $mode eq 'get';

my $debug = $options{debug};
my $file = $options{file};

die "Sorry, you need to specify an existing netrc file (with or without a .gpg extension) with -f AUTHFILE"
unless defined $file;

unless (-f $file) {
    print STDERR "Sorry, the specified netrc $file is not accessible\n" if $debug;
    exit 0;
}

my @data;

if ($file =~ m/\.gpg$/) {
    @data = load('-|', qw(gpg --decrypt), $file)
}
else {
    @data = load('<', $file);
}

chomp @data;

unless (scalar @data) {
    print STDERR "Sorry, we could not load data from [$file]\n" if $debug;
    exit;
}

# the query: start with every token with no value
my %q = map { $_ => undef } values(%{$options{tmap}});

while (<STDIN>) {
    next unless m/([^=]+)=(.+)/;

    my ($token, $value) = ($1, $2);
    die "Unknown search token $1" unless exists $q{$token};
    $q{$token} = $value;
}

# build reverse token map
my %rmap;
foreach my $k (keys %{$options{tmap}}) {
    push @{$rmap{$options{tmap}->{$k}}}, $k;
}

# there are CPAN modules to do this better, but we want to avoid
# dependencies and generally, complex netrc-style files are rare

if ($debug) {
    printf STDERR "searching for %s = %s\n", $_, $q{$_} || '(any value)'
     foreach sort keys %q;
}

LINE: foreach my $line (@data) {

    print STDERR "line [$line]\n" if $debug;
    my @tok;
    # gratefully stolen from Net::Netrc
    while (length $line &&
	   $line =~ s/^("((?:[^"]+|\\.)*)"|((?:[^\\\s]+|\\.)*))\s*//) {
	(my $tok = $+) =~ s/\\(.)/$1/g;
	push(@tok, $tok);
    }

    # skip blank lines, comments, etc.
    next LINE unless scalar @tok;

    my %tokens;
    while (@tok) {
	my ($k, $v) = (shift @tok, shift @tok);
	next unless defined $v;
	next unless exists $options{tmap}->{$k};
	$tokens{$options{tmap}->{$k}} = $v;
    }

    foreach my $check (sort keys %q) {
	if (exists $tokens{$check} && defined $q{$check}) {
	    print STDERR "comparing [$tokens{$check}] to [$q{$check}] in line [$line]\n" if $debug;
	    next LINE unless $tokens{$check} eq $q{$check};
	}
	else {
	    print STDERR "we could not find [$check] but it's OK\n" if $debug;
	}
    }

    print STDERR "line has passed all the search checks\n" if $debug;
 TOKEN:
    foreach my $token (sort keys %rmap) {
	print STDERR "looking for useful token $token\n" if $debug;
	next unless exists $tokens{$token}; # did we match?

	foreach my $rctoken (@{$rmap{$token}}) {
	    next TOKEN if defined $q{$rctoken};           # don't re-print given tokens
	}

	print STDERR "FOUND: $token=$tokens{$token}\n" if $debug;
	printf "%s=%s\n", $token, $tokens{$token};
    }

    last;
}

sub load {

    # this supports pipes too
    my $io = new IO::File(@_) or die "Could not open [@_]: $!\n";
    return <$io>;                          # whole file
}

I had a feeling that this may start out from contrib/ but will soon prove to be fairly important to be part of the Git proper.

I thought that we tend to avoid Emacs/Vim formatting cruft left in the file. Do we have any in existing file outside contrib/?

Indent is done with a Tab and indent level is 8 places (check add--interactive.perl and imitate it, perhaps?).

Thanks.

On Mon, 04 Feb 2013 11:06:16 -0800 Junio C Hamano <[email protected]> wrote:

JCH> I had a feeling that this may start out from contrib/ but will soon JCH> prove to be fairly important to be part of the Git proper.

Cool!

JCH> I thought that we tend to avoid Emacs/Vim formatting cruft left in JCH> the file. Do we have any in existing file outside contrib/?

No, but it's a nice way to express the settings so no one is guessing what the project prefers. At least for me it's not an issue anymore, since I understand your criteria better now, so let me know if you want me to express it in the CodingGuidelines, in a dir-locals.el file, or somewhere else.

JCH> Indent is done with a Tab and indent level is 8 places (check add--interactive.perl JCH> and imitate it, perhaps?).

Yup, got it. My mistake on the size-4 indents.

I found this helpful, at least while I was indenting, for anyone else who might want to indent Perl appropriately to imitate existing Perl code:

# -*- mode: cperl; tab-width: 8; cperl-indent-level: 8; indent-tabs-mode: t; -*-

I'll resubmit now.

Thanks
Ted

Historically we treated this from CodingGuidelines a sufficient clue:

As for more concrete guidelines, just imitate the existing code (this is a good guideline, no matter which project you are contributing to). It is always preferable to match the _local_ convention. New code added to git suite is expected to match the overall style of existing code. Modifications to existing code is expected to match the style the surrounding code already uses (even if it doesn't match the overall style of existing code).

but over time people wanted more specific guidelines and added language specific style guides there. We have sections that cover C, shell and Python, and I do not think adding Perl would not hurt.

On Mon, 04 Feb 2013 15:10:45 -0800 Junio C Hamano <[email protected]> wrote:

JCH> Ted Zlatanov <[email protected]> writes:
JCH> I thought that we tend to avoid Emacs/Vim formatting cruft left in JCH> the file. Do we have any in existing file outside contrib/?

JCH> Historically we treated this from CodingGuidelines a sufficient JCH> clue:

JCH>     As for more concrete guidelines, just imitate the existing code
JCH>     (this is a good guideline, no matter which project you are
JCH>     contributing to). It is always preferable to match the _local_
JCH>     convention. New code added to git suite is expected to match
JCH>     the overall style of existing code. Modifications to existing
JCH>     code is expected to match the style the surrounding code already
JCH>     uses (even if it doesn't match the overall style of existing code).

JCH> but over time people wanted more specific guidelines and added JCH> language specific style guides there. We have sections that cover JCH> C, shell and Python, and I do not think adding Perl would not hurt.

The following is how I have interpreted the Perl guidelines. I hope it's OK to include Emacs-specific settings; they make it much easier to reindent code to be acceptable.

I will submit as a patch if you think this is reasonable at all.

The org-mode markers around the code are just a suggestion.

For Perl 5 programs:

- Most of the C guidelines above apply.

- We try to support Perl 5.8 and later ("use Perl 5.008").

- use strict and use warnings are strongly preferred.

- As in C (see above), we avoid using braces unnecessarily (but Perl forces braces around if/unless/else/foreach blocks, so this is not always possible).

- Don't abuse statement modifiers (unless $youmust).

- We try to avoid assignments inside if().

- Learn and use Git.pm if you need that functionality.

- For Emacs, it's useful to put the following in GIT_CHECKOUT/.dir-locals.el, assuming you use cperl-mode:

#+begin_src lisp
((nil . ((indent-tabs-mode . t)

              (tab-width . 8)
              (fill-column . 80)))

(cperl-mode . ((cperl-indent-level . 8)
(cperl-extra-newline-before-brace . nil) (cperl-merge-trailing-else . t))))
#+end_src

Is it ever (as opposed to "not always") possible to omit braces?

It sounds as if we encourage the use of statement modifiers, which certainly is not what I want to see.

You probably would want to mention that opening braces for "if/else/elsif" do not sit on their own line, and closing braces for them will be followed the next "else/elseif" on the same line instead, but that is part of "most of the C guidelines above apply" so it may be redundant.

It does not make a useful guidance to leave $youmust part unspecified.

Incidentally, your sentence is a good example of where use of statement modifiers is appropriate: $youmust is rarely true.

In general:

	... do something ...
	do_this() unless (condition);
        ... do something else ...

is easier to follow the flow of the logic than

	... do something ...
	unless (condition) {
		do_this();
	}
        ... do something else ...

*only* when condition is extremely rare, iow, when do_this() is expected to be almost always called.

On Wed, 06 Feb 2013 08:29:30 -0800 Junio C Hamano <[email protected]> wrote:

JCH> Is it ever (as opposed to "not always") possible to omit braces?

Oh yes! Not that I recommend it, and I'm not even going to touch on Perl Golf :)

JCH> It sounds as if we encourage the use of statement modifiers, which JCH> certainly is not what I want to see.

Yup. I think I captured that in the patch, but please feel free to revise it after applying or throw it back to me.

JCH> You probably would want to mention that opening braces for JCH> "if/else/elsif" do not sit on their own line, and closing braces for JCH> them will be followed the next "else/elseif" on the same line JCH> instead, but that is part of "most of the C guidelines above apply" JCH> so it may be redundant.

OK; done.

JCH> It does not make a useful guidance to leave $youmust part JCH> unspecified.

JCH> Incidentally, your sentence is a good example of where use of JCH> statement modifiers is appropriate: $youmust is rarely true.

I was trying to be funny, honestly. But OK; reworded.

Ted

I'd suggest to just drop that "try to write without braces" entirely.

It wasn't a useful guidance, but it _was_ funny.

On Wed, 06 Feb 2013 10:16:21 -0800 Junio C Hamano <[email protected]> wrote:

JCH> I'd suggest to just drop that "try to write without braces" entirely.

OK, I'll do it on the reroll, or you can just make the change directly.

I agree it was not going anywhere :)

Ted

index 951d74c..857f4e2 100644
@@ -187,10 +187,6 @@ For Perl 5 programs:
 
  - use strict and use warnings are strongly preferred.
 
- - As in C (see above), we avoid using braces unnecessarily (but Perl forces
-   braces around if/unless/else/foreach blocks, so this is not always possible).
-   At least make sure braces do not sit on their own line, like with C.
-
  - Don't abuse statement modifiers--they are discouraged.  But in general:
 
        ... do something ...

I think you are wrong. Can you provide an example?

Larry specifically wanted to avoid the "dangling else" problem that C suffers from, and made it so that blocks are mandatory. The only exception is statement modifiers, which are not only allowed to omit the braces but also the parens on the condition.

Yves

perl -Mre=debug -e "/just|another|perl|hacker/"

On Wed, 6 Feb 2013 19:25:43 +0100 demerphq <[email protected]> wrote:

JCH> Is it ever (as opposed to "not always") possible to omit braces?

d> I think you are wrong. Can you provide an example?

d> Larry specifically wanted to avoid the "dangling else" problem that C d> suffers from, and made it so that blocks are mandatory. The only d> exception is statement modifiers, which are not only allowed to omit d> the braces but also the parens on the condition.

Oh, perhaps I didn't state it correctly. You can avoid braces, but not if you want to use if/elsif/else/unless/etc. which require them:

condition && do_this();
condition || do_this();
condition ? do_this() : do_that();

(and others I can't recall right now)

But my point was only that it's always possible to get around these artificial restrictions; it's more important to ask for legible sensible code. Sorry if that was unclear!

Ted

Ah ok. Right, at a low level:

if (condition) { do_this() }

is identical to

condition && do_this();

IOW, Perl allows logical operators to act as control flow statements.

I hope your document include something that says that using logical operators as control flow statements should be used sparingly, and generally should be restricted to low precedence operators and should never involve more than one operator.

Yves

perl -Mre=debug -e "/just|another|perl|hacker/"

On Wed, 6 Feb 2013 19:44:16 +0100 demerphq <[email protected]> wrote:

d> Ah ok. Right, at a low level:

d> if (condition) { do_this() }

d> is identical to

d> condition && do_this();

d> IOW, Perl allows logical operators to act as control flow statements.

d> I hope your document include something that says that using logical d> operators as control flow statements should be used sparingly, and d> generally should be restricted to low precedence operators and should d> never involve more than one operator.

I'd stay away from wording it so tightly, but instead just say

"Make your code readable and sensible, and don't try to be clever."

But this is good C and shell advice too, so I'd put it under "General Guidelines" and leave it for Junio to decide if it's appropriate.

Ted

Sounds sensible.

Update the coding guidelines for Perl 5.

---
Changes since PATCHv1:
- removed brace guidelines
- add "don't try to be clever" at beginning

 Documentation/CodingGuidelines |   42 ++++++++++++++++++++++++++++++++++++++++
 1 files changed, 42 insertions(+), 0 deletions(-)

index 1d7de5f..166c141 100644
@@ -18,6 +18,8 @@ code.  For Git in general, three rough rules are:
    judgement call, the decision based more on real world
    constraints people face than what the paper standard says.
 
+For any programming language below, make your code readable and sensible, and
+don't try to be clever.
 
 As for more concrete guidelines, just imitate the existing code
 (this is a good guideline, no matter which project you are

@@ -179,6 +181,46 @@ For C programs:
  - Use Git's gettext wrappers to make the user interface
    translatable. See "Marking strings for translation" in po/README.
 
+For Perl 5 programs:
+
+ - Most of the C guidelines above apply.
+
+ - We try to support Perl 5.8 and later ("use Perl 5.008").
+
+ - use strict and use warnings are strongly preferred.
+
+ - Don't abuse statement modifiers--they are discouraged.  But in general:
+
+	... do something ...
+	do_this() unless (condition);
+        ... do something else ...
+
+   should be used instead of
+
+	... do something ...
+	unless (condition) {
+		do_this();
+	}
+        ... do something else ...
+
+   *only* when when the condition is so rare that do_this() will be called
+   almost always.
+
+ - We try to avoid assignments inside if().
+
+ - Learn and use Git.pm if you need that functionality.
+
+ - For Emacs, it's useful to put the following in
+   GIT_CHECKOUT/.dir-locals.el, assuming you use cperl-mode:
+
+    ;; note the first part is useful for C editing, too
+    ((nil . ((indent-tabs-mode . t)
+                  (tab-width . 8)
+                  (fill-column . 80)))
+     (cperl-mode . ((cperl-indent-level . 8)
+                    (cperl-extra-newline-before-brace . nil)
+                    (cperl-merge-trailing-else . t))))
+
 Writing Documentation:
 
  Every user-visible change should be reflected in the documentation.

1.7.9.rc2

Update the coding guidelines for Perl 5.

---
 Documentation/CodingGuidelines |   44 ++++++++++++++++++++++++++++++++++++++++
 1 files changed, 44 insertions(+), 0 deletions(-)

index 1d7de5f..951d74c 100644
@@ -179,6 +179,50 @@ For C programs:
  - Use Git's gettext wrappers to make the user interface
    translatable. See "Marking strings for translation" in po/README.
 
+For Perl 5 programs:
+
+ - Most of the C guidelines above apply.
+
+ - We try to support Perl 5.8 and later ("use Perl 5.008").
+
+ - use strict and use warnings are strongly preferred.
+
+ - As in C (see above), we avoid using braces unnecessarily (but Perl forces
+   braces around if/unless/else/foreach blocks, so this is not always possible).
+   At least make sure braces do not sit on their own line, like with C.
+
+ - Don't abuse statement modifiers--they are discouraged.  But in general:
+
+	... do something ...
+	do_this() unless (condition);
+        ... do something else ...
+
+   should be used instead of
+
+	... do something ...
+	unless (condition) {
+		do_this();
+	}
+        ... do something else ...
+
+   *only* when when the condition is so rare that do_this() will be called
+   almost always.
+
+ - We try to avoid assignments inside if().
+
+ - Learn and use Git.pm if you need that functionality.
+
+ - For Emacs, it's useful to put the following in
+   GIT_CHECKOUT/.dir-locals.el, assuming you use cperl-mode:
+
+    ;; note the first part is useful for C editing, too
+    ((nil . ((indent-tabs-mode . t)
+                  (tab-width . 8)
+                  (fill-column . 80)))
+     (cperl-mode . ((cperl-indent-level . 8)
+                    (cperl-extra-newline-before-brace . nil)
+                    (cperl-merge-trailing-else . t))))
+
 Writing Documentation:
 
  Every user-visible change should be reflected in the documentation.

1.7.9.rc2

Only in a statement modifier.

As you mention below statement modifiers have their place. For instance

next if $whatever;

Is considered preferable to

if ($whatever) {
next;
}

Similarly

open my $fh, ">", $filename
or die "Failed to open '$filename': $!";

Is considered preferable by most Perl programmers to:

my $fh;
if ( not open $fh, ">", $filename ) {
die "Failed to open '$filename': $!";
}

"unless" often leads to maintenance errors as the expression gets more complicated over time, more branches need to be added to the statement, etc. Basically people are bad at doing De Morgans law in their head.

if (not $condition) {
do_this();
}

Is much less error prone in terms of maintenance than

unless ($condition) {
do_this();
}

Similarly

do_this() if not $condition;

leads to less maintenance errors than

do_this() unless $condition;

So if you objective is maintainability I would just ban "unless" outright.

Cheers,
Yves

perl -Mre=debug -e "/just|another|perl|hacker/"

Yeah, and that is for the same reason. When you are trying to get a birds-eye view of the codeflow, the former makes it clear that "we do something, and then we open, and then we ...", without letting the error handling (which also is rare case) distract us.

That might also be true, but my comment was not an endorsement for (or suggestion against) use of unless. I was commenting on statement modifiers, which some people tend to overuse (or abuse) and make the resulting code harder to follow.

perldoc perlstyle has language which explains this well if you want to crib a description from somewhere.

That's also my point about unless. They tend to get abused and then lead to maint devs making errors, and people misunderstanding the code. The only time that unless IMO is "ok" (ish) is when it really is a very simple statement. As soon as it mentions more than one var it should be converted to an if. This applies even more so to the modifier form.

Yves

perl -Mre=debug -e "/just|another|perl|hacker/"

On Wed, 6 Feb 2013 18:45:56 +0100 demerphq <[email protected]> wrote:

d> So if you objective is maintainability I would just ban "unless" outright.

Please consider me opposed to such a ban.

Ted

Grr, sorry for the bad formatting. First time doing format-patch.

Ted

---
 contrib/credential/netrc/git-credential-netrc |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

index 92fc306..a47a223 100755
@@ -192,6 +192,9 @@ LINE: foreach my $line (@data)
   push(@tok, $tok);
  }
 
+ # skip blank lines, comments, etc.
+ next LINE unless scalar @tok;
+
  my %tokens;
  while (@tok)
  {

1.7.9.rc2

---
 contrib/credential/netrc/git-credential-netrc |   38 +++++++++++++------------
 1 files changed, 20 insertions(+), 18 deletions(-)

index a47a223..0e35918 100755
@@ -3,8 +3,6 @@
 use strict;
 use warnings;
 
-use Data::Dumper;
-
 use Getopt::Long;
 use File::Basename;
 

@@ -58,7 +56,7 @@ if ($options{help})
 
 $0 [-f AUTHFILE] [-d] get
 
-Version $VERSION by tzz\@lifelogs.com.  License: any use is OK.
+Version $VERSION by tzz\@lifelogs.com.  License: BSD.
 
 Options:
   -f AUTHFILE: specify a netrc-style file

@@ -129,31 +127,36 @@ my $file = $options{file};
 die "Sorry, you need to specify an existing netrc file (with or without a .gpg extension) with -f AUTHFILE"
  unless defined $file;
 
-die "Sorry, the specified netrc $file is not accessible"
- unless -f $file;
+unless (-f $file)
+{
+ print STDERR "Sorry, the specified netrc $file is not accessible\n" if $debug;
+ exit 0;
+}
 
+my @data;
 if ($file =~ m/\.gpg$/)
 {
- $file = "gpg --decrypt $file|";
+ @data = load('-|', qw(gpg --decrypt), $file)
+}
+else
+{
+ @data = load('<', $file);
 }
 
-my @data = load($file);
 chomp @data;
 
-die "Sorry, we could not load data from [$file]"
- unless (scalar @data);
-
-# the query
-my %q;
-
-foreach my $v (values %{$options{tmap}})
+unless (scalar @data)
 {
- undef $q{$v};
+ print STDERR "Sorry, we could not load data from [$file]\n" if $debug;
+ exit;
 }
 
+# the query: start with every token with no value
+my %q = map { $_ => undef } values(%{$options{tmap}});
+
 while (<STDIN>)
 {
- next unless m/([a-z]+)=(.+)/;
+ next unless m/([^=]+)=(.+)/;
 
  my ($token, $value) = ($1, $2);
  die "Unknown search token $1" unless exists $q{$token};

@@ -237,9 +240,8 @@ LINE: foreach my $line (@data)
 
 sub load
 {
- my $file = shift;
  # this supports pipes too
- my $io = new IO::File($file) or die "Could not open $file: $!\n";
+ my $io = new IO::File(@_) or die "Could not open [@_]: $!\n";
 
  return <$io>;                          # whole file
 }

1.7.9.rc2

Especially because this is an initial submission, please equash three patches into one, instead of sending three "here is my first attempt with many problems I know I do not want to be there", "one small improvement", "another one to fix remaining issues".

Otherwise you will waste reviewers' time, getting distracted by undesirable details they find in an earlier patch while reviewing, without realizing that some of them are fixed in a later one.

Thanks.

On Mon, 04 Feb 2013 09:27:46 -0800 Junio C Hamano <[email protected]> wrote:

JCH> Especially because this is an initial submission, please equash JCH> three patches into one, instead of sending three "here is my first JCH> attempt with many problems I know I do not want to be there", "one JCH> small improvement", "another one to fix remaining issues".

JCH> Otherwise you will waste reviewers' time, getting distracted by JCH> undesirable details they find in an earlier patch while reviewing, JCH> without realizing that some of them are fixed in a later one.

OK, thanks. I wasn't sure, since Jeff already reviewed it, if it was better to squash or not. Ignore this same question in my other reply to you, and thanks for your patience.

Ted

On Sun, 3 Feb 2013 14:41:49 -0500 Jeff King <[email protected]> wrote:

JK> No, git does not do anything special with respect to credential helpers JK> and ttys (nor should it, since one use of helpers is to get credentials JK> when there is no tty). I think it is GPG's problem to deal with, though. JK> We will invoke it, and it is up to it to decide whether it can acquire JK> the passphrase or not (either through the tty, or possibly from JK> gpg-agent). So it would be wrong to do the tty check yourself.

JK> I haven't tested GPG, but I assume it properly tries to read from JK> /dev/tty and not stdin. Your helper's stdio is connected to git and JK> speaking the credential-helper protocol, so GPG reading from stdin would JK> either steal your input (if run before you read it), or just get EOF (if JK> you have read all of the pipe content already). If GPG isn't well JK> behaved, it may be worth redirecting its stdin from /dev/null as a JK> safety measure.

In my testing GPG did the right thing, so I think this is OK.

JK> Overall looks sane to me, though my knowledge of .netrc is not JK> especially good. Usually we try to send patches inline in the email JK> (i.e., as generated by git-format-patch), and include a "Signed-off-by" JK> line indicating that content is released to the project; see JK> Documentation/SubmittingPatches.

OK, thanks. I will fire that off.

JK> I don't see it used here. Leftover from debugging?

It's part of my Perl new script skeleton, sorry.

JK> Cute, I haven't seen that one before.

Heh heh. I've had to explain that one in code review many times. "See, it's the precursor to the modern horse..."

JK> I don't know if we have a particular policy for items in contrib/, but JK> this license may be too vague. In particular, it does not explicitly JK> allow redistribution, which would make Junio shipping a release with it JK> a copyright violation.

JK> Any objection to just putting it under some well-known simple license JK> (GPL, BSD, or whatever)?

No, I didn't know what Git requires, and I'd like it to be the least restrictive, so BSD is OK. Stated in -h now.

JK> Does this need to quote $file, since the result will get passed to the JK> shell? It might be easier to just use the list form of open(), like:

JK> my @data = $file =~ /\.gpg$/ ?
JK> load('-|', qw(gpg --decrypt), $file) :
JK> load('<', $file);

JK> (and then obviously update load to just dump all of @_ to open()).

Yes, thanks. Done.

JK> Probably not that interesting a corner case, but this means we die on an JK> empty .netrc, whereas it might be more sensible for it to behave as "no JK> match".

JK> For the same reason, it might be worth silently exiting when we don't JK> find a .netrc (or any of its variants). That lets people who share their JK> dot-files across machines configure git globally, even if they don't JK> necessarily have a netrc on every machine.

OK; done.

JK> Just my personal style, but I find the intent more obvious with "map" (I JK> know some people find it unreadable, though):

JK> my %q = map { $_ => undef } values(%{$options{tmap}});

Yes, changed.

JK> We don't currently have any exotic tokens that this would not match, nor JK> do I plan to add them, but the credential documentation defines a valid JK> line as /^([^=]+)=(.+)/.

JK> It's also possible for the value to be empty, but I do not think JK> off-hand that current git will ever send such an empty value.

Yes, changed.

JK> The rest of it looks fine to me. I don't think any of my comments are JK> show-stoppers. Tests would be nice, but integrating contrib/ stuff with JK> the test harness is kind of a pain.

"I tested it on AIX, it works great!" :)

It's pretty easy to write a local Makefile with a test target, if you think it worthwhile.

Ted

As far as I understand, there could be a git-credential helper that reads ~/.authinfo and than git-send-email would just call “git credential fill”, right?

I've noticed though, that git-credential does not support port argument, which makes it slightly incompatible with ~/.authinfo.

Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +----<email/xmpp: [email protected]>--------------ooO--(_)--Ooo--

On Mon, 04 Feb 2013 17:33:58 +0100 Michal Nazarewicz <[email protected]> wrote:

MN> As far as I understand, there could be a git-credential helper that MN> reads ~/.authinfo and than git-send-email would just call “git MN> credential fill”, right?

MN> I've noticed though, that git-credential does not support port argument, MN> which makes it slightly incompatible with ~/.authinfo.

My proposed netrc credential helper does this :)

The token mapping I use:

port, protocol        => protocol
machine, host         => host
path                  => path

login, username, user => username
password => password

I think that's sensible.

Ted

Technically you can speak a particular protocol on an alternate port:

https://example.com:31337/repo.git

In this case, git will send you the host as:

example.com:31337

You might want to map this to "port" in .autoinfo separately if it's available.

-Peff

On Mon, 4 Feb 2013 15:10:40 -0500 Jeff King <[email protected]> wrote:

JK> Technically you can speak a particular protocol on an alternate port:

JK> https://example.com:31337/repo.git

JK> In this case, git will send you the host as:

JK> example.com:31337

JK> You might want to map this to "port" in .autoinfo separately if it's JK> available.

That would create the following possibilities:

* host example.com:31337, protocol https
* host example.com:31337, protocol unspecified
* host example.com, protocol https
* host example.com, protocol unspecified

How would you like each one to be handled? My preference would be to make the user say "host example.com:31337" in the netrc file (the current situation); that's what we do in Emacs and it lets applications request credentials for a logical service no matter what the port is.

It means that example.com credentials won't be used for example.com:31337. In practice, that has not been a problem for us.

Ted

Possibilities for .netrc, or for git? Git will always specify the protocol.

Yeah, I think that is a good thing. The credentials used for example.com:31337 are not necessarily the same as for the main site. It's less convenient, but a more secure default.

What I was more wondering (and I know very little about .netrc, so this might not be a possibility at all) is a line like:

host example.com port 5001 protocol https username foo password bar

To match git's representation on a token-by-token basis, you would have to either split out git's "host:port" pair, or combine the .netrc's representation to "example.com:5001".

-Peff

On Mon, 4 Feb 2013 15:59:11 -0500 Jeff King <[email protected]> wrote:

JK> On Mon, Feb 04, 2013 at 03:28:52PM -0500, Ted Zlatanov wrote:
JK> You might want to map this to "port" in .autoinfo separately if it's JK> available.

JK> Possibilities for .netrc, or for git? Git will always specify the JK> protocol.

Possibilities for the netrc data. How clever do we want to be with taking 31337 and mapping it to the "protocol"? My preference is to be very simple here.

JK> What I was more wondering (and I know very little about .netrc, so this JK> might not be a possibility at all) is a line like:

JK> host example.com port 5001 protocol https username foo password bar

JK> To match git's representation on a token-by-token basis, you would have JK> to either split out git's "host:port" pair, or combine the .netrc's JK> representation to "example.com:5001".

Currently, we map both the "port" and "protocol" netrc tokens to the credential helper protocol's "protocol". So this will have undefined results. To do what you specify could be pretty simple: we could do a preliminary scan of the tokens, looking for "host X port Y" where Y is an integer, and rewriting the host to be "X:Y". That would be clean and simple, unless the user breaks it with "host x:23 port 22". Let me know if you agree and I'll do.

Ted

I think simple is OK, as we can iterate on it as specific use-cases come up. The important thing is to make sure we err on the side of "does not match" and not "oops, we accidentally sent your plaintext credentials to the wrong server".

Yeah, I think that is simple and obvious. If the user is saying "host x:23 port 22", that is nonsensical.

-Peff

JK> Yeah, I think that is simple and obvious. If the user is saying "host JK> x:23 port 22", that is nonsensical.

OK; added.

Ted

I see a lot of rerolls on the credential helper front, but is there anybody working on hooking send-email to the credential framework?

Not answering the question, but git-remote-mediawiki supports the credential framework. It is written in perl, and the credential support is rather cleanly written and doesn't have dependencies on the wiki part, so the way to go for send-email is probably to libify the credential support in git-remote-mediawiki, and to use it in send-email.

Matthieu Moy
http://www-verimag.imag.fr/~moy/

On Wed, 06 Feb 2013 09:11:17 +0100 Matthieu Moy <[email protected]> wrote:

MM> Not answering the question, but git-remote-mediawiki supports the MM> credential framework. It is written in perl, and the credential support MM> is rather cleanly written and doesn't have dependencies on the wiki MM> part, so the way to go for send-email is probably to libify the MM> credential support in git-remote-mediawiki, and to use it in send-email.

I looked and that's indeed very useful. If it's put in a library, I'd use credential_read() and credential_write() in my netrc credential helper. But I would formalize it a little more about the token names and output, and I wouldn't necessarily die() on error. Maybe this can be merged with the netrc credential helper's read_credential_data_from_stdin() and print_credential_data()?

Let me know if you'd like me to libify this... I'm happy to leave it to Matthieu or Michal, or anyone else interested.

Ted

Can you elaborate on this? The idea of the Perl code was to mimick a call to the C API, keeping essentially the same names.

Sure, die()ing in a library is bad.

I don't know about the netrc credential helper, but I guess that's another layer. The git-remote-mediawiki code is the code to call the credential C API, that in turn may (or may not) call a credential helper.

I'd happily let you do the job, but I can help if needed. One thing to be careful about: git-remote-mediawiki is currently a standalone script, so it can be installed with a plain "cp git-remote-mediawiki $somewhere/". One consequence of libification is that it adds a dependency on the library (e.g. Git.pm). We should be carefull to keep it easy for the user to install it (e.g. some kind of "make install", or update the doc).

Matthieu Moy
http://www-verimag.imag.fr/~moy/

On Wed, 06 Feb 2013 16:10:12 +0100 Matthieu Moy <[email protected]> wrote:

MM> Ted Zlatanov <[email protected]> writes:
MM> [...] so the way to go for send-email is probably to libify the MM> credential support in git-remote-mediawiki, and to use it in send-email.

MM> Can you elaborate on this? The idea of the Perl code was to mimick a MM> call to the C API, keeping essentially the same names.

None of these are a big deal, and Michal said he's working on libifying this anyhow:

- making 'fill' a special operation is weird
- anchor the key regex to beginning of line (not strictly necessary)
- sort the output tokens (after 'url' is extracted) so the output is consistent and testable

MM> Sure, die()ing in a library is bad.

MM> I don't know about the netrc credential helper, but I guess that's MM> another layer. The git-remote-mediawiki code is the code to call the MM> credential C API, that in turn may (or may not) call a credential MM> helper.

Yup. But what you call "read" and "write" are, to the credential helper, "write" and "read" but it's the same protocol :) So maybe the names should be changed to reflect that, e.g. "query" and "response."

MM> One thing to be careful about: git-remote-mediawiki is currently a MM> standalone script, so it can be installed with a plain "cp MM> git-remote-mediawiki $somewhere/". One consequence of libification MM> is that it adds a dependency on the library (e.g. Git.pm). We should MM> be carefull to keep it easy for the user to install it (e.g. some MM> kind of "make install", or update the doc).

I don't know--it's up to the `git-remote-mediawiki' maintainers... But I think anywhere you have Git, you also have Git.pm, right? Maybe? But then you also have to look at whether Git.pm has the functionality you need... so I better go quiet :)

Ted

Is that true? As a user of the credential system, git-remote-mediawiki would want to "write" to git-credential, then "read" the response. As a helper, git-credential-netrc would want to "read" the query then "write" the response. The order is different, but the operations should be the same in both cases.

The big difference is that mediawiki would want an additional function to open a pipe to "git credential" and operate on that, whereas the helper will be reading/writing stdio.

-Peff

On Wed, 6 Feb 2013 16:57:24 -0500 Jeff King <[email protected]> wrote:

JK> On Wed, Feb 06, 2013 at 10:58:13AM -0500, Ted Zlatanov wrote:
MM> I don't know about the netrc credential helper, but I guess that's MM> another layer. The git-remote-mediawiki code is the code to call the MM> credential C API, that in turn may (or may not) call a credential MM> helper.

JK> Is that true? As a user of the credential system, git-remote-mediawiki JK> would want to "write" to git-credential, then "read" the response. As a JK> helper, git-credential-netrc would want to "read" the query then JK> "write" the response. The order is different, but the operations should JK> be the same in both cases.

Logically they are different steps (query and response), even though the data protocol is the same. But it's really not a big deal, I know what it means either way.

JK> The big difference is that mediawiki would want an additional function JK> to open a pipe to "git credential" and operate on that, whereas the JK> helper will be reading/writing stdio.

Yup.

Ted

Yes, but if you rename write() to query(), then on the helper side, you'll have to call query() to send the response, and response() to read the query. Much worse than keeping read/write.

Plus, read/write has already been used for a while in the C API, so I'd rather keep the same names for the Perl equivalent.

Matthieu Moy
http://www-verimag.imag.fr/~moy/

On Thu, 07 Feb 2013 08:08:59 +0100 Matthieu Moy <[email protected]> wrote:

MM> Plus, read/write has already been used for a while in the C API, so I'd MM> rather keep the same names for the Perl equivalent.

That makes perfect sense.

Ted

Well, 'fill' is the only operation that mutates the credential structure (i.e. the only one for which "git credential" emits an output to be parsed), so you don't have much choice.

Right. The greedyness of * ensures correction, but I like explicit anchors ^...$ too.

Why not, if you want to use the output of credential_write in tests. But credential_write is essentially used to talk to "git credential", so the important information is the content of the hash before credential_write and after credential_read. They are unordered, but consistent and testable.

I don't think that would be a better naming. Maybe "serialize" and "parse" would be better, but "query" would sound like it establishes the connection and possibly reads the response to me.

That is, me ;-).

Yes, but you have to find out where it is installed. Git's Makefile hardcodes the path to Git.pm at build time, inserting one line in the perl script:

use lib (split(/:/, $ENV{GITPERLLIB} || "$INSTLIBDIR"));

The same needs to be done for git-remote-mediawiki. As much as possible, I'd rather avoid copy-pasting from Git's Makefile, so this means extracting the perl part of Git's Makefile and make it available in contrib/.

I'll try a patch in this direction.

If git-remote-mediawiki is installed from Git's source, I think it's OK to assume that Git.pm will be up to date, but that would be even better if we can issue a clean error message when the functions to be called do not exist.

Matthieu Moy
http://www-verimag.imag.fr/~moy/

The very final goal is to be able to move painlessly (credential) code from git-remote-mediawiki to Git.pm, but then it's nice for the user to be able to say just "cd contrib/mw-to-git && make install" and let the Makefile set perl's library path just like other Git commands written in perl.

This series does this while trying to minimize code duplication, and to make it easy for future other tools in contrib to do the same.

Matthieu Moy (4):
Makefile: extract perl-related rules to make them available from other
dirs
perl.mak: introduce $(GIT_ROOT_DIR) to allow inclusion from other
directories
Makefile: factor common configuration in git-default-config.mak
git-remote-mediawiki: use Git's Makefile to build the script

 Makefile                                           | 108 +--------------------
 contrib/mw-to-git/.gitignore                       |   1 +
 contrib/mw-to-git/Makefile                         |  45 ++++++---
 ...-remote-mediawiki => git-remote-mediawiki.perl} |   0
 default-config.mak                                 |  61 ++++++++++++
 perl.mak                                           |  52 ++++++++++
 6 files changed, 145 insertions(+), 122 deletions(-)
 create mode 100644 contrib/mw-to-git/.gitignore
 rename contrib/mw-to-git/{git-remote-mediawiki => git-remote-mediawiki.perl} (100%)
 create mode 100644 default-config.mak
 create mode 100644 perl.mak

1.8.1.2.526.gf51a757

The final goal is to make it easy to write Git commands in perl in the contrib/ directory. It is currently possible to do so, but without the benefits of Git's Makefile: adapt first line with $(PERL_PATH), hardcode the path to Git.pm, ...

We make the perl-related part of the Makefile available from directories other than the toplevel so that:

* Developers can include it, to avoid code duplication

* Users can get a consistent behavior of "make install"

---
 Makefile | 46 +---------------------------------------------
 perl.mak | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 45 deletions(-)
 create mode 100644 perl.mak

index 731b6a8..f39d4a9 100644
@@ -573,14 +573,10 @@ BINDIR_PROGRAMS_NO_X += git-cvsserver
 ifndef SHELL_PATH
 	SHELL_PATH = /bin/sh
 endif
-ifndef PERL_PATH
-	PERL_PATH = /usr/bin/perl
-endif
 ifndef PYTHON_PATH
 	PYTHON_PATH = /usr/bin/python
 endif
 
-export PERL_PATH
 export PYTHON_PATH
 
 LIB_FILE = libgit.a

@@ -1441,10 +1437,6 @@ ifeq ($(TCLTK_PATH),)
 NO_TCLTK = NoThanks
 endif
 
-ifeq ($(PERL_PATH),)
-NO_PERL = NoThanks
-endif
-
 ifeq ($(PYTHON_PATH),)
 NO_PYTHON = NoThanks
 endif

@@ -1522,7 +1514,6 @@ prefix_SQ = $(subst ','\'',$(prefix))
 gitwebdir_SQ = $(subst ','\'',$(gitwebdir))
 
 SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
-PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
 PYTHON_PATH_SQ = $(subst ','\'',$(PYTHON_PATH))
 TCLTK_PATH_SQ = $(subst ','\'',$(TCLTK_PATH))
 DIFF_SQ = $(subst ','\'',$(DIFF))

@@ -1715,9 +1706,6 @@ $(SCRIPT_LIB) : % : %.sh GIT-SCRIPT-DEFINES
 	$(QUIET_GEN)$(cmd_munge_script) && \
 	mv $@+ $@
 
-ifndef NO_PERL
-$(patsubst %.perl,%,$(SCRIPT_PERL)): perl/perl.mak
-
 perl/perl.mak: perl/PM.stamp
 
 perl/PM.stamp: FORCE

@@ -1728,39 +1716,7 @@ perl/PM.stamp: FORCE
 perl/perl.mak: GIT-CFLAGS GIT-PREFIX perl/Makefile perl/Makefile.PL
 	$(QUIET_SUBDIR0)perl $(QUIET_SUBDIR1) PERL_PATH='$(PERL_PATH_SQ)' prefix='$(prefix_SQ)' $(@F)
 
-$(patsubst %.perl,%,$(SCRIPT_PERL)): % : %.perl GIT-VERSION-FILE
-	$(QUIET_GEN)$(RM) $@ $@+ && \
-	INSTLIBDIR=`MAKEFLAGS= $(MAKE) -C perl -s --no-print-directory instlibdir` && \
-	sed -e '1{' \
-	    -e '	s|#!.*perl|#!$(PERL_PATH_SQ)|' \
-	    -e '	h' \
-	    -e '	s=.*=use lib (split(/$(pathsep)/, $$ENV{GITPERLLIB} || "'"$$INSTLIBDIR"'"));=' \
-	    -e '	H' \
-	    -e '	x' \
-	    -e '}' \
-	    -e 's/@@GIT_VERSION@@/$(GIT_VERSION)/g' \
-	    [email protected] >$@+ && \
-	chmod +x $@+ && \
-	mv $@+ $@
-
-
-.PHONY: gitweb
-gitweb:
-	$(QUIET_SUBDIR0)gitweb $(QUIET_SUBDIR1) all
-
-git-instaweb: git-instaweb.sh gitweb GIT-SCRIPT-DEFINES
-	$(QUIET_GEN)$(cmd_munge_script) && \
-	chmod +x $@+ && \
-	mv $@+ $@
-else # NO_PERL
-$(patsubst %.perl,%,$(SCRIPT_PERL)) git-instaweb: % : unimplemented.sh
-	$(QUIET_GEN)$(RM) $@ $@+ && \
-	sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
-	    -e 's|@@REASON@@|NO_PERL=$(NO_PERL)|g' \
-	    unimplemented.sh >$@+ && \
-	chmod +x $@+ && \
-	mv $@+ $@
-endif # NO_PERL
+include perl.mak
 
 ifndef NO_PYTHON
 $(patsubst %.py,%,$(SCRIPT_PYTHON)): GIT-CFLAGS GIT-PREFIX GIT-PYTHON-VARS

1.8.1.2.526.gf51a757

The goal may be worthy, but the split does not look quite right.

What business do contrib/ scripts have knowing how gitweb and git-instaweb are built and what they depend on, for example?

perl.mak uses relative path, which is OK when called from the toplevel, but won't be anymore if one includes it from elsewhere. It is now possible to include the file using:

GIT_ROOT_DIR=<whatever>
include $(GIT_ROOT_DIR)/perl.mak

---
 perl.mak | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

index 8bbeef3..a2b8717 100644
@@ -1,5 +1,9 @@
 # Rules to build Git commands written in perl
 
+ifndef GIT_ROOT_DIR
+	GIT_ROOT_DIR = .
+endif
+
 ifndef PERL_PATH
 	PERL_PATH = /usr/bin/perl
 endif

@@ -11,12 +15,11 @@ NO_PERL = NoThanks
 endif
 
 ifndef NO_PERL
-$(patsubst %.perl,%,$(SCRIPT_PERL)): perl/perl.mak
-
+$(patsubst %.perl,%,$(SCRIPT_PERL)): $(GIT_ROOT_DIR)/perl/perl.mak
 
-$(patsubst %.perl,%,$(SCRIPT_PERL)): % : %.perl GIT-VERSION-FILE
+$(patsubst %.perl,%,$(SCRIPT_PERL)): % : %.perl $(GIT_ROOT_DIR)/GIT-VERSION-FILE
 	$(QUIET_GEN)$(RM) $@ $@+ && \
-	INSTLIBDIR=`MAKEFLAGS= $(MAKE) -C perl -s --no-print-directory instlibdir` && \
+	INSTLIBDIR=`MAKEFLAGS= $(MAKE) -C $(GIT_ROOT_DIR)/perl -s --no-print-directory instlibdir` && \
 	sed -e '1{' \
 	    -e '	s|#!.*perl|#!$(PERL_PATH_SQ)|' \
 	    -e '	h' \

1.8.1.2.526.gf51a757

Similarly to the extraction of perl-related code in perl.mak, we extract general default configuration from the Makefile to make it available from directories other than the toplevel.

This is required to make perl.mak usable because it requires $(pathsep) to be set.

---
 Makefile           | 62 +-----------------------------------------------------
 default-config.mak | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+), 61 deletions(-)
 create mode 100644 default-config.mak

index f39d4a9..9649a41 100644
@@ -346,67 +346,7 @@ GIT-VERSION-FILE: FORCE
 	@$(SHELL_PATH) ./GIT-VERSION-GEN
 -include GIT-VERSION-FILE
 
-# CFLAGS and LDFLAGS are for the users to override from the command line.
-
-CFLAGS = -g -O2 -Wall
-LDFLAGS =
-ALL_CFLAGS = $(CPPFLAGS) $(CFLAGS)
-ALL_LDFLAGS = $(LDFLAGS)
-STRIP ?= strip
-
-# Among the variables below, these:
-#   gitexecdir
-#   template_dir
-#   mandir
-#   infodir
-#   htmldir
-#   sysconfdir
-# can be specified as a relative path some/where/else;
-# this is interpreted as relative to $(prefix) and "git" at
-# runtime figures out where they are based on the path to the executable.
-# This can help installing the suite in a relocatable way.
-
-prefix = $(HOME)
-bindir_relative = bin
-bindir = $(prefix)/$(bindir_relative)
-mandir = share/man
-infodir = share/info
-gitexecdir = libexec/git-core
-mergetoolsdir = $(gitexecdir)/mergetools
-sharedir = $(prefix)/share
-gitwebdir = $(sharedir)/gitweb
-localedir = $(sharedir)/locale
-template_dir = share/git-core/templates
-htmldir = share/doc/git-doc
-ETC_GITCONFIG = $(sysconfdir)/gitconfig
-ETC_GITATTRIBUTES = $(sysconfdir)/gitattributes
-lib = lib
-# DESTDIR =
-pathsep = :
-
-export prefix bindir sharedir sysconfdir gitwebdir localedir
-
-CC = cc
-AR = ar
-RM = rm -f
-DIFF = diff
-TAR = tar
-FIND = find
-INSTALL = install
-RPMBUILD = rpmbuild
-TCL_PATH = tclsh
-TCLTK_PATH = wish
-XGETTEXT = xgettext
-MSGFMT = msgfmt
-PTHREAD_LIBS = -lpthread
-PTHREAD_CFLAGS =
-GCOV = gcov
-
-export TCL_PATH TCLTK_PATH
-
-SPARSE_FLAGS =
-
-
+include default-config.mak
 
 ### --- END CONFIGURATION SECTION ---
 

1.8.1.2.526.gf51a757

I really think this is going in a wrong direction. Whatever you happen to have chosen in this patch will be available to others, while whatever are left out will not be. When adding new things, people need to ask if it needs to be sharable or not, and the right answer to that question will even change over time.

My feeling is that Git's toplevel Makefile has become too large to remain completely monolithic, and splitting is good to organize it (and yes, splitting code into several files imply that future added code will have to be added in the right file, but that's not very different from splitting C code into several .c files to me). But that is another matter, and ...

This ended up being very simple to implement (essentially, the Makefile already knows how to do this, so this just means adding convenience build-perl-script target to the toplevel), so 2 new patches follow doing this, with a ridiculously small new version of mw-to-git/Makefile.

Matthieu Moy
http://www-verimag.imag.fr/~moy/

Git's Makefile provides a few nice features for script build and installation (substitute the first line with the right path, hardcode the path to Git library, ...).

The Makefile already knows how to process files outside the toplevel directory with e.g.

make SCRIPT_PERL=path/to/file.perl path/to/file

but we can make it simpler for callers by exposing build, install and clean rules as .PHONY targets.

---
The goal of this series is to use perl, but it is as easy to do it with sh and python too, so I did it for them too. I tested a manual "make -C ../../" in contrib/subtree and contrib/hg-to-git/ to check that it actually works.

 Makefile | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

index 5a2e02d..b4af30d 100644
@@ -480,9 +480,38 @@ SCRIPT_PERL += git-svn.perl
 SCRIPT_PYTHON += git-remote-testpy.py
 SCRIPT_PYTHON += git-p4.py
 
-SCRIPTS = $(patsubst %.sh,%,$(SCRIPT_SH)) \
-	  $(patsubst %.perl,%,$(SCRIPT_PERL)) \
-	  $(patsubst %.py,%,$(SCRIPT_PYTHON)) \
+# Generated files for scripts
+SCRIPT_SH_GEN = $(patsubst %.sh,%,$(SCRIPT_SH))
+SCRIPT_PERL_GEN = $(patsubst %.perl,%,$(SCRIPT_PERL))
+SCRIPT_PYTHON_GEN = $(patsubst %.py,%,$(SCRIPT_PYTHON))
+
+# Individual rules to allow e.g.
+# "make -C ../.. SCRIPT_PERL=contrib/foo/bar.perl build-perl-script"
+# from subdirectories like contrib/*/
+.PHONY: build-perl-script build-sh-script build-python-script
+build-perl-script: $(SCRIPT_PERL_GEN)
+build-sh-script: $(SCRIPT_SH_GEN)
+build-python-script: $(SCRIPT_PYTHON_GEN)
+
+.PHONY: install-perl-script install-sh-script install-python-script
+install-sh-script: $(SCRIPT_SH_GEN)
+	$(INSTALL) $(SCRIPT_SH_GEN) '$(DESTDIR_SQ)$(gitexec_instdir_SQ)'
+install-perl-script: $(SCRIPT_PERL_GEN)
+	$(INSTALL) $(SCRIPT_PERL_GEN) '$(DESTDIR_SQ)$(gitexec_instdir_SQ)'
+install-python-script: $(SCRIPT_PYTHON_GEN)
+	$(INSTALL) $(SCRIPT_PYTHON_GEN) '$(DESTDIR_SQ)$(gitexec_instdir_SQ)'
+
+.PHONY: clean-perl-script clean-sh-script clean-python-script
+clean-sh-script:
+	$(RM) $(SCRIPT_SH_GEN)
+clean-perl-script:
+	$(RM) $(SCRIPT_PERL_GEN)
+clean-python-script:
+	$(RM) $(SCRIPT_PYTHON_GEN)
+
+SCRIPTS = $(SCRIPT_SH_GEN) \
+	  $(SCRIPT_PERL_GEN) \
+	  $(SCRIPT_PYTHON_GEN) \
 	  git-instaweb
 
 ETAGS_TARGET = TAGS

1.8.1.2.530.g3cc16e4.dirty

This makes the Makefile simpler, while providing more features, and more consistency (the exact same rules with the exact same configuration as Git official commands are applied with the new version).

---
 contrib/mw-to-git/.gitignore                       |  1 +
 contrib/mw-to-git/Makefile                         | 64 ++++++----------------
 ...-remote-mediawiki => git-remote-mediawiki.perl} |  0
 3 files changed, 18 insertions(+), 47 deletions(-)
 create mode 100644 contrib/mw-to-git/.gitignore
 rewrite contrib/mw-to-git/Makefile (96%)
 rename contrib/mw-to-git/{git-remote-mediawiki => git-remote-mediawiki.perl} (100%)

new file mode 100644
index 0000000..b919655
@@ -0,0 +1 @@
+git-remote-mediawiki

similarity index 100%
rename from contrib/mw-to-git/git-remote-mediawiki
rename to contrib/mw-to-git/git-remote-mediawiki.perl

1.8.1.2.530.g3cc16e4.dirty

The configuration of the install directory is not reused from the toplevel Makefile: we assume Git is already built, hence just call "git --exec-path". This avoids too much surgery in the toplevel Makefile.

git-remote-mediawiki.perl can now "use Git;".

---
 contrib/mw-to-git/.gitignore                       |  1 +
 contrib/mw-to-git/Makefile                         | 45 ++++++++++++++--------
 ...-remote-mediawiki => git-remote-mediawiki.perl} |  0
 3 files changed, 30 insertions(+), 16 deletions(-)
 create mode 100644 contrib/mw-to-git/.gitignore
 rename contrib/mw-to-git/{git-remote-mediawiki => git-remote-mediawiki.perl} (100%)

new file mode 100644
index 0000000..b919655
@@ -0,0 +1 @@
+git-remote-mediawiki

similarity index 100%
rename from contrib/mw-to-git/git-remote-mediawiki
rename to contrib/mw-to-git/git-remote-mediawiki.perl

1.8.1.2.526.gf51a757

Continuing to the comment on 3/4, I wonder if it would be a lot simpler and more maintainable if you replaced 1/4 to 3/4 with a smaller patch to the top-level Makefile to teach it to munge arbitrary path/to/foo.perl to path/to/foo the same way as we do to other path/tool.perl that are known to the top-level Makefile (similarly, another target to install the resulting path/to/foo at an arbitrary place). Then do something like

	all::
		$(MAKE) -C ../.. \
			PERL_SCRIPT=contrib/mw-to-git/git-remote-mediawiki.perl \
                        build-perl-script
	install::
		$(MAKE) -C ../.. \
			PERL_SCRIPT=contrib/mw-to-git/git-remote-mediawiki.perl \
                        install-perl-script

in this step.

That seems much cleaner to me. If done right, it could also let people put:

CONTRIB_PERL += contrib/mw-to-git/git-remote-mediawiki

or similar into their config.mak, and just get specific contrib bits built and installed along with the rest of git.

-Peff

Actually, you can already do this:

SCRIPT_PERL += contrib/mw-to-git/git-remote-mediawiki.perl

probably not by design, but it works!

Matthieu Moy
http://www-verimag.imag.fr/~moy/

So putting:

  ROOT=contrib/mw-to-git
  git-remote-mediawiki: FORCE
          @make -C ../.. SCRIPT_PERL=$(ROOT)/[email protected] $(ROOT)/$@

in contrib/mw-to-git/Makefile would already work? Neat.

-Peff

That essentially is what [v2 2/2] does, no?

Yes (this one was cc'd to me, but the others were not, so I read it in isolation). I think Matthieu's series is nicer than just that, though, because it handles the single-file case installation, too, which requires more support from the parent Makefile.

-Peff

On Wed, 06 Feb 2013 17:41:01 +0100 Matthieu Moy <[email protected]> wrote:

MM> Why not, if you want to use the output of credential_write in tests. But MM> credential_write is essentially used to talk to "git credential", so the MM> important information is the content of the hash before credential_write MM> and after credential_read. They are unordered, but consistent and MM> testable.

I like testing output (especially when it's part of an API), so we should make the externally observable output consistent and testable.

The change is tiny, just sort the keys instead of calling each(), so I hope it makes it in the final version.

MM> I don't think that would be a better naming. Maybe "serialize" and MM> "parse" would be better, but "query" would sound like it establishes the MM> connection and possibly reads the response to me.

I'm OK with anything unambiguous.

Thanks!
Ted

I assumed someone had, but if not I can take a stab at it. I'm not sure however how should I map server, server-port, and user to credential key-value pairs. I'm leaning towards

	protocol=smtp
	host=<smtp-server>:<smtp-port>
	user=<user>

and than netrc/authinfo helper splitting host to host name and port number, unless port is not in host in which case protocol is assumed as port.

Best regards,                                         _     _
.o. | Liege of Serenely Enlightened Majesty of      o' \,=./ `o
..o | Computer Science,  Michał “mina86” Nazarewicz    (o o)
ooo +----<email/xmpp: [email protected]>--------------ooO--(_)--Ooo--

On Wed, 06 Feb 2013 14:26:46 +0100 Michal Nazarewicz <[email protected]> wrote:

MN> I assumed someone had, but if not I can take a stab at it. I'm not sure MN> however how should I map server, server-port, and user to credential MN> key-value pairs. I'm leaning towards

MN> protocol=smtp
MN> host=<smtp-server>:<smtp-port>
MN> user=<user>

MN> and than netrc/authinfo helper splitting host to host name and port MN> number, unless port is not in host in which case protocol is assumed as MN> port.

That would work (with my PATCHv6 of the netrc credential helper) as follows:

1) just host

host=H

maps to

machine H login Y password Z

2) host + protocol smtp

host=H
protocol=smtp

maps to any of:

machine H port smtp login Y password Z
machine H protocol smtp login Y password Z

3) host:port + protocol smtp

host=H:25
protocol=smtp

maps to any of:

machine H port 25 protocol smtp login Y password Z machine H:25 port smtp login Y password Z machine H:25 protocol smtp login Y password Z

That's my understanding of what we discussed with Peff and Junio about token mapping. Note we don't split the input host, but instead say "if token 'port' is numeric, append it to the host token" on the netrc side.

Does that sound reasonable? If yes, I can add it to the testing Makefile for the netrc credential helper, to make sure it's clearly stated and tested.

Ted

On Tue, 29 Jan 2013 11:53:19 -0800 Junio C Hamano <[email protected]> wrote:

JCH> Makes one wonder why .authinfo and not .netrc;

JCH> http://www.gnu.org/software/emacs/manual/html_node/auth/Help-for-users.html

JCH> phrases it amusingly:

JCH>         “Netrc” files are usually called .authinfo or .netr
JCH>         nowadays .authinfo seems to be more popular and the
JCH>         auth-source library encourages this confusion by accepting
JCH>         both

I wrote this and the auth-source.el library in Emacs (I'm glad it was amusing :). The confusion is further perpetuated by our (in Emacs) encouragement to use a .authinfo.gpg file, which is then decrypted on the fly by Emacs through GPG. The format is the same; by the time auth-source.el sees the contents, they are plain text since the decoding happens at the file handler level.

I think it makes sense to write the code to support both `git-send-email' and credentials. I have had it in my TODO list for almost 2 years now to work on credential support, and to support the ~/.authinfo.gpg decoding specifically. Ideally this would also support the other formats... Michal, would you be interested in that feature? I promise to get off my rear and help out.

JCH> It is rather strange to require a comma-separated-values parser to JCH> read a file format this simple, isn't it?

I'd recommend a hand-crafted parser. Among other things, you should accept both "strings" and 'strings' if possible (I've seen both formats in the wild), and the format is simple enough to avoid the module dependency.

JCH> Perhaps you can convert at least some popular ones yourself? After JCH> all, the user may be using an _existing_ .authinfo/.netrc that she JCH> has been using with other programs that do understand symbolic port JCH> names. Rather than forcing all such users to update their files, JCH> the patch can work a bit harder for them and the world will be a JCH> better place, no?

I agree, "port imap" is a nice self-documenting token. Maybe it can be interpreted by the program that requests the token with a services lookup, where supported.

Ted