What's cooking in git.git (Jun 2013, #02; Tue, 4)

Here are the topics that have been cooking. Commits prefixed with '-' are only in 'pu' (proposed updates) while commits prefixed with '+' are in 'next'.

We are in the post-1.8.3 cycle. As promised, 'next' has been rewound. A few stalled topics have been ejected and bunch of new topics that have been cooking are now in it. I expect these on 'next' to graduate to 'master' soonish, as I picked relatively easy ones in this round.

You can find the changes described here in the integration branches of the repositories listed at

http://git-blame.blogspot.com/p/git-public-repositories.html

--------------------------------------------------
[Graduated to "master"]

* kb/status-ignored-optim-2 (2013-06-02) 1 commit (merged to 'next' on 2013-06-02 at 88ee588)
+ dir.c: fix ignore processing within not-ignored directories

Fix 1.8.3 regressions in the .gitignore path exclusion logic.

--------------------------------------------------
[New Topics]

* ar/wildmatch-foldcase (2013-06-02) 1 commit
(merged to 'next' on 2013-06-04 at 3180bcc)
+ wildmatch: properly fold case everywhere

The wildmatch engine did not honor WM_CASEFOLD option correctly.

Will merge to 'master'.

* cr/git-work-tree-sans-git-dir (2013-06-03) 1 commit (merged to 'next' on 2013-06-04 at bebedca)
+ git.txt: remove stale comment regarding GIT_WORK_TREE

These days, "git --work-tree=there cmd" without specifying an explicit --git-dir=here will do the usual discovery, but we had a description of older behaviour in the documentation.

Will merge to 'master'.

* fc/do-not-use-the-index-in-add-to-index (2013-06-03) 2 commits (merged to 'next' on 2013-06-04 at 94e7b60)
+ read-cache: trivial style cleanups
+ read-cache: fix wrong 'the_index' usage

Will merge to 'master'.

* fc/sequencer-skip-quiet (2013-06-03) 8 commits
- revert/cherry-pick: add --skip option
- revert/cherry-pick: add --quiet option
- sequencer: run post-rewrite hook
- cherry-pick: store rewritten commits
- SQUASH???
- cherry-pick: add --skip-empty option
- sequencer: trivial fix
- sequencer: remove useless indentation

I think the post-rewrite hook should not apply to revert, and revert should be taught about --skip-empty. The "copy-notes" change was nak'ed, and I agree with Thomas that the external interface to the mechanism should be aligned with existing notes.rewrite.<command>.

Waiting for a reroll.

$gmane/225676, $gmane/226263, $gmane/226271

* js/test-ln-s-add (2013-06-02) 11 commits
- t6035: use test_ln_s_add to remove SYMLINKS prerequisite
- t3509, t4023, t4114: use test_ln_s_add to remove SYMLINKS prerequisite
- t3100: use test_ln_s_add to remove SYMLINKS prerequisite
- t3030: use test_ln_s_add to remove SYMLINKS prerequisite
- t2100: use test_ln_s_add to remove SYMLINKS prerequisite
- t0000: use test_ln_s_add to remove SYMLINKS prerequisite
- tests: use test_ln_s_add to remove SYMLINKS prerequisite (trivial cases)
- tests: introduce test_ln_s and test_ln_s_add
- t3010: modernize style
- t2100: modernize style and unroll a loop of test cases
- test-chmtime: Fix exit code on Windows

Many tests that check the behaviour of symbolic links stored in the index or the tree objects do not have to be skipped on a filesystem that lack symbolic link support.

There seem to be some misconversion, mostly around the use of the new test_ln_s helper.

Waiting for responses to reviews.
$gmane/226417 and others.

* mt/send-email-cc-match-fix (2013-06-03) 6 commits
- t/send-email: test suppress-cc=self with non-ascii
- t/send-email: add test with quoted sender
- send-email: make --suppress-cc=self sanitize input
- t/send-email: test suppress-cc=self on cccmd
- send-email: fix suppress-cc=self on cccmd
- t/send-email.sh: add test for suppress-cc=self

It may want to have an additional test case for --from='"A U. Thor" <[email protected]>' to make sure we do not doubly escape what is already escaped.

Some changes in patch 2/6 and a later patch may need to be flipped around.

* rr/complete-difftool (2013-06-03) 2 commits (merged to 'next' on 2013-06-04 at 01c7611)
+ completion: clarify ls-tree, archive, show completion
+ completion: difftool takes both revs and files

Update command line completion (in contrib/) to use a better named completion helper function for commands that take revisions and paths.

Will merge to 'master'.

* rr/diffcore-pickaxe-doc (2013-06-03) 2 commits (merged to 'next' on 2013-06-04 at 67d1fc7)
+ diffcore-pickaxe doc: document -S and -G properly
+ diffcore-pickaxe: make error messages more consistent

Update the low-level diffcore documentation on -S/-G and --pickaxe-all.

Will merge to 'master'.

* tr/sha1-file-silence-loose-object-info-under-prune-race (2013-06-03) 1 commit (merged to 'next' on 2013-06-04 at e891bb8)
+ sha1_file: silence sha1_loose_object_info

Will merge to 'master'.

* bp/mediawiki-credential (2013-06-04) 1 commit
- git-remote-mediawiki: use git.pm functions for credentials

The bridge to MediaWiki has been updated to use the credential helper interface in Git.pm, losing its own and the original implementation the former was based on.

Minor review comments sent.

* mz/rebase-tests (2013-06-03) 7 commits
- tests: move test for rebase messages from t3400 to t3406
- t3406: modernize style
- add tests for rebasing merged history
- add tests for rebasing root
- add tests for rebasing of empty commits
- add tests for rebasing with patch-equivalence present
- add simple tests of consistency across rebase types

--------------------------------------------------
[Stalled]

* mg/more-textconv (2013-05-10) 7 commits
- grep: honor --textconv for the case rev:path
- grep: allow to use textconv filters
- t7008: demonstrate behavior of grep with textconv
- cat-file: do not die on --textconv without textconv filters
- show: honor --textconv for blobs
- diff_opt: track whether flags have been set explicitly
- t4030: demonstrate behavior of show with textconv

Make "git grep" and "git show" pay attention to --textconv when dealing with blob objects.

I thought this was pretty well designed and executed, but it seems there are some doubts on the list; kicking back to 'pu'.

* mh/multimail (2013-04-21) 1 commit
- git-multimail: a replacement for post-receive-email

Waiting for the initial history to pull from.
$gmane/223564

* jc/format-patch (2013-04-22) 2 commits
- format-patch: --inline-single
- format-patch: rename "no_inline" field

A new option to send a single patch to the standard output to be appended at the bottom of a message. I personally have no need for this, but it was easy enough to cobble together. Tests, docs and stripping out more MIMEy stuff are left as exercises to interested parties.

Not ready for inclusion.

Will discard unless we hear from anybody who is interested in tying its loose ends.

* jk/gitweb-utf8 (2013-04-08) 4 commits
- gitweb: Fix broken blob action parameters on blob/commitdiff pages
- gitweb: Don't append ';js=(0|1)' to external links
- gitweb: Make feed title valid utf8
- gitweb: Fix utf8 encoding for blob_plain, blobdiff_plain, commitdiff_plain, and patch

Various fixes to gitweb.

Drew Northup volunteered to take a look into this. $gmane/226216

* jk/commit-info-slab (2013-04-19) 3 commits
- commit-slab: introduce a macro to define a slab for new type
- commit-slab: avoid large realloc
- commit: allow associating auxiliary info on-demand (this branch is used by jc/show-branch.)

Technology demonstration to show a way we could use unbound number of flag bits on commit objects.

* jc/show-branch (2013-05-21) 5 commits
- show-branch: use commit slab to represent bitflags of arbitrary width
- show-branch.c: remove "all_mask"
- show-branch.c: abstract out "flags" operation
- show-branch.c: lift all_mask/all_revs to a global static
- show-branch.c: update comment style
(this branch uses jk/commit-info-slab.)

Waiting for the final step to lift the hard-limit before sending it out.

--------------------------------------------------
[Cooking]

* fc/completion-less-ls-remote (2013-06-02) 1 commit (merged to 'next' on 2013-06-03 at 6624f0b)
+ completion: avoid ls-remote in certain scenarios

Will merge to 'master'.

* jk/test-exit-code-by-signal (2013-06-02) 1 commit (merged to 'next' on 2013-06-03 at 25af892)
+ t0005: test git exit code from signal death

Will merge to 'master'.

* nd/make-wildmatch-default (2013-06-02) 1 commit
- Makefile: promote wildmatch to be the default fnmatch implementation

Will merge to 'next'.

* rr/remove-contrib-some (2013-06-02) 1 commit
- contrib: remove continuous/ and patches/

Will merge to 'next'.

* rs/unpack-trees-plug-leak (2013-06-02) 7 commits
(merged to 'next' on 2013-06-03 at 97e7b6d)
+ unpack-trees: free cache_entry array members for merges
+ diff-lib, read-tree, unpack-trees: mark cache_entry array paramters const
+ diff-lib, read-tree, unpack-trees: mark cache_entry pointers const
+ unpack-trees: create working copy of merge entry in merged_entry
+ unpack-trees: factor out dup_entry
+ read-cache: mark cache_entry pointers const
+ cache: mark cache_entry pointers const

Will merge to 'master'.

* tr/test-commit-only-on-orphan (2013-06-02) 1 commit (merged to 'next' on 2013-06-03 at b1864fd)
+ Test 'commit --only' after 'checkout --orphan'

Will merge to 'master'.

* ap/diff-ignore-blank-lines (2013-05-29) 1 commit
- diff: add --ignore-blank-lines option

"git diff" learned a mode that ignores hunks whose change consists only of additions and removals of blank lines, which is the same as "diff -B" (ignore blank lines) of GNU diff.

Will be rerolled.
$gmane/226394

* fc/show-branch-in-rebase-am (2013-05-29) 1 commit (merged to 'next' on 2013-06-03 at 176f6b7)
+ prompt: fix for simple rebase

The bash prompt code (in contrib/) displayed the name of the branch being rebased when "rebase -i/-m/-p" modes are in use, but not the plain vanilla "rebase".

Will merge to 'master'.

* ks/difftool-dir-diff-copy-fix (2013-05-29) 1 commit (merged to 'next' on 2013-06-03 at ca0cae0)
+ difftool --dir-diff: allow changing any clean working tree file

"difftool --dir-diff" did not copy back changes made by the end-user in the diff tool backend to the working tree in some cases.

Will merge to 'master'.

* rr/push-head (2013-05-29) 3 commits
(merged to 'next' on 2013-06-03 at ecd5be7)
+ push: make push.default = current use resolved HEAD
+ push: fail early with detached HEAD and current
+ push: factor out the detached HEAD error message

"git push $there HEAD:branch" did not resolve HEAD early enough, so it was easy to flip it around while push is still going on and push out a branch that the user did not originally intended when the command was started.

Will merge to 'master'.

* sb/archive-zip-double-assignment-fix (2013-05-29) 1 commit (merged to 'next' on 2013-06-03 at c316eec)
+ archive-zip:write_zip_entry: Remove second reset of size variable to zero.

Will merge to 'master'.

* rj/mingw-cygwin (2013-05-08) 2 commits (merged to 'next' on 2013-06-04 at 308fdb4)
+ cygwin: Remove the CYGWIN_V15_WIN32API build variable
+ mingw: rename WIN32 cpp macro to GIT_WINDOWS_NATIVE

Update build for Cygwin 1.[57]. Torsten Bögershausen reports that this is fine with Cygwin 1.7 ($gmane/225824) so let's try moving it ahead.

Will merge to 'master'.

* rr/rebase-autostash (2013-05-29) 7 commits (merged to 'next' on 2013-06-04 at 16f7c54)
+ rebase: implement --[no-]autostash and rebase.autostash
+ rebase --merge: return control to caller, for housekeeping
+ rebase -i: return control to caller, for housekeeping
+ am: return control to caller, for housekeeping
+ rebase: prepare to do generic housekeeping
+ rebase -i: don't error out if $state_dir already exists
+ am: tighten a conditional that checks for $dotest

Will merge to 'master'.

* nd/urls-doc-no-file-hyperlink-fix (2013-05-24) 1 commit (merged to 'next' on 2013-06-03 at 54903b2)
+ urls.txt: avoid auto converting to hyperlink

Will merge to 'master'.

* cb/log-follow-with-combined (2013-05-28) 1 commit (merged to 'next' on 2013-06-04 at d5bf4f3)
+ fix segfault with git log -c --follow

Will merge to 'master'.

* fc/cleanups (2013-05-28) 3 commits
(merged to 'next' on 2013-06-03 at 527cf93)
+ test: rebase: fix --interactive test
+ test: trivial cleanups
+ remote: trivial style cleanup

Will merge to 'master'.

* fc/makefile (2013-05-26) 5 commits
(merged to 'next' on 2013-06-03 at d1074e4)
+ build: do not install git-remote-testpy
+ build: add NO_INSTALL variable
+ build: cleanup using $<
+ build: cleanup using $^
+ build: trivial simplification
(this branch is used by fc/remote-helpers-use-specified-python.)

Will merge to 'master'.

* fc/remote-helpers-use-specified-python (2013-05-28) 4 commits
- remote-helpers: add exec-path links
- remote-helpers: allow direct test execution
- remote-helpers: rename tests
- remote-helpers: generate scripts
(this branch uses fc/makefile.)

I do not particularly think the second from the bottom is a good change, but it takes the remainder of the series hostage.

Waiting for a reroll.

* fc/send-email-chainreplyto-warning (2013-05-28) 1 commit (merged to 'next' on 2013-06-03 at e04764f)
+ send-email: remove warning about unset chainreplyto

An overdue removal od "behaviour changed at 1.7.0; if you were living in a cave, here is what you can adjust to it" message.

Will merge to 'master'.

* nd/prune-packed-dryrun-verbose (2013-05-28) 1 commit (merged to 'next' on 2013-06-03 at 3445b27)
+ prune-packed: avoid implying "1" is DRY_RUN in prune_packed_objects()

Will merge to 'master'.

* rj/mingw-compat-st-mode-bits (2013-05-29) 1 commit (merged to 'next' on 2013-06-03 at 2efe84c)
+ path: Fix a sparse warning

Will merge to 'master'.

* rs/commit-m-no-edit (2013-05-28) 1 commit
(merged to 'next' on 2013-06-03 at 14329fa)
+ commit: don't start editor if empty message is given with -m

"git commit --allow-empty-message -m ''" should not start an editor.

Will merge to 'master'.

* xq/credential-osxkeychain (2013-05-28) 1 commit
(merged to 'next' on 2013-06-04 at a4ee0e0)
+ credential-osxkeychain: support more protocols

Will merge to 'master'.

* jc/core-checkstat (2013-05-06) 1 commit
(merged to 'next' on 2013-06-03 at 2166cb3)
+ deprecate core.statinfo at Git 2.0 boundary
(this branch is used by jc/core-checkstat-2.0.)

Will merge to 'master'.

* mh/reflife (2013-06-02) 25 commits
- refs: document the lifetime of the args passed to each_ref_fn
- register_ref(): make a copy of the bad reference SHA-1
- exclude_existing(): set existing_refs.strdup_strings
- string_list_add_refs_by_glob(): add a comment about memory management
- string_list_add_one_ref(): rename first parameter to "refname"
- show_head_ref(): rename first parameter to "refname"
- show_head_ref(): do not shadow name of argument
- add_existing(): do not retain a reference to sha1
- do_fetch(): clean up existing_refs before exiting
- do_fetch(): reduce scope of peer_item
- object_array_entry: fix memory handling of the name field
- find_first_merges(): remove unnecessary code
- find_first_merges(): initialize merges variable using initializer
- fsck: don't put a void*-shaped peg in a char*-shaped hole
- object_array_remove_duplicates(): rewrite to reduce copying
- revision: use object_array_filter() in implementation of gc_boundary()
- object_array: add function object_array_filter()
- revision: split some overly-long lines
- cmd_diff(): make it obvious which cases are exclusive of each other
- cmd_diff(): rename local variable "list" -> "entry"
- cmd_diff(): use an object_array for holding trees
- builtin_diff_tree(): make it obvious that function wants two entries
- add_rev_cmdline(): make a copy of the name argument
- fetch: make own copies of refnames
- describe: make own copy of refname

Define memory ownership and lifetime rules for what for-each-ref feeds to its callbacks (in short, "you do not own it, so make a

 copy if you want to keep it").

Will merge to 'next'.

* th/bisect-skip-report-range-fix (2013-05-22) 1 commit (merged to 'next' on 2013-06-03 at 7bd4656)
+ bisect: Fix log output for multi-parent skip ranges

Fix for an additional bisect log comments.

Will merge to 'master'.

* mm/mediawiki-https-fail-message (2013-05-29) 1 commit (merged to 'next' on 2013-06-04 at fb2671c)
+ git-remote-mediawiki: better error message when HTTP(S) access fails

Hint users when https:// connection failed to check the certificate; it is a good hint if we assumie that it is common error for the end users to make.

Will merge to 'master'.

* tg/maint-zsh-svn-remote-prompt (2013-05-22) 1 commit (merged to 'next' on 2013-06-03 at 32a45c0)
+ prompt: fix show upstream with svn and zsh

zsh prompt script that borrowed from bash prompt script did not work due to slight differences in array variable notation between these two shells.

Will merge to 'master'.

* tr/push-no-verify-doc (2013-05-23) 1 commit
(merged to 'next' on 2013-06-03 at 01737d6)
+ Document push --no-verify

"git push --[no-]verify" was not documented.

Will merge to 'master'.

* dm/unbash-subtree (2013-05-21) 1 commit
(merged to 'next' on 2013-06-03 at 2c9d2fb)
+ contrib/git-subtree: Use /bin/sh interpreter instead of /bin/bash

It turns out that git-subtree script does not have to be run with bash.

Will merge to 'master'.

* fc/transport-helper-no-refspec (2013-05-21) 2 commits (merged to 'next' on 2013-06-03 at 8763bda)
+ transport-helper: check if the dry-run is supported
+ transport-helper: barf when user tries old:new

With "export" remote-helper protocol, (1) a push that tries to update a remote ref whose name is different from the pushing side does not work yet, and (2) the helper may not know how to do --dry-run, so detect such problematic cases and disable them for now.

Will merge to 'master'.

* rr/die-on-missing-upstream (2013-06-02) 2 commits (merged to 'next' on 2013-06-03 at 00847ea)
+ sha1_name: fix error message for @{<N>}, @{<date>}
+ sha1_name: fix error message for @{u}

When a reflog notation is used for implicit "current branch", we did not say which branch and worse said "branch ''".

Will merge to 'master'.

* fc/remote-bzr (2013-05-28) 8 commits (merged to 'next' on 2013-06-04 at a603082)
+ remote-bzr: add fallback check for a partial clone
+ remote-bzr: reorganize the way 'wanted' works
+ remote-bzr: trivial cleanups
+ remote-bzr: change global repo
+ remote-bzr: delay cloning/pulling
+ remote-bzr: simplify get_remote_branch()
+ remote-bzr: fix for files with spaces
+ remote-bzr: recover from failed clones

Will merge to 'master'.

* jx/clean-interactive (2013-06-03) 15 commits
- test: add t7301 for git-clean--interactive
- git-clean: add documentation for interactive git-clean
- git-clean: add ask each interactive action
- git-clean: add select by numbers interactive action
- git-clean: add filter by pattern interactive action
- git-clean: use a git-add-interactive compatible UI
- git-clean: add colors to interactive git-clean
- git-clean: show items of del_list in columns
- git-clean: add support for -i/--interactive
- git-clean: refactor git-clean into two phases
- Refactor write_name_quoted_relative, remove unused params
- Refactor quote_path_relative, remove unused params
- quote.c: remove path_relative, use relative_path instead
- path.c: refactor relative_path(), not only strip prefix
- test: add test cases for relative_path

* tr/test-v-and-v-subtest-only (2013-05-16) 6 commits
- test-lib: support running tests under valgrind in parallel
- test-lib: allow prefixing a custom string before "ok N" etc.
- test-lib: valgrind for only tests matching a pattern
- test-lib: verbose mode for only tests matching a pattern
- test-lib: refactor $GIT_SKIP_TESTS matching
- test-lib: enable MALLOC_* for the actual tests

Allows N instances of tests run in parallel, each running 1/N parts of the test suite under Valgrind, to speed things up.

The tip one may be useful in practice but is a tad ugly ;-)

There seem to be some miscounting by toggling the verbose/valgrind mode at wrong places? Cf. $gmane/225735

Waiting for a reroll.

* rr/zsh-color-prompt (2013-05-17) 3 commits
(merged to 'next' on 2013-06-03 at d011a76)
+ prompt: colorize ZSH prompt
+ prompt: factor out gitstring coloring logic
+ prompt: introduce GIT_PS1_STATESEPARATOR

Will merge to 'master'.

* fc/contrib-related (2013-06-03) 4 commits
- contrib: related: parse committish like format-patch
- contrib: related: add option to parse from committish
- contrib: related: add support for multiple patches
- Add new git-related helper to contrib

Waiting for the design review to settle.

* fc/remote-hg (2013-05-28) 50 commits (merged to 'next' on 2013-06-04 at 9ee7dab)
+ remote-hg: add support for --force
+ remote-hg: add support for --dry-run
+ remote-hg: check if a fetch is needed
+ remote-hg: trivial cleanup
+ remote-helpers: improve marks usage
+ remote-hg: add check_push() helper
+ remote-hg: add setup_big_push() helper
+ remote-hg: remove files before modifications
+ remote-hg: improve lightweight tag author
+ remote-hg: use remote 'default' not local one
+ remote-hg: improve branch listing
+ remote-hg: simplify branch_tip()
+ remote-hg: check diverged bookmarks
+ remote-hg: pass around revision refs
+ remote-hg: implement custom checkheads()
+ remote-hg: implement custom push()
+ remote-hg: only update necessary revisions
+ remote-hg: force remote bookmark push selectively
+ remote-hg: reorganize bookmark handling
+ remote-hg: add test for failed double push
+ remote-hg: add test for big push
+ remote-hg: add test for new bookmark special
+ remote-hg: add test for bookmark diverge
+ remote-hg: add test for diverged push
+ remote-hg: add test to push new bookmark
+ remote-hg: add remote tests
+ remote-hg: update bookmarks when using a remote
+ remote-hg: add check_bookmark() test helper
+ remote-bzr: simplify test checks
+ remote-hg: add tests for 'master' bookmark
+ remote-hg: always point HEAD to master
+ remote-hg: improve progress calculation
+ remote-hg: trivial cleanups
+ remote-hg: ensure remote rebasing works
+ remote-hg: upgrade version 1 marks
+ remote-hg: switch from revisions to SHA-1 noteids
+ remote-hg: add version checks to the marks
+ remote-hg: improve node traversing
+ remote-hg: shuffle some code
+ remote-hg: use a shared repository store
+ remote-hg: load all extensions
+ remote-hg: test: simplify previous branch checkout
+ remote-helpers: test: simplify remote URLs
+ remote-helpers: tests: general improvements
+ remote-helpers: test: cleanup style
+ remote-helpers: test: cleanup white-spaces
+ remote-hg: trivial reorganization
+ remote-hg: test: be a little more quiet
+ remote-hg: tests: fix hg merge
+ remote-helpers: tests: use python directly

Will merge to 'master'.

* hv/config-from-blob (2013-05-12) 5 commits
- do not die when error in config parsing of buf occurs
- teach config --blob option to parse config from database
- config: make parsing stack struct independent from actual data source
- config: drop cf validity check in get_next_char()
- config: factor out config file stack management

Waiting for a reroll.
$gmane/223964

* nd/clone-connectivity-shortcut (2013-05-28) 4 commits (merged to 'next' on 2013-06-03 at 812bd80)
+ clone: open a shortcut for connectivity check
+ index-pack: remove dead code (it should never happen)
+ fetch-pack: prepare updated shallow file before fetching the pack
+ clone: let the user know when check_everything_connected is run

Will merge to 'master'.

* kb/full-history-compute-treesame-carefully-2 (2013-05-16) 15 commits
- revision.c: make default history consider bottom commits
- revision.c: don't show all merges for --parents
- revision.c: discount side branches when computing TREESAME
- revision.c: add BOTTOM flag for commits
- simplify-merges: drop merge from irrelevant side branch
- simplify-merges: never remove all TREESAME parents
- t6012: update test for tweaked full-history traversal
- revision.c: Make --full-history consider more merges
- Documentation: avoid "uninteresting"
- rev-list-options.txt: correct TREESAME for P
- t6111: add parents to tests
- t6111: allow checking the parents as well
- t6111: new TREESAME test set
- t6019: test file dropped in -s ours merge
- decorate.c: compact table when growing

Major update to a very core part of the system to improve culling of irrelevant parents while traversing a mergy history.

Will not be a 1.8.3 material, but is an important topic.

Will merge to 'next'.

* mm/color-auto-default (2013-05-15) 2 commits
- make color.ui default to 'auto'
- config: refactor management of color.ui's default value

Flip the default for color.ui to 'auto', which is what many tutorials recommend new users to do. The updated code claims the switch happened at Git 2.0 in the past tense, but we might want to expedite it, as this change is not all that important to deserve a major version bump.

I'd vote for merging this without waiting for 2.0. Comments?

Waiting for a reroll.

* jh/shorten-refname (2013-05-07) 4 commits
- t1514: refname shortening is done after dereferencing symbolic refs
- shorten_unambiguous_ref(): Fix shortening refs/remotes/origin/HEAD to origin
- t1514: Demonstrate failure to correctly shorten "refs/remotes/origin/HEAD"
- t1514: Add tests of shortening refnames in strict/loose mode

When remotes/origin/HEAD is not a symbolic ref, "rev-parse --abbrev-ref remotes/origin/HEAD" ought to show "origin", not "origin/HEAD", which is fixed with this series (if it is a symbolic ref that points at remotes/origin/something, then it should show "origin/something" and it already does).

Expecting a reroll, as an early part of a larger series.

* nd/warn-ambiguous-object-name (2013-05-29) 1 commit (merged to 'next' on 2013-06-04 at e87c9d1)
+ get_sha1: warn about full or short object names that look like refs

"git cmd <name>", when <name> happens to be a 40-hex string, directly uses the 40-hex string as an object name, even if a ref "refs/<some hierarchy>/<name>" exists. This disambiguation order is unlikely to change, but we should warn about the ambiguity just like we warn when more than one refs/ hierachies share the same name.

Will merge to 'master'.

* jk/packed-refs-race (2013-05-06) 4 commits
- for_each_ref: load all loose refs before packed refs
- get_packed_refs: reload packed-refs file when it changes
- add a stat_validity struct
- resolve_ref: close race condition for packed refs

What is the status of this thing?

* fc/at-head (2013-05-08) 13 commits
(merged to 'next' on 2013-06-04 at f334a2a)
+ sha1_name: compare variable with constant, not constant with variable
+ Add new @ shortcut for HEAD
+ sha1_name: refactor reinterpret()
+ sha1_name: check @{-N} errors sooner
+ sha1_name: reorganize get_sha1_basic()
+ sha1_name: don't waste cycles in the @-parsing loop
+ sha1_name: remove unnecessary braces
+ sha1_name: remove no-op
+ tests: at-combinations: @{N} versus HEAD@{N}
+ tests: at-combinations: increase coverage
+ tests: at-combinations: improve nonsense()
+ tests: at-combinations: check ref names directly
+ tests: at-combinations: simplify setup

Instead of typing four capital letters "HEAD", you can say "@" instead.

Will merge to 'master'.

* jk/submodule-subdirectory-ok (2013-04-24) 3 commits
- submodule: fix quoting in relative_path()
- submodule: drop the top-level requirement
- rev-parse: add --prefix option

Allow various subcommands of "git submodule" to be run not from the top of the working tree of the superproject.

Waiting for a reroll.

* jl/submodule-mv (2013-04-23) 5 commits
- submodule.c: duplicate real_path's return value
- rm: delete .gitmodules entry of submodules removed from the work tree
- Teach mv to update the path entry in .gitmodules for moved submodules
- Teach mv to move submodules using a gitfile
- Teach mv to move submodules together with their work trees

"git mv A B" when moving a submodule A does "the right thing", inclusing relocating its working tree and adjusting the paths in the .gitmodules file.

Waiting for a reroll.

* jn/add-2.0-u-A-sans-pathspec (2013-04-26) 1 commit
- git add: -u/-A now affects the entire working tree

Will cook in 'next' until Git 2.0.

* jc/core-checkstat-2.0 (2013-05-06) 1 commit
- core.statinfo: remove as promised in Git 2.0
(this branch uses jc/core-checkstat.)

Will cook in 'next' until Git 2.0.

* jc/push-2.0-default-to-simple (2013-04-03) 1 commit
- push: switch default from "matching" to "simple"

Will cook in 'next' until Git 2.0.

* jc/add-2.0-ignore-removal (2013-04-22) 1 commit
- git add <pathspec>... defaults to "-A"

Updated endgame for "git add <pathspec>" that defaults to "--all" aka "--no-ignore-removal".

Will cook in 'next' until Git 2.0.

Am 6/5/2013 1:45, schrieb Junio C Hamano:

I haven't gotten around to run this new test on Windows. I've reason to believe that it won't pass as is. Please don't let it graduate, yet.

-- Hannes

Yeah, I sort of assumed that we would need to check for either "3" or "131" on Windows, but I wasn't sure which. I don't think there is any rush on it.

-Peff

From: Johannes Sixt <[email protected]>

The test case depends on that test-sigchain can commit suicide by a call to raise(SIGTERM) in a way that run-command.c::wait_or_whine() can detect as death through a signal. There are no POSIX signals on Windows, and a sufficiently close emulation is not available in the Microsoft C runtime (and probably not even possible).

The particular deficiency is that when a signal is raise()d whose SIG_DFL action will cause process death (SIGTERM in this case), the implementation of raise() just calls exit(3).

We could check for exit code 3 in addition to 143, but that would miss the point of the test entirely. Hence, just skip it on Windows.

---
 t/t0005-signals.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index ad9e604..981437b 100755
@@ -20,7 +20,7 @@ test_expect_success 'sigchain works' '
 	test_cmp expect actual
 '
 
-test_expect_success 'signals are propagated using shell convention' '
+test_expect_success !MINGW 'signals are propagated using shell convention' '
 	# we use exec here to avoid any sub-shell interpretation
 	# of the exit code
 	git config alias.sigterm "!exec test-sigchain" &&

1.8.3.1489.g15123b5

Thanks. I wasn't quite clear on how the signal handling worked on Windows, but from your description, I agree there is not any point in running the test at all.

-Peff

Shouldn't we clarify that Git exit codes only work on UNIX-like operating systems?

Felipe Contreras

Clarify where? My impression is that this issue is well-known in the msys world, and it is a platform issue, not a git issue. If somebody wants to write a note somewhere in the git documentation, that's fine with me, but I'm not clear on exactly what it would even say.

-Peff

I actually was scratching my head while reading "the implementation of raise() just calls exit(3)." part, in this:

After a bit of web searching, it seems to me that this behaviour of raise() is in msvcrt, and compat/mingw.c::mingw_raise() just calls that. In other words, "the implementation of raise()" is at an even lower level than mingw/msys, and I would agree that it is a platform issue.

I agree with both points. I can suggest to clarify the log message a bit with "the implementation of raise() in msvcrt (Microsoft C Runtime library) just calls exit(3)", but that does not address the end-user documentation issue.

I tried to summarize the issue for end-user documentation and came up with this:

The Git implementation on MinGW exits with status code 3 upon receiving an uncaught process-terminating signal, just like any program that link with msvcrt (Microsoft C Runtime library) whose raise() implementation just calls exit(3). This is different from Git on POSIX, which reports a death by receiving a signal with the exit status code (128 + signal number).

But when stated this way, it feels that it belongs to Msysgit documentation, not ours, at least to me.

Yeah, if it were mingw_raise responsible for this, I would suggest using the POSIX shell "128+sig" instead. We could potentially check for SIG_DFL[1] mingw_raise and intercept and exit there. I don't know if that would create headaches or confusion for other msys programs, though. I'd leave that up to the msysgit people to decide whether it is worth the trouble.

[1] I'd use sigaction to do that on POSIX, but I would not be surprised to find that there is no support for it in msys. :)

Yeah, I think I agree.

-Peff

Am 6/6/2013 19:40, schrieb Jeff King:

Even if we move the signal emulation closer to POSIX in the way that you suggested (if that were possible, I haven't checked), the emulation is still not complete, because we would have addressed only raise(). Therefore, personally I would like to delay the issue until there is a user (script) that depends on POSIXly exit codes.

As far as t0005.2 is concerned, its the best to just concede that Windows does not have POSIX-like behavior as far as signals are concerned, and skip the test.

We could also sweep the issue under the rug with the patch below, which works because SIGALRM does not exist in MSVCRT and is handled entirely by compat/mingw.c. But it goes into the wrong direction, IMO.

index ad9e604..68b6c3b 100755
@@ -12,9 +12,8 @@ EOF
 test_expect_success 'sigchain works' '
 	test-sigchain >actual
 	case "$?" in
-	143) true ;; # POSIX w/ SIGTERM=15
-	271) true ;; # ksh w/ SIGTERM=15
-	  3) true ;; # Windows
+	142) true ;; # POSIX w/ SIGALRM=14
+	270) true ;; # ksh w/ SIGTERM=14
 	  *) false ;;
 	esac &&
 	test_cmp expect actual

@@ -23,8 +22,8 @@ test_expect_success 'sigchain works' '
 test_expect_success 'signals are propagated using shell convention' '
 	# we use exec here to avoid any sub-shell interpretation
 	# of the exit code
-	git config alias.sigterm "!exec test-sigchain" &&
-	test_expect_code 143 git sigterm
+	git config alias.sigalrm "!exec test-sigchain" &&
+	test_expect_code 142 git sigalrm
 '
 
 test_done

index 42db234..0233a39 100644
@@ -14,9 +14,9 @@ X(three)
 #undef X
 
 int main(int argc, char **argv) {
-	sigchain_push(SIGTERM, one);
-	sigchain_push(SIGTERM, two);
-	sigchain_push(SIGTERM, three);
-	raise(SIGTERM);
+	sigchain_push(SIGALRM, one);
+	sigchain_push(SIGALRM, two);
+	sigchain_push(SIGALRM, three);
+	raise(SIGALRM);
 	return 0;
 }

...and here's the code to do just that:

index b295e2f..8b3c1b4 100644
@@ -1573,7 +1573,8 @@ static HANDLE timer_event;
 static HANDLE timer_thread;
 static int timer_interval;
 static int one_shot;
-static sig_handler_t timer_fn = SIG_DFL, sigint_fn = SIG_DFL;
+static sig_handler_t timer_fn = SIG_DFL, sigint_fn = SIG_DFL,
+    sigterm_fn = SIG_DFL;
 
 /* The timer works like this:
  * The thread, ticktack(), is a trivial routine that most of the time

@@ -1688,6 +1689,10 @@ sig_handler_t mingw_signal(int sig,

sig_handler_t handler)
sigint_fn = handler;
break;

+ case SIGTERM:
+ sigterm_fn = handler;
+ break;
+
default:
return signal(sig, handler);
}
@@ -1715,6 +1720,13 @@ int mingw_raise(int sig)
sigint_fn(SIGINT);
return 0;

+ case SIGTERM:
+ if (sigterm_fn == SIG_DFL)
+ exit(128 + SIGTERM);
+ else if (sigterm_fn != SIG_IGN)
+ sigterm_fn(SIGTERM);
+ return 0;
+
default:
return raise(sig);
}

Am 6/7/2013 12:12, schrieb Erik Faye-Lund:

That's not entirely true. On Windows, there's only *one* way to generate SIGTERM; "signal(SIGTERM)". Ctrl+C does not generate SIGTERM. We generate SIGINT on Ctrl+C in mingw_fgetc, but the default Control+C handler routine calls ExitProcess():
http://msdn.microsoft.com/en-us/library/windows/desktop/ms683242(v=vs.85).aspx

So I believe this *does* entirely fix SIGTERM (as we currently know it) on Windows. SIGINT is still not entirely clean, but we can fix that on a case-by-case basis.

Am 6/7/2013 14:00, schrieb Erik Faye-Lund:

But a call to signal(SIGTERM, my_handler) should divert Ctrl+C to my_handler. The unpatched version does, because MSVCRT now knows about my_handler and sets things up so that the event handler calls my_handler. But your patched version bypasses MSVCRT, and the default (whatever MSVCRT has set up) happens, and my_handler is not called.

-- Hannes

No, it does not:
--->8---
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

void my_handler(int signum)

{
        printf("signal: %d\n", signum);
        exit(1);
}

int main()

{
        signal(SIGTERM, my_handler);
        while (1);
        return 0;
}

--->8---

This quietly kills the process on Windows with MSVCRT's signal-implementation. In fact SIGTERM isn't raised on Linux either. Ctrl+C raises SIGINT, not SIGTERM.

Am 6/7/2013 14:46, schrieb Erik Faye-Lund:

<action type="slap" destination="forehead"/>

You are right. Your change would "fix" SIGTERM as it can be raised only via raise() on Windows nor can it be caught when a process is killed via mingw_kill(...,SIGTERM) by another process.

But then the current handling of SIGINT in compat/mingw.c is broken. The handler is not propagated to MSVCRT, and after a SIGINT handler is installed, Ctrl+C still terminates the process. No?

BTW, isn't mingw_signal() bogus in that it returns the SIGALRM handler even if a SIGINT handler is installed?

-- Hannes

Yeah, probably. I wasn't aware that it handled SIGINT, but yeah it does. So this was indeed a regression.

Yep, that's a bug. Thanks for noticing.

I've pushed out a branch here that tries to address these issues, but I haven't had time to test them. I'll post the series when I have. In the mean time:

https://github.com/kusma/git/tree/win32-signal-raise

From: Erik Faye-Lund <[email protected]>

Returning the SIGALRM handler for SIGINT is not very useful.

---
Am 6/7/2013 16:20, schrieb Erik Faye-Lund:

This is your patch to address it.

Concerning the other two patches:

* SIGINT: perhaps handle only the SIG_DFL case (for the exit code) and forward all other cases to MSVCRT?

* SIGTERM: it papers only over a general issue and should be dropped.

IMO.

-- Hannes

 compat/mingw.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index b295e2f..bb92c43 100644
@@ -1677,14 +1677,16 @@ int sigaction(int sig, struct sigaction *in,

struct sigaction *out)
#undef signal
sig_handler_t mingw_signal(int sig, sig_handler_t handler) {
- sig_handler_t old = timer_fn;
+ sig_handler_t old;

switch (sig) {
case SIGALRM:
+ old = timer_fn;
timer_fn = handler;
break;

case SIGINT:
+ old = sigint_fn;
sigint_fn = handler;
break;

1.8.3.1504.g78dbf7a

Thanks.

Perhaps. I'll have to think a bit about it, but it might very well be the sanest approach, as long as it doesn't break git_terminal_prompt (which was the reason for the change in the first place). I can't of the top of my head understand why it should, though.

Fair enough.

Yes. Unless the goal is to give Windows port pratically the same signal semantics as ports on other platforms, I do not think special casing SIGTERM (unless it is a very common signal on Windows and others are unlikely to be useful) buys us much.

Am 6/9/2013 22:31, schrieb Junio C Hamano:

I'm thinking the same. And, no, SIGTERM is not very common on Windows.

-- Hannes

I have no strong feelings on SIGTERM, but my knee-jerk reaction is the same. AFAIK, the only issue we've seen with it has been this one, which is synthetic.

That's not accurate at all. MinGW is essentially a compiler for Windows. It doesn't matter if you use MinGW, Visual C++, or any other compiler for Windows, the result would be the same.

Are we saying then that Git doesn't support Windows? That wouldn't be wise considering it's one of the most widely used reasons to argue that Git is not a good choice of a SCM; lack of Windows support.

The truth of the matter is that exit codes are platform-dependent. Period.

Felipe Contreras

Documentation/technical/api-run-command.txt

That the exit code is not the same in Windows (not msys).

Felipe Contreras

Huh? We do "exit(128 + sigint);" in mingw_raise these days, no?

Or is the signal triggered from a non-git process?

Argh, I'm blind. Yeah, SIGTERM, not SIGINT...

As people may have seen in the discussion on the earlier iteration, something like this (there may be a room for bikeshedding the name, though) that takes either a range of changes or set of patches and finds people who may be able to review them may be a good addition to our official toolchest.

http://thread.gmane.org/gmane.comp.version-control.git/221728/focus=221796

Right now, "related" is in contrib/ primarily because its design review phase is not yet finished and because it is in Ruby, which the rest of the system does not depend on.

I have some administrative comments on two issues as the maintainer.

* Do we want to add Ruby dependency?
* Do we want to keep expanding contrib/?

These have been triggered by "related", but the comments in this message are not limited to the specific topic (e.g. you can read it with s/Ruby/<any language we currently do not depend on>/).

On Ruby:

Assuming "related" is a good idea, to make it as the proper part of the system out of contrib/ when its design review phase is finished, one of these things has to happen:

1. Find a volunteer to rewrite it in one of the languages that we know the platforms our current users use already support, which means either C (not a good match), POSIX shell (not the best match), or Perl.

2. Promote Ruby to the first-class citizen status, which involves making sure people on platforms that matter do not have problem adding dependency on it (I am primarily worried about MinGW folks), and also making sure core developers do not mind reviewing code written in it.

As long as we can get as high quality reviews on changes written in Ruby as we do for the current codebase, it is OK to go route #2, and that may hopefully happen in the longer term as and there will be some people, among competent Ruby programmers, who have understood how the pieces of entire Git are designed to fit together by the time it happens.

I however do not know how much extra burden it would place to add dependencies to platform folks, so obviously the safer approach is 1 at least in the immediate future. My understanding is that msysgit folks are already having trouble with Python, and we do not want to go route #2 at least for now. Having to ship a variant of Git with NO_PYTHON is already bad enough. And that is why the option 1 above does not list Python as a possible candidate.

On contrib/:

Back when Git was very young, it made sense to bundle third-party tools in our tree's "contrib/" section to give them visibility and users convenience. Now Git ecosystem has grown to have many users who know Git and who do not necessarily come to this list, and with easy-to-use hosting sites where anybody can publish their ware and collaborate with their contributors, "giving more visibility" angle of contrib/ has outlived its usefulness. When there are multiple third-party tools that address similar needs, there is not much point picking one at random and ship it over others, and shipping all of them is simply crazy. In an ecosystem with flourishing third-party add-ons, their products should and will stand on their own.

As the maintainer, I've been thinking about closing contrib/ area for new stuff, and shrinking existing ones, either by moving stuff that are only useful within the context of Git to main part of the tree (e.g. "contrib/workdir" may move to a new directory "addons/", some of remote-helpers in contrib/ may move to "remote-helpers/", etc.), and removing others from contrib/, for this reason. Of course, interested folks can take the last version of the removed ones and continue improving them as standalone projects.

And that is why the list of possible actions in the previous part does not have "3. Keep it in contrib/ forever" as an option.

That is all for the "administrative comments" as the maintainer.

The rest is just a personal opinion.

If we were looking at a compelling and sizeable web application that depends on Rails, it is very likely that it would not make much sense to rewrite it in other languages only to avoid a new language dependency on Ruby.

But "related" is "read and extract some info out of text files, spawn a 'blame' (or two) based on that info, read to collect further info and summarize", for which Ruby does not especially shine compared to Perl, which is the language we already depend on. Because of this, I am moderately reluctant to add Ruby dependency only for this script. Unless I know people who regularly give us high quality reviews, and those who support various platforms, are fine with it, that is.

In the shorter term (read: up to 2.0), I am inclined to vote that we should go route #1 (i.e. rewrite in Perl once the design settles).

My "personal opinion" above of course assumes that everybody agrees that "related" is a good addition. If not, there is "3. not add it to contrib/ and leave it as an out-of-tree third-party project" option.

I don't have an opinion on allowing Ruby into the core, except to say that I would personally prefer *some* alternative that is more capable than shell and more modern and self-consistent than Perl. Python, Ruby, and Lua would seem to be the obvious candidates, with the latter being easiest for packagers.

For completeness, let me point out two other small advantages of contrib:

* a tool in contrib can assume that it is being bundled with the corresponding version of Git, and therefore doesn't necessarily have to go to the effort of supporting older versions of Git.

* at the source-code level, a tool in contrib can take advantage of some of the Git build/test infrastructure, though I don't know whether they currently do.

But my main point is that I think it would be easier to phase out contrib/ if there were a good alternate way of providing visibility to "satellite" projects. The relevant Git wiki page [1] is the most likely candidate, but it is a bit overwhelming due to its size, it has fallen into disuse because it was broken for such a long time, and it is not prominently linked to from git-scm.com. If it were curated a bit, it would help users find the best ancillary tools quickly. Perhaps ranking the tools based on the results of the Git user surveys would help bring the most popular to the top of each category.

Michael

[1] https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools

Michael Haggerty
[email protected]
http://softwareswirl.blogspot.com/

It is true that in-tree stuff can go in-sync with the rest, but I think that is irrelevant, as we are discussing a tool in contrib/; if it is part of the core, it deserves that benefit over tools developed out-of-tree (that need to worry about utilizing new features after a version check). After moving tools that we want to keep as a part of core out of contrib/, they will still be in-sync.

For those that alternative third-party designs and implementations for solving the non-core problems they try to solve (e.g. ciabot, continuous, blameview) can exist, it would be better for the ecosystem of they compete with their alternatives on the same ground.

That is a very good point.

One idea here could be to mirror what the libgit2 project [1] (and many others) are doing on GitHub. Use the organization unit [2] as an umbrella for the contrib projects. If necessary, put a pretty web-page on top [3].

Of course you don't have to tie it to GitHub, but they do have some nice mechanisms for showing off popularity (stars and forks).

I heard that clojure/contrib [4] went through a big clean-up recently, although I'm not sure if there was an equivalent reasoning behind it. But their guide-lines on what should go into contrib may have some good ideas [5].

[1] https://github.com/libgit2
[2] https://github.com/git
[3] http://libgit2.github.com/
[4] http://dev.clojure.org/display/design/Where+Did+Clojure.Contrib+Go
[5] http://dev.clojure.org/pages/viewpage.action?pageId=5767464

They do not do much AFAICT. For example, contrib/subtree/t/Makefile is essentially copy-pasted from Git's equivalent.

But they can do to some extend, for example "make install" in contrib/mw-to-git/ re-uses Git's Makefile to hardcode the PERL_PATH in the file, find Git's exec-path & so. I'd love to be able to use Documentation/Makefile and t/Makefile too for external programs meant to be used as a Git command.

That does not strictly imply that these commands be maintained within git.git, as we could imagine:

* Ask the user to build external programs with

make GIT_ROOT=where/git/lives/

* or, ask users to checkout the external program as a subdirectory of git.git to build it (for example, clang's build installation ask you to put clang as a subdirectory of LLVM's tree).

I think this is the most important point. A good example would be
git-multimail: for now, the shell version in contrib/ is somehow considered as the official hook to send emails, just because it is in contrib, while git-multimail is clearly superior (unless you don't have a python interpreter on your server).

I also see contrib/ as a "safe" place to live, in that the likeliness for the project to be abandonned is rather small. Especially for small pieces of code, it's easy to create a repo and throw the code somewhere on GitHub, but maintaining it is harder. Take again the example of post-receive-email, the code was originally written by Andy Parkins, but the community took over it (Andy's last commit on the script was in 2008). I'm not sure this would have been so easy with a script hosted on an arbitrary repo.

I'm not opposed to Junio's proposal to restrict contrib/ (although a bit reluctant), but I think this should be done with care, at least to give potential users a way to chose which tool to use (really, nobody want to go use https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools
to pick the right tool. It's a great list, but not a guide).

Matthieu Moy
http://www-verimag.imag.fr/~moy/

I was envisioning to sift what are in contrib/ into these four categories:

(1) Ones that deserve to be Git subcommands;

(2) Ones that are useful only in the context of using Git

     (e.g. hooks, completion scripts, credential and remote helpers);

(3) Ones that are no longer useful;

(4) Ones that primarily _use_ Git, not the other way around (i.e. opposite of category (2) which help use of Git).

The first category will live next to git-am.sh (i.e. in the longer term when we restructure the source tree into src/, lib/, etc., candidates for new scripted subcommands move with the scripted Porcelains).

The second category will be in a separate hierarchy (perhaps addons/, hooks/, ..., but I am fine if we decide to keep them in contrib/addons, contrib/hooks, etc.).

The last two categories will be removed; people are welcome to decide which category between (3) and (4) each piece belongs to, and pick up to start a standalone third-party project.

The multimail tool can be in the second category. It helps use of Git more than it is helped by using Git.

This rests on the assumption that Ruby would be as difficult to distribute as Python, which might not be the case.

This does make sense, however, I do think some parts of Git might be more maintainable if they have their own Makefile (e.g. bash completion), where it's clear where they should be installed by default.

Either way, the user might want to do 'install-all' or 'install-addons', to install all these things, and I think a good rule of thumb is that if we don't want 'install-all' to install certain script (eventually), then that script probably doesn't belong in 'contrib' (or anywhere in Git).

That might make sense for the shorter term, but in longer term I see Perl as declining in favor of other languages. It's only a matter of time before Ruby surpasses Perl in popularity, and soon enough new contributors to the Git project will have problems trying to improve Git because parts of it are written in a language they are not familiar with, and have trouble learning (isn't that already happening?).

The Ruby vs. Python is another question altogether, I could go into detail about why I think Ruby is a better choice, but my point right now is that Perl is not a good choice for the future.

Felipe Contreras

Good thing you are being objective and leaving out the Python 3.0 mess, the long legacy of backwards compatibility in the Perl community, the active community behind it, its extensive portability support, and fail to mention the lack of an equivalent to CPAN. We wouldn't want facts to get in the way of a personal bias would we?

Just thought I'd push back on the FUD. People have been saying Perl is going away for decades...

Yves

None of that has anything to do with Perl's popularity.

Perl has been going away for the last decade [1], and will continue to go away. Perl is going away, and that an undeniable fact, and if you are not interested in discussing on the basis of reality, I'm not interested in discussing with you.

[1] http://www.tiobe.com/content/paperinfo/tpci/images/tpci_trends.png

Felipe Contreras

The linchpin of your argument is that Perl is dying. Let's assume that the TIOBE index is a reliable basis for making business decisions--it's not, but let's pretend--the graph you linked to doesn't even seem to support your conclusion (or am I missing something?). It looks like Perl's popularity has pretty much been constant for at least two years. It's apparently not increasing in popularity, but this isn't an electrocardiogram (i.e. flat-lining is not dead or even dying). The same graph shows that Ruby's popularity also hasn't changed very much since 2007 after its initial surge.

Now, it's probably too off-topic to pick apart TIOBE's methodology here, but suffice it to say that, like any trend indicator, it's only as useful as your knowledge of its limitations, and this has been discussed enough elsewhere.

It's true that Perl isn't soon going to win any trendiness awards, but the same reasons that made Perl a good choice for git so many years ago are still there and then some. You would probably also be surprised at the number of new kids learning Perl.

I guess I just denied the "undeniable fact" that Perl is going away, so maybe I'm one of those with whom you do not want to discuss this, but, for my part, I am willing to consider other evidence for the claim. As I pointed out, the evidence shown so far (one reference to the TIOBE index) isn't nearly enough to settle the matter. I also apologize for dragging this out if this thread is judged to not be worth a whole lot.

Charles McGarvey

I don't think the usefulness of a language should be judged by hits on a web site.

Personally I would like the Git client to be packaged with as few dependencies as possible. Right now that seems to require Shell, Sed, Awk and Perl. The documentation has other requirements, but a prebuild tar file is available.

I would have the rest of the distribution be bundled as something like "git-utils" which could have a subdirectory for each support language. Then one could even make available alternative implementations of higher level utilities and people could decide if support of a specific language was useful to them.

Most such extension code is simple, although more complex than suitable for just Shell/Sed/Awk. People in each language community could provide code which meets the needs of their community, and the Git project itself would not need to make (Solomon like) decisions about what extension languages to support.

--
Barry Fishman

Nobody is judging the usefulness of a language, I have plenty of arguments for that, but this is about popularity.

I would be perfectly fine with replacing shell, sed, awk and perl with ruby. But that's not what you are arguing, is it?

Felipe Contreras

I used "usefulness" in its general vague sense. It is useful to be popular, I don't make choices solely on that or I would be writing everything in Java.

I'm talking about porcelain code and not core functionality which should be left in C. I'm saying that you should be free to provide Ruby implementations of all such superstructure. And the same can be done by (but not required by) the Perl, Python, Tcl and even C, Haskel, Guile and whatever communities. Most such higher level code is fairly trivial, and if the file names are kept the same, the same test procedures could be run.

I don't think the cost of duplication of code functionality is that significant, since it would bring new people to the project. After all this is a free project and not a commerical venture. It certainly helps porting to new platforms. Separate language communities would be maintaining their own contributions, with their own experimental directories.

Translating the same functionality to multiple languages requires careful reading which can help identify some hidden bugs.

Barry Fishman

Straw man.

I don't know what you are saying, but it clearly has nothing to do with the point.

Perl is declining, and it would be wise to use another language instead of it.

Felipe Contreras

You want a simple statement. I don't particulary like Perl, but it has worked well for the project.

If you have a better solution, then write all the code to replace it, and demonstrate with a significant number of active users that your solution works out better in practice.

Wasn't that how Git started?

Barry Fishman

It would serve it less and less as the years go by.

False dichotomy fallacy. I don't need to do that.

Felipe Contreras

As someone who builds minimalist builds (firewalls, openwrt, raspberry pi, etc), having to pull in a full ruby install to get git installed would not be something I'd like to see.

Yes, openwrt (and I) can build our own version, but that's a pain. I tend to build my tight systems from Debian and it's nice to be able to use stock packages.

I tend to use git for sysadmin type functions as much as for development, so it's very useful even on such small and slow platforms.

If you can, you should leave just enough of a stub in place so that people who don't know about the change and try to run the stuff that used to be in contrib/ get a message pointing them to the new home.

David Lang

You wouldn't _have_ to, just like you don't _have_ to install Python right now.

Felipe Contreras

As one of the people who helps maintain git packages in pkgsrc, my initial reaction is negative to adding a ruby dependency. There are several not-entirely-related reasons:

git is a core tool that people use on almost the smallest of boxes, perhaps even replacing rcs for managing local config files. On such machines, even perl may be large, but a second scripting language seems excessive. On a NetBSD 6 i386 system, the size of the ruby193-base binary package (as installed) is 25 MB (vs 15 MB for the git base package, which lacks gitk and docs). (Presently, the git base package defaults to requiring python and installing the git_remote_helpers, but I think that's a bug.) perl is 54 MB.

I am unclear on how mature/stable ruby is. perl has a good track record over the last many years. In particular, no one in pkgsrc has felt the need to support multiple concurrent versions of perl. But there presently exists both 1.8 and 1.9 in pkgsrc (and there are multiple python verions). So given how critical git is on many systems, I'd ask if the ruby requirement is for a stable vs old vs bleeding-edge version, and how that is expected to function over the next 5 years. (With perl, the answer seems to be "any half-way modern version of 5.x is fine, from unreasonably old to the latest release".) By stable, I don't mean that a particular ruby release works well. I mean the experience of having code that depends on ruby over many years, and whether one can just use whatever ruby happens to be there, or whether it's effort to manage having an acceptable version.

From a packaging viewpoint, dependencies are costly, because they force build and installation of them before the package can be built. In a source-centric packaging environment (where it's normal to build, rather than only having pre-built packages), the question is if the git package needs ruby, which is a different question than whether something in git which may be optional needs ruby. So if ruby, or something else, is needed for optional components, it would be really nice if the build system were such that it was simple (via arguments to configure, selecting subdirs, or something functionally similar) to build the main part, and then the ruby part as a separate build. Then, it would be pretty easy to have git-ruby package that has the ruby parts. But if the ruby part isn't considered optional, that won't work. (Note that the usual GNU/Linux approach of split binary packages doesn't really address this, because as I understand it you need the union of the dependencies installed to build once, and then tar up the resulting bits separately. So that fixes the problem for people that install binaries, but doesn't help building from source.)

tcl/tk is another dependency, but it seems limited to gitk. pkgsrc has a separate scmgit-gitk package, which is relatively easy to maintain because it just involves selecting subdirs to build. So that's an example of a good way to do it, from the source-based packaging viewpoint.

Finally, I realize that most people on this list will build git directly From sources. While that of course has to be smooth, I think great weight should be given to how packaging systems use releases and how that impacts packaging effort and the eventual user experience. I would guess that over 99% of git users are running binaries built from a packaging system.

You can compile Git without any of them.

That's only the default, if the default doesn't suit you, don't use the default.

Besides, that doesn't carry any weight if Perl code is replaced with Ruby code, or Python.

It is quite possible to slowly rewrite the Perl scripts, preferably move as much code as possible to C, but the rest to shell, or Ruby. For Ruby, we could maintain both versions at the same time until the new versions are ready, and then the Perl dependency gets deprecated. In this interim time, people that don't want Ruby could use the Perl versions. But I think this is overkill. Yes, ideally we wouldn't want to depend on both Ruby and Perl, but I think it's OK to do that for a while, until the Perl scripts are rewritten.

In the end my point remains unchanged; Perl is declining, so it would be wise for the future to use another scripting language instead.

Felipe Contreras

Perl use may or may not be declining (depending on how you measure it), but are you really willing to take on the task of re-writing everything that's in Perl into another language and force all developers of scripts to learn that other language? what's the ROI of this?

Perl isn't going to disappear any time soon. What makes you think that whatever language you pick to replace Perl is going to be more stable than Perl is?

and, like the parent poster, by 'stable' I mean from the compatibility point of view.

What are the odds that the 'newer' language that you pick is going to pull a "python 3" on you?

There have been a very large number of scripting languages show up, make a lot of press, and then fade in favor of other languages while Perl has continued. It's not the sexy languange nowdays, but it's there, reliable, and used so heavily that there's really no chance of it dissapearing in the forseable future.

David Lang

Let's not talk hypotheticals. git-svn.perl (+ perl/SVN/*.pl) is absolutely massive. It's an incredibly useful tool in that it actually works, and that there is nothing replacing it in the foreseeable future. This monster was written almost entirely by one brilliant person, and nobody is going to rewrite it. We don't start a huge discussion about what languages are "approved" before accepting such a contribution: if the contributor wants to write something in a dominant language (Perl in this case), and it's going to be useful, we merge it. End of story.

All this planning is a colossal waste of time, in my opinion.

Why are we discussing something that is indeterminate? It is impossible to foresee the future, but that is no reason to freeze _present_ development.

Various programming languages have different philosophies, and have grown in different ways. What matters is that some of them have a large number of users, and we're talking about one such example.

This has to be a rhetorical, because I don't imagine you expect anyone to predict the future. As Felipe already pointed out Ruby 2.0 is a good sign.

Nobody claimed that "press coverage" is a good metric. We can only talk about facts, and Felipe already showed you a TIOBE index graph. If you have overwhelming _evidence_ that Ruby is a weak language that will die soon, share it: otherwise, I see no value in this discussion.

Well, Felipe is saying that Perl is dieing and we should re-write everything that exists in Perl to Ruby.

Part of the reason for the discussion now is because not having similar discussions in the past have caused problems.

and it's not a reason to throw away existing stuff based on the argument that Perl is dieing

TIOBE index graph is "press coverage" as far as I'm concerned.

I'm not saying that Ruby in particular has a fatal flaw, I'm just questioning the "Perl is dead, re-write everything in Ruby" mantra.

The language that you choose to use when writing a new application is related to things related to that type of application.

Ruby is not an extremely common language for sysadmins to use.

Perl remains a common language for these sorts of tasks, even if it's not used for user visible applications.

Arguing that Perl is dieing, we need to abandon it is just wrong.

David Lang

I don't agree with that opinion. More generally, I think the entire discussion on what _should_ or _should not_ be done is rubbish. What _will_ and _will not_ happen depends on the patches contributors send.
If a contributor sends a patch rewriting git-svn in ruby, then we
have a discussion: is anyone bored enough to pick up the task in the first place?

Well, that's your definition of "press coverage" then. TIOBE index is generated from scraping the web to figure out which languages are "living", based on discussions between programmers (and yes, "press" articles). I do not have conclusive or "undeniable" proof that perl is dying, but the trends are indicative of a decline.

I think Felipe is using the argument that perl is declining to answer the question "why didn't you write git-related in perl instead?"; that's it.

A question which nobody asked, I presume?

I think we heard enough from packaging folks that a new dependency is unwelcome. Also we heard from no regular/high-value reviewers that they feel comfortable reviewing additions in Ruby.

So at least for now, the conclusion is to take approach #1, i.e. somebody who finds "related" a good addition rewrites it in Perl to promote it out of contrib/ once the design issues settle (of course it is still a possibility that no such volunteer appears).

We'll think about it if and when we get major ruby patches, and see perl patches declining. One git-related.rb doesn't say anything.

What are you talking about? Which are these "packaging folks" we heard from?

Correction; *current* regular/high-value reviewers.

Regardless of what you conclude, Perl still continues to die, and by clinging to a dying language, all we do is hurt the project.

There's tons of people that are familiar with Git and Ruby, but these people are not in this mailing list because there's nothing for them to discuss, because Git doesn't use Ruby. By making conclusions based on the comments from people who do follow the mailing list *currently*, you are being biased towards the status quo.

It's no surprise that you decided to keep the status quo.

We could change, and we would probably receive a big influx of fresh contributors happy that they can contribute in their favorite language. But we won't do that, why? Because you already decided that's not going to happen, because you are making the false assumption that things in the future can only be like things have been in the past.

Felipe Contreras

Dscho is one of the primary people behind msysgit effort, and I consulted with others from the circle with an draft before I sent the message to the list for sanity checking (fearing that I may be worrying about adding new dependencies needlessly). Jonathan packages git for Debian and he is negative on adding new dependency needlessly. It was unexpected that we hear from a pkgsrc person but the response was also negative.

That is exactly what I meant.

The code review is not only about following best practices in the implementation language. If somebody who is an expert in a language we do not currently depend on, but who does not know how the parts of Git are supposed to fit together enough to judge the soundness of the design of new code written in that new language, or does not know how the tests, documentation and log messages are supposed to written around here, that person cannot be the only reviewer for changes written in that language to ensure quality standard.

The reviewer pool for code written in a new language _must_ be seeded by some from the current set of reviewers whose judgement I/we can trust.

He said he won't do it, but I said I would. Doesn't that solve the problem?

I don't see any comment from Jonathan.

You mean Greg Troxel? He is only one of the persons that help, and I did shut down his argument, didn't I?

By that standard nothing will ever change. Ever.

Even twenty years from now, you will still only trust people that are familiar with shell, Perl, and C. Because the only way to gain your trust, is by being proficient in shell, Perl, and C.

Felipe Contreras

I don't see why a trusted person cannot learn a new language and convince the community to give it a try (well given that enough reviewers support the new language, which was Junio's point). --
Duy

I do. Raise your hand if you are interested in giving a try to Ruby for Git's core given that somebody gives convincing reasons?

How many hands do you expect?

Felipe Contreras

Personally, no additional runtime dependency > Ruby > Python. I don't think Ruby is available on SunOS and I prefer not to build and install Python nor Ruby myself to be able to use Git. So no hands from me.

If not many hands show up, the Git community clearly is not ready to adopt Ruby. Maybe ask again next year when Ruby is getting more popular?
--
Duy

You forgot to list the current ones; shell, perl, python.

It doesn't surprise me that you stopped at an assumption, instead of making sure.

And they will never be. Nor Ruby nor anything else, which was precisely my point.

You will stop again with another assumption, without ever giving it a chance.

Felipe Contreras

Correct. The opinions of inactive community members and non-contributors are less useful.

Okay, so here's the deal: commit a lot of good ruby code to contrib*, and attract users/ contributors. Eventually, if you're right about git.git growing a healthy ruby ecosystem, we'll get ruby in core. As I've said multiple times, this agenda-based approach just sucks, because we can't predict the future.

* I'll help out in whatever little way I can

Oh, by the way: I have a project idea. There's this really popular project called hub[1] that has an implementation of the GitHub API in ruby. Unfortunately, it's a terrible piece of software because it creates an extra layer of indirection by putting a ruby wrapper on top of git, and this slows git down: I cannot tolerate even a microsecond of delay in git. Maybe it's worth ripping out the GitHub API implementation and writing some useful subcommands?

[1]: https://github.com/defunkt/hub

Hi Ram,

I humbly suggest to treat other people's contribution with the same respect you want yours' to be treated.

Just a suggestion,
J

What?! When did I disrespect other people's contributions? git.git is what it is today because of everyone's contributions: if I disrespected them, why would I work on improving git?

My opinion has nothing to do with me, or my contributions. I have already stated multiple times on the list that I take no pride in my contributions whatsoever*: I have no ego to speak of. I said "the opinions of inactive community members and non-contributors are less useful [than those of active contributors]", and I'm still scratching my head over what you inferred. Do you think that the opinions of inactive community members and non-contributors are _more_ valuable than those of active contributors, or am I missing something?

* You'd know that if you read emails on the list. But you don't, for some mysterious unstated reason.

I am not Dscho, but it probably is worth saying this anyway.

6d297f81373e (Status update on merge-recursive in C, 2006-07-08) stole merge-recursive.c from git-merge-recursive.py with an explicit purpose of making sure that those without a working Python can perform such a core operation like "merge" with Git without extra forking.

The person who worked on it, as long as he knows that the project not just accepted the patch and kept using the code but also that the project understood the rationale behind that change, does not necessarily have a reason to appear every week to interject comments in discussions on any part of the system, even to proposed changes to merge-recursive.c, as long as the original thing the change meant to address is not broken (e.g. removing merge-recursive.c and add it as a merge strategy written in Python or Ruby might trigger "huh", but ditching merge-recursive.c and replacing it with merge-replay.c as long as it works would be a "meh" for him).

When otherwise silent old-timers feel a need to come during a discussion that might affect the course of the project in a major way, we should pay more attention, not less, to what they say (I am not saying "we should blindly follow"). They can explain why some things are as they are, why some changes that may look like a good idea did not work out and how they failed, etc.

Certainly the opinions from them are no less valuable.

I say we should pay attention to the arguments, not the people, that's ad hominem.

Felipe Contreras

No, I said we should try to move away from Perl. Move stuff to C, shell scripts, and yeah, Ruby.

Who said anything about throwing away code?

Who said we need a language commonly used by sysadmins for our Git core?

Ruby is pretty much a replacement for Perl. For every task Perl is good, Ruby also is. Ruby's syntax even borrows from Perl.

The difference is; Ruby is better for many more tasks that suck in Perl.

Straw man. Nobody is arguing that.

I said we should try to avoid it, not abandon it immediately.

Felipe Contreras

But that's exactly what we are asking the newer generations of developers; to learn another language. Fewer and fewer new contributors will come with knowledge of Perl.

Ruby 2 speaks volumes on that front.

Yet it's declining, more and more every year. And the more the time goes by, the more we hurt ourselves.

Felipe Contreras

That ignores the 99% of people who use packaged versions. The question is really "Should the standared approach for building be to use or not use dependency X?". Really this should be expressed in the README, and it creates expectations for someone who just installs the git package in terms of whether pieces of functionality are there. Packagers generally should be reading the README and including required/recommended dependencies and not including optional dependencies (in the main package). The information in INSTALL is pretty reasonable, but it doesn't really clearly say "if you hand someone git built without perl, it is { perfectly ok but missing a fringe optional feature | deficient
because "git add -p" won't work }. I'm leaning towards the "deficient"
camp.

So "you can compile git without X" should really translate into "when one runs the default build following the instructions, and does not take affirmative steps to use X, X should not be used or depended on". If it doesn't mean that, it doesn't help the packaging/expectations discussion.

It's of course fine that one can hand-compile a smaller than standard but still useful subset. But that's entirely different from the definition of normal.

It's not about what I want. It's about making choices that affect other people, and trying to find a plan that will be overall reasonable; that's the essence of stewardship in packaging. Compiling for just myself is far easier.

Have you asked the SBCL or Google-Chrome package maintainers what they have to deal with? I believe they're packaging nightmares. GHC/ Haskell projects aren't far behind. Git is probably the _last_ thing to be complaining about when it comes to packaging.

Sure, people want to run Git on embedded devices like Rpi. The core is already in C and Bourne Shell, and I don't see anyone rewriting that in Ruby. No cause for concern.

So whom is this extra dependency affecting, if 99% of users are using packaged versions? So, it's just extra burden for the package maintainers (and users on source-based distributions)? git-svn and git-send-email are already separate packages on Ubuntu/Debian because they depend on lots of CPAN packages that can be non-trivial to install for new users. If we do get one important Ruby script, ship it as a separate package: done?

At the moment, there's just contrib/git-related that depends on ruby. Can we just stop planning centuries in advance, and tackle the problem when it arises? It remains to be determined whether or not git.git will grow a healthy ruby sub-ecosystem. If it does, package maintainers will be inconvenienced slightly. Otherwise, we'll just throw out the ruby code that's rotting in our tree, and get on with our lives.

The direction of the project is not decided on the basis of some vague future packaging expectations. It's decided on the basis of what patches come in from contributors, and everything else is secondary. If people want to write ruby, they will write ruby. Whether or not the git project welcomes their contributions.

It would be nice if contrib/ files supported the usual "make; make install; make clean" targets. That's missing functionality that does matter to at least one packager.

It would be nice if the dependencies associated to each piece of functionality or makefile flag were documented more clearly. Currently when e.g. features of gitweb gain dependencies I don't notice until the testsuite fails.

Jonathan

Here's a good example how to do that:
https://github.com/felipec/git/blob/fc/master/contrib/remote-helpers/Makefile

Felipe Contreras

The 99% of people who use packaged versions wouldn't care about the additional dependency.

It is exactly about what you want.

You use the argument that 99% of the people use packaged versions, yet you ignore the fact that 99% of the people don't care about a single extra dependency (specially one that would be transitory). It is all about 1% of the users, in fact, not even that, because of this 1% of users who dread extra dependencies, most of them would be happy that it's only temporary, and eventually a heavier dependency would be replaced with a lighter one. It is for all intents and purposes only you, the person you are speaking on behalf of.

If the Git project considers a new dependency that would be needed in addition to Perl for a finite period of time, your argument does absolutely nothing to block this route.

Felipe Contreras

Hi Greg,

My initial reaction, too. It was hard enough to get Perl included with Git for Windows (because of that pesky Subversion dependency).

As you can see from the commit history, I was the primary force behind trying to get everything "core" in Git away from requiring scripting languages (I think it is an awesome thing to provide APIs for as many languages as possible, but a not-so-cool thing to use more than one language in the core code). It does not seem that anybody picked up that task when I left, though.

Ciao,
Johannes

Nevertheless, we had to do it, and we did it. We will do it again, if we get enough important code written in Ruby.

Rewriting everything in C? Is anyone bored enough to pick up this task? Bourne shell is a great language for prototyping; git-rebase.sh (and friends), git-stash.sh, git-pull.sh are doing just fine. Sure, it makes sense to do heavy-lifting in C, and this is happening as it has always been happening (remember git-commit.sh?). If you followed the list emails, you'd know that Felipe is looking into delegating large portions of the work done by git-rebase.sh to sequencer.c.

Anyway, all this talk about some hypothetical ideas just bores me. What matters is what is currently happening. And nobody is actively rewriting the "core in Git" in C, so I don't see the point of discussing anything but patches.

Hi Ram,

That is not quite correct. *I* did it. Not *we*. And I will not do it again.

I am a bit bored by this hypothetical talk. This hypothetical "we will do it again", to be precise. Given my experience, it would be very painful if "enough important code" was written in Ruby. Nobody would help me "do it again". Just like nobody helps right now to upgrade to a newer Perl. Feel free to prove me wrong. Until that time, I will firmly believe that there is no "we will do it again".

So here is a chance to prevent that: not repeat the mistake, and stay away from language hell by avoiding to require yet another language.

As you know, there are very good reasons why I do not follow those mails.

Exactly. Nobody really cares about keeping Git portable enough. Hence my impression that this idea to start requiring yet another language for core parts of Git is a bit misguided, and only logical from the point of view:
"If you don't like it, why don't you install Linux?" (which, just in case you wondered, is a pretty naive way of looking at the real world).

Ciao,
Johannes

When I say "we", I mean the git community. You were incidentally part of the git community when we got perl included in git-for-windows. Don't make laughable assumptions about your indispensability: the community will run fine without you (as it will run fine without me, fc, or even jc). There are enough and more smart diverse people to replace the inactive people.

What is this "mistake" that you're talking about? We never should have merged git-svn.perl in our tree, and lost out on an incredibly useful piece of software?

Let me make one thing clear: contributors are priority #1, and everything else is secondary. git.git is the sum of contributor wishes, and we exist to keep contributors productive. Yes, we have to force some boring work onto contributors: nobody likes to write documentation, and we have guidelines to make sure that it gets done, for instance.

If enough people are interested in porting git-related to Windows, they will figure out how to ship ruby with git-for-windows. It is not rocket science.

No, I don't know the reason. I would guess that you are disinterested.

I am personally disinterested in git-for-windows, but I wouldn't make such a sweeping statement: there are some people interested in git-for-windows.

I think it should be "the Git for Windows community", and my feeling is that the community developing Git for POSIX systems is far more active than the one making it work for Windows (although we may now have more windows users than unix users).

Reading Git for Windows's FAQ
( https://github.com/msysgit/msysgit/wiki/Frequently-Asked-Questions ), it seems rather clear that the TODO-list is already long to have a correct Perl support (I'm quite admirative of the work done already). The POSIX guys shouldn't move faster than the Windows guys can follow.

Matthieu Moy
http://www-verimag.imag.fr/~moy/

That is a very good summary.

It does not mean everybody must always crawl at the same pace as the slowest people. But it is one of the important things we should consider, when we have choices to make (e.g. "do we write this in Python", "do we write it using this Perl module we haven't depended on", etc.), to pick the one that does not make others work harder than necessary---it affects the trade-offs.

Ha! So, it's subversion and perlxs. I was wondering what was so non-trivial about shipping a perl interpreter with Windows when dscho mentioned it.

Hopefully, I've done a little bit to improve the situation by pushing svnrdump into core (I don't know if it has any users). On the git side, git-remote-testsvn is still a toy (which is a product of many months of work by db + 2x SoCs) compared to git-svn.perl. Yeah, supporting subversion is extraordinarily painful.

If I can be excused for being frank, and saying something potentially blasphemous:

I think he way forward on Windows is an implementation like libgit2 or git# with some sort of gui/ide integration. I never understood why users on Windows want to use something as POSIX'y as git.git. Wouldn't they prefer some visual-studio integration thing? *scratches head*

Why is there only one way forward? Why do you get to pick it, given that you've said you're not interested in working on it?

[...]

Plenty of users on Windows use a command line. I have even been such a user from time to time. I'm quite grateful for Dscho et al's work on making that less painful.

Jonathan

I agree completely. It's rare that I use Windows now, but a few years ago I installed Cygwin on any machine that I would use in any serious capacity. I haven't needed to do this since I started to use Git; the Windows installer ships all of the POSIX utilities that I need (with the possible exception of 'tree', and the caveat that scp can't handle files >= 2GB).

I'm very appreciative of the work that's gone in to Git for Windows, and from my perspective it's a pleasant coincidence that it includes a POSIX shell and associated tools.

MMR...

Whether it's based on POSIX is an implementation detail for the user. The real question is more command-line Vs GUI than POSIX/Win32. Some Linux users like GUI, some windows users use command-line. I tried IDE integration with EGIT, and quite frankly I ended-up doing all the Git stuff in a terminal next to Eclipse.

Visual Studio now has official Git support from MS (based on libgit2 if I understood correctly). That's cool, but not a reason to kill msysgit IMHO ;-).

Matthieu Moy
http://www-verimag.imag.fr/~moy/

Oh, I'm not interested in killing anything. If people want msysgit, they will work on it: I'm nobody to say otherwise. I was just curious to know why msysgit is suffering from a lack of attention: fewer users?

I see. But isn't it possible to implement a CLI in libgit2 too, no?

Yes (there have actually been several attempts at this like
https://github.com/Romain-Geissler/git2 and
https://github.com/vfr-nl/git2/), but there are a *lot* of stuff that are in git.git and not in libgit2.

I'd love to see Git re-implemented on top of libgit2, but that's not going to happen tomorrow :-\.

Matthieu Moy
http://www-verimag.imag.fr/~moy/

Specially not if we *always* keep the status quo, and don't make better use of C and scripting languages. One of the advantages of Ruby is that it can be very easily extended from C. I have never seen an easier way to write bindings than in Ruby. If we allowed Ruby to be in the core, it would make sense to create official Ruby bindings, and that would create an enormous motivation for libigit/libgit2 to be complete.

But if we always keep the status quo, there will never be that much motivation for such an effort.

Felipe Contreras

That's fine, I can do it. I bet it will be easy.

While at it, why not re-evaluate the whole msysgit approach? I bet we don't need a whole separate project just to create a Windows installer. I've written Windows installers before, it's very easy to do from Linux.

To the detriment on the project.

Felipe Contreras

Yeah, taking the pain out of msysgit packaging would be a great way to counter this new-dependency-fud. The main problem, as mm pointed out is subversion + perlxs [1]. Any idea how to tackle that?

[1]: https://github.com/msysgit/msysgit/wiki/Frequently-Asked-Questions

Hi Ram,

Yeah, this is the main problem, and you probably will end up with a much better (Linux-based) solution than the people who contributed to the Git for Windows project in all those years since August 2007.

Surprise me, with code,
J

Nobody seems to mention it yet. There's another reason behind the C rewrite effort: fork is costly on Windows. The C rewrite allows us to run with one process (most of the time). This applies for shell, perl and even ruby scripts because libgit.a is never meant to be used outside git.c context (unlike libgit2). In this regard, ruby is just as bad as currently supported non-C languages.
--
Duy

Are you sure?

---
#!/bin/sh

cat > /tmp/test <<EOF
require './git'

(1..100).each do |e|
puts \git rev-parse HEAD~#{e}\
end
EOF

strace -o /tmp/log -e fork,clone ruby /tmp/test
cat /tmp/log
---

---
clone(child_stack=0x7f84131dbff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f84131dc9d0, tls=0x7f84131dc700,
child_tidptr=0x7f84131dc9d0) = 17455
+++ exited with 0 +++
---

I wrote a simple Ruby extension to access Git builtins so git rev-parse actually calls cmd_rev_parse directly. I don't know of any other language that supports so much extensibility. Of course, as soon as one command does exit(), the script ends too. It could be useful to do experiments though.

Felipe Contreras

#include <builtin.h>
#include <cache.h>
#include <fcntl.h>

#undef NORETURN
#undef PATH_SEP

#include <ruby.h>

static VALUE shellwords;

struct cmd_struct {
	const char *cmd;
	int (*fn)(int, const char **, const char *);
	int option;
};

#define RUN_SETUP (1 << 0)
#define RUN_SETUP_GENTLY (1 << 1)
#define USE_PAGER (1 << 2)
#define NEED_WORK_TREE (1 << 3)

static struct cmd_struct commands[] = {
	{ "rev-parse", cmd_rev_parse },
	{ "show", cmd_show, RUN_SETUP },
};

static VALUE git_rb_backticks(int o_argc, VALUE *o_argv, VALUE ctx)

{
	int argc, i, old;
	int pipefd[2];
	const char **argv;
	char buf[0x1000];
	VALUE command;
	int do_read;
	struct cmd_struct *cmd = NULL;
	const char *prefix = NULL;

	if (strncmp(RSTRING_PTR(o_argv[0]), "git ", 4)) {
		VALUE port, result;
		port = rb_funcall(rb_cIO, rb_intern("popen"), 1, o_argv[0]);
		result = rb_funcall(port, rb_intern("read"), 0);
		rb_funcall(result, rb_intern("chomp!"), 0);
		rb_io_close(port);
		return result;
	}

	command = rb_funcall(shellwords, rb_intern("shellsplit"), 1, o_argv[0]);
	rb_ary_shift(command);

	argc = RARRAY_LEN(command);
	argv = xcalloc(sizeof(*argv), argc);

	VALUE *rarray = RARRAY_PTR(command);
	for (i = 0; i < argc; i++)
		argv[i] = RSTRING_PTR(rarray[i]);

	old = dup(1);
	i = pipe(pipefd);
	dup2(pipefd[1], 1);
	close(pipefd[1]);

	for (i = 0; i < ARRAY_SIZE(commands); i++) {
		struct cmd_struct *p = &commands[i];
		if (strcmp(p->cmd, argv[0]))
			continue;
		cmd = p;
	}

	if (!cmd)
		rb_raise(rb_eArgError, "unknown command: %s", argv[0]);

	if (cmd->option & RUN_SETUP)
		prefix = setup_git_directory();

	i = cmd->fn(argc, argv, prefix);
	rb_last_status_set(i, getpid());

	fflush(stdout);
	dup2(old, 1);

	i = read(pipefd[0], buf, sizeof(buf));
	if (buf[i - 1] == '\n')
		i -= 1;
	buf[i] = '\0';

	return rb_str_new(buf, i);
}

void Init_git(void)

{
	rb_require("shellwords");
	shellwords = rb_define_module("Shellwords");
	rb_define_global_function("`", git_rb_backticks, -1);
}

I'm not saying you can't. I'm saying it's not meant to be used that way. Which means there may be problems lurking around. You can write a ruby extension to access libgit.a, sure, but how many people on this list understand git design and limits _and_ ruby's good enough to spot the bugs? If a bug is found and requires major restructuring in libgit.a, how are you sure it's worth the effort and does not destablize the rest of git? A better way to do it is linking against libgit2.

--
Duy

Code is code. If something is not meant to be used in certain way, you fix it.

I'm not using libgit.a, I'm using the builtin commands. This is exactly the same code you run when you type 'git foo'.

Now you are changing the subject. Does that mean that you accept that 'fork' wouldn't be a problem when writing Ruby scripts?

As for the people that know Git and Ruby; they can learn. Didn't you just said that you didn't see any problem with the community learning a new language?

There is no need to destabilize anything. I just showed you 100 lines of code that are able to run git commands without forks, and without changing anything in libgit.a.

I would rather use what the rest of Git uses. It doesn't make any sense fragment even more the code, and make Ruby scripts 2nd class citizens along the way. Plus, any script that tries to use libgit2, would certainly need more than 100 lines.

Felipe Contreras

Code is code. Bugs can be hard and easy. Hard bugs take a lot of time and may not be worth it after all.

There are a lot of static variables in builtin/ (and outside too), which make it non-entrant, or at least not safe. fork provides a process space isolation, some depend on that. And there are die() everywhere. Good luck controlling them.

I said nothing about the community being ready _now_, did I? When you have the support for Ruby in Git, sure go ahead.

And how do you deal with, for example die(), or thread safety?

Duy

So?

Process space isolation from what?

Done.

@@ -1,6 +1,7 @@
 #include <builtin.h>
 #include <cache.h>
 #include <fcntl.h>
+#include <ucontext.h>
 
 #undef NORETURN
 #undef PATH_SEP

@@ -8,6 +9,8 @@
 #include <ruby.h>
 
 static VALUE shellwords;
+static ucontext_t main_context;
+static int status;
 
 struct cmd_struct {
 	const char *cmd;

@@ -73,7 +76,14 @@ static VALUE git_rb_backticks(int o_argc, VALUE

*o_argv, VALUE ctx)
if (cmd->option & RUN_SETUP)
prefix = setup_git_directory();

- i = cmd->fn(argc, argv, prefix);
+ getcontext(&main_context);
+ if (status == 0) {
+ status += 1;
+ i = cmd->fn(argc, argv, prefix);
+ } else {
+ i = 1;
+ }
+ status = 0;
rb_last_status_set(i, getpid());

fflush(stdout);
@@ -87,9 +97,19 @@ static VALUE git_rb_backticks(int o_argc, VALUE *o_argv, VALUE ctx)
return rb_str_new(buf, i);
}

+static void bye(void)
+{
+ if (status != 1)
+ return;
+ status += 1;
+ setcontext(&main_context);
+}
+
void Init_git(void)
{
rb_require("shellwords");
shellwords = rb_define_module("Shellwords"); rb_define_global_function("`", git_rb_backticks, -1);
+
+ atexit(bye);
}

If they can learn Ruby five years from now, then can learn it now.

You are going in circles.

See above for die(), and I don't see many perl or shell scripts with multiple threads, why should the Ruby scripts have more than one thread?

Felipe Contreras

Manipulation of global variables. Here are a few examples off the top of my head:

Try running "git diff" from your Ruby hook, then try running "git diff-files" within the same process. I believe the latter will start respecting porcelain diff config like diff.mnemonicprefix. To clear state you need to reset a list of global variables back to their initial states (some of which are the BSS-default zero, but some of which are not).

Try running "git log" followed by another "git log". The log family of commands does not clear its marks from the commit objects, since it expects to exit after the traversal. The second log will sometimes give wrong answers if its traversal overlaps with the first (e.g., commits marked SEEN or UNINTERESTING that should not be). You can add a call to clear them at the end of the process, but that does not cover any cases where we die().

These are problems that can be solved. But there is a lot of work involved in finding these subtle bugs and coming up with fixes. I think you would be better off working on an implementation of git that was designed from scratch to work in-process, like libgit2. And it even has an actively developed and maintained Ruby binding[1].

libgit2 doesn't have feature parity with regular git yet, but there are many clients based around it that use the library internally for speed, and then exec regular git to fill in the gaps.

-Peff

[1] https://github.com/libgit2/rugged

So you are in favor of never ever having an official Git library. Got it.

There's a reason why the Git project doesn't use libgit2, and for the same reason the official Ruby scripts should not use it.

As history indicates, the Git project will never have any pressure to fix it's re-entrancy and re-run issues, so these issues will remain there forever.

Only if you allow code that exposes those issues will there ever be any pressure to fix them.

Felipe Contreras

No, I didn't say that at all.

I do think that it would be more work to try to slowly massage the git code into a library-ready form than it would be to simply start with more library-friendly code and pull in bits of git.git as appropriate.

That is what the libgit2 project is doing. Perhaps one day that project will reach a point where we start building git.git commands off of it or sometihng like it (for that matter, there is no reason you could not build external commands off of libgit2 right now). Would it be the "official" Git library then? I don't know. It is not clear to me what that even means.

In the meantime, I think it cannot be a bad thing for libgit2 to proceed along its path, and I don't see a good reason for people not to use it.

But hey, you don't need to listen to me. If you think it would be easier to make the git.git code into a library, go ahead and work on it. But I think you will find that there are a large number of hard-to-find bugs caused by implicit assumptions about global state, how file descriptors are used, and so forth.

What reason is that?

I think it is a matter of critical mass. If you were to start linking against libgit.a and 90% of it worked, you might have a reason to fix the other 10%. But I suspect it is more the other way around.

-Peff

Then you truly think libgit2 will ever reach the point where it can replace libgit.a?

It won't. But decreeing that both projects should remain isolated, and that libgit.a should never be a library, you are effectively condemning the effort to fail, knowingly or not.

How many years has libgit2 been brewing? Do you think it's closer for merging so it can be used by Git's core? No, it doesn't, and it will not in the future, because it was never meant for that.

It might be more effort, but the results are guaranteed by our extensive testing infrastructure and huge user-base. Slowly but steadily we'll get there.

Waiting for libgit2 to switch directions and reach some hypothetical point is waiting for hell to freeze.

It won't happen. There's no incentive nor reason for it to happen.

It means 'make install' installs a shared library with a clearly defined and stable API that other projects can depend on, and it can be used for all sort of purposes, including the git binary, and it's builtins.

Its path will never end as an official Git library, not unless we do something.

That's impossible. Specially since moving irrelevant code out of libgit.a is not permitted.

You tell me. Why isn't Git using libgit2?

It doesn't matter if it's 90% or 10%, it's the only thing we have.

Unless you are in favor of including libgit2 and start using it for Git's core *right now*, the only way forward is to improve libgit.a.

Felipe Contreras

I don't know. It may. Or something like it may. It is certainly not ready to do so yet, but perhaps one day it will be.

Oh, I see, you were not actually interested in my answer and were just being rhetorical.

Huh? When did I decree anything? You asked Duy what kinds of problems you would run into with running multiple git commands in the same process space. I answered with concrete examples, and gave my opinions on what the path of least work would be to reach a re-entrant library. You don't have to agree (didn't I even say "you don't have to listen to me" in the last email?).

There has been about 2 years of active development, and there's been quite a lot of improvement in that time. Closer than what? Than it was 2 years ago? Yes, I think it is. But it still has a ways to go.

I do not think there will be a flag day where we throw away git.git's code and start using libgit2. But we could slowly start replacing underlying bits with libgit2 bits, if that implementation proves to be robust and clean enough to do so.

I'm not even sure what your second sentence means.

But it seems to me that the first step would be cleaning up the internal code so that it is more friendly to library callers (both in interface and in being re-entrant), with those first sets of callers being the existing code in git.git. Such cleanups would be a good thing for the modularity of the code, even without an intended library step.

And then you can start to pull out individual interfaces that are known to be safe for library use, and think about making a coherent library out of them.

And please don't tell me about "not permitted". You are free to fork and work on this. But do not expect people who have already said "that does not seem like a fruitful path to me" to jump into it with you. If you think it is worth doing and that you can come up with initial results to convince others, go for it.

Wait, you indicated you had such a reason in mind, but now you won't tell me? Is it a secret?

That seems like a false choice to me. You obviously feel that libgit2 is some kind of dead end. I don't agree. Whatever.

I have very little interest in discussing this further with you, as it is not leading in a productive direction. In my opinion, the productive things to do would be one (or both) of:

1. Work on libgit2.

2. Clean up non-reentrant bits of git.git, hopefully making the code more readable and more modular (and taking care not to make it worse in other ways, like maintainability or performance).

-Peff

Perhaps one day we would end poverty and hunger, and perhaps one day we will live in peace, but I wouldn't hold on my breath. I fact, I'll do the opposite, I bet it won't happen anytime soon.

When you said in your opinion we should wait until libgit2 is ready, and not improve libgit.a.

Why is it closer? In what ways is it a better fit now than 2 years ago? What is missing before merging to be used in Git's core?

And what are we waiting for then? Shouldn't we copy the whole libgit2 code and start migrating?

It means this:
http://article.gmane.org/gmane.comp.version-control.git/226752

I move code that doesn't belong in a libgit library out of libgit.a, and the change gets rejected.

That is the second step. It doesn't make sense to make code re-entrant, if that code will only be used by builtin commands. First step is to move irrelevant code out of libgit.a.

I did not. I made the assumption that there was a reason, if there's no reason to stay clear of libgit2, then let's merge it already.

I never said so. It is a dead end *if* we don't do an effort to have a proper libgit library, which is the path we are taking.

You forgot the first step of 2.: move irrelevant code out of libgit.a.

Felipe Contreras

Hi Duy,

I think you should have said "on Windows" when you said "fork is costly". Oh wait, you did.

It seems that at least some people participating in this discussion are not overly keen on supporting the platform that -- according to statistics, i.e. facts -- is the most prevalent.

I am glad that Junio still seems to be interested in giving us poor folks trying to make the Git experience on Windows a less sucky one enough credit, even if we only got a little over 900 commits into git.git. But that does not count because the commits are older than one year. That makes them useless to some people, apparently.

Hopefully Junio will have more mercy on us poor Windows folks than others would,
Dscho