Skip to contentSkip to conversation

backup or mirror a repository

DISCUSSION Dan Farina 4 participants 9 messages Jun 15, 2016

lore
Share & tools
lore

Dan Farina

author
#0 lore raw

Hello list,

I have been poking around on IRC and gmane, but haven't yet seen a solution to my problem:

What I really want is something semantically identical to a recursive copy of my entire git repository on one machine sent to another machine. Right now I can't simply use "rsync -ar --delete ..." because the pack changes names, thus fooling rsync and sending a bunch of bits over the network.

I saw git-mirror on gmane and asked around in IRC, but the former seems to have died sometime last November and the latter told me I'd probably have to write some glue to process git show-ref and git ls-remote. This seems viable, but suboptimal. Another user chimed in that they also had the same use case as myself.

fdr

(Justifications: I am super paranoid, so I want to push my local branches to another disk to be safe, but then they proliferate over time since deleted local branches are not also deleted on the target repository. I also want to work on multiple machines seamlessly, and as such doing a bunch of "git checkout -r ..." when I need to change branches is also suboptimal. As I said, I can get what I want by simply using rsync, so this is entirely about efficiency by exploiting git's structure)

#1 lore raw

Hi,

Quoted 2 lines · Dan Farina

I have been poking around on IRC and gmane, but haven't yet seen a solution to my problem:

I guess you're looking for "git remote add --mirror origin <url>". AFAIK this is not in any released version yet, though.

Ciao,
Dscho

author
#2 lore raw
Quoted 2 lines · Johannes Schindelin

I guess you're looking for "git remote add --mirror origin <url>". AFAIK this is not in any released version yet, though.

I just tried a bleeding edge git with --mirror.

It's close, except (unless there is some other provision to take care of this) remote branch deletes. (Which, after more gmane trolling, seemed to be "git remote rm", which doesn't seem present just yet).

Of course, it is completely possible I'm using it incorrectly. In brief, I did "git init" "git remote add --mirror backup <url>" "git pull" (perhaps I should have used "git remote update"?) And the behavior of this is more or less what I wanted, as far as sending branches. Convenient removal of branches and propagating that removal still stumps me. Is there a preferred way to do this yet?

Thanks,
fdr

#3 lore raw

Hi,

Quoted 9 lines · Dan Farina
Quoted 2 lines · Johannes Schindelin

I guess you're looking for "git remote add --mirror origin <url>". AFAIK this is not in any released version yet, though.

I just tried a bleeding edge git with --mirror.

It's close, except (unless there is some other provision to take care of this) remote branch deletes. (Which, after more gmane trolling, seemed to be "git remote rm", which doesn't seem present just yet).

git remote rm is about removing the remote nick, not a remote branch.

Quoted 2 lines

Of course, it is completely possible I'm using it incorrectly. In brief, I did "git init" "git remote add --mirror backup <url>" "git pull"

"git pull" without further parameters accesses the remote "origin", not "backup", and it merges _one_ branch into the current branch.

What you should have done is "git fetch backup".

Or use "origin" instead of "backup" right from the start, and then use "git fetch".

But no pull.

Quoted 4 lines

(perhaps I should have used "git remote update"?) And the behavior of this is more or less what I wanted, as far as sending branches. Convenient removal of branches and propagating that removal still stumps me. Is there a preferred way to do this yet?

You want to remove branches from the local repo which are no longer there on the remote side?

Then "git remote prune" is for you. For details, please see the man page.

Ciao,
Dscho

author
#4 lore raw
Quoted 5 lines · Johannes Schindelin

Hi,

On Thu, 27 Sep 2007, Dan Farina wrote:

git remote rm is about removing the remote nick, not a remote branch.

Ah, I thought for some reason there'd be a "remove" (since the other operations have long names) and, much like "ls-remote" there'd be some "rm-remote." An insane presupposition of mine from having too little information.

Quoted 6 lines

What you should have done is "git fetch backup".

Or use "origin" instead of "backup" right from the start, and then use "git fetch".

But no pull.

I did use "pull backup" (my error for being sloppy and omitting it from my mail) and it does work provided I do the dirty thing of doing a subsequent reset --hard (I thought perhaps things were not yet fully-there). Fetch does it perfectly, as you say, thanks!

Quoted 5 lines

You want to remove branches from the local repo which are no longer there on the remote side?

Then "git remote prune" is for you. For details, please see the man page.

I did look at prune and update, but my problem is the opposite: I want something that will remove branches from the remote repo when they no longer exist locally. As-is over time I will proliferate little local branches unless I occasionally sit down and delete branches by operating directly on the bare backup repository. (and then use prune on the remote nodes)

Thanks,
fdr

#5 lore raw
Quoted 6 lines · Dan Farina

I did look at prune and update, but my problem is the opposite: I want something that will remove branches from the remote repo when they no longer exist locally. As-is over time I will proliferate little local branches unless I occasionally sit down and delete branches by operating directly on the bare backup repository. (and then use prune on the remote nodes)

The "git remote add --mirror" setup is about setting up the local repository _AS_ the backup of the remote. In other words, the contents come from the remote by fetching from it and safely kept away from disaster on the local side. And for that, "remote prune" is a perfect thing to do.

I think what you are asking for is an opposite, a backup remote site you would push into. That is not what "remote add --mirror" is about.

You can almost do it with

	git push --all $remote

except there is no way to automagically remove the branch you removed from the local repository. For that, we would need a new --mirror option to "git-push".

I think it is trivial to do for native transports, as we first get the list of all refs from the remote side before starting the transfer. You need to change the last parameter called 'all' to remote.c::match_refs() into an enum ('push_all' being one of choices), introduce another enum 'push_mirror', and teach it to "match" the remote (i.e. dst) ref that does not have corresponding entry on our side (i.e. src) with an empty object name to mark it removed. Then the part marked as "Finally, tell the other end!" in send-pack.c::send_pack() will take care of the actual removal.

author
#6 lore raw
Quoted 7 lines · Junio C Hamano

You can almost do it with

	git push --all $remote

except there is no way to automagically remove the branch you removed from the local repository. For that, we would need a new --mirror option to "git-push".

Yup! I actually was looking for such a thing in the git-push man page, but was unsuccessful. I was living with git-push --all for a little while before I thought I'd ask. Unfortunately, the preferred setup (having the backup machine actively perform fetch and prune) is not very nice for me due to firewalls.

Quoted 11 lines

I think it is trivial to do for native transports, as we first get the list of all refs from the remote side before starting the transfer. You need to change the last parameter called 'all' to remote.c::match_refs() into an enum ('push_all' being one of choices), introduce another enum 'push_mirror', and teach it to "match" the remote (i.e. dst) ref that does not have corresponding entry on our side (i.e. src) with an empty object name to mark it removed. Then the part marked as "Finally, tell the other end!" in send-pack.c::send_pack() will take care of the actual removal.

I should have a look, but if someone else wants to work on this they shouldn't block on me: work already made me get hit by a bus on one project (luckily I wasn't important), and the storm is not over.

Thanks,
fdr

#7 lore raw
In reply to Junio C Hamano
Quoted 5 lines · Junio C Hamano

The "git remote add --mirror" setup is about setting up the local repository _AS_ the backup of the remote. In other words, the contents come from the remote by fetching from it and safely kept away from disaster on the local side. And for that, "remote prune" is a perfect thing to do.

I have tried to do that but I am getting a warning:

$ git remote prune origin
Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*
and no branch is removed.

I suspect that the change that introduced --mirror option for the 'add' command did not adjust the prune procedure to handle the new situation properly. Or is just me doing something wrong?

Anyway, because the official released git still does not have --mirror and it could be difficult to convince the admin to use the bleading edge version of Git for secure backup, I wrote a simple script instead, which provides the same functionality but using the released version of Git. Maybe, someone else finds it useful.

$ cat git-mirror
#!/bin/sh
usage() {
	echo >&2 "Usage: $0 URL"
	exit 1
}

set -e

test $# -eq 1 || usage
URL="$1"

# Initialize Git bare directory
git --bare init

# Initialize the mirror
git remote add origin "$URL"
git fetch
rmdir refs/heads
ln -s remotes/origin refs/heads

# Adding creating a script that will be run by cron cat <<EOF > git-mirror-sync
cd "$PWD"
git fetch
git remote prune origin
EOF
chmod +x git-mirror-sync

# Adding git-mirror-sync to cron
echo "Please, add $PWD/git-mirror-sync to cron"

#########################################################

Dmitry

#8 lore raw

Hi,

Quoted 7 lines · Dmitry Potapov
Quoted 5 lines · Junio C Hamano

The "git remote add --mirror" setup is about setting up the local repository _AS_ the backup of the remote. In other words, the contents come from the remote by fetching from it and safely kept away from disaster on the local side. And for that, "remote prune" is a perfect thing to do.

Quoted 8 lines

I have tried to do that but I am getting a warning:

$ git remote prune origin
Warning: unrecognized mapping in remotes.origin.fetch: +refs/*:refs/*
and no branch is removed.

I suspect that the change that introduced --mirror option for the 'add' command did not adjust the prune procedure to handle the new situation properly. Or is just me doing something wrong?

No, you're right. I did not anticipate git-remote to be written the way it is. After fiddling with it for several hours, I am giving up for now. IMHO the script is too married to the idea that the remote branches live in refs/remotes/<remote>/*. Probably it would be very easy by now to implement it as a builtin, using remote.[ch].

Ciao,
Dscho