how to reduce disk usage for large .git dirs?
How can I reduce the disk usage for multiple copies of the same repo?
Up to now I just made copies like this, but since .git alone is already 2GB it becomes expensive:
# git clone git://host/repo.git repo-master # cp -a repo-master repo-branchA # cd repo-branchA # git checkout -b branchA origin/branchA # cd - # cp -a repo-master repo-branchB # cd repo-branchB # git checkout -b branchB origin/branchB # cd - # cp -a repo-master repo-branchB-feature # cd repo-branchB-feature # git checkout -b branchB-feature origin/branchB # cd -
Since each .git is almost identical I wonder if there is a reliable way to "share" it. The "git clone" man page mentions --shared as a dangerous way to do things. It does not give an advice how to manage such cloned trees.
So how can I reduce the disk usage needed for the four .git dirs above? I looked around in the docs that came with my git-2.1.3 package, but found nothing that answers my question. Maybe we can workout something and add it to one of the existing docs.
Thanks!
Olaf
Quoted 20 lines · Olaf Hering
How can I reduce the disk usage for multiple copies of the same repo?
Up to now I just made copies like this, but since .git alone is already 2GB it becomes expensive:
# git clone git://host/repo.git repo-master # cp -a repo-master repo-branchA # cd repo-branchA # git checkout -b branchA origin/branchA # cd - # cp -a repo-master repo-branchB # cd repo-branchB # git checkout -b branchB origin/branchB # cd - # cp -a repo-master repo-branchB-feature # cd repo-branchB-feature # git checkout -b branchB-feature origin/branchB # cd -
Quoted 4 lines
Since each .git is almost identical I wonder if there is a reliable way to "share" it. The "git clone" man page mentions --shared as a dangerous way to do things. It does not give an advice how to manage such cloned trees.
But you're not using clone you are using cp .
The clone man page also says this:-
--local, -l
When the repository to clone from is on a local machine, this flag bypasses the normal "Git aware" transport mechanism and clones the repository by making a copy of HEAD and everything under objects and refs directories. The files under .git/objects/ directory are hardlinked to save space when possible.
If the repository is specified as a local path (e.g., /path/to/repo), this is the default, and --local is essentially a no-op. If the repository is specified as a URL, then this flag is ignored (and we never use the local optimizations). Specifying --no-local will override the default when /path/to/repo is given, using the regular Git transport instead.
Note the first sentence of the second paragraph.
eg:
# git clone git://host/repo.git repo-master # git clone repo-master repo-branchA # cd repo-branchA # git checkout -b branchA origin/branchA # cd - # git clone repo-master repo-branchB # cd repo-branchB # git checkout -b branchB origin/branchB # cd - # git clone repo-master repo-branchB-feature # cd repo-branchB-feature # git checkout -b branchB-feature origin/branchB # cd -
Should work better for you. And there is probably a way to do it less commands too.
Signature · 1 line
Roger Gammans <[email protected]>
Quoted 6 lines · Roger Gammans
Note the first sentence of the second paragraph.
eg:# git clone git://host/repo.git repo-master # git clone repo-master repo-branchA # cd repo-branchA # git checkout -b branchA origin/branchA
It fails right here because in this dir only "master" exists, but branchA is expected.
So far the sequence of commands is:
# git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd - # git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchB
Next step will be:
# $do_work ; git commit -avs ; git push
Will that work as expected? Will find out after lunch..
Olaf
Quoted 2 lines · Olaf Hering
How can I reduce the disk usage for multiple copies of the same repo?
You can use --local och --shared. As you say --shared can be dangerous. If you don't understand the man page enough to know how you should manage your clones you should probably not use it.
--local seems to be what you're looking for.
However as a side note I'm curious about what your use case is. Why do you need this many repos?
Your setup looks familiar to me for a subversion user switching to git and trying to use git as subversion. The common usecase is not to have multiple worktrees but to do a checkout to the worktree you need to work on. This is possible with git since it's very fast and I recommend you to try to use one worktree.
Signature · 5 lines
Med vänlig hälsning Fredrik Gustafsson tel: 0733-608274 e-post: [email protected]
Quoted 7 lines · Fredrik Gustafsson
Quoted 2 lines · Olaf Hering
How can I reduce the disk usage for multiple copies of the same repo?
You can use --local och --shared. As you say --shared can be dangerous. If you don't understand the man page enough to know how you should manage your clones you should probably not use it.
Perhaps its more a lack of doc how to use the result.
Quoted 1 line
--local seems to be what you're looking for.
I will try this.
Quoted 2 lines
However as a side note I'm curious about what your use case is. Why do you need this many repos?
Because I do work in each copy, poke around, or do commits and push them.
Quoted 5 lines
Your setup looks familiar to me for a subversion user switching to git and trying to use git as subversion. The common usecase is not to have multiple worktrees but to do a checkout to the worktree you need to work on. This is possible with git since it's very fast and I recommend you to try to use one worktree.
Switching branches will invalidate timestamps, causing a full rebuild.
Olaf
W dniu 2014-11-13 13:03, Olaf Hering pisze:
Quoted 1 line · Fredrik Gustafsson
[...]
Quoted 7 lines
Quoted 5 lines
Your setup looks familiar to me for a subversion user switching to git and trying to use git as subversion. The common usecase is not to have multiple worktrees but to do a checkout to the worktree you need to work on. This is possible with git since it's very fast and I recommend you to try to use one worktree.
Switching branches will invalidate timestamps, causing a full rebuild.
Wouldn't a better way of solving "full rebuild" issue be to use ccache or similar solution?
Anyway, switching branches invalidates timestamps only on those files that change between branches -- it is to avoid unnecessary rebuilds.
You can always clone with --reference, and use alternates (alternate object store). Just don't delete objects in repository that other repositories borrow from; GitHub uses refs/borrowers/ namespace for that, IIRC.
HTH
Signature · 1 line
Jakub Narębski
Quoted 4 lines · Olaf Hering
So how can I reduce the disk usage needed for the four .git dirs above? I looked around in the docs that came with my git-2.1.3 package, but found nothing that answers my question. Maybe we can workout something and add it to one of the existing docs.
While playing around with this I made some notes, this is the result:
Manage multiple branches as separate copies
To preserve disk space for each clone, use a master copy of the reop and do local clones from such copy of a remote repository.
First clone the remote repository as usual. Then create a local branch for each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd -
Now clone each work branch into its own directory. The work dir references the master repo. All changes come from and go into this repo, instead of the remote repo.
# git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchB
To make changs in a work dir, commit as usual. The changes will be pushed from the work copy into the local master repo. Its required to have some other branch than branchA active in repo-master, or push from work copy to repo-master will fail.
# cd repo-master # git checkout master # cd - # cd repo-branchA # git commit -avs # git push origin branchA # cd -
To publish the outstanding changes its required to do this from the master repo. First checkout the work branch, then pull the local changes and finally push them to the remote repo.
# cd repo-master # git checkout branchA # git pull # git push origin branchA # cd -
To receive changes from the remote repo its required to do this from the master repo. First checkout the work branch, then pull the outstanding remote changes into the local branch. And finally pull them into the work dir.
# cd repo-master # git fetch --all (optional) # git checkout branchB # git pull # cd - # cd repo-branchB # git pull # cd -
# vim: set tw=72 et
Thanks for sharing your notes! A few comments:
Quoted 11 lines · Olaf Hering
First clone the remote repository as usual. Then create a local branch for each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd -Now clone each work branch into its own directory. The work dir references the master repo. All changes come from and go into this repo, instead of the remote repo.
Quoted 7 lines
# git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchBTo make changs in a work dir, commit as usual. The changes will be pushed from the work copy into the local master repo. Its required to have some other branch than branchA active in repo-master, or push from work copy to repo-master will fail.
That's one of the reason it's not recommended to push into a non-bare repository. You should clone your repo-master with the --bare option to avoid having a work dir there.
Quoted 8 lines
To publish the outstanding changes its required to do this from the master repo. First checkout the work branch, then pull the local changes and finally push them to the remote repo.
# cd repo-master # git checkout branchA # git pull # git push origin branchA # cd -
It's not. You could just add your remote repository as a remote to each of your clones of your master repo and push directly from them. It would be much simplier and it would allow you to directly fetch changes from your remote into your branches as well.
(however, I'm not sure but I think, that this will slowly increase the difference between your repositories when you develop. So that they won't change any new data since to local clone was made).
Signature · 5 lines
Med vänlig hälsning Fredrik Gustafsson tel: 0733-608274 e-post: [email protected]
Quoted 35 lines · Fredrik Gustafsson
Thanks for sharing your notes! A few comments:
Quoted 11 lines · Olaf Hering
First clone the remote repository as usual. Then create a local branch for each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd -Now clone each work branch into its own directory. The work dir references the master repo. All changes come from and go into this repo, instead of the remote repo.
Quoted 7 lines
# git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchBTo make changs in a work dir, commit as usual. The changes will be pushed from the work copy into the local master repo. Its required to have some other branch than branchA active in repo-master, or push from work copy to repo-master will fail.
That's one of the reason it's not recommended to push into a non-bare repository. You should clone your repo-master with the --bare option to avoid having a work dir there.
Quoted 8 lines
To publish the outstanding changes its required to do this from the master repo. First checkout the work branch, then pull the local changes and finally push them to the remote repo.
# cd repo-master # git checkout branchA # git pull # git push origin branchA # cd -
Quoted 8 lines
It's not. You could just add your remote repository as a remote to each of your clones of your master repo and push directly from them. It would be much simplier and it would allow you to directly fetch changes from your remote into your branches as well.
(however, I'm not sure but I think, that this will slowly increase the difference between your repositories when you develop. So that they won't change any new data since to local clone was made).
Can you not do this much simpler with --reference? Like this:
$ git clone --bare git://host/repo.git repo-master $ git clone -b branchA --reference repo-master git://host/repo.git repo-branchA $ git clone -b branchB --reference repo-master git://host/repo.git repo-branchB
All three repos now push/pull directly to/from git://host/repo.git,
but repo-branchA and repo-branchB reference objects from within the bare repo-master. You have to make use to never delete objects from repo-master (if those objects happen to be referenced from repo-branchA|B). If you want to prevent the repos growing in size, you must devise a way to add new objects into repo-master before repo-branchA|B. (e.g. a nightly cron-job in repo-master that fetches from origin), so that when repo-branchA|B pulls, they will find most objects are already present in repo-master.
...Johan
Signature · 2 lines
Johan Herland, <[email protected]> www.herland.net
Quoted 8 lines · Johan Herland
Can you not do this much simpler with --reference? Like this:
$ git clone --bare git://host/repo.git repo-master $ git clone -b branchA --reference repo-master git://host/repo.git repo-branchA $ git clone -b branchB --reference repo-master git://host/repo.git repo-branchB
Quoted 4 lines
All three repos now push/pull directly to/from
git://host/repo.git,
but repo-branchA and repo-branchB reference objects from within the bare repo-master. You have to make use to never delete objects from repo-master
I think the "never delete" part is why we usually warn people off of using alternates. I think at the least you would have to "git config gc.auto 0" in the bare repository (otherwise your nightly fetches risk pruning). Of course you'd probably want to repack eventually for performance reasons. So maybe setting gc.pruneExpire is a better option (to something like "20.years.ago").
Quoted 5 lines
If you want to prevent the repos growing in size, you must devise a way to add new objects into repo-master before repo-branchA|B. (e.g. a nightly cron-job in repo-master that fetches from origin), so that when repo-branchA|B pulls, they will find most objects are already present in repo-master.
You can also fetch from the children into repo-master periodically. Like:
cd repo-master &&
for i in branchA branchB; do
git fetch ../$i +refs/*:refs/remotes/$i/*
done
after which it is actually safe to run "git gc" in the master (assuming there isn't simultaneous activity in the children). This is how we manage fork networks on GitHub (we take in objects to individual forks via push, and then migrate them to the master repo via fetch).
-Peff
Quoted 17 lines · Fredrik Gustafsson
Thanks for sharing your notes! A few comments:
Quoted 14 lines · Olaf Hering
First clone the remote repository as usual. Then create a local branch for each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd -Now clone each work branch into its own directory. The work dir references the master repo. All changes come from and go into this repo, instead of the remote repo.
# git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchB
Quoted 8 lines
Quoted 4 lines
To make changs in a work dir, commit as usual. The changes will be pushed from the work copy into the local master repo. Its required to have some other branch than branchA active in repo-master, or push from work copy to repo-master will fail.
That's one of the reason it's not recommended to push into a non-bare repository. You should clone your repo-master with the --bare option to avoid having a work dir there.
So my repo-master is now "bare". I pushed from repo-branchA into repo-master and see my commits in both repos. But pushing from repo-master to the remote fails because repo-master does not have outstanding remote commits. However, git fetch doesnt do anything:
Fetching origin
From host:/remote/dir
* branch HEAD -> FETCH_HEAD
Obviously I miss something. The man page of git clone or fetch does not mention how "bare" is supposed to be handled.
Olaf
Quoted 4 lines · Olaf Hering
So my repo-master is now "bare". I pushed from repo-branchA into repo-master and see my commits in both repos. But pushing from repo-master to the remote fails because repo-master does not have outstanding remote commits. However, git fetch doesnt do anything:
Are you mixing up your branches? So that you're updating one branch in your master repo but trying to push an other branch to your remote repo?
Signature · 5 lines
Med vänlig hälsning Fredrik Gustafsson tel: 0733-608274 e-post: [email protected]
Quoted 8 lines · Fredrik Gustafsson
Quoted 4 lines · Olaf Hering
So my repo-master is now "bare". I pushed from repo-branchA into repo-master and see my commits in both repos. But pushing from repo-master to the remote fails because repo-master does not have outstanding remote commits. However, git fetch doesnt do anything:
Are you mixing up your branches? So that you're updating one branch in your master repo but trying to push an other branch to your remote repo?
I dont think so. I have branchA in repo-branchA, and a 'git push origin branchA' puts it into repo-master.
How is a bare repo supposed to be updated? Is a simple 'git fetch --all' supposed to work?
Olaf
Quoted 8 lines · Olaf Hering
Quoted 6 lines · Fredrik Gustafsson
Quoted 4 lines · Olaf Hering
So my repo-master is now "bare". I pushed from repo-branchA into repo-master and see my commits in both repos. But pushing from repo-master to the remote fails because repo-master does not have outstanding remote commits. However, git fetch doesnt do anything:
Quoted 7 lines
Quoted 2 lines
Are you mixing up your branches? So that you're updating one branch in your master repo but trying to push an other branch to your remote repo?
I dont think so. I have branchA in repo-branchA, and a 'git push origin branchA' puts it into repo-master.
How is a bare repo supposed to be updated? Is a simple 'git fetch --all' supposed to work?
Is there s a slim chance that I have to fetch in repo-master before doing a git push in repo-branchA? git remote show origin shows that some branches are out of date.
Olaf
Quoted 3 lines · Fredrik Gustafsson
That's one of the reason it's not recommended to push into a non-bare repository. You should clone your repo-master with the --bare option to avoid having a work dir there.
Even if I do a fresh clone with --bare, the result can not be updated anymore with git fetch. What I'm doing wrong?
Olaf
Quoted 2 lines · Olaf Hering
Even if I do a fresh clone with --bare, the result can not be updated anymore with git fetch. What I'm doing wrong?
A --bare clone has no connection to its origin (there are no remotes). You want a --mirror.
Andreas.
Signature · 3 lines
Andreas Schwab, [email protected] GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Quoted 7 lines · Andreas Schwab
Quoted 2 lines · Olaf Hering
Even if I do a fresh clone with --bare, the result can not be updated anymore with git fetch. What I'm doing wrong?
A --bare clone has no connection to its origin (there are no remotes). You want a --mirror.
Using --mirror for this purpose is dangerous because it will most likely overwrite changes on the remote side. Fortunately I used 'git push --dry-run origin' and the output was like:
To git://host/repo.git
+ abbrev1..abbrev2 branchA -> branchA (forced update)
Before that I pushed already to the remote repo without realizing where the push goes to.
Looks like using the --bare option for cloning the master and then doing something like "git clone --origin local_bare -b branchA --reference repo-master git://host/repo.git repo-branchA" will work better.
Olaf
Quoted 4 lines · Olaf Hering
Since each .git is almost identical I wonder if there is a reliable way to "share" it. The "git clone" man page mentions --shared as a dangerous way to do things. It does not give an advice how to manage such cloned trees.
If you know what you are doing, you can try git-new-workdir in contrib/workdir. A safe and reliable version of that is being worked on, hopefully it'll be released in 2.3.0.
Signature · 1 line
Duy