fetch2/git: questions on read-tree and checkout-index
Hi Ke,
As I ran into http://bugzilla.yoctoproject.org/show_bug.cgi?id=1089 today working with Saul to validate bug 1029, I spent some time reading through the fetch2/git source and the commit history. I had a couple questions regarding the rationale for the use of "read-tree" and "checkout-index" in the unpack routine:
runfetchcmd("git clone -s -n %s %s" % (ud.clonedir, destdir), d)
if not ud.nocheckout:
os.chdir(destdir)
runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d) runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
As I understand it this would be equivalent to checking out HEAD and then overwriting everything in the tree with the contents of the repository at ud.revisions[ud.names[0]]. This results in all the modifications listed with git status but doesn't add any of the changes back to the index, so the log still appears to be at HEAD (with a lot of local changes). This seems unnecessary for the majority of use cases. The one where it seems potentially useful would be the subdir case. Is that the only motivator for using this method?
If so, what is the motivation for checkout out of a subdir - as opposed to just changing the recipe to build within that subdir?
From what I could tell we were not using the ability to merge multiple trees with the read-tree, which I could see as being useful to generate a tree like the linux-yocto tree with the meta branch included in the bsp source - but even that could be done using standard merge commands.
Unless I'm missing a use-case (quite likely as there are next to no comments articulating the rationale and approach taken) I think it would make more sense to just checkout the required hash into a detached head for the build and update any recipes that make use of the subdir option to build within than subdir. Alternatively, doing a checkout for the non subdir case would make most of the recipes get this behavior while allowing the subdir users to remain untouched.
Thanks,
Signature · 3 lines
Darren Hart Intel Open Source Technology Center Yocto Project - Linux Kernel
Quoted 11 lines · Darren Hart
As I ran into http://bugzilla.yoctoproject.org/show_bug.cgi?id=1089 today working with Saul to validate bug 1029, I spent some time reading through the fetch2/git source and the commit history. I had a couple questions regarding the rationale for the use of "read-tree" and "checkout-index" in the unpack routine:
runfetchcmd("git clone -s -n %s %s" % (ud.clonedir, destdir), d)
if not ud.nocheckout:
os.chdir(destdir)
runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d)runfetchcmd("%s checkout-index -q -f -a" % ud.basecmd, d)
Quoted 7 lines
As I understand it this would be equivalent to checking out HEAD and then overwriting everything in the tree with the contents of the repository at ud.revisions[ud.names[0]]. This results in all the modifications listed with git status but doesn't add any of the changes back to the index, so the log still appears to be at HEAD (with a lot of local changes). This seems unnecessary for the majority of use cases. The one where it seems potentially useful would be the subdir case. Is that the only motivator for using this method?
You're asking why this is as it is an the best answer I can give you is history. We now use fetch2 and there were some changes in concept there and the overall changes were discussed on the mailing list in detail before they were implemented.
One key change was that we switched to preserving the SCM metadata with the checked out source code when we used to throw it away. This allowed several optimisations in the way we mirror data and "unpack" code.
If you imagine the .git directory isn't there at all and we were just generating tarballs of checked out data, the fetcher code code starts to make a lot more sense. The code needed to check out a specific revision, it didn't matter what state the index/tree were in with reference to branches and so forth. The fragment listed above does exactly that.
History wise, that fetcher has also been around since git was very new. Back then many modern git commands didn't even exist so it does things that now are considered more "internal" use of git.
Quoted 2 lines
If so, what is the motivation for checkout out of a subdir - as opposed to just changing the recipe to build within that subdir?
Some projects are huge with subprojects within projects as a hangover from svn source control for example (matchbox, clutter, bsd spring to mind with that structure). I can see a use case for it, particularly with the way the fetcher used to work as mentioned above.
How do we marry this up against our desire to keep the SCM metadata around now? Good question...
Quoted 2 lines
Unless I'm missing a use-case (quite likely as there are next to no comments articulating the rationale and approach taken)
There are several design discussions on the mailing list for fetch2 itself. We've concentrated on the overall architecture rather than the individual specific fetchers as I think they flow from the former and one needed to be got right before the other.
I don't think there is detailed comments about the specific commands themselves although hopefully above you can see how we've arrived here.
Quoted 3 lines
I think it would make more sense
to just checkout the required hash into a detached head for the build and update any recipes that make use of the subdir option to build within than subdir.
We need to be very careful about changing/breaking API but I agree the benefits we once had with subdir have become more minimal now we use the git metadata rather than the taring and untaring archives of source code for the unpack stage.
Quoted 2 lines
Alternatively, doing a checkout for the non subdir case would make most of the recipes get this behavior while allowing the subdir users to remain untouched.
I'd like to see your proposed checkout commands. We do need to be 100% sure that the given checkout matches 100% with the revision the system is trying to obtain which I believe is why the code has been left the way it has for so long.
Cheers,
Richard