Clone from shallow bundle bug
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle. The initial shallow clone has a .git/shallow file that identifies it as a shallow clone ( and I guess keeps things from complaining about the missing history ), but the the repo cloned from the bundle does not, so gitk run in that clone complains about not being able to find the objects referenced by the oldest commit in the repo.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJVGw03AAoJENRVrw2cjl5R4aQIAKxddA+eneEEchuygYwA8zFK 4O+LuEXbJ09JR196Zj535jbJ3NLre8KLX8l7gxVRFQ5FscG0+ylvawAZ2VCUrl+6 dfTXOTfzmu36GGKJ+wG7SFIIEAzjFyLk8bj5qtJgF3F3PZqkgxmBGmQskGK1Dlet VUqXL0IndMTDnb5//pFGyF2L5aPvBfXcC6pZAUBjKz4dl7MfFVXbuzCMe2TqN+l6 dPPjqANGb8MiAdhoVQ69c0uI2XH9dvmpRJTOX1Yr1IMW/AvKKM/w/A9MDKkhsab3 ccgeYDUjtUioaplnMcDwkSj4jQfQedgmIUdHeMbT0WOiKvHRArlroRVcjQS8pps= =kPAv
-----END PGP SIGNATURE-----
Quoted 8 lines · Phillip Susi
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle. The initial shallow clone has a .git/shallow file that identifies it as a shallow clone ( and I guess keeps things from complaining about the missing history ), but the the repo cloned from the bundle does not,
You made me worry a bit. We have checks in clone and fetch to make sure the result is "good" (i.e. gitk should not complain, clone/fetch should report it instead). Unfortunately I tested and it seemed to work as expected (i.e. clone fails)
$ LANG=C ./git clone ./shallow.bundle shallow2 Cloning into 'shallow2'... Receiving objects: 100% (2813/2813), 5.33 MiB | 0 bytes/s, done. Resolving deltas: 100% (250/250), done. Checking connectivity... error: Could not read 50a3ba22454e2989424d5de489de9c0f68fed5ec fatal: Failed to traverse parents of commit c73a8a63134734ddf7077d09355a10a0077ed2ca fatal: remote did not send all necessary objects
Quoted 3 lines
so gitk
run in that clone complains about not being able to find the objects referenced by the oldest commit in the repo.
Signature · 1 line
Duy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 16 lines · Duy Nguyen
Quoted 8 lines · Phillip Susi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle. The initial shallow clone has a .git/shallow file that identifies it as a shallow clone ( and I guess keeps things from complaining about the missing history ), but the the repo cloned from the bundle does not,
You made me worry a bit. We have checks in clone and fetch to make sure the result is "good" (i.e. gitk should not complain, clone/fetch should report it instead). Unfortunately I tested and it seemed to work as expected (i.e. clone fails)
Quoted 7 lines
$ LANG=C ./git clone ./shallow.bundle shallow2 Cloning into 'shallow2'... Receiving objects: 100% (2813/2813), 5.33 MiB | 0 bytes/s, done. Resolving deltas: 100% (250/250), done. Checking connectivity... error: Could not read 50a3ba22454e2989424d5de489de9c0f68fed5ec fatal: Failed to traverse parents of commit c73a8a63134734ddf7077d09355a10a0077ed2ca fatal: remote did not send all necessary objects
Strange; it works fine for me using git 1.9.4.msysgit.1, and then I just get the complaints from gitk. I created the bundle with no prereq argument, i.e. "git bundle create shallow.bundle". Did you use a prereq argument?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJVG+pKAAoJENRVrw2cjl5RLkMH/j0IlFsf5oEsFejLqD2fxFAJ 8r7pZCtYFvFqMqLQivLCdU/aYCd/5F99VUtusH3NphJvxkmCaLyRwLyA1KR/AozQ slEXc5gmjbUg9yEBffYQ/xFPAGrizb2BblSzl6hcAZGtscLNyKvOjHttvJL+xM1+ uY0dwHcQ97m5p3DlehjLSAHolJF+waEhS6MarACZuSbi2JBTvo3OOagyt0o46sjp t5v4kfRfTYD6DTlY+VPTUC56unBaVItLDfxY5d+iHGDY2o5rhl4AFWLbCh3v6ltl OAuVs+UPKn0kV9tIQ6EBhQKf5CooCZtbr45OVGpVLPipjQFq2QqUjVWrLMEskhw= =Fem/
-----END PGP SIGNATURE-----
Quoted 28 lines · Phillip Susi
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 16 lines · Duy Nguyen
Quoted 8 lines · Phillip Susi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle. The initial shallow clone has a .git/shallow file that identifies it as a shallow clone ( and I guess keeps things from complaining about the missing history ), but the the repo cloned from the bundle does not,
You made me worry a bit. We have checks in clone and fetch to make sure the result is "good" (i.e. gitk should not complain, clone/fetch should report it instead). Unfortunately I tested and it seemed to work as expected (i.e. clone fails)
Quoted 7 lines
$ LANG=C ./git clone ./shallow.bundle shallow2 Cloning into 'shallow2'... Receiving objects: 100% (2813/2813), 5.33 MiB | 0 bytes/s, done. Resolving deltas: 100% (250/250), done. Checking connectivity... error: Could not read 50a3ba22454e2989424d5de489de9c0f68fed5ec fatal: Failed to traverse parents of commit c73a8a63134734ddf7077d09355a10a0077ed2ca fatal: remote did not send all necessary objects
Quoted 4 lines
Strange; it works fine for me using git 1.9.4.msysgit.1, and then I just get the complaints from gitk. I created the bundle with no prereq argument, i.e. "git bundle create shallow.bundle". Did you use a prereq argument?
No, just your command plus a branch name. I tried v1.9.4.msysgit.1 (but on linux, not windows), clone rejected too.
Signature · 1 line
Duy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 7 lines · Duy Nguyen
Quoted 4 lines
Strange; it works fine for me using git 1.9.4.msysgit.1, and then I just get the complaints from gitk. I created the bundle with no prereq argument, i.e. "git bundle create shallow.bundle". Did you use a prereq argument?
No, just your command plus a branch name. I tried v1.9.4.msysgit.1 (but on linux, not windows), clone rejected too.
Even stranger... when I use a branch name, the clone fails, but not when I use a detached HEAD.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJVG/LlAAoJENRVrw2cjl5RKtQH/2TTJ31PtG+3MQjlKjyMhhNZ Xl/A7QSvruDEUF8V7kdyCYlN4I3EQIelMHeG0tZPWw/qnOPMMvpMvxI2xu3na0xf L4AopC3XPFCe2kRG4EV17Nf0QAR8zx+ARGhCzf+PzLgGlFdmMsN3TZ/8Oe3yJHSZ /eu+CPvmvE0N5PeC1EnPoYJwoTcBHFFI5he736OHI5PA8WtekJPQ2SZ17pZttN0p jmZSeqoUZAe4Jeu+xfE0hYuuoVKtlkat/2GmOKrYcglyuw1+RaU31op+mKkhtGyw x8yq0LmJ+zMAtj3Ab5fqv0rRrjbNPYeO5uTT+lgHSU62QRPVclv8lkugRyotcMk= =owOK
-----END PGP SIGNATURE-----
Quoted 9 lines · Phillip Susi
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 5 lines · Duy Nguyen
Quoted 4 lines
Strange; it works fine for me using git 1.9.4.msysgit.1, and then I just get the complaints from gitk. I created the bundle with no prereq argument, i.e. "git bundle create shallow.bundle". Did you use a prereq argument?
Quoted 5 lines
Quoted 2 lines
No, just your command plus a branch name. I tried v1.9.4.msysgit.1 (but on linux, not windows), clone rejected too.
Even stranger... when I use a branch name, the clone fails, but not when I use a detached HEAD.
Thank you. I can reproduce it now. We need to plug this hole.
Signature · 1 line
Duy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 1 line · Duy Nguyen
Thank you. I can reproduce it now. We need to plug this hole.
I'd much rather it not refuse to clone so that I can end up with a proper shallow clone. At least the way it is now, when I clone the detached head, I can manually add the .git/shallow file and everything is fine.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJVG/trAAoJENRVrw2cjl5RTTIIAJkdte4gWrrOGA49CI0xivX7 1FXH3tPp/Nhd7gG3MXNfozm78DS3ZWqoU4l2SUhoE3La9UJ81T2rVo9GjcR/yXeS V0In+JyoQX3spZdtvH18qzKCFczyeUlu260EG7mQsBFgnAHsAJW3BPA6DWEPpfJS U3RPGt4S7KKy2+XJAGZJgvhvwM9vndgx161Kgwwpdocv2uWmv0AZEcMzOppZQy3y RqWSO5iY3qRwpMiRRh9YsQsuVNpXGxwPqV5oXXFLD7yaAMqCF5qdUYz8fWNQQ1+V 49RpGzMNHA60FP9BrvlHMCaJgFEvBT4nrpN4MCQgkelp8LOELX1kfuq9MWl1irA= =BmpO
-----END PGP SIGNATURE-----
Quoted 2 lines · Phillip Susi
I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle.
I think the introdution of shallow clone feature broke git bundle create by not teaching it that its shallow boundaries are prerequisite commits to unbundle its contents. IOW, the bundle created from the shallow clone is broken, I would think.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 9 lines · Junio C Hamano
Quoted 2 lines · Phillip Susi
I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle.
I think the introdution of shallow clone feature broke git bundle create by not teaching it that its shallow boundaries are prerequisite commits to unbundle its contents. IOW, the bundle created from the shallow clone is broken, I would think.
It seems to me that it isn't exactly broken; it just needs to put something in the bundle noting that it was built from a shallow clone, and then when cloning from the shallow bundle, the new clone needs its .git/shallow file.
In other words, the bundle contains all of the objects in the shallow clone, so a new shallow clone can be correctly constructed from the bundle, it's just that the new clone doesn't *know* it too is shallow and things like git log and gitk need to stop following the history chain when they reach the shallow point rather than complain that the rest of the history is missing ( which is intentional ).
For that matter, if you do create a depth limited bundle from a non shallow repository, then try to clone from it, the cloned repo should automatically become shallow rather than complain about missing history.
In other words, any time you clone from a bundle, the clone process should check if the full history is in the bundle, and if not, automatically make the new repo shallow so as to avoid the error messages about the missing history.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBCgAGBQJVG15kAAoJENRVrw2cjl5RnvgH/iMyN/1U2zg+ju/teVEQIMRL prvC60S9/yLxSp6RmiXpN2xuGHMkn7i2y6XpM9RQdE6ETeGaIw7UaDan3r7BPTSD +Q9DrAM0g17IGNxvmGPiJZP7j0t2e43oTA9KM8alf6icMU/mWJgQsbtc9QFVfVkd jsYevK1GR6ysrAHBAV6GxKfNw5yw3N+kTf9s2rKXWIFaArD0rcKJZVxiygMlhVSa hT4j3+n5f3n0WMDVxFzhwOaW+yrPiXF3gs1pKFX8GT5g1BtvOAEnyskSgA5nZsNB G53ncyyefinaaBqCvPSbcTLXmWLV8QuLBExc13BWXiVUD2rRNQr7u1ihbxWKyvU= =D90B
-----END PGP SIGNATURE-----
Quoted 6 lines · Phillip Susi
Quoted 5 lines · Junio C Hamano
Quoted 2 lines · Phillip Susi
I made a shallow clone of my repo, then used git bundle create to pack it all into a bundle file, then cloned from that bundle.
Quoted 7 lines
Quoted 4 lines
I think the introdution of shallow clone feature broke git bundle create by not teaching it that its shallow boundaries are prerequisite commits to unbundle its contents. IOW, the bundle created from the shallow clone is broken, I would think.
It seems to me that it isn't exactly broken; it just needs to put something in the bundle noting ...
I think you are saying the same thing as I did without realizing ;-)
The only way a bundle can record "something" "noting" that it is an incomplete history, while allowing it to be read by existing implementations of "git bundle unbundle" is to list the commits, behind which there is no history available in the bundle, as bundle's pre-requisites. I said that the addition of shallow repository support did not enhance "git bundle create" to do so, and you are saying "it just needs to put", implying that it currently does not.
Now, the only way the existing implementations of "git bundle unbundle" can extract from such a bundle with prerequisites is to unbundle into a repository that satisifies the prerequisites (these incomplete bundle files are designed as a medium for sneaker-net incremental updates). I however suspect "git clone --shallow" from such a bundle _might_ work (I never tried it), but I wouldn't be surprised if the existing implementations of bundle unpacking code refused to unbundle into a repository that lacks prerequiste commits (i.e. making the repository a shallow-clone).
Allowing "git clone" (without "--shallow") from an incomplete bundle file create a shallow clone, instead of outright refusing to work or silently creating a corrupt repository, might be an interesting project, perhaps the right size for GSoC or something like that.
Patches welcome, but one thing whoever is doing such a patch needs to be careful about is that the pack data in the bundle is likely to be a thin one---truly requiring the repository the bundle is unpacked into to have the prerequisite objects.
Quoted 7 lines · Junio C Hamano
The only way a bundle can record "something" "noting" that it is an incomplete history, while allowing it to be read by existing implementations of "git bundle unbundle" is to list the commits, behind which there is no history available in the bundle, as bundle's pre-requisites. I said that the addition of shallow repository support did not enhance "git bundle create" to do so, and you are saying "it just needs to put", implying that it currently does not.
Alternatively, we can record SHA-1 in the shallow file as refs whose name is always ".shallow". This way "unbundle" can recreate the shallow file if it wants. Having this "remote" shallow file would fit well in our fetch pipeline. It's harder to recreate shallow file if we record prerequisite instead: if commit A is in the shallow file, the ones in prerequisite category would be A's parents. So we would need to go over the bundle to look for commits whose parents are all in prerequisite list. It takes more time.
Quoted 9 lines
Now, the only way the existing implementations of "git bundle unbundle" can extract from such a bundle with prerequisites is to unbundle into a repository that satisifies the prerequisites (these incomplete bundle files are designed as a medium for sneaker-net incremental updates). I however suspect "git clone --shallow" from such a bundle _might_ work (I never tried it), but I wouldn't be surprised if the existing implementations of bundle unpacking code refused to unbundle into a repository that lacks prerequiste commits (i.e. making the repository a shallow-clone).
Signature · 1 line
Duy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 11 lines · Duy Nguyen
Quoted 8 lines · Junio C Hamano
The only way a bundle can record "something" "noting" that it is an incomplete history, while allowing it to be read by existing implementations of "git bundle unbundle" is to list the commits, behind which there is no history available in the bundle, as bundle's pre-requisites. I said that the addition of shallow repository support did not enhance "git bundle create" to do so, and you are saying "it just needs to put", implying that it currently does not.
Quoted 8 lines
Alternatively, we can record SHA-1 in the shallow file as refs whose name is always ".shallow". This way "unbundle" can recreate the shallow file if it wants. Having this "remote" shallow file would fit well in our fetch pipeline. It's harder to recreate shallow file if we record prerequisite instead: if commit A is in the shallow file, the ones in prerequisite category would be A's parents. So we would need to go over the bundle to look for commits whose parents are all in prerequisite list. It takes more time.
Right; you can't rely only on the pre-requisets as objects in the bundle may be deltaed against them and so they can not be unpacked without them. The idea here is that all required objects ( and their delta bases ) are present, either in the bundle, or in the local repository, but if you walk the history chain you arrive at a parent pointer to an object you don't have. That point should automatically be recorded in .git/shallow.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
iQEcBAEBAgAGBQJVG+wIAAoJENRVrw2cjl5Rj2UH/0AXyy63MDYhg0C7t4ljsb0U pHo5vZyfurO2k4vd2kiUySDhIaZ7gmhkPySbPlphzqHGtvdPtyLwkYPVgqBVv7uA fDodTsxt64MbdFN3CNk5zh5BLDs6q1+1IjscvTlsmjCQbTz+ys+Qw1QS0zS9hSWD +jGDCa1x5zETniI0wJiXSSiCF6ZtFHuEJwZp5MSj257tAidibi/a0U+AHYdFwgSf jtoUWXR2t9Xl/eN1Xkw3bjE5xQUogZYox0IuUWPvv4c4rmgxhU6SYzYftWD7Lkof vL2t+y1x1IhmgLaIoPz+/p/Dfeupivf6F8dS6cfyG++QIshPGq6ACZ49NRymcy0= =TctP
-----END PGP SIGNATURE-----
Quoted 3 lines · Duy Nguyen
Alternatively, we can record SHA-1 in the shallow file as refs whose name is always ".shallow". This way "unbundle" can recreate the shallow file if it wants.
Wouldn't the extracting end care about such a name ".shallow" being duplicates (if you have multiple boundaries) or not check-ref-format kosher?
I was hoping we will have a solution for a new bundle command to create a bundle out of a shallow repository that is somehow usable by existing 'git bundle' implementations. You may take a bundle out of a shallow repository to preserve your work on top since you made the shallow clone originally, and then send it to your friend with existing 'git bundle' implementation to extract it into his full clone to sneaker-net.
And that is why I did not suggest such an alternative.
Quoted 9 lines · Junio C Hamano
Quoted 3 lines · Duy Nguyen
Alternatively, we can record SHA-1 in the shallow file as refs whose name is always ".shallow". This way "unbundle" can recreate the shallow file if it wants.
Wouldn't the extracting end care about such a name ".shallow" being duplicates (if you have multiple boundaries) or not check-ref-format kosher?
I think we are careful to ignore funny ref names, so it should not be a problem. But I haven't tested this.
Quoted 7 lines
I was hoping we will have a solution for a new bundle command to create a bundle out of a shallow repository that is somehow usable by existing 'git bundle' implementations. You may take a bundle out of a shallow repository to preserve your work on top since you made the shallow clone originally, and then send it to your friend with existing 'git bundle' implementation to extract it into his full clone to sneaker-net.
OK two additional options on top of what we already have:
- save .have and add extra prerequisite SHA-1.
- create a bundle that does not hit shallow boundary in the first
place, roughly speaking it's "max depth minus one". This one does not have extra .have or prerequisites
Signature · 1 line
Duy
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Quoted 6 lines · Duy Nguyen
OK two additional options on top of what we already have:
- save .have and add extra prerequisite SHA-1. - create a bundle that does not hit shallow boundary in the first place, roughly speaking it's "max depth minus one". This one does not have extra .have or prerequisites
Huh? If it is one less deep that doesn't help: the new clone will still be missing history. AFAICS, the only way to keep the new clone kosher is for it to have a .git/shallow file that identifies it as a shallow clone so that history walks stop short instead of complaining about the missing history.
Thus, all that is needed is for the clone, when it notices that history is missing, is to create the shallow file instead of erroring out, telling you to fsck yourself, and rudely deleting the new clone.
It is one thing to tell you that there is missing history, but quite
another to delete the repository it just created due to it missing history.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBCgAGBQJVHJraAAoJENRVrw2cjl5RABEH/RW+J5eFNRL80qMVSSnYI4Wb RjCb5Lb1pp4PBQtGK6yJ/7lzDYptAN6aLcpMVrGiyIIJDm7KfZ6rGvhUegn37ImH dCelZs+XHeR1dVd05Lbn9FGgB3mg873JOb5+i/hMuuudrXhNjRy6hhFGBnVulpPP lySfaPMscbSH7lzqr1zxgdu4GzRLlLPzKv1ojiWGyy97iRAsN6bRy6I1/wsddKMn hESUlv7AdTNQxu3b3NsLGS20a9QHMpKjBxBLvOYE6ftr4yyHMkxum/+BnoASY2UB h5LenKnQzRwFNGiw1BDeQESqYDXpQ4yA7lpd91gsINwgVsRqg82iSen9/fszzq0= =shT5
-----END PGP SIGNATURE-----
Quoted 6 lines · Duy Nguyen
OK two additional options on top of what we already have:
- save .have and add extra prerequisite SHA-1.
- create a bundle that does not hit shallow boundary in the first
place, roughly speaking it's "max depth minus one". This one does not have extra .have or prerequisites
Too fast.. the second option does need extra prerequisites.. Ignore that one.
Signature · 1 line
Duy