Skip to contentSkip to conversation

partial checkouts

DISCUSSION Chani 6 participants 9 messages Jun 15, 2016 1 message not in the archive

lore
Share & tools
lore

Chani

author
#0 lore raw

so I'm trying to convert KDE's translation helper scripts to support git repositories for kde modules. the translators themselves are staying in svn for now, but we want to start migrating kde modules over to git.

I've hit a bit of a problem, though.
inside the translators' folder, there's a folder called documentation/ - and inside that is a bunch of svn externals pointing to the doc/ folder of every kde module. (this is, then, all of kde's english documentation.) translators occasionally need to run a script that reads from the documentation/ folder.

now, what happens when one of those modules switches to git? first, the svn external goes away. that's ok, I can work around that by having the update_xml script (the one translators need to run) fetch the doc folder itself, right? but where does it fetch it from? doc/ is now a subdirectory in a git repository, and git doesn't support checking out only one folder, not even if all I want to do is read the files in it.

checking out the entire git repo isn't really feasible, because once KDE finishes switching to git that'll mean checking out all of KDE. I don't think that turning every doc folder into a submodule is feasible, either - they sound kinda awkward to work with, and I can imagine lots of people getting confused and messing them up...

right now all I've thought of is one ugly hack: have a server that checks out all the kde git repos, pulls daily, copies all the doc/ folders into a documentation folder, and offers that folder up on the interwebs so that update_xml can rsync from it or download a tgz of it or something. there appear to be lots of images in the documentation, so it's not a small download - 200mb and growing. it still hasn't finished downloading all the externals...

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo that only tracks the doc/ folder for a module - but I've no idea whether it'd have to be recreated from scratch every time someone changes something in the real repo's doc/

can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?

Signature · 2 lines
This message brought to you by eevil bananas and the number 3.
www.chani3.com
#1 lore raw
Quoted 9 lines · Chani

[...]

right now all I've thought of is one ugly hack: have a server that checks out all the kde git repos, pulls daily, copies all the doc/ folders into a documentation folder, and offers that folder up on the interwebs so that update_xml can rsync from it or download a tgz of it or something. there appear to be lots of images in the documentation, so it's not a small download - 200mb and growing. it still hasn't finished downloading all the externals...

Do you need the doc/ folders from _all_ kde git repos, or just from those repos that you have currently checked out? In the latter case, you could solve this by adding symlinks to all the doc/ folders inside the documentation/ folder, and then make sure the software that traverse the documentation/ folder recognize and skips symlinks. Of course, this won't work if the translations project need _all_ doc/ folders accessible, but not all the kde git repos.

Quoted 8 lines

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo that only tracks the doc/ folder for a module - but I've no idea whether it'd have to be recreated from scratch every time someone changes something in the real repo's doc/

can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?

Check out git-subtree. It can split out a subdirectory into its own repo, and re-integrate it back into the "parent" repo at a later date. git-subtree has been posted as a patch to this list a couple of times without much response, but it looks like an interesting alternative to submodules:
http://alumnit.ca/~apenwarr/log/?m=200904#30

If a lot of people find git-subtree useful, who knows, it might be included in a future git version.

Have fun! :)

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
author
#2 lore raw
Quoted 11 lines · On May 23, 2009 16:34:53 Johan Herland wrote:
Quoted 9 lines · Chani

[...]

right now all I've thought of is one ugly hack: have a server that checks out all the kde git repos, pulls daily, copies all the doc/ folders into a documentation folder, and offers that folder up on the interwebs so that update_xml can rsync from it or download a tgz of it or something. there appear to be lots of images in the documentation, so it's not a small download - 200mb and growing. it still hasn't finished downloading all the externals...

Quoted 7 lines

Do you need the doc/ folders from _all_ kde git repos, or just from those repos that you have currently checked out? In the latter case, you could solve this by adding symlinks to all the doc/ folders inside the documentation/ folder, and then make sure the software that traverse the documentation/ folder recognize and skips symlinks. Of course, this won't work if the translations project need _all_ doc/ folders accessible, but not all the kde git repos.

nope, the translators may not have checked out *any* of them but the script they want to run needs *all* the docs. :( however, I've been told they also don't want to have to change their workflow in any way at all no matter how small, so we may be stuck in svn-land anyways, because you can't make an svn external out of something that's not in svn, and having anything other than svn externals would change their workflow :(

Quoted 10 lines
Quoted 9 lines

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo that only tracks the doc/ folder for a module - but I've no idea whether it'd have to be recreated from scratch every time someone changes something in the real repo's doc/

can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?

Quoted 8 lines

Check out git-subtree. It can split out a subdirectory into its own repo, and re-integrate it back into the "parent" repo at a later date. git-subtree has been posted as a patch to this list a couple of times without much response, but it looks like an interesting alternative to
submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30

If a lot of people find git-subtree useful, who knows, it might be included in a future git version.

looks interesting. might have been a solution until I heard about this requirement to not change workflow at all. :/

however, my friend told me about a project to make a git-svnserver that serves git repos as svn repos, and *that* would allow the translators to stay where they are without holding everyone else there too. know anything about that?

mm, google turns up an email from someone claiming they have a partial implementation in python...

Signature · 2 lines
This message brought to you by eevil bananas and the number 3.
www.chani3.com
#3 lore raw
Quoted 11 lines · Chani
Quoted 10 lines · On May 23, 2009 16:34:53 Johan Herland wrote:
Quoted 8 lines · Chani

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo that only tracks the doc/ folder for a module - but I've no idea whether it'd have to be recreated from scratch every time someone changes something in the real repo's doc/

can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?

Quoted 8 lines
Quoted 5 lines

Check out git-subtree. It can split out a subdirectory into its own repo, and re-integrate it back into the "parent" repo at a later date. git-subtree has been posted as a patch to this list a couple of times without much response, but it looks like an interesting alternative to
submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30

looks interesting. might have been a solution until I heard about this requirement to not change workflow at all. :/

Here's something that might work. Write a script that runs on a server somewhere. It checks out *all* of the git-managed KDE components, including documentation (it's okay to do this from *one* place, right?). Then it loops through all the documentation subdirectories and splits them out into individual projects/branches with 'git subtree split'.

You could then make a *new* project, kde-docs for example, that pulls those new doc projects back together into a single project, using 'git subtree merge'.

People could then checkout the single kde-docs project and get *all* the docs, then make changes and push them back upstream to you. As far as the average person is concerned, this is probably even easier than svn:externals - they don't have to remember to commit in each subdirectory separately, as git-subtree rips it all apart later. Also note that *end users* never have to operate the git subtree command; all that happens behind the scenes on your server.

Later, you use 'git subtree split' to rip it apart back into the individual projects, and 'git subtree merge' those back into their original git repositories.

Since git-subtree is my own little toy and it's designed for exactly this kind of workflow, I'd be happy to help you debug a script for this if you like. Probably I'll learn something.

Disclaimer: git-subtree hasn't been used by too many people yet and we may end up finding some bugs.

Have fun,

Avery

author
#4 lore raw
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

Check out git-subtree. It can split out a subdirectory into its own repo, and re-integrate it back into the "parent" repo at a later date. git-subtree has been posted as a patch to this list a couple of times without much response, but it looks like an interesting alternative to
submodules: http://alumnit.ca/~apenwarr/log/?m=200904#30

looks interesting. might have been a solution until I heard about this requirement to not change workflow at all. :/

Quoted 7 lines

Here's something that might work. Write a script that runs on a server somewhere. It checks out *all* of the git-managed KDE components, including documentation (it's okay to do this from *one* place, right?). Then it loops through all the documentation subdirectories and splits them out into individual projects/branches with 'git subtree split'.

I like this idea. I like the idea of git subtree in general. :) however, we've decided to go the "copy all documentation into svn" route for now (so that the translators can keep using only svn).
I'll be back when we run into the next svn-external issue (there's sure to be more). :)

Signature · 2 lines
This message brought to you by eevil bananas and the number 3.
www.chani3.com
#5 lore raw
In reply to Avery Pennarun

If all you're looking for is a "single set of read-only docs" conglomeration of all the KDE submodules projects documentation so you can translate them, what about using "archive"? Since archive takes a "tree-ish", and work on remote upload-pack servers (if it's enabled), you can get an un-versioned "remote partial checkout" with something like do things like:

	git archive --format=tar --prefix=git/Documentation/ --remote=git://repo.or.cz/git.git HEAD:Documentation | tar tf - | head
	git archive --format=tar --prefix=/opt/git/share/man/ --remote=git://repo.or.cz/git.git man | tar tf -

So, this could be simply scripted to give you a repo with *all* modules docs/ inside a single repo...
for m in $MODULES
do
git archive --format=tar --prefix="$m/$DOCDIR/" --remote=$REMOTE_BASE/$m $BRANCH:$DOCDIR | tar xf - git add $m/$DOCDIR
git commit $m/$DOCDIR -m "$m: Automatic import"
done

Do that in a central location an dlet translators pull that... Everythings "duplicated", but git's object store will help out there... You loose the history of the original docs, but for translations, you you're not translating the history anyways, your'e translation the docs at a point in time...

Or just dump all thd docs in a .gitignore'd subdirctory and have translators run a script to update them.

a.

Quoted 46 lines · Chani
Quoted 10 lines · On May 23, 2009 16:34:53 Johan Herland wrote:
Quoted 9 lines · Chani
Quoted 8 lines

I'm kinda wondering if there'd be a way to use git-filter-branch to make a repo that only tracks the doc/ folder for a module - but I've no idea whether it'd have to be recreated from scratch every time someone changes something in the real repo's doc/

can anyone think of a less ugly solution?
what are the chances of git supporting this kind of partial checkout someday?

Here's something that might work.  Write a script that runs on a server somewhere.  It checks out all of the git-managed KDE components, including documentation (it's okay to do this from one place, right?).  Then it loops through all the documentation subdirectories and splits them out into individual projects/branches with 'git subtree split'.

You could then make a new project, kde-docs for example, that pulls those new doc projects back together into a single project, using 'git subtree merge'.

People could then checkout the single kde-docs project and get all the docs, then make changes and push them back upstream to you.  As far as the average person is concerned, this is probably even easier than svn:externals - they don't have to remember to commit in each subdirectory separately, as git-subtree rips it all apart later.  Also note that *end users* never have to operate the git subtree command; all that happens behind the scenes on your server.

Later, you use 'git subtree split' to rip it apart back into the individual projects, and 'git subtree merge' those back into their original git repositories.

Since git-subtree is my own little toy and it's designed for exactly this kind of workflow, I'd be happy to help you debug a script for this if you like.  Probably I'll learn something.

Disclaimer: git-subtree hasn't been used by too many people yet and we may end up finding some bugs.

Have fun,

Avery

#6 lore raw
In reply to Chani
Quoted 5 lines · Chani

checking out the entire git repo isn't really feasible, because once KDE finishes switching to git that'll mean checking out all of KDE. I don't think that turning every doc folder into a submodule is feasible, either - they sound kinda awkward to work with, and I can imagine lots of people getting confused and messing them up...

If you don't mind cloning the entire git repo (which may be quite large in KDE case, I guess), then you may want to try "sparse checkout". The patch series is available as a topic branch "tp/sco" in this branch:

http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst

Or I can send the patch series to you. It lets you specify what part of the tree you want to checkout.

Signature · 1 line
Duy
#7 lore raw

2009/5/24 Nguyen Thai Ngoc Duy <[email protected]>:

Quoted 12 lines · Chani
Quoted 5 lines

checking out the entire git repo isn't really feasible, because once KDE finishes switching to git that'll mean checking out all of KDE. I don't think that turning every doc folder into a submodule is feasible, either - they sound kinda awkward to work with, and I can imagine lots of people getting confused and messing them up...

If you don't mind cloning the entire git repo (which may be quite large in KDE case, I guess), then you may want to try "sparse checkout". The patch series is available as a topic branch "tp/sco" in this branch:

Quoted 4 lines

http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst

Or I can send the patch series to you. It lets you specify what part of the tree you want to checkout.

I recall following this series when you first announced it. Do you have any plans to try and get it included in mainline? ISTR there was some more work needed to be done on it before then.

-- Thomas Adam

#8 lore raw
Quoted 13 lines · Thomas Adam

2009/5/24 Nguyen Thai Ngoc Duy <[email protected]>:

Quoted 12 lines · Chani
Quoted 5 lines

checking out the entire git repo isn't really feasible, because once KDE finishes switching to git that'll mean checking out all of KDE. I don't think that turning every doc folder into a submodule is feasible, either - they sound kinda awkward to work with, and I can imagine lots of people getting confused and messing them up...

If you don't mind cloning the entire git repo (which may be quite large in KDE case, I guess), then you may want to try "sparse checkout". The patch series is available as a topic branch "tp/sco" in this branch:

Quoted 8 lines
Quoted 4 lines

http://repo.or.cz/w/git/pclouds.git?a=shortlog;h=refs/heads/inst

Or I can send the patch series to you. It lets you specify what part of the tree you want to checkout.

I recall following this series when you first announced it.  Do you have any plans to try and get it included in mainline?  ISTR there was some more work needed to be done on it before then.

It was pulled off because of big, intrusive and not well-designed patches.

http://mid.gmane.org/[email protected]

I may submit it again when I get code insertions down to ~500 lines or something like that ;-) Anyway you could always help by giving more use cases to test against the current design.

Signature · 1 line
Duy