support git+mosh for unreliable connections
Hi,
When working with big projects over a slow, unreliable connection,
currently there is no way to resume a clone or pull when the connection breaks. mosh is a better replacement for ssh over unreliable connections. supporting git+mosh protocol will go a long way in supporting people who work with unreliable, mobile networks, especially in developed countries (I personally have to try many times when working with large projects as my 3g mobile connection keeps dropping. I recently discovered mosh and it works like a charm. More about mosh
https://mosh.mit.edu/
Thanks
Praveen
Quoted 11 lines · Pirate Praveen
Hi,
When working with big projects over a slow, unreliable connection,
currently there is no way to resume a clone or pull when the connection breaks. mosh is a better replacement for ssh over unreliable connections. supporting git+mosh protocol will go a long way in supporting people who work with unreliable, mobile networks, especially in developed countries (I personally have to try many times when working with large projects as my 3g mobile connection keeps dropping. I recently discovered mosh and it works like a charm. More about mosh
https://mosh.mit.edu/
mosh isn't a generic transport though, it's a udp-based session state synchronization protocol. I don't think it can be used as a git transport.
Signature · 2 lines
Dennis Kaarsemaker www.kaarsemaker.net
Hi Praveen "Arrrr",
Quoted 10 lines · Pirate Praveen
When working with big projects over a slow, unreliable connection,
currently there is no way to resume a clone or pull when the connection breaks. mosh is a better replacement for ssh over unreliable connections. supporting git+mosh protocol will go a long way in supporting people who work with unreliable, mobile networks, especially in developed countries (I personally have to try many times when working with large projects as my 3g mobile connection keeps dropping. I recently discovered mosh and it works like a charm. More about mosh
https://mosh.mit.edu/
Quoted 3 lines
From https://github.com/keithw/mosh:
Mosh does not support X forwarding or the non-interactive uses of SSH, including port forwarding.
In particular it "does not support [...] the non-interactive uses of SSH", which the git+mosh transport would require, though.
That means that you would have to invest quite a bit of effort into enhancing mosh to *support* the non-interactive uses of SSH before you could start implementing git-remote-mosh...
Ciao,
Johannes
Quoted 11 lines · Johannes Schindelin
From https://github.com/keithw/mosh:
Quoted 1 line
Mosh does not support X forwarding or the non-interactive uses of SSH, including port forwarding.
In particular it "does not support [...] the non-interactive uses of SSH", which the git+mosh transport would require, though.
That means that you would have to invest quite a bit of effort into enhancing mosh to *support* the non-interactive uses of SSH before you could start implementing
git-remote-mosh...Ciao,
Johannes
Q: Are the mosh principles relevant to other network applications?
We think so. The design principles that Mosh stands for are
conservative: warning the user if the state being displayed is out of date, serializing and checkpointing all transactions so that if there are no warnings, the user knows every prior transaction has succeeded, and handling expected events (like roaming from one WiFi network to another) gracefully.
Can the ideas be used to resume a pull, push or clone operation? Especially serializing and checkpointing.
Quoted 10 lines · Pirate Praveen
Q: Are the mosh principles relevant to other network applications?
We think so. The design principles that Mosh stands for are
conservative: warning the user if the state being displayed is out of date, serializing and checkpointing all transactions so that if there are no warnings, the user knows every prior transaction has succeeded, and handling expected events (like roaming from one WiFi network to another) gracefully.
Quoted 2 lines
Can the ideas be used to resume a pull, push or clone operation? Especially serializing and checkpointing.
Well, it is possible to write a remote helper and serverside program that internally handles connection unreliability, so Git itself (upload-archive, upload-pack, receive-pack, archive, fetch-pack and send-pack) sees a reliable (full-duplex, half-closeable, stream) channel.
Suitably done, that can "resume" (from Git POV, nothing special happened) across things like IP address changes.
However, that is quite difficult to do in practice. Not because interface to Git is complicated, but because the transport problem itself is complicated (however, it still seems way easier than making Git internally be able to resume interrupted operations).
Mosh needs to solve at least most of that, it just doesn't provode the right kind of interface.
-Ilari
Quoted 23 lines · Ilari Liusvaara
Quoted 10 lines · Pirate Praveen
Q: Are the mosh principles relevant to other network applications?
We think so. The design principles that Mosh stands for are
conservative: warning the user if the state being displayed is out of date, serializing and checkpointing all transactions so that if there are no warnings, the user knows every prior transaction has succeeded, and handling expected events (like roaming from one WiFi network to another) gracefully.Quoted 2 lines
Can the ideas be used to resume a pull, push or clone operation? Especially serializing and checkpointing.
Well, it is possible to write a remote helper and serverside program that internally handles connection unreliability, so Git itself (upload-archive, upload-pack, receive-pack, archive, fetch-pack and send-pack) sees a reliable (full-duplex, half-closeable, stream) channel.
Suitably done, that can "resume" (from Git POV, nothing special happened) across things like IP address changes.
Quoted 7 lines
However, that is quite difficult to do in practice. Not because interface to Git is complicated, but because the transport problem itself is complicated (however, it still seems way easier than making Git internally be able to resume interrupted operations).
Mosh needs to solve at least most of that, it just doesn't provode the right kind of interface.
I have requested mosh team to fix these issues
https://github.com/keithw/mosh/issues/597
Pirate Praveen venit, vidit, dixit 15.04.2015 15:07:
Quoted 11 lines
Hi,
When working with big projects over a slow, unreliable connection,
currently there is no way to resume a clone or pull when the connection breaks. mosh is a better replacement for ssh over unreliable connections. supporting git+mosh protocol will go a long way in supporting people who work with unreliable, mobile networks, especially in developed countries (I personally have to try many times when working with large projects as my 3g mobile connection keeps dropping. I recently discovered mosh and it works like a charm. More about mosh
https://mosh.mit.edu/
What would that require git to do, beyond taking whatever you tell it (using GIT_SSH or _GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
Michael
Quoted 5 lines · Michael J Gruber
What would that require git to do, beyond taking whatever you tell it (using
GIT_SSHor_GIT_SSH_COMMAND) to use as a drop in replacement for ssh?Michael
May be support git+mosh as a protocol, since it is not a drop in replacement. It is redesigned remote shell. The ideas it uses for session resumption needs to be reimplemented. This will need support from git, because it needs server side to be modified. Use SSP to return the the current progress for a particular session (it uses AES session ids).
So when a client connect with a session id, git server side can respond with the current state, how many objects received in that session, and client can continue from where it stopped. Client also will need to store session information.
Hi Praveen,
Quoted 9 lines · Pirate Praveen
Quoted 2 lines · Michael J Gruber
What would that require git to do, beyond taking whatever you tell it (using
GIT_SSHor_GIT_SSH_COMMAND) to use as a drop in replacement for ssh?May be support git+mosh as a protocol, since it is not a drop in replacement. It is redesigned remote shell. The ideas it uses for session resumption needs to be reimplemented. This will need support from git, because it needs server side to be modified. Use SSP to return the the current progress for a particular session (it uses AES session ids).
It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
Quoted 4 lines
So when a client connect with a session id, git server side can respond with the current state, how many objects received in that session, and client can continue from where it stopped. Client also will need to store session information.
No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
Ciao,
Johannes
Quoted 7 lines · Johannes Schindelin
Hi Praveen,
Quoted 4 lines · Pirate Praveen
Quoted 2 lines · Michael J Gruber
What would that require git to do, beyond taking whatever you tell it (using
GIT_SSHor_GIT_SSH_COMMAND) to use as a drop in replacement for ssh?
Quoted 7 lines
Quoted 5 lines
May be support git+mosh as a protocol, since it is not a drop in replacement. It is redesigned remote shell. The ideas it uses for session resumption needs to be reimplemented. This will need support from git, because it needs server side to be modified. Use SSP to return the the current progress for a particular session (it uses AES session ids).
It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
is that really relevent? mosh doesn't support things like X forwarding or port forwarding, but it certainly does support ssh <host> <command> and then doing IO. It might not support not doing terminal emulation stuff, but that seems like a simple thing to change in principal at which point I think it would support enough of ssh's functionality its a drop in replacement as far as git is concerned. Seems to me mosh is close enough on its own its worth experimentation by someone who cares.
Trev
Quoted 13 lines
Quoted 4 lines
So when a client connect with a session id, git server side can respond with the current state, how many objects received in that session, and client can continue from where it stopped. Client also will need to store session information.
No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
Ciao,
Johannes
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Trevor,
Quoted 13 lines · Trevor Saunders
Quoted 12 lines · Johannes Schindelin
Quoted 9 lines · Pirate Praveen
Quoted 2 lines · Michael J Gruber
What would that require git to do, beyond taking whatever you tell it (using
GIT_SSHor_GIT_SSH_COMMAND) to use as a drop in replacement for ssh?May be support git+mosh as a protocol, since it is not a drop in replacement. It is redesigned remote shell. The ideas it uses for session resumption needs to be reimplemented. This will need support from git, because it needs server side to be modified. Use SSP to return the the current progress for a particular session (it uses AES session ids).
Quoted 5 lines
Quoted 1 line
It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
is that really relevent? mosh doesn't support things like X forwarding or port forwarding, but it certainly does support ssh <host> <command> and then doing IO.
Ah, so mosh's README lied to me!
If mosh <user>@<host> <command> works, then a simple GIT_SSH=mosh should work out of the box, too. Have you tried it?
Ciao,
Johannes
It might not support not doing terminal emulation
Quoted 24 lines
stuff, but that seems like a simple thing to change in principal at which point I think it would support enough of ssh's functionality its a drop in replacement as far as git is concerned. Seems to me mosh is close enough on its own its worth experimentation by someone who cares.
Trev
Quoted 5 lines
Quoted 4 lines
So when a client connect with a session id, git server side can respond with the current state, how many objects received in that session, and client can continue from where it stopped. Client also will need to store session information.
Quoted 8 lines
No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
Ciao,
Johannes
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Trevor Saunders
but it certainly does support ssh <host> <command> and then doing IO.
Yes, in interactive sessions. mosh synchronizes terminal state, it doesn't allow random I/O between client and server.
Quoted 4 lines
Ah, so mosh's README lied to me!
If
mosh <user>@<host> <command>works, then a simpleGIT_SSH=moshshould work out of the box, too. Have you tried it?
It does not and cannot work. The way mosh works, is that it uses ssh to log in and launch a mosh-server daemon. This daemon and the mosh client then communicate via a custom UDP protocol. The SSH connection is closed after the mosh-server has been launched as it is no longer needed.
The communication between the mosh client and server synchronizes terminal state, somewhat like what screen/tmux do.
Signature · 2 lines
Dennis Kaarsemaker www.kaarsemaker.net
On Wed, 15 Apr 2015 21:25:44 +0000, Dennis Kaarsemaker wrote:
...
Quoted 7 lines
It does not and cannot work. The way mosh works, is that it uses ssh to log in and launch a mosh-server daemon. This daemon and the mosh client then communicate via a custom UDP protocol. The SSH connection is closed after the mosh-server has been launched as it is no longer needed.
The communication between the mosh client and server synchronizes terminal state, somewhat like what screen/tmux do.
I object to the 'can not' part a bit. There is (1) the terminal state prediction and (2) the reliable-over-reconnects communication, and for a noninteractive usage you'd need only (2).
Once upon a time I implemented a simple UDP server and client; the client to be used as a ProxyCommand in ssh, and the server just talks to the local ssh server. This pretty much does what the OP wants, and it works just as a transport for ssh, so all ssh features are there (but of course there is no terminal prediction). Unfortunately it needs to be ported to libev/libuv before it could be released. It's *much* simpler than mosh, although the use-ssh-to-start-server trick would be nice.)
Andreas
Signature · 3 lines
"Totally trivial. Famous last words." From: Linus Torvalds <torvalds@*.org> Date: Fri, 22 Jan 2010 07:29:21 -0800
Quoted 16 lines · Johannes Schindelin
Hi Trevor,
Quoted 13 lines · Trevor Saunders
Quoted 12 lines · Johannes Schindelin
Quoted 9 lines · Pirate Praveen
Quoted 2 lines · Michael J Gruber
What would that require git to do, beyond taking whatever you tell it (using
GIT_SSHor_GIT_SSH_COMMAND) to use as a drop in replacement for ssh?May be support git+mosh as a protocol, since it is not a drop in replacement. It is redesigned remote shell. The ideas it uses for session resumption needs to be reimplemented. This will need support from git, because it needs server side to be modified. Use SSP to return the the current progress for a particular session (it uses AES session ids).
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
is that really relevent? mosh doesn't support things like X forwarding or port forwarding, but it certainly does support ssh <host> <command> and then doing IO.
Ah, so mosh's README lied to me!
I wouldn't say it lied, its just not really clear what is "interactive" I'd say git's use of ssh is kind of interactive compared to things like port forwarding.
Quoted 1 line
If
mosh <user>@<host> <command>works, then a simpleGIT_SSH=moshshould work out of the box, too. Have you tried it?
it does work, I just tried mosh $host cat and then typing stuff and having it printed back at me. However it clears the terminal before hand and prints a message on exit. I tried GIT_SSH=mosh git clone and it failed, but I haven't really dug into why. SO I suspect this can be made to work with some work on the mosh side, but I'm not sure exactly how ssh and mosh are behaving differently here.
Trev
Quoted 32 lines
Ciao,
Johannes
It might not support not doing terminal emulationQuoted 20 lines
stuff, but that seems like a simple thing to change in principal at which point I think it would support enough of ssh's functionality its a drop in replacement as far as git is concerned. Seems to me mosh is close enough on its own its worth experimentation by someone who cares.
Trev
Quoted 5 lines
Quoted 4 lines
So when a client connect with a session id, git server side can respond with the current state, how many objects received in that session, and client can continue from where it stopped. Client also will need to store session information.
Quoted 8 lines
No, the protocol can stay exactly the same, once you have a way to communicate non-interactively via mosh.
Ciao,
Johannes
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlQuoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe git" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Trevor Saunders venit, vidit, dixit 15.04.2015 20:59:
Quoted 31 lines · Johannes Schindelin
Quoted 16 lines
Hi Trevor,
Quoted 13 lines · Trevor Saunders
Quoted 12 lines · Johannes Schindelin
Quoted 9 lines · Pirate Praveen
Quoted 2 lines · Michael J Gruber
What would that require git to do, beyond taking whatever you tell it (using
GIT_SSHor_GIT_SSH_COMMAND) to use as a drop in replacement for ssh?May be support git+mosh as a protocol, since it is not a drop in replacement. It is redesigned remote shell. The ideas it uses for session resumption needs to be reimplemented. This will need support from git, because it needs server side to be modified. Use SSP to return the the current progress for a particular session (it uses AES session ids).
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
It will need support from Git alright, but not as much as from mosh, see my other reply: Mosh was not designed for non-interactive use. That support would have to be added before we can go any further.
is that really relevent? mosh doesn't support things like X forwarding or port forwarding, but it certainly does support ssh <host> <command> and then doing IO.
Ah, so mosh's README lied to me!
I wouldn't say it lied, its just not really clear what is "interactive" I'd say git's use of ssh is kind of interactive compared to things like port forwarding.
Quoted 1 line
If
mosh <user>@<host> <command>works, then a simpleGIT_SSH=moshshould work out of the box, too. Have you tried it?
Quoted 8 lines
it does work, I just tried mosh $host cat and then typing stuff and having it printed back at me. However it clears the terminal before hand and prints a message on exit. I tried GIT_SSH=mosh git clone and it failed, but I haven't really dug into why. SO I suspect this can be made to work with some work on the mosh side, but I'm not sure exactly how ssh and mosh are behaving differently here.
Trev
First thing you see on mosh.mit.edu:
"Mosh is a replacement for SSH."
I guess that needs a footnote...
Michael