Secure XML-RPC for Xend
RFC PATCH 1/1 Secure XML-RPC for Xend
Hi,
The following patch implements a secure XML-RPC protocol for Xend. Instead of using HTTPS with basic authentication and dealing with all that nasty OpenSSL/PAM integration, it just uses SSH. This gives you all the properties you want (great security and PAM integration) with very little code.
There are some minor issues so I'd rather it not be applied immediately. I'd like to get some feedback from people as to whether this approach is reasonable. A user-facing change is that now you can use the XM_SERVER environmental variable to specific an XML-RPC URI.
For instance:
XM_SERVER='ssh://[email protected]/RPC2' xm list
Runs xm list on a local machine but does all of the RPCs over a secure connection (prompting for passwords).
Thoughts?
Regards,
Anthony Liguori
# HG changeset patch
# User [email protected]
# Node ID 4de241a7e91a1e59b6db965f5d2ef10014f764e5
# Parent 4f1e39ec05d6ec711f9ba4a66a3653ed3e168311
Add support secure XML-RPC. This is done by multiplexing multiple SSH sessions over a single session (to avoid multiple password entries). Here are the changes:
1) Add support to xmlrpclib2.ServerProxy for ssh:// protocol
2) Add an xm serve command which proxies XML-RPC over stdio
3) Make xm look at the XM_SERVER variable to determine which XML-RPC protocol to use
There are some issues that need to be addressed before inclusion. Namely:
1) Python moans about tempnam(). I don't think there's a better solution though.
2) A command *must* be executed to cleanup the ssh session on exit. I currently use __del__() which doesn't seem to make Python happy in certain cases.
3) I have done basic testing but not regression testing with xm-test
tools/python/xen/util/xmlrpclib2.py +126 -3
@@ -24,14 +24,117 @@ import types from httplib import HTTPConnection, HTTP -from xmlrpclib import Transport +from xmlrpclib import Transport, getparser + from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler import xmlrpclib, socket, os, stat import SocketServer -import xen.xend.XendClient from xen.xend.XendLogging import log +import os, commands, getpass, termios, fcntl + +class SSH: + def __init__(self, host, user=None, askpass=None): + """Constructor for SSH object. + + This constructor allows you to specify a hostname, a username, + and an askpass program. username will default to the current + user and if askpass is not specified, the password will be + read (if necessary) from the controlling terminal.""" + + self.sock = os.tempnam() + self.host = host + sock = self.sock + if user == None: user = getpass.getuser() + self.user = user + + pid = os.fork() + if pid == 0: + if askpass: + f = open('/dev/tty', 'w') + os.environ['SSH_ASKPASS'] = askpass + fcntl.ioctl(f.fileno(), termios.TIOCNOTTY) + f.close() + + os.execvp('/usr/bin/ssh', ['ssh', '-f', '-2', '-N', '-M', '-S', + sock, '-a', '-x', '-l', user, host]) + + p,s = os.waitpid(pid, 0) + if s != 0: + raise OSError(s, 'Failed to start ssh server') + + def close(self): + """Closes an SSH object destroying the SSH server. + + This command must be called when the object is no longer + needed to ensure that the SSH server is destroyed properly.""" + + if self.sock: + commands.getstatusoutput('ssh -q -t -S %s -l %s -O exit %s' % + (self.sock, self.user, self.host)) + + # is this the best way to deal with this?? + def __del__(self): + self.close() + + def getcmd(self, cmd): + """Returns a command expanded to include SSH options. + + This function will add the appropriate ssh command and + options to the passed in command string. This is useful + if commands.getstatusoutput is not an appropriate exec + interface or if you want to handle error conditions in + your own way.""" + + return 'ssh -q -t -S %s -l %s %s %s' % (self.sock, self.user, + self.host, cmd) + + def runcmd(self, cmd, data=None): + """Runs a command using an existing SSH connection. + + This function will run the passed in command on a remote + machine and either return the output or raise an OSError + if the command exits with a non-zero status (or some + other failure occurs).""" + + cmdline = self.getcmd(cmd) + if data: + f = open("/tmp/stuff.txt", "w") + f.write(data) + f.close() + cmdline = "cat /tmp/stuff.txt | %s" % cmdline + else: + cmdline = cmdline + + o,s = commands.getstatusoutput(cmdline) + if o != 0: + raise OSError(o,s) + return s + +class SSHTransport(object): + def __init__(self, ssh): + self.ssh = ssh + + def request(self, host, handler, request_body, verbose=0): + p, u = getparser() + s = self.ssh.runcmd("xm serve", + """POST /%s HTTP/1.0 +User-Agent: Xen +Host: %s +Content-Type: text/xml +Content-Length: %d + +%s""" % (handler, host, len(request_body), request_body)) + lines = s.split('\n') + for i in range(len(lines)): + if lines[i] == '' or lines[i] == '\r': + s = '\n'.join(lines[(i + 1):]) + break + + p.feed(s) + p.close() + return u.close() # A new ServerProxy that also supports httpu urls. An http URL comes in the # form:
@@ -68,13 +171,32 @@ class ServerProxy(xmlrpclib.ServerProxy): + def process_ssh_uri(self, uri, askpass): + uri = uri[6:] + parts = uri.split('@') + if len(parts) > 1: + user = parts[0] + uri = '@'.join(parts[1:]) + else: + user = None + parts = uri.split('/') + if len(parts) < 2: + raise ValueError("Invalid ssh:// URL '%s'" % uri) + host = parts[0] + path = '/'.join(parts[1:]) + ssh = SSH(host, user, askpass=askpass) + transport = SSHTransport(ssh) + return transport, 'http://%s/%s' % (host, path) + def __init__(self, uri, transport=None, encoding=None, verbose=0, - allow_none=1): + allow_none=1, askpass=None): if transport == None: (protocol, rest) = uri.split(':', 1) if protocol == 'httpu': uri = 'http:' + rest transport = UnixTransport() + elif protocol == 'ssh': + transport, uri = self.process_ssh_uri(uri, askpass) xmlrpclib.ServerProxy.__init__(self, uri, transport, encoding, verbose, allow_none)
@@ -121,6 +243,7 @@ except xmlrpclib.Fault, fault: response = xmlrpclib.dumps(fault) except Exception, exn: + import xen.xend.XendClient log.exception(exn) response = xmlrpclib.dumps( xmlrpclib.Fault(xen.xend.XendClient.ERROR_INTERNAL, str(exn)))
tools/python/xen/xend/XendClient.py +5 -1
@@ -18,6 +18,7 @@ #============================================================================ from xen.util.xmlrpclib2 import ServerProxy +import os XML_RPC_SOCKET = "/var/run/xend/xmlrpc.sock"
@@ -25,4 +26,7 @@ ERROR_GENERIC = 2 ERROR_INVALID_DOMAIN = 3 -server = ServerProxy('httpu:///var/run/xend/xmlrpc.sock') +if os.environ.has_key('XM_SERVER'): + server = ServerProxy(os.environ['XM_SERVER']) +else: + server = ServerProxy('httpu:///var/run/xend/xmlrpc.sock')
tools/python/xen/xm/main.py +34 -1
@@ -124,6 +124,7 @@ loadpolicy_help = "loadpolicy <policy> Load binary policy into hypervisor" makepolicy_help = "makepolicy <policy> Build policy and create .bin/.map files" labels_help = "labels [policy] [type=DOM|..] List <type> labels for (active) policy." +serve_help = "serve Proxy Xend XML-RPC over stdio" short_command_list = [ "console",
@@ -171,7 +172,8 @@ host_commands = [ "dmesg", "info", - "log" + "log", + "serve", ] scheduler_commands = [
@@ -833,6 +835,36 @@ arg_check(args, "log", 0) print server.xend.node.log() + +def xm_serve(args): + arg_check(args, "serve", 0) + + try: + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + s.connect(xen.xend.XendClient.XML_RPC_SOCKET) + f = sys.stdin + + content_length = 0 + line = f.readline() + headers = '' + while len(line) != 0: + if line.lower().startswith('content-length: '): + content_length = int(line[16:]) + headers += line + if line in ['\r\n', '\n']: + break + line = f.readline() + + buf = f.read(content_length) + s.sendall(headers + buf) + + buf = s.recv(4096) + while len(buf) != 0: + sys.stdout.write(buf) + buf = s.recv(4096) + s.close() + except Exception, ex: + print ex def parse_dev_info(info): def get_info(n, t, d):
@@ -1072,6 +1104,7 @@ "dmesg": xm_dmesg, "info": xm_info, "log": xm_log, + "serve": xm_serve, # scheduler "sched-bvt": xm_sched_bvt, "sched-bvt-ctxallow": xm_sched_bvt_ctxallow,
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 5 lines · Anthony Liguori
The following patch implements a secure XML-RPC protocol for Xend. Instead of using HTTPS with basic authentication and dealing with all that nasty OpenSSL/PAM integration, it just uses SSH. This gives you all the properties you want (great security and PAM integration) with very little code.
Are there any plans to make the XML-RPC interface easily accessable for things other than xm? Although HTTPS (I'd use client certs rather than basic auth, but that's even worse from a PAM integration PoV) is more overhead, it's at least platform-independent. I've been doing a bit of poking into using XML-RPC to control Xend from Ruby, and it's a hassle. Adding an SSH tunnel layer over the top is going to be even more of a nightmare.
My workaround at the moment is to create a higher-level interface to control Xend -- it's SOAP over HTTPS (with client certs to perform authentication and, eventually, authorization). I'm using SOAP simply because it's got WSDL. The interface is a lot simpler (very, very few S-expressions to deal with), so it'll probably still exist even if Xend gets a more advanced XML-RPC layer, but it'd save me some fiddling if I could poke Xend's XML-RPC securely without needing to grub around for a Unix socket.
- Matt
Quoted 3 lines · Anthony Liguori
Add support secure XML-RPC. This is done by multiplexing multiple SSH sessions over a single session (to avoid multiple password entries). Here are the changes:
I like the general idea, comments inline.
Quoted 10 lines
1) Add support to xmlrpclib2.ServerProxy for ssh:// protocol
2) Add an xm serve command which proxies XML-RPC over stdio
3) Make xm look at theXM_SERVERvariable to determine which XML-RPC protocol to useThere are some issues that need to be addressed before inclusion. Namely:
1) Python moans about
tempnam(). I don't think there's a better solution though.
I don't like the dependency on directly calling ssh multiplexing, as it requires a relatively modern OpenSSH (>3.9) and the above race condition is introduced. A newer feature in OpenSSH is to let the ControlPath consist of "%h,%p,%r" wildcards which fill in the host/user/port being connected to in a socket pathname, which solves that particular race.
Why not just do the SSH every time, and let the user either set up connection multiplexing or ssh agent in their local environment instead? That way it will work for old OpenSSH versions and you don't have to deal with all the quirks.
Quoted 6 lines
2) A command *must* be executed to cleanup the ssh session on exit. I currently use
__del__()which doesn't seem to make Python happy in certain cases.
3) I have done basic testing but not regression testing with xm-testdiff -r
4f1e39ec05d6-r4de241a7e91atools/python/xen/util/xmlrpclib2.py
...
Quoted 4 lines
tools/python/xen/util/xmlrpclib2.py +4
-0+ def runcmd(self, cmd, data=None): + """Runs a command using an existing SSH connection. + + This function will run the passed in command on a remote
tools/python/xen/util/xmlrpclib2.py line 106
Quoted 10 lines
tools/python/xen/util/xmlrpclib2.py +10
-0+ machine and either return the output or raise an OSError + if the command exits with a non-zero status (or some + other failure occurs).""" + + cmdline = self.getcmd(cmd) + if data: + f = open("/tmp/stuff.txt", "w") + f.write(data) + f.close() + cmdline = "cat /tmp/stuff.txt | %s" % cmdline
Ouch, this bit definitely needs to be fixed at least :)
Signature · 2 lines
Anil Madhavapeddy http://anil.recoil.org University of Cambridge http://www.cl.cam.ac.uk
Quoted 10 lines · Anil Madhavapeddy
Quoted 2 lines
1) Python moans about
tempnam(). I don't think there's a better solution though.I don't like the dependency on directly calling ssh multiplexing, as it requires a relatively modern OpenSSH (>3.9) and the above race condition is introduced. A newer feature in OpenSSH is to let the ControlPath consist of "%h,%p,%r" wildcards which fill in the host/user/port being connected to in a socket pathname, which solves that particular race.
Quoted 3 lines
Why not just do the SSH every time, and let the user either set up connection multiplexing or ssh agent in their local environment instead?
SSH authentication is really expensive especially when you compare to
other cost in the XML-RPC. I would really like some persistency of the connection if possible, especially for operations like monitoring, it's okay to reopen from time to time, but without reuse it would just not work.
Daniel
Signature · 3 lines
Daniel Veillard | Red Hat http://redhat.com/ [email protected] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Quoted 6 lines · Daniel Veillard
SSH authentication is really expensive especially when you compare to
other cost in the XML-RPC. I would really like some persistency of the connection if possible, especially for operations like monitoring, it's okay to reopen from time to time, but without reuse it would just not work.
Yes, but the right place to do it is not in Xend. The auth caching can be set up outside of Xend much more robustly depending on your version of OpenSSH. If done in Xend, then it definitely needs to use the wildcard support in ControlPath to avoid the authentication race condition, and an OpenSSH version check.
As Ian says, stunnel/SSL is probably easier from the client's point of view (although I do like the easier SSH key management this patch allows).
Anil
Quoted 6 lines
Why not just do the SSH every time, and let the user either set up connection multiplexing or ssh agent in their local environment instead? That way it will work for old OpenSSH versions and you don't have to deal with all the quirks.
Another option is just to make sure that the XML-RPC supports keep-alive and reuse the same xm serve session. I think that's probably the most compatible approach.
Quoted 18 lines
Quoted 7 lines
2) A command *must* be executed to cleanup the ssh session on exit. I currently use
__del__()which doesn't seem to make Python happy in certain cases.
3) I have done basic testing but not regression testing with xm-testdiff -r
4f1e39ec05d6-r4de241a7e91atools/python/xen/util/xmlrpclib2.py...
Quoted 9 lines
tools/python/xen/util/xmlrpclib2.py +9
-0+ def runcmd(self, cmd, data=None): + """Runs a command using an existing SSH connection. + + This function will run the passed in command on a remote + machine and either return the output or raise an OSError + if the command exits with a non-zero status (or some + other failure occurs).""" + + cmdline = self.getcmd(cmd)
Quoted 9 lines
tools/python/xen/util/xmlrpclib2.py line 106
Quoted 6 lines
tools/python/xen/util/xmlrpclib2.py +5
-0+ if data: + f = open("/tmp/stuff.txt", "w") + f.write(data) + f.close() + cmdline = "cat /tmp/stuff.txt | %s" % cmdlineOuch, this bit definitely needs to be fixed at least :)
Yikes, didn't know that was still there :-)
Regards,
Anthony Liguori
Quoted 8 lines · Anil Madhavapeddy
Quoted 6 lines · Daniel Veillard
SSH authentication is really expensive especially when you compare to
other cost in the XML-RPC. I would really like some persistency of the connection if possible, especially for operations like monitoring, it's okay to reopen from time to time, but without reuse it would just not work.
Quoted 5 lines
Yes, but the right place to do it is not in Xend. The auth caching can be set up outside of Xend much more robustly depending on your version of OpenSSH. If done in Xend, then it definitely needs to use the wildcard support in ControlPath to avoid the authentication race condition, and an OpenSSH version check.
I think Daniel is suggesting that we use HTTP Keep-Alive which I also think is a really good idea. I think this will come in handy regardless of whether we use SSH.
This makes my patch a lot nicer though. I just would make sure the client uses Keep-Alive and then you get the same 1-time auth without any of the SSH trickery.
I'm investigating this right now. I seem to recall the HTTP server in python providing support for Keep-Alive...
Quoted 4 lines
As Ian says, stunnel/SSL is probably easier from the client's point of view (although I do like the easier SSH key management this patch allows).
There doesn't have to be one solution. The only real code that's needed here is xm serve which is not more than 100 lines. The client code is more of an example. I see no reason why we couldn't support all of these protocols (httpu, http, https, ssh).
Regards,
Anthony Liguori
Quoted 1 line
Anil
Quoted 16 lines · Anthony Liguori
Quoted 8 lines · Anil Madhavapeddy
Quoted 6 lines · Daniel Veillard
SSH authentication is really expensive especially when you compare to
other cost in the XML-RPC. I would really like some persistency of the connection if possible, especially for operations like monitoring, it's okay to reopen from time to time, but without reuse it would just not work.Quoted 5 lines
Yes, but the right place to do it is not in Xend. The auth caching can be set up outside of Xend much more robustly depending on your version of OpenSSH. If done in Xend, then it definitely needs to use the wildcard support in ControlPath to avoid the authentication race condition, and an OpenSSH version check.
Quoted 3 lines
I think Daniel is suggesting that we use HTTP Keep-Alive which I also think is a really good idea. I think this will come in handy regardless of whether we use SSH.
Activating Keep-Alive would be a really good idea in any case,
local or remote, direct auth or tunnelling ! IIRC the main question was about activating it at the Python level, that's something we discussed on IRC but never formally I guess :-)
Quoted 3 lines
This makes my patch a lot nicer though. I just would make sure the client uses Keep-Alive and then you get the same 1-time auth without any of the SSH trickery.
Is that just client side ?
Quoted 2 lines
I'm investigating this right now. I seem to recall the HTTP server in python providing support for Keep-Alive...
Okay, maybe I'm off base :-)
Quoted 9 lines
Quoted 4 lines
As Ian says, stunnel/SSL is probably easier from the client's point of view (although I do like the easier SSH key management this patch allows).
There doesn't have to be one solution. The only real code that's needed here is xm serve which is not more than 100 lines. The client code is more of an example. I see no reason why we couldn't support all of these protocols (httpu, http, https, ssh).
Agreed, those are layered features, they should not have to conflict.
Daniel
Signature · 3 lines
Daniel Veillard | Red Hat http://redhat.com/ [email protected] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Quoted 8 lines · Daniel Veillard
Quoted 3 lines
I think Daniel is suggesting that we use HTTP Keep-Alive which I also think is a really good idea. I think this will come in handy regardless of whether we use SSH.
Activating Keep-Alive would be a really good idea in any case,
local or remote, direct auth or tunnelling ! IIRC the main question was about activating it at the Python level, that's something we discussed on IRC but never formally I guess :-)
I've got Keep-Alive working. I'll post the patches in a few hours when I get home and have a real email client.
The Python BaseHTTPServer code supports Keep-Alive quite nicely but the XML-RPC has an ugly bug where it always shutdowns the connection explicitly after a POST. After overriding the offending method, and subclassing the RequestHandler to enabled HTTP/1.1 as the default protocol, it appears to Just Work.
I'll do an xm-test run this afternoon and hopefully have a well-tested patch (for keep-alive) by the evening.
Quoted 5 lines
Quoted 3 lines
This makes my patch a lot nicer though. I just would make sure the client uses Keep-Alive and then you get the same 1-time auth without any of the SSH trickery.
Is that just client side ?
I'm showing my ignorance a bit. Defaulting to HTTP/1.1, Keep-Alive is enabled by default. I just have to cleanup the SSH transport now to actually make use of it.
I don't think it's worth having the httpu/http protocols use Keep-Alive by default but I'm sure we'd want to make use of it force https.
Regards,
Anthony Liguori
Quoted 18 lines
Quoted 2 lines
I'm investigating this right now. I seem to recall the HTTP server in python providing support for Keep-Alive...
Okay, maybe I'm off base :-)
Quoted 9 lines
Quoted 4 lines
As Ian says, stunnel/SSL is probably easier from the client's point of view (although I do like the easier SSH key management this patch allows).
There doesn't have to be one solution. The only real code that's needed here is xm serve which is not more than 100 lines. The client code is more of an example. I see no reason why we couldn't support all of these protocols (httpu, http, https, ssh).
Agreed, those are layered features, they should not have to conflict.
Daniel
Quoted 8 lines · Anthony Liguori
Hi,
The following patch implements a secure XML-RPC protocol for Xend. Instead of using HTTPS with basic authentication and dealing with all that nasty OpenSSL/PAM integration, it just uses SSH. This gives you all the properties you want (great security and PAM integration) with very little code.
Quoted 4 lines
There are some minor issues so I'd rather it not be applied immediately. I'd like to get some feedback from people as to whether this approach is reasonable. A user-facing change is that now you can use the
XM_SERVERenvironmental variable to specific an XML-RPC URI.
I'm with Ian -- I'd rather see the SSL/PAM solution done properly than this. That said, I don't see why we can't have this transport as well -- it's not a big patch.
What happens if SSH isn't installed? I don't see any nice diagnostic of that, so I'm guessing that it just splats out an "execv failed" exception (unless I've missed something).
Ewan.
Quoted 10 lines · Ewan Mellor
Quoted 6 lines · Anthony Liguori
Hi,
The following patch implements a secure XML-RPC protocol for Xend. Instead of using HTTPS with basic authentication and dealing with all that nasty OpenSSL/PAM integration, it just uses SSH. This gives you all the properties you want (great security and PAM integration) with very little code.
There are some minor issues so I'd rather it not be applied immediately. I'd like to get some feedback from people as to whether this approach is reasonable. A user-facing change is that now you can use the
XM_SERVERenvironmental variable to specific an XML-RPC URI.
Quoted 8 lines
I'm with Ian -- I'd rather see the SSL/PAM solution done properly than this. That said, I don't see why we can't have this transport as well -- it's not a big patch.
What happens if SSH isn't installed? I don't see any nice diagnostic of that, so I'm guessing that it just splats out an "execv failed" exception (unless I've missed something).
In the current code, Popen throws an OSError.
I really don't like catching exceptions and doing an sys.exit within the command handler. I'd rather introduce a new exception type for use in xm and rethrow the OSError with a friendly message. This will make localization quite a bit easier.
What do you think of this?
Regards,
Anthony Liguori
Quoted 2 lines
Ewan.
Quoted 30 lines · Anthony Liguori
Quoted 23 lines · Ewan Mellor
Quoted 8 lines · Anthony Liguori
Hi,
The following patch implements a secure XML-RPC protocol for Xend. Instead of using HTTPS with basic authentication and dealing with all that nasty OpenSSL/PAM integration, it just uses SSH. This gives you all the properties you want (great security and PAM integration) with very little code.
Quoted 5 lines
There are some minor issues so I'd rather it not be applied immediately. I'd like to get some feedback from people as to whether this approach is reasonable. A user-facing change is that now you can use the
XM_SERVERenvironmental variable to specific an XML-RPC URI.I'm with Ian -- I'd rather see the SSL/PAM solution done properly than this.
That said, I don't see why we can't have this transport as well -- it's not a
big patch.Quoted 5 lines
What happens if SSH isn't installed? I don't see any nice diagnostic of that,
so I'm guessing that it just splats out an "execv failed" exception (unless I've missed something).
Quoted 8 lines
In the current code, Popen throws an OSError.
I really don't like catching exceptions and doing an sys.exit within the command handler. I'd rather introduce a new exception type for use in xm and rethrow the OSError with a friendly message. This will make localization quite a bit easier.
What do you think of this?
Sure, diagnose and rethrow all the way to the top level -- that's what main.py does now for most things, and it makes it easier to integrate main.py into larger applications too.
Ewan.