Skip to contentSkip to conversation

Add support for XML-RPC over SSH (take two)

PATCH Applied Anthony Liguori 2 participants 9 messages Jun 14, 2006 – Jun 21, 2006

lore
Share & tools
lore
#0 lore raw

cover Add support for XML-RPC over SSH (take two)

This series is a repost of the previous series taking into account Ewan's feedback wrt error reporting. Additionally, to make things a bit saner, I changed the ssh invocation from being always done in the ctor to being done on demand. This means that xm commands like help won't actually require a password if the ssh transport is being used.

I've also included the HTTP/1.1 Keep-Alive patch as it's required for this patch set.

Regards,

Anthony Liguori

#1 lore raw

PATCH 1/4 Fix Xend's HTTP/1.1 Keep-Alive support

Repost of earlier patch

Regards,

Anthony Liguori

# HG changeset patch
# User [email protected]
# Node ID 9ad475c664bff5e668cebbf864861807583d657d # Parent 4f1e39ec05d6ec711f9ba4a66a3653ed3e168311 Add support to Xend XML-RPC server for HTTP/1.1 Keep-Alive.

This patch fixes a few bugs in the Python SimpleXMLRPC server and enables HTTP/1.1 by default. This allows a client to use Keep-Alive. Keep-Alive improves performance by eliminating the overhead of connection setup and, more importantly, avoids credential caching when executing multiple commands over a secure connection.

tools/python/xen/util/xmlrpclib2.py +31 -2
diff -r 4f1e39ec05d6 -r 9ad475c664bf tools/python/xen/util/xmlrpclib2.py
@@ -39,6 +39,31 @@
 # httpu:///absolute/path/to/socket.sock
 #
 # It assumes that the RPC handler is /RPC2.  This probably needs to be improved
+
+# We're forced to subclass the RequestHandler class so that we can work around
+# some bugs in Keep-Alive handling and also enabled it by default
+class XMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
+    protocol_version = "HTTP/1.1"
+
+    # this is inspired by SimpleXMLRPCRequestHandler's do_POST but differs
+    # in a few non-trivial ways
+    # 1) we never generate internal server errors.  We let the exception
+    #    propagate so that it shows up in the Xend debug logs
+    # 2) we don't bother checking for a _dispatch function since we don't
+    #    use one
+    # 3) we never shutdown the connection.  This appears to be a bug in
+    #    SimpleXMLRPCServer.py as it breaks HTTP Keep-Alive
+    def do_POST(self):
+        data = self.rfile.read(int(self.headers["content-length"]))
+        rsp = self.server._marshaled_dispatch(data)
+
+        self.send_response(200)
+        self.send_header("Content-Type", "text/xml")
+        self.send_header("Content-Length", str(len(rsp)))
+        self.end_headers()
+
+        self.wfile.write(rsp)
+        self.wfile.flush()
 
 class HTTPUnixConnection(HTTPConnection):
     def connect(self):
@@ -94,6 +119,10 @@
 class TCPXMLRPCServer(SocketServer.ThreadingMixIn, SimpleXMLRPCServer):
     allow_reuse_address = True
 
+    def __init__(self, addr, requestHandler=XMLRPCRequestHandler,
+                 logRequests=1):
+        SimpleXMLRPCServer.__init__(self, addr, requestHandler, logRequests)
+
     def _marshaled_dispatch(self, data, dispatch_method = None):
         params, method = xmlrpclib.loads(data)
         try:
@@ -131,10 +160,10 @@
 # It implements proper support for allow_reuse_address by
 # unlink()'ing an existing socket.
 
-class UnixXMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
+class UnixXMLRPCRequestHandler(XMLRPCRequestHandler):
     def address_string(self):
         try:
-            return SimpleXMLRPCRequestHandler.address_string(self)
+            return XMLRPCRequestHandler.address_string(self)
         except ValueError, e:
             return self.client_address[:2]
 

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#2 lore raw

PATCH 2/4 Add support for an xm serve command

In reply to Anthony Liguori

Repost of earlier patch.

Regards,

Anthony Liguori

# HG changeset patch
# User [email protected]
# Node ID cf8e253723daf8b95b54b24f6988a8e74c6cc0aa # Parent a0212dab2954807a979058d91b246564a7bf2cc2 Add an xm serve command.

This command proxies the Xend XML-RPC over stdio. This is similiar to mercurial's hg serve --stdio command.

The most obvious use of this command is remote XML-RPC invocation over SSH.

tools/python/xen/xm/main.py +31 -1
diff -r a0212dab2954 -r cf8e253723da tools/python/xen/xm/main.py
@@ -41,6 +41,7 @@
 import xen.xend.XendClient
 from xen.xend.XendClient import server
 from xen.util import security
+from select import select
 
 # getopt.gnu_getopt is better, but only exists in Python 2.3+.  Use
 # getopt.getopt if gnu_getopt is not available.  This will mean that options
@@ -124,6 +125,7 @@
 loadpolicy_help = "loadpolicy <policy>              Load binary policy into hypervisor"
 makepolicy_help = "makepolicy <policy>              Build policy and create .bin/.map files"
 labels_help     = "labels [policy] [type=DOM|..]    List <type> labels for (active) policy."
+serve_help      = "serve                            Proxy Xend XML-RPC over stdio"
 
 short_command_list = [
     "console",
@@ -171,7 +173,8 @@
 host_commands = [
     "dmesg",
     "info",
-    "log"
+    "log",
+    "serve",
     ]
 
 scheduler_commands = [
@@ -833,6 +836,32 @@
     arg_check(args, "log", 0)
     
     print server.xend.node.log()
+
+def xm_serve(args):
+    arg_check(args, "serve", 0)
+
+    from fcntl import fcntl, F_SETFL
+    
+    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+    s.connect(xen.xend.XendClient.XML_RPC_SOCKET)
+    fcntl(sys.stdin, F_SETFL, os.O_NONBLOCK)
+
+    while True:
+        iwtd, owtd, ewtd = select([sys.stdin, s], [], [])
+        if s in iwtd:
+            data = s.recv(4096)
+            if len(data) > 0:
+                sys.stdout.write(data)
+                sys.stdout.flush()
+            else:
+                break
+        if sys.stdin in iwtd:
+            data = sys.stdin.read(4096)
+            if len(data) > 0:
+                s.sendall(data)
+            else:
+                break
+    s.close()
 
 def parse_dev_info(info):
     def get_info(n, t, d):
@@ -1072,6 +1101,7 @@
     "dmesg": xm_dmesg,
     "info": xm_info,
     "log": xm_log,
+    "serve": xm_serve,
     # scheduler
     "sched-bvt": xm_sched_bvt,
     "sched-bvt-ctxallow": xm_sched_bvt_ctxallow,

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#3 lore raw

PATCH 2/4 Client support for XML-RPC over SSH

In reply to Anthony Liguori

This incorporates Ewan's error reporting suggestion and uses lazy evaluation to initialize the ssh connection.

Regards,

Anthony Liguori

# HG changeset patch
# User [email protected]
# Node ID ee7eb0713498591c50f43cabfcf48608b0b7aa32
# Parent 84596b81600e09c83e85eb8c7517747d57d1da98
This patch adds client side support for XML-RPC over ssh. It differs from the previous submission in that it now uses lazy evaluation to connect to the remote server and provides more user friendly error messages if ssh is not available.

Signed-off-by Anthony Liguori
tools/python/xen/util/xmlrpclib2.py +74 -4
diff -r 84596b81600e -r ee7eb0713498 tools/python/xen/util/xmlrpclib2.py
@@ -24,13 +24,67 @@
 import types
 
 from httplib import HTTPConnection, HTTP
-from xmlrpclib import Transport
+from xmlrpclib import Transport, getparser, Fault
 from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
-import xmlrpclib, socket, os, stat
+from subprocess import Popen, PIPE
+from getpass import getuser
+from fcntl import ioctl
+import xmlrpclib, socket, os, stat, termios, errno
 import SocketServer
 
-import xen.xend.XendClient
 from xen.xend.XendLogging import log
+
+class SSHTransport(object):
+    def __init__(self, host, user, askpass=None):
+        self.host = host
+        self.user = user
+        self.askpass = askpass
+        self.ssh = None
+
+    def getssh(self):
+        if self.ssh == None:
+            if self.askpass:
+                f = open('/dev/tty', 'w')
+                try:
+                    os.environ['SSH_ASKPASS'] = self.askpass
+                    ioctl(f.fileno(), termios.TIOCNOTTY)
+                finally:
+                    f.close()
+
+            cmd = ['ssh', '%s@%s' % (self.user, self.host), 'xm serve']
+            try:
+                self.ssh = Popen(cmd, bufsize=0, stdin=PIPE, stdout=PIPE)
+            except OSError, (err, msg):
+                if err == errno.ENOENT:
+                    raise Fault(0, "ssh executable not found!")
+                raise
+        return self.ssh
+
+    def request(self, host, handler, request_body, verbose=0):
+        p, u = getparser()
+        ssh = self.getssh()
+        ssh.stdin.write("""POST /%s HTTP/1.1
+User-Agent: Xen
+Host: %s
+Content-Type: text/xml
+Content-Length: %d
+
+%s""" % (handler, host, len(request_body), request_body))
+        ssh.stdin.flush()
+
+        content_length = 0
+        line = ssh.stdout.readline()
+        if line.split()[1] != '200':
+            raise Fault(0, 'Server returned %s' % (' '.join(line[1:])))
+        
+        while line not in ['', '\r\n', '\n']:
+            if line.lower().startswith('content-length:'):
+                content_length = int(line[15:].strip())
+            line = ssh.stdout.readline()
+        content = ssh.stdout.read(content_length)
+        p.feed(content)
+        p.close()
+        return u.close()
 
 
 # A new ServerProxy that also supports httpu urls.  An http URL comes in the
@@ -100,9 +154,24 @@
             if protocol == 'httpu':
                 uri = 'http:' + rest
                 transport = UnixTransport()
+            elif protocol == 'ssh':
+                if not rest.startswith('//'):
+                    raise ValueError("Invalid ssh URL '%s'" % uri)
+                rest = rest[2:]
+                user = getuser()
+                path = 'RPC2'
+                if rest.find('@') != -1:
+                    (user, rest) = rest.split('@', 1)
+                if rest.find('/') != -1:
+                    (host, rest) = rest.split('/', 1)
+                    if len(rest) > 0:
+                        path = rest
+                else:
+                    host = rest
+                transport = SSHTransport(host, user)
+                uri = 'http://%s/%s' % (host, path)
         xmlrpclib.ServerProxy.__init__(self, uri, transport, encoding,
                                        verbose, allow_none)
-
 
     def __request(self, methodname, params):
         response = xmlrpclib.ServerProxy.__request(self, methodname, params)
@@ -150,6 +219,7 @@
         except xmlrpclib.Fault, fault:
             response = xmlrpclib.dumps(fault)
         except Exception, exn:
+            import xen.xend.XendClient
             log.exception(exn)
             response = xmlrpclib.dumps(
                 xmlrpclib.Fault(xen.xend.XendClient.ERROR_INTERNAL, str(exn)))

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#4 lore raw

PATCH 4/4 Add support for XM_SERVER variable to choose XML-RPC URI

In reply to Anthony Liguori

Repost of earlier patch.

Regards,

Anthony Liguori

# HG changeset patch
# User [email protected]
# Node ID f8fcaee58fc1f7e13ae8443336a80de4c4e50538
# Parent 03bbc156d3e8fb01eee2af990176bb64e012c4e1
Use an environmental variable (XM_SERVER) to determine XML-RPC URI.

This allows users to invoke xm against a remote server. This is much more useful though for exercising XML-RPC transports with xm-test though since xm-test will continue to (mostly) function if the URI is localhost.

tools/python/xen/xend/XendClient.py +6 -1
diff -r 03bbc156d3e8 -r f8fcaee58fc1 tools/python/xen/xend/XendClient.py
@@ -18,6 +18,7 @@
 #============================================================================
 
 from xen.util.xmlrpclib2 import ServerProxy
+import os
 
 XML_RPC_SOCKET = "/var/run/xend/xmlrpc.sock"
 
@@ -25,4 +26,8 @@
 ERROR_GENERIC = 2
 ERROR_INVALID_DOMAIN = 3
 
-server = ServerProxy('httpu:///var/run/xend/xmlrpc.sock')
+uri = 'httpu:///var/run/xend/xmlrpc.sock'
+if os.environ.has_key('XM_SERVER'):
+    uri = os.environ['XM_SERVER']
+
+server = ServerProxy(uri)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Applied
#5 lore raw
In reply to Anthony Liguori
Quoted 8 lines · Anthony Liguori

This series is a repost of the previous series taking into account Ewan's feedback wrt error reporting. Additionally, to make things a bit saner, I changed the ssh invocation from being always done in the ctor to being done on demand. This means that xm commands like help won't actually require a password if the ssh transport is being used.

I've also included the HTTP/1.1 Keep-Alive patch as it's required for this patch set.

I've applied this whole set. Thanks Anthony. Could we have some documentation patches too?

Cheers,

Ewan.

#6 lore raw
Quoted 8 lines · Ewan Mellor
Quoted 6 lines · Anthony Liguori

This series is a repost of the previous series taking into account Ewan's feedback wrt error reporting. Additionally, to make things a bit saner, I changed the ssh invocation from being always done in the ctor to being done on demand. This means that xm commands like help won't actually require a password if the ssh transport is being used.

Quoted 5 lines
Quoted 2 lines

I've also included the HTTP/1.1 Keep-Alive patch as it's required for this patch set.

I've applied this whole set. Thanks Anthony. Could we have some documentation patches too?

I've just noticed that these patches only work on Python 2.4, because they use subprocess. I've split SSHTransport out to its own file (which is reasonable in it's own right anyway), and if the import fails, it now disables the XML-RPC over SSH altogether.

If there's anyone out there who wants XML-RPC over SSH on Python 2.3, then a patch that used popen2.popen2 or xen.util.xpopen.xPopen3 instead of subprocess.Popen would be welcome.

Ewan.

#7 lore raw
Quoted 10 lines · Ewan Mellor
Quoted 7 lines · Ewan Mellor
Quoted 4 lines · Anthony Liguori

This series is a repost of the previous series taking into account Ewan's feedback wrt error reporting. Additionally, to make things a bit saner, I changed the ssh invocation from being always done in the ctor to being done on demand. This means that xm commands like help won't actually require a password if the ssh transport is being used.

I've also included the HTTP/1.1 Keep-Alive patch as it's required for this patch set.

Quoted 9 lines
Quoted 3 lines

I've applied this whole set. Thanks Anthony. Could we have some documentation patches too?

I've just noticed that these patches only work on Python 2.4, because they use subprocess. I've split SSHTransport out to its own file (which is reasonable in it's own right anyway), and if the import fails, it now disables the XML-RPC over SSH altogether.

Ah, sorry, didn't realize that.

Quoted 4 lines

If there's anyone out there who wants XML-RPC over SSH on Python 2.3, then a patch that used popen2.popen2 or xen.util.xpopen.xPopen3 instead of subprocess.Popen would be welcome.

I'll put this on my TODO. For future reference, what are the official versions of Python that we support and do you happen to know which distros have the older versions?

Regards,

Anthony Liguori

Quoted 2 lines

Ewan.

#8 lore raw
Quoted 3 lines · Anthony Liguori

For future reference, what are the official
versions of Python that we support and do you happen to know which distros have the older versions?

2.3 is a must -- that's Debian stable, SLES 9.2, RHEL 4, and things of that age. There's still support in the codebase for Python 2.2, and as far as I know we still support that. That's RHEL 3 and SLES 8.

Ewan.