Skip to contentSkip to conversation

nvmetcli: remote configuation

RFC PATCH In review Hannes Reinecke 2 participants 6 messages Feb 12, 2021 – Jul 13, 2021

Subsystems (2)
lore
Share & tools
lore
#1 lore raw

PATCH 1/3 nvmetcli: add 'merge' parameter to set_config()

In reply to Hannes Reinecke

Add a parameter 'merge' to set_config() to allow for merge a given configuration with the existing one.

---
 nvmet/nvme.py | 8 +++++---
 nvmetcli      | 4 ++--
 2 files changed, 7 insertions(+), 5 deletions(-)
nvmet/nvme.py +5 -3
index 83fd75b..22aef5a 100644
@@ -331,17 +331,19 @@ class Root(CFSNode):
         for h in self.hosts:
             h.delete()
 
-    def restore(self, config, clear_existing=False, abort_on_error=False):
+    def restore(self, config, clear_existing=False, abort_on_error=False, merge=False):
         '''
         Takes a dict generated by dump() and reconfigures the target to match.
         Returns list of non-fatal errors that were encountered.
         Will refuse to restore over an existing configuration unless
-        clear_existing is True.
+        clear_existing or merge is True. If clear_existing is True the
+        existing configuration will be cleared. If merge is True the
+        given configuration will be attempted to be merged.
         '''
         if clear_existing:
             self.clear_existing()
         else:
-            if any(self.subsystems):
+            if any(self.subsystems) and not merge:
                 raise CFSError("subsystems present, not restoring")
 
         errors = []
nvmetcli +2 -2
index 8ee8590..f2b59bb 100755
@@ -104,11 +104,11 @@ class UIRootNode(UINode):
         UIPortsNode(self)
         UIHostsNode(self)
 
-    def ui_command_restoreconfig(self, savefile=None, clear_existing=False):
+    def ui_command_restoreconfig(self, savefile=None, clear_existing=False, merge=False):
         '''
         Restores configuration from a file.
         '''
-        errors = self.cfnode.restore_from_file(savefile, clear_existing)
+        errors = self.cfnode.restore_from_file(savefile, clear_existing, merge)
         self.refresh()
 
         if errors:
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#0 lore raw

cover nvmetcli: remote configuation

Hi all,

I've discovered that SPDK has an JSON RPC interface which allows for remote configuration. And it turns out that the SPDK configuration itself is pretty similar to the in-kernel NVMe-oF target (surprise, surprise).
And to not let SPDK have an upper edge here's a simple JSON-RPC proxy for nvmet, together with a JSON-RPC client 'nvmetadm'. This allows a full remote configuration of an NVMe-oF target:

# nvmetadm nvmf_create_subsystem --nqn=nqn.nvmf-test1
# nvmetadm nvmf_subsystem_add_port --nqn=nqn.nvmf-test1 \
  --traddr=<target-ip> --trtrype=tcp --adrfam=ipv6
# nvmetadm bdev_create_file --file-name=Testfile1 --size=1G
# nvmetadm nvmf_subsystem_add_ns --nqn=nqn.nvmf-test1 \
  --bdev=Testfile1
# nvmetadm nvmf_subsystem_add_host --nqn=nqn.nvmf-test1 \
  --host=<hostnqn>
# nvme discover --transport=tcp --traddr=<target-ip> \
  --trsvcid=4420

With this it should be possible to convert blktests nvme suite to use this interface, which will allow blktest to run across arbitrary transports instead of just nvme-loop.

As usual, comments and reviews are welcome.

Hannes Reinecke (3):
nvmetcli: add 'merge' parameter to set_config()
nvmetproxy: add a JSON-RPC proxy daemon
nvmetadm: add JSON-RPC client for remote configuration

 Documentation/Makefile       |  24 +-
 Documentation/nvmetproxy.txt | 111 ++++++++
 nvmet/__init__.py            |   4 +-
 nvmet/bdev.py                |  72 +++++
 nvmet/nvme.py                |   8 +-
 nvmet/rpc.py                 | 495 +++++++++++++++++++++++++++++++++++
 nvmetadm                     | 259 ++++++++++++++++++
 nvmetcli                     |   4 +-
 nvmetproxy                   | 197 ++++++++++++++
 setup.py                     |   2 +-
 10 files changed, 1157 insertions(+), 19 deletions(-)
 create mode 100644 Documentation/nvmetproxy.txt
 create mode 100644 nvmet/bdev.py
 create mode 100644 nvmet/rpc.py
 create mode 100755 nvmetadm
 create mode 100755 nvmetproxy
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#2 lore raw

PATCH 2/3 nvmetproxy: add a JSON-RPC proxy daemon

Add a JSON-RPC proxy daemon to allow for remote configuration of the NVMe-over-Fabrics target.

---
 Documentation/Makefile       |  24 +-
 Documentation/nvmetproxy.txt | 111 ++++++++
 nvmet/__init__.py            |   4 +-
 nvmet/bdev.py                |  72 +++++
 nvmet/rpc.py                 | 495 +++++++++++++++++++++++++++++++++++
 nvmetproxy                   | 197 ++++++++++++++
 setup.py                     |   2 +-
 7 files changed, 891 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/nvmetproxy.txt
 create mode 100644 nvmet/bdev.py
 create mode 100644 nvmet/rpc.py
 create mode 100755 nvmetproxy
Documentation/Makefile +12 -12
index 8e0281c..2778429 100644
@@ -1,31 +1,31 @@
 PKGNAME = nvmetcli
-MANPAGE = ${PKGNAME}.8
-HTMLFILE = ${PKGNAME}.html
-XMLFILE = ${PKGNAME}.xml
 INSTALL ?= install
 PREFIX := /usr
 
 ASCIIDOC = asciidoc
 XMLTO = xmlto --skip-validation
 
-DOCDATA = ${XMLFILE} ${HTMLFILE}
- 
-${MANPAGE}: ${DOCDATA}
+DOCFILES = ${PKGNAME}.txt nvmetproxy.txt
+MANPAGES = ${DOCFILES:.txt=.8}
+HTMLFILES = ${DOCFILES:.txt=.html}
+XMLFILES = ${DOCFILES:.txt=.xml}
+
+%.8: %.xml
 	${XMLTO} man $<
- 
+
 %.xml: %.txt
 	${ASCIIDOC} -b docbook -d manpage -o $@ $<
- 
+
 %.html: %.txt
 	${ASCIIDOC} -a toc -o $@ $<
- 
+
 installdoc: man8
 
-man8:
-	${INSTALL} -m 644 ${MANPAGE} ${PREFIX}/share/man/man8
+man8: ${MANPAGES}
+	${INSTALL} -m 644 $< ${PREFIX}/share/man/man8
 
 uninstalldoc:
 	-rm -f ${PREFIX}/share/man/man8/${MANPAGE}
 
 clean:
-	-rm -f ${MANPAGE} ${HTMLFILE} ${XMLFILE}
+	-rm -f ${MANPAGES} ${HTMLFILES} ${XMLFILES}
Documentation/nvmetproxy.txt new file +111 -0

111 lines not on this page · raw message

nvmet/__init__.py +3 -1
index cf172bd..730393b 100644
@@ -1,2 +1,4 @@
-from .nvme import Root, Subsystem, Namespace, Port, Host, Referral, ANAGroup,\
+from .bdev import BackingFile
+from .rpc import JsonRPC
+from .nvme import CFSError, Root, Subsystem, Namespace, Port, Host, Referral, ANAGroup,\
     DEFAULT_SAVE_FILE
nvmet/bdev.py new file +72 -0

72 lines not on this page · raw message

nvmet/rpc.py new file +495 -0

495 lines not on this page · raw message

nvmetproxy new file +197 -0

197 lines not on this page · raw message

setup.py +1 -1
index 1956d95..f15e5b0 100755
@@ -27,5 +27,5 @@ setup(
     maintainer_email = '[email protected]',
     test_suite='nose2.collector.collector',
     packages = ['nvmet'],
-    scripts=['nvmetcli']
+    scripts=['nvmetcli', 'nvmetproxy']
     )
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#3 lore raw

PATCH 3/3 nvmetadm: add JSON-RPC client for remote configuration

In reply to Hannes Reinecke

Add a JSON-RPC client for remote configuration of an NVMe-over-Fabrics target.

---
 nvmetadm | 259 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 setup.py |   2 +-
 2 files changed, 260 insertions(+), 1 deletion(-)
 create mode 100755 nvmetadm
nvmetadm new file +259 -0
new file mode 100755
index 0000000..74479ff
@@ -0,0 +1,259 @@
+#!/usr/bin/python3
+
+import sys
+import argparse
+import requests
+import json
+
+class rpc_client():
+    def bdev_file_create(self, args):
+        self.parser.add_argument('--filename', required=True)
+        self.parser.add_argument('--size', required=True)
+        self.parser.add_argument('--pool')
+        a = self.parser.parse_args(args)
+        s = a.size.lower().rfind('g')
+        if s != -1:
+            size = int(a.size.lower()[:-s]) * 1024 * 1024 * 1024
+        else:
+            s = a.size.lower().rfind('m')
+            if s != -1:
+                size = int(a.size.lower()[:-s]) * 1024 * 1024
+            else:
+                s = a.size.lower().rfind('k')
+                if s != -1:
+                    size = int(a.size.lower()[:-s]) * 1024
+                else:
+                    size = int(a.size)
+        params = {'file_name': a.filename, 'size': size}
+        if a.pool:
+            params['pool'] = a.pool
+        return params
+
+    def bdev_file_delete(self, args):
+        self.parser.add_argument('--filename', required=True)
+        self.parser.add_argument('--pool')
+        a = self.parser.parse_args(args)
+        params = {'filename': a.filename }
+        if a.pool:
+            params['pool'] = a.pool
+        return params
+
+    def nvmf_set_config(self, args):
+        self.parser.add_argument('--cfg-file', required=True)
+        a = self.parser.parse_args(args)
+        try:
+            cfg = open(a.cfg_file, "r")
+        except OSError:
+            sys.exit(1)
+        params = json.load(cfg)
+        cfg.close()
+        return params
+
+    def nvmf_create_transport(self, args):
+        self.parser.add_argument('--trtype', required=True)
+        a = self.parser.parse_args(args)
+        if not a.trtype:
+            return None
+        return {"trtype": a.trtype}
+
+    def nvmf_create_subsystem(self, args):
+        self.parser.add_argument('--nqn')
+        a = self.parser.parse_args(args)
+        params = None
+        if not a.nqn:
+            return None
+        return {"nqn": a.nqn}
+
+    def nvmf_delete_subsystem(self, args):
+        self.parser.add_argument('--nqn', required=True)
+        a = self.parser.parse_args(args)
+        return {"nqn": a.nqn}
+
+    def nvmf_subsystem_add_ns(self, args):
+        self.parser.add_argument('--nqn', required=True)
+        self.parser.add_argument('--bdev-name', required=True)
+        self.parser.add_argument('--nsid')
+        self.parser.add_argument('--nguid')
+        self.parser.add_argument('--eui64')
+        self.parser.add_argument('--uuid')
+        a = self.parser.parse_args(args)
+        ns = { 'bdev_name': a.bdev_name }
+        if a.nsid:
+            ns['nsid'] = a.nsid
+        if a.nguid:
+            ns['nguid'] = a.nguid
+        if a.eui64:
+            ns['eui64'] = a.eui64
+        if a.uuid:
+            ns['uuid'] = a.uuid
+        return {"nqn": a.nqn, 'namespace': ns}
+
+    def nvmf_subsystem_remove_ns(self, args):
+        self.parser.add_argument('--nqn', required=True)
+        self.parser.add_argument('--nsid', required=True, type=int)
+        a = self.parser.parse_args(args)
+        return {'nqn': a.nqn, 'nsid': a.nsid }
+
+    def nvmf_subsystem_port(self, args):
+        self.parser.add_argument('--nqn', required=True)
+        self.parser.add_argument('--portid', type=int)
+        self.parser.add_argument('--trtype', required=True)
+        self.parser.add_argument('--traddr', required=True)
+        self.parser.add_argument('--adrfam')
+        self.parser.add_argument('--trscvid')
+        self.parser.add_argument('--host-traddr')
+        a = self.parser.parse_args(args)
+        port = { 'trtype': a.trtype, 'traddr': a.traddr }
+        if a.portid:
+            port['portid'] = a.portid
+        if a.adrfam:
+            port['adrfam'] = a.adrfam
+        if a.trscvid:
+            port['trsvcid'] = a.trscvid
+        if a.host_traddr:
+            port['host_traddr'] = a.host_traddr
+        return {"nqn": a.nqn, 'port': port}
+
+    def nvmf_subsystem_host(self, args):
+        self.parser.add_argument('--nqn', required=True)
+        self.parser.add_argument('--host', required=True)
+        a = self.parser.parse_args(args)
+        return {'nqn': a.nqn, 'host': a.host }
+
+    def nvmf_port_ana(self, args):
+        self.parser.add_argument('--portid', required=True)
+        self.parser.add_argument('--grpid')
+        self.parser.add_argument('--ana_state')
+        a = self.parser.parse_args(args)
+        params = { 'portid': a.portid }
+        if a.grpid:
+            params['grpid'] = a.grpid
+        if a.ana_state:
+            params['ana_state'] = a.ana_state
+        return params
+
+    def method_no_param(self, args = None):
+        a = self.parser.parse_args(args)
+        return
+
+    _rpc_methods = dict(bdev_file_list_pools=method_no_param,
+                        bdev_file_create=bdev_file_create,
+                        bdev_file_delete=bdev_file_delete,
+                        nvmf_get_config=method_no_param,
+                        nvmf_set_config=nvmf_set_config,
+                        nvmf_get_interfaces=method_no_param,
+                        nvmf_create_transport=nvmf_create_transport,
+                        nvmf_get_transports=method_no_param,
+                        nvmf_create_subsystem=nvmf_create_subsystem,
+                        nvmf_delete_subsystem=nvmf_delete_subsystem,
+                        nvmf_subsystem_add_ns= nvmf_subsystem_add_ns,
+                        nvmf_subsystem_remove_ns=nvmf_subsystem_remove_ns,
+                        nvmf_subsystem_add_port=nvmf_subsystem_port,
+                        nvmf_subsystem_remove_port=nvmf_subsystem_port,
+                        nvmf_subsystem_add_host=nvmf_subsystem_host,
+                        nvmf_subsystem_remove_host=nvmf_subsystem_host,
+                        nvmf_port_add_ana=nvmf_port_ana,
+                        nvmf_port_set_ana=nvmf_port_ana,
+                        nvmf_port_remove_ana=nvmf_port_ana,
+                        nvmf_get_subsystems=method_no_param)
+
+    def __init__(self, method):
+        if method not in self._rpc_methods:
+            print("Invalid rpc method '%s'; must be one of" % method)
+            print(", ".join([x for x in list(self._rpc_methods)]))
+            sys.exit(1)
+                
+        self.parser = argparse.ArgumentParser(prog="nvmetadm %s" % method)
+        self._method = self._rpc_methods[method]
+
+    def _error(self, err):
+        data = None
+        if 'code' not in err:
+            code = -32000
+            message = "Server error"
+            data = []
+            data.append("Invalid JSON RPC response")
+        else:
+            code = err['code']
+        if 'message' in err:
+            message = err['message']
+        else:
+            if code == -32700:
+                message = "Parse error"
+            elif code == -32600:
+                message = "Invalid Request"
+            elif code == -32601:
+                message = "Method not found"
+            elif code == -32602:
+                message = "Invalid params"
+            elif code == -32603:
+                message = "Internal error"
+            elif code <= -32000 and code > -32100:
+                message = "Server error"
+            else:
+                data = []
+                data.append("Invalid JSON RPC error code %d" & code)
+                code = -32001
+                message = "Server error"
+        if 'data' in err:
+            data = err['data']
+        return (code, message, data)
+
+    def call(self, args):
+        return self._method(self, args)
+
+    def response(self, response):
+        if 'error' in response:
+            (code, message, data) = self._error(response['error'])
+            if data:
+                sys.exit("JSON RPC error %d: %s\n%s" % (code, message, json.dumps(data)))
+            else:
+                sys.exit("JSON RPC error %d: %s" % (code, message))
+        elif 'result' not in response:
+            sys.exit("JSON RPC error: Not a valid response\n%s", response)
+        elif response['result']:
+            return response['result']
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-H', '--host', help='Hostname to connect to',
+                        required=True)
+    parser.add_argument('-p', '--port', type=int, default=4260,
+                        help='Port number for the connection')
+    parser.add_argument('-U', '--username')
+    parser.add_argument('-P', '--password')
+    parser.add_argument('-u', '--uri', default='nvmet',
+                        help='URI of the JSON-RPC namespace')
+    (a, args) = parser.parse_known_args()
+    if a.username:
+        if not a.password:
+            sys.exit("Need to specify both username and password")
+        auth = (a.username, a.password)
+    elif not a.password:
+        auth = None
+    else:
+        sys.exit("Need to specify both username and password")
+    url = "http://%s:%d/%s" % (a.host, a.port, a.uri)
+
+    if not args or not len(args):
+        sys.exit("No method given")
+
+    method = args.pop(0)
+    payload = {
+        "method": method,
+        "jsonrpc": "2.0",
+        "id": 0,
+    }
+    rpc = rpc_client(method)
+    params = rpc.call(args)
+    if params:
+        payload['params'] = params
+    response = requests.post(url, auth=auth,json=payload)
+    response.raise_for_status()
+    msg = rpc.response(response.json())
+    if msg:
+        print("%s" % json.dumps(msg, indent=2))
+
+
+if __name__ == "__main__":
+    main()
setup.py +1 -1
index f15e5b0..998ff78 100755
@@ -27,5 +27,5 @@ setup(
     maintainer_email = '[email protected]',
     test_suite='nose2.collector.collector',
     packages = ['nvmet'],
-    scripts=['nvmetcli', 'nvmetproxy']
+    scripts=['nvmetcli', 'nvmetproxy', 'nvmetadm']
     )
Signature · 7 lines
2.29.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#5 lore raw
Quoted 3 lines · Christoph Hellwig

So this doesn't look horrible to me, but do you plan to actually have users for this? Test cases would also be really helpful.

Well, idea was to enable things like blktest to configure remote systems such that we can expand the test matrix beyond nvme-loop to 'real' hardware like nvme-fc or nvme-rdma.

But then I found that jsonrpc is actually defunct as google switched to gRPC (and go in general), making the use of jsonrpc ever so slightly awkward.

So not sure; I'd be happy to leave it for now, and I'll see to tie in blktests with this.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme