nvmetcli: remote configuation
Subsystems (2)
cover nvmetcli: remote configuation
Hi all,
I've discovered that SPDK has an JSON RPC interface which allows for remote configuration. And it turns out that the SPDK configuration itself is pretty similar to the in-kernel NVMe-oF target (surprise, surprise).
And to not let SPDK have an upper edge here's a simple JSON-RPC proxy for nvmet, together with a JSON-RPC client 'nvmetadm'. This allows a full remote configuration of an NVMe-oF target:
# nvmetadm nvmf_create_subsystem --nqn=nqn.nvmf-test1 # nvmetadm nvmf_subsystem_add_port --nqn=nqn.nvmf-test1 \ --traddr=<target-ip> --trtrype=tcp --adrfam=ipv6 # nvmetadm bdev_create_file --file-name=Testfile1 --size=1G # nvmetadm nvmf_subsystem_add_ns --nqn=nqn.nvmf-test1 \ --bdev=Testfile1 # nvmetadm nvmf_subsystem_add_host --nqn=nqn.nvmf-test1 \ --host=<hostnqn> # nvme discover --transport=tcp --traddr=<target-ip> \ --trsvcid=4420
With this it should be possible to convert blktests nvme suite to use this interface, which will allow blktest to run across arbitrary transports instead of just nvme-loop.
As usual, comments and reviews are welcome.
Hannes Reinecke (3):
nvmetcli: add 'merge' parameter to set_config()
nvmetproxy: add a JSON-RPC proxy daemon
nvmetadm: add JSON-RPC client for remote configuration
Documentation/Makefile | 24 +- Documentation/nvmetproxy.txt | 111 ++++++++ nvmet/__init__.py | 4 +- nvmet/bdev.py | 72 +++++ nvmet/nvme.py | 8 +- nvmet/rpc.py | 495 +++++++++++++++++++++++++++++++++++ nvmetadm | 259 ++++++++++++++++++ nvmetcli | 4 +- nvmetproxy | 197 ++++++++++++++ setup.py | 2 +- 10 files changed, 1157 insertions(+), 19 deletions(-) create mode 100644 Documentation/nvmetproxy.txt create mode 100644 nvmet/bdev.py create mode 100644 nvmet/rpc.py create mode 100755 nvmetadm create mode 100755 nvmetproxy
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 1/3 nvmetcli: add 'merge' parameter to set_config()
Add a parameter 'merge' to set_config() to allow for merge a given configuration with the existing one.
--- nvmet/nvme.py | 8 +++++--- nvmetcli | 4 ++-- 2 files changed, 7 insertions(+), 5 deletions(-)
nvmet/nvme.py +5 -3
@@ -331,17 +331,19 @@ class Root(CFSNode): for h in self.hosts: h.delete() - def restore(self, config, clear_existing=False, abort_on_error=False): + def restore(self, config, clear_existing=False, abort_on_error=False, merge=False): ''' Takes a dict generated by dump() and reconfigures the target to match. Returns list of non-fatal errors that were encountered. Will refuse to restore over an existing configuration unless - clear_existing is True. + clear_existing or merge is True. If clear_existing is True the + existing configuration will be cleared. If merge is True the + given configuration will be attempted to be merged. ''' if clear_existing: self.clear_existing() else: - if any(self.subsystems): + if any(self.subsystems) and not merge: raise CFSError("subsystems present, not restoring") errors = []
nvmetcli +2 -2
@@ -104,11 +104,11 @@ class UIRootNode(UINode): UIPortsNode(self) UIHostsNode(self) - def ui_command_restoreconfig(self, savefile=None, clear_existing=False): + def ui_command_restoreconfig(self, savefile=None, clear_existing=False, merge=False): ''' Restores configuration from a file. ''' - errors = self.cfnode.restore_from_file(savefile, clear_existing) + errors = self.cfnode.restore_from_file(savefile, clear_existing, merge) self.refresh() if errors:
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/3 nvmetproxy: add a JSON-RPC proxy daemon
In reply to Hannes ReineckeAdd a JSON-RPC proxy daemon to allow for remote configuration of the NVMe-over-Fabrics target.
--- Documentation/Makefile | 24 +- Documentation/nvmetproxy.txt | 111 ++++++++ nvmet/__init__.py | 4 +- nvmet/bdev.py | 72 +++++ nvmet/rpc.py | 495 +++++++++++++++++++++++++++++++++++ nvmetproxy | 197 ++++++++++++++ setup.py | 2 +- 7 files changed, 891 insertions(+), 14 deletions(-) create mode 100644 Documentation/nvmetproxy.txt create mode 100644 nvmet/bdev.py create mode 100644 nvmet/rpc.py create mode 100755 nvmetproxy
Documentation/Makefile +12 -12
@@ -1,31 +1,31 @@ PKGNAME = nvmetcli -MANPAGE = ${PKGNAME}.8 -HTMLFILE = ${PKGNAME}.html -XMLFILE = ${PKGNAME}.xml INSTALL ?= install PREFIX := /usr ASCIIDOC = asciidoc XMLTO = xmlto --skip-validation -DOCDATA = ${XMLFILE} ${HTMLFILE} - -${MANPAGE}: ${DOCDATA} +DOCFILES = ${PKGNAME}.txt nvmetproxy.txt +MANPAGES = ${DOCFILES:.txt=.8} +HTMLFILES = ${DOCFILES:.txt=.html} +XMLFILES = ${DOCFILES:.txt=.xml} + +%.8: %.xml ${XMLTO} man $< - + %.xml: %.txt ${ASCIIDOC} -b docbook -d manpage -o $@ $< - + %.html: %.txt ${ASCIIDOC} -a toc -o $@ $< - + installdoc: man8 -man8: - ${INSTALL} -m 644 ${MANPAGE} ${PREFIX}/share/man/man8 +man8: ${MANPAGES} + ${INSTALL} -m 644 $< ${PREFIX}/share/man/man8 uninstalldoc: -rm -f ${PREFIX}/share/man/man8/${MANPAGE} clean: - -rm -f ${MANPAGE} ${HTMLFILE} ${XMLFILE} + -rm -f ${MANPAGES} ${HTMLFILES} ${XMLFILES}
Documentation/nvmetproxy.txt new file +111 -0
111 lines not on this page · raw message
nvmet/__init__.py +3 -1
@@ -1,2 +1,4 @@ -from .nvme import Root, Subsystem, Namespace, Port, Host, Referral, ANAGroup,\ +from .bdev import BackingFile +from .rpc import JsonRPC +from .nvme import CFSError, Root, Subsystem, Namespace, Port, Host, Referral, ANAGroup,\ DEFAULT_SAVE_FILE
nvmet/bdev.py new file +72 -0
72 lines not on this page · raw message
nvmet/rpc.py new file +495 -0
495 lines not on this page · raw message
nvmetproxy new file +197 -0
197 lines not on this page · raw message
setup.py +1 -1
@@ -27,5 +27,5 @@ setup( maintainer_email = '[email protected]', test_suite='nose2.collector.collector', packages = ['nvmet'], - scripts=['nvmetcli'] + scripts=['nvmetcli', 'nvmetproxy'] )
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 3/3 nvmetadm: add JSON-RPC client for remote configuration
In reply to Hannes ReineckeAdd a JSON-RPC client for remote configuration of an NVMe-over-Fabrics target.
--- nvmetadm | 259 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ setup.py | 2 +- 2 files changed, 260 insertions(+), 1 deletion(-) create mode 100755 nvmetadm
nvmetadm new file +259 -0
@@ -0,0 +1,259 @@ +#!/usr/bin/python3 + +import sys +import argparse +import requests +import json + +class rpc_client(): + def bdev_file_create(self, args): + self.parser.add_argument('--filename', required=True) + self.parser.add_argument('--size', required=True) + self.parser.add_argument('--pool') + a = self.parser.parse_args(args) + s = a.size.lower().rfind('g') + if s != -1: + size = int(a.size.lower()[:-s]) * 1024 * 1024 * 1024 + else: + s = a.size.lower().rfind('m') + if s != -1: + size = int(a.size.lower()[:-s]) * 1024 * 1024 + else: + s = a.size.lower().rfind('k') + if s != -1: + size = int(a.size.lower()[:-s]) * 1024 + else: + size = int(a.size) + params = {'file_name': a.filename, 'size': size} + if a.pool: + params['pool'] = a.pool + return params + + def bdev_file_delete(self, args): + self.parser.add_argument('--filename', required=True) + self.parser.add_argument('--pool') + a = self.parser.parse_args(args) + params = {'filename': a.filename } + if a.pool: + params['pool'] = a.pool + return params + + def nvmf_set_config(self, args): + self.parser.add_argument('--cfg-file', required=True) + a = self.parser.parse_args(args) + try: + cfg = open(a.cfg_file, "r") + except OSError: + sys.exit(1) + params = json.load(cfg) + cfg.close() + return params + + def nvmf_create_transport(self, args): + self.parser.add_argument('--trtype', required=True) + a = self.parser.parse_args(args) + if not a.trtype: + return None + return {"trtype": a.trtype} + + def nvmf_create_subsystem(self, args): + self.parser.add_argument('--nqn') + a = self.parser.parse_args(args) + params = None + if not a.nqn: + return None + return {"nqn": a.nqn} + + def nvmf_delete_subsystem(self, args): + self.parser.add_argument('--nqn', required=True) + a = self.parser.parse_args(args) + return {"nqn": a.nqn} + + def nvmf_subsystem_add_ns(self, args): + self.parser.add_argument('--nqn', required=True) + self.parser.add_argument('--bdev-name', required=True) + self.parser.add_argument('--nsid') + self.parser.add_argument('--nguid') + self.parser.add_argument('--eui64') + self.parser.add_argument('--uuid') + a = self.parser.parse_args(args) + ns = { 'bdev_name': a.bdev_name } + if a.nsid: + ns['nsid'] = a.nsid + if a.nguid: + ns['nguid'] = a.nguid + if a.eui64: + ns['eui64'] = a.eui64 + if a.uuid: + ns['uuid'] = a.uuid + return {"nqn": a.nqn, 'namespace': ns} + + def nvmf_subsystem_remove_ns(self, args): + self.parser.add_argument('--nqn', required=True) + self.parser.add_argument('--nsid', required=True, type=int) + a = self.parser.parse_args(args) + return {'nqn': a.nqn, 'nsid': a.nsid } + + def nvmf_subsystem_port(self, args): + self.parser.add_argument('--nqn', required=True) + self.parser.add_argument('--portid', type=int) + self.parser.add_argument('--trtype', required=True) + self.parser.add_argument('--traddr', required=True) + self.parser.add_argument('--adrfam') + self.parser.add_argument('--trscvid') + self.parser.add_argument('--host-traddr') + a = self.parser.parse_args(args) + port = { 'trtype': a.trtype, 'traddr': a.traddr } + if a.portid: + port['portid'] = a.portid + if a.adrfam: + port['adrfam'] = a.adrfam + if a.trscvid: + port['trsvcid'] = a.trscvid + if a.host_traddr: + port['host_traddr'] = a.host_traddr + return {"nqn": a.nqn, 'port': port} + + def nvmf_subsystem_host(self, args): + self.parser.add_argument('--nqn', required=True) + self.parser.add_argument('--host', required=True) + a = self.parser.parse_args(args) + return {'nqn': a.nqn, 'host': a.host } + + def nvmf_port_ana(self, args): + self.parser.add_argument('--portid', required=True) + self.parser.add_argument('--grpid') + self.parser.add_argument('--ana_state') + a = self.parser.parse_args(args) + params = { 'portid': a.portid } + if a.grpid: + params['grpid'] = a.grpid + if a.ana_state: + params['ana_state'] = a.ana_state + return params + + def method_no_param(self, args = None): + a = self.parser.parse_args(args) + return + + _rpc_methods = dict(bdev_file_list_pools=method_no_param, + bdev_file_create=bdev_file_create, + bdev_file_delete=bdev_file_delete, + nvmf_get_config=method_no_param, + nvmf_set_config=nvmf_set_config, + nvmf_get_interfaces=method_no_param, + nvmf_create_transport=nvmf_create_transport, + nvmf_get_transports=method_no_param, + nvmf_create_subsystem=nvmf_create_subsystem, + nvmf_delete_subsystem=nvmf_delete_subsystem, + nvmf_subsystem_add_ns= nvmf_subsystem_add_ns, + nvmf_subsystem_remove_ns=nvmf_subsystem_remove_ns, + nvmf_subsystem_add_port=nvmf_subsystem_port, + nvmf_subsystem_remove_port=nvmf_subsystem_port, + nvmf_subsystem_add_host=nvmf_subsystem_host, + nvmf_subsystem_remove_host=nvmf_subsystem_host, + nvmf_port_add_ana=nvmf_port_ana, + nvmf_port_set_ana=nvmf_port_ana, + nvmf_port_remove_ana=nvmf_port_ana, + nvmf_get_subsystems=method_no_param) + + def __init__(self, method): + if method not in self._rpc_methods: + print("Invalid rpc method '%s'; must be one of" % method) + print(", ".join([x for x in list(self._rpc_methods)])) + sys.exit(1) + + self.parser = argparse.ArgumentParser(prog="nvmetadm %s" % method) + self._method = self._rpc_methods[method] + + def _error(self, err): + data = None + if 'code' not in err: + code = -32000 + message = "Server error" + data = [] + data.append("Invalid JSON RPC response") + else: + code = err['code'] + if 'message' in err: + message = err['message'] + else: + if code == -32700: + message = "Parse error" + elif code == -32600: + message = "Invalid Request" + elif code == -32601: + message = "Method not found" + elif code == -32602: + message = "Invalid params" + elif code == -32603: + message = "Internal error" + elif code <= -32000 and code > -32100: + message = "Server error" + else: + data = [] + data.append("Invalid JSON RPC error code %d" & code) + code = -32001 + message = "Server error" + if 'data' in err: + data = err['data'] + return (code, message, data) + + def call(self, args): + return self._method(self, args) + + def response(self, response): + if 'error' in response: + (code, message, data) = self._error(response['error']) + if data: + sys.exit("JSON RPC error %d: %s\n%s" % (code, message, json.dumps(data))) + else: + sys.exit("JSON RPC error %d: %s" % (code, message)) + elif 'result' not in response: + sys.exit("JSON RPC error: Not a valid response\n%s", response) + elif response['result']: + return response['result'] + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('-H', '--host', help='Hostname to connect to', + required=True) + parser.add_argument('-p', '--port', type=int, default=4260, + help='Port number for the connection') + parser.add_argument('-U', '--username') + parser.add_argument('-P', '--password') + parser.add_argument('-u', '--uri', default='nvmet', + help='URI of the JSON-RPC namespace') + (a, args) = parser.parse_known_args() + if a.username: + if not a.password: + sys.exit("Need to specify both username and password") + auth = (a.username, a.password) + elif not a.password: + auth = None + else: + sys.exit("Need to specify both username and password") + url = "http://%s:%d/%s" % (a.host, a.port, a.uri) + + if not args or not len(args): + sys.exit("No method given") + + method = args.pop(0) + payload = { + "method": method, + "jsonrpc": "2.0", + "id": 0, + } + rpc = rpc_client(method) + params = rpc.call(args) + if params: + payload['params'] = params + response = requests.post(url, auth=auth,json=payload) + response.raise_for_status() + msg = rpc.response(response.json()) + if msg: + print("%s" % json.dumps(msg, indent=2)) + + +if __name__ == "__main__": + main()
setup.py +1 -1
@@ -27,5 +27,5 @@ setup( maintainer_email = '[email protected]', test_suite='nose2.collector.collector', packages = ['nvmet'], - scripts=['nvmetcli', 'nvmetproxy'] + scripts=['nvmetcli', 'nvmetproxy', 'nvmetadm'] )
Signature · 7 lines
2.29.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerSo this doesn't look horrible to me, but do you plan to actually have users for this? Test cases would also be really helpful.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 3 lines · Christoph Hellwig
So this doesn't look horrible to me, but do you plan to actually have users for this? Test cases would also be really helpful.
Well, idea was to enable things like blktest to configure remote systems such that we can expand the test matrix beyond nvme-loop to 'real' hardware like nvme-fc or nvme-rdma.
But then I found that jsonrpc is actually defunct as google switched to gRPC (and go in general), making the use of jsonrpc ever so slightly awkward.
So not sure; I'd be happy to leave it for now, and I'll see to tie in blktests with this.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme