nvme-fabrics: TCP transport
Subsystems (2)
cover nvme-fabrics: TCP transport
Experimental proof of concept NVMe Fabrics transport over TCP, intended to provide good performance over any network.
Initial performance tests using both ramdisks and NVMe SSDs show latency and throughput broadly comparable to a RoCE transport. Further optimisation is needed, but we welcome input and feedback on our approach and goal of adding a TCP transport to the NVMe Fabrics specification.
Using this requires a modified version of the nvme-cli tool, available at
https://github.com/solarflarecommunications/nvme-cli/
IPv6 support is not yet implemented, but will be added using Sagi's new inet_pton_with_scope.
Bert Kenward (2):
nvme-fabrics: add target support for TCP transport
nvme-fabrics: add tcp to trtype list for configfs
Ripduman Sohan (2):
nvme-fabrics: add TCP as valid NVMe transport type
nvme-fabrics: add host support for TCP transport
drivers/nvme/host/Kconfig | 14 + drivers/nvme/host/Makefile | 3 + drivers/nvme/host/tcp.c | 1616 ++++++++++++++++++++++++++++++++++++++++ drivers/nvme/target/Kconfig | 12 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/configfs.c | 10 + drivers/nvme/target/tcp.c | 1186 +++++++++++++++++++++++++++++ include/linux/nvme.h | 1 + 8 files changed, 2844 insertions(+) create mode 100644 drivers/nvme/host/tcp.c create mode 100644 drivers/nvme/target/tcp.c
Signature · 1 line
2.7.4
RFC PATCH 1/4 nvme-fabrics: add TCP as valid NVMe transport type
From: Ripduman Sohan <[email protected]>
--- include/linux/nvme.h | 1 + 1 file changed, 1 insertion(+)
include/linux/nvme.h +1 -0
@@ -49,6 +49,7 @@ enum {enum {NVMF_TRTYPE_RDMA =1 ,/* RDMA */ NVMF_TRTYPE_FC =2 ,/* Fibre Channel */ +NVMF_TRTYPE_TCP =3 ,/* TCP */ NVMF_TRTYPE_LOOP =254 ,/* Reserved for host usage */ NVMF_TRTYPE_MAX , };
Signature · 1 line
2.7.4
RFC PATCH 2/4 nvme-fabrics: add target support for TCP transport
In reply to Bert KenwardIntroduce the target side of an experimental proof of concept NVMe Fabrics transport running directly over TCP. This is intended to provide good performance over any interface.
At present most work is deferred to workqueues for simplicity, but it may be possible to avoid this in some situations and thus reduce the latency.
Memory is currently allocated as zero-order pages. It's possible that using higher order pages may be desirable, trading potential allocation latency for less fragmented SKBs on transmit.
--- drivers/nvme/target/Kconfig | 12 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/tcp.c | 1186 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1200 insertions(+) create mode 100644 drivers/nvme/target/tcp.c
drivers/nvme/target/Kconfig +12 -0
@@ -58,3 +58,15 @@ config NVME_TARGET_FCLOOP to test NVMe-FC transport interfaces. If unsure, say N. + +config NVME_TARGET_TCP + tristate "NVMe over Fabrics TCP target support" + depends on NVME_TARGET + select NVME_CORE + select NVME_FABRICS + select SG_POOL + help + This enables the NVMe TCP target support, which allows exporting + NVMe devices over TCP. + + If unsure, say N.
drivers/nvme/target/Makefile +2 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_NVME_TARGET_LOOP) += nvme-loop.o obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o obj-$(CONFIG_NVME_TARGET_FC) += nvmet-fc.o obj-$(CONFIG_NVME_TARGET_FCLOOP) += nvme-fcloop.o +obj-$(CONFIG_NVME_TARGET_TCP) += nvmet-tcp.o nvmet-y += core.o configfs.o admin-cmd.o io-cmd.o fabrics-cmd.o \ discovery.o
@@ -11,3 +12,4 @@ nvme-loop-y += loop.o nvmet-rdma-y += rdma.o nvmet-fc-y += fc.o nvme-fcloop-y += fcloop.o +nvmet-tcp-y += tcp.o
drivers/nvme/target/tcp.c new file +1186 -0
1,186 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 2/4 nvme-fabrics: add target support for TCP transport
Hi Bert,
some initial comments inline
Quoted 8 lines · Bert Kenward
Introduce the target side of an experimental proof of concept NVMe Fabrics transport running directly over TCP. This is intended to provide good performance over any interface.
At present most work is deferred to workqueues for simplicity, but it may be possible to avoid this in some situations and thus reduce the latency.
Quoted 8 lines
Memory is currently allocated as zero-order pages. It's possible that using higher order pages may be desirable, trading potential allocation latency for less fragmented SKBs on transmit.
Cc Ripduman Sohan <rip.sohan at verrko.com>Cc Lucian Carata <lucian.carata at cl.cam.ac.uk>Signed-off-by Bert Kenward <bkenward at solarflare.com>---
[...]
Can you avoid the forward declarations?
Quoted 16 lines
drivers/nvme/target/tcp.c new file +16
-0+static int nvmet_tcp_add_port (struct nvmet_port *nvmet_port); +static void nvmet_tcp_remove_port (struct nvmet_port *nvmet_port); +static void nvmet_tcp_queue_response (struct nvmet_req *req); +static void nvmet_tcp_delete_ctrl (struct nvmet_ctrl *ctrl); + +static struct nvmet_fabrics_ops nvmet_tcp_ops = { + .owner =THIS_MODULE , + .type =NVMF_TRTYPE_TCP , + .sqe_inline_size =NVMET_TCP_MAX_REQ_DATA_SIZE , + .msdbd =1 , + .has_keyed_sgls =0 , + .add_port = nvmet_tcp_add_port, + .remove_port = nvmet_tcp_remove_port, + .queue_response = nvmet_tcp_queue_response, + .delete_ctrl = nvmet_tcp_delete_ctrl, +};
[...]
Quoted 6 lines
drivers/nvme/target/tcp.c new file +6
-0+static int request_create_sgl (struct nvmet_tcp_request *request,u32 len, +int *nents) +{ +struct scatterlist *sg; +struct page *page; +int page_count;
drivers/nvme/target/tcp.c line 316
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+int i; + + page_count =DIV_ROUND_UP (len,PAGE_SIZE ); + +if (!page_count) { + request->scatterlist =NULL ; +return 0 ; + } + + sg =kmalloc_array (page_count,sizeof (struct scatterlist ),GFP_KERNEL );
Why aren't you using an sg pool here?
Quoted 4 lines
drivers/nvme/target/tcp.c new file +4
-0+if (!sg) +return -ENOMEM ; + +sg_init_table (sg, page_count);
[...]
drivers/nvme/target/tcp.c line 484
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+pr_err ("command %#x received but req_init_failed\n" , + request->cmd.common.opcode); +} + +
Nit, double newline
Quoted 3 lines
drivers/nvme/target/tcp.c new file +3
-0+static int connection_recv_from_sock (struct nvmet_tcp_connection *connection, +void *buf,size_t buf_len) +{
[...]
drivers/nvme/target/tcp.c line 597
Quoted 9 lines
drivers/nvme/target/tcp.c new file +9
-0+static void connection_rwork (struct work_struct *work) +{ +struct nvmet_tcp_connection *connection; +int count =0 ; +int rc; + + connection = container_of(work, struct nvmet_tcp_connection, rwork); + +while (test_bit (NVMET_ACCEPT_REQUESTS , &connection->flags)) {
See comment below
[...]
drivers/nvme/target/tcp.c line 658
Quoted 11 lines
drivers/nvme/target/tcp.c new file +11
-0+static void connection_swork (struct work_struct *work) +{ +struct nvmet_tcp_connection *connection; +struct nvmet_tcp_request *request =NULL ; +struct scatterlist *sg; +struct nvmet_req *req; +int count =0 ; +int rc; + + connection = container_of(work, struct nvmet_tcp_connection, swork); +
Emulating kthreads with workqueues?
Quoted 2 lines
drivers/nvme/target/tcp.c new file +2
-0+while (test_bit (NVMET_SOCK_CONNECTED , &connection->flags)) { +if (!request)
[...]
drivers/nvme/target/tcp.c line 606
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+ +if (count++ >10 ) { +cond_resched (); + count =0 ; + }
Please re-queue yourself and don't do this cond_reched() tricks.
Quoted 4 lines
drivers/nvme/target/tcp.c new file +4
-0+ } + +connection_put (connection); +}
[...]
Quoted 9 lines
drivers/nvme/target/tcp.c new file +9
-0+static struct nvmet_tcp_connection *connection_create ( +struct nvmet_tcp_listener *listener, +struct socket *socket) +{ +struct nvmet_tcp_connection *connection =NULL ; +u16 wq_index; +int rc =0 ; + + connection =kzalloc (sizeof (*connection),GFP_KERNEL );
drivers/nvme/target/tcp.c line 796
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+if (!connection) +return ERR_PTR (-ENOMEM ); + +mutex_lock (&nvmet_tcp_connection_mutex); + wq_index = listener->conn_wq_index++; +mutex_unlock (&nvmet_tcp_connection_mutex); + + connection->workqueue =alloc_ordered_workqueue ("nvmet-tcp-l%04x-c%04x" , +0 , listener->wq_index, wq_index); +if (!connection->workqueue) {
rc = ERR_PTR(-ENOMEM);
goto err_free_connection;
drivers/nvme/target/tcp.c line 805
Quoted 7 lines
drivers/nvme/target/tcp.c new file +7
-0+ } + +listener_get (listener); + + rc =nvmet_sq_init (&connection->nvme_sq); +if (rc) + goto err;
goto err_destroy_wq?
[...]
err_destroy_wq:
Review
Quoted 1 line
(quoted diff) +1
-0+ destroy_workqueue(connection->workqueue);
err_free_connection:
Quoted 6 lines
drivers/nvme/target/tcp.c new file +6
-0+kfree (connection); +listener_put (listener); + +return ERR_PTR (rc); +} +
[...]
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+static void listener_rwork (struct work_struct *work) +{ +struct nvmet_tcp_listener *listener; +int count =0 ; +int rc;
drivers/nvme/target/tcp.c line 889
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+ + listener = container_of(work, struct nvmet_tcp_listener, rwork); + +while (1 ) { + rc =listener_accept (listener); +if (rc) +break ; + +if (count++ >10 ) { +cond_resched ();
Why are you only calling cond_resched() every 10th accept?
[...]
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+static void listener_sock_destroy (struct nvmet_tcp_listener *listener) +{ +if (!listener->sock) +return ; + +kernel_sock_shutdown (listener->sock,SHUT_RDWR ); +sock_release (listener->sock); + + listener->sock =NULL ; +}
[...]
drivers/nvme/target/tcp.c line 936
Quoted 1 line
drivers/nvme/target/tcp.c new file +1
-0+ rc =kstrtou16 (disc_addr->trsvcid,0 , &port_in);
Do you plan to allocate a well known port for NVMe over TCP?
[...]
Quoted 4 lines
drivers/nvme/target/tcp.c new file +4
-0+/* +* NVMET ops +*/ +static int nvmet_tcp_add_port (struct nvmet_port *nvmet_port)
drivers/nvme/target/tcp.c line 1024
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+{ +struct nvmet_tcp_listener *listener; +static u16 wq_index; +int rc; + +if (!try_module_get (THIS_MODULE )) +return -EINVAL ; + + listener =kzalloc (sizeof (*listener),GFP_KERNEL ); +if (!listener) {
goto put_module;
drivers/nvme/target/tcp.c line 1034
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+module_put (THIS_MODULE ); +return -ENOMEM ; + } + + listener->wq_index = wq_index++; + listener->workqueue =alloc_ordered_workqueue ("nvmet-tcp-l%04x" ,0 , + listener->wq_index); +if (!listener->workqueue) { + rc = -ENOMEM ; + goto err;
goto free_listener;
drivers/nvme/target/tcp.c line 1045
Quoted 11 lines
drivers/nvme/target/tcp.c new file +11
-0+ } + +INIT_WORK (&listener->rwork, listener_rwork); +INIT_WORK (&listener->relwork, listener_release_work); +kref_init (&listener->kref); + listener->nvmet_port = nvmet_port; + nvmet_port->priv = listener; + + rc =listener_sock_create (listener); +if (rc <0 ) + goto err;
goto destroy_listener_sock;
drivers/nvme/target/tcp.c line 1053
Quoted 8 lines
drivers/nvme/target/tcp.c new file +8
-0+ +mutex_lock (&nvmet_tcp_listener_mutex); +list_add_tail (&listener->list, &nvmet_tcp_listener_list); +mutex_unlock (&nvmet_tcp_listener_mutex); + +return 0 ; + +err:
destroy_listener_sock:
Review
Quoted 1 line
(quoted diff) +1
-0+ listener_sock_destroy(listener);
free_listener:
Review
Quoted 1 line
(quoted diff) +1
-0+ kfree(listener);
put_module:
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+module_put (THIS_MODULE ); + +return rc; +} +
[...]
drivers/nvme/target/tcp.c line 1111
Quoted 13 lines
drivers/nvme/target/tcp.c new file +13
-0+if (cmpxchg (&request->state,NVMET_TCP_REQ_AWAITING_RESPONSE , +NVMET_TCP_REQ_SENDING_RESPONSE ) == +NVMET_TCP_REQ_AWAITING_RESPONSE ) + goto valid; + +if (cmpxchg (&request->state,NVMET_TCP_REQ_AWAITING_CMD , +NVMET_TCP_REQ_SENDING_RESPONSE ) == +NVMET_TCP_REQ_AWAITING_CMD ) + goto valid; + +pr_err ("Unexpected request state %d\n" , request->state); +return ; +
Can you please invert the above logic and jump to an invalid label?
[...]
drivers/nvme/target/tcp.c line 1160
Quoted 4 lines
drivers/nvme/target/tcp.c new file +4
-0+static int __init nvmet_tcp_init (void ) +{ + request_cache =kmem_cache_create ("nvmet_tcp_request" , +sizeof (struct nvmet_tcp_request ),0 ,0 ,NULL );
if (!request_cache) return -ENOMEM;
Signature · 6 lines
Johannes Thumshirn Storage jthumshirn at suse.de +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Jane Smithard, Graham Norton HRB 21284 (AG N?rnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
RFC PATCH 2/4 nvme-fabrics: add target support for TCP transport
Quoted 3 lines · Johannes Thumshirn
Hi Bert,
some initial comments inline
Thanks for all these Johannes.
Quoted 13 lines · Bert Kenward
Quoted 12 lines
Introduce the target side of an experimental proof of concept NVMe Fabrics transport running directly over TCP. This is intended to provide good performance over any interface.
At present most work is deferred to workqueues for simplicity, but it may be possible to avoid this in some situations and thus reduce the latency.
Memory is currently allocated as zero-order pages. It's possible that using higher order pages may be desirable, trading potential allocation latency for less fragmented SKBs on transmit.
Quoted 6 lines
Quoted 4 lines
Cc Ripduman Sohan <rip.sohan at verrko.com>Cc Lucian Carata <lucian.carata at cl.cam.ac.uk>Signed-off-by Bert Kenward <bkenward at solarflare.com>---
Can you avoid the forward declarations?
I'll see what I can move around to do so.
Quoted 1 line
Why aren't you using an sg pool here?
An excellent question, with no good answer.... I shall change this.
Quoted 7 lines
drivers/nvme/target/tcp.c line 484
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+pr_err ("command %#x received but req_init_failed\n" , + request->cmd.common.opcode); +} + +Nit, double newline
Thanks.
Quoted 5 lines
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+static void connection_swork (struct work_struct *work) +{ +struct nvmet_tcp_connection *connection; +struct nvmet_tcp_request *request =NULL ; +struct scatterlist *sg;
Quoted 8 lines
drivers/nvme/target/tcp.c line 658
Quoted 6 lines
drivers/nvme/target/tcp.c new file +6
-0+struct nvmet_req *req; +int count =0 ; +int rc; + + connection = container_of(work, struct nvmet_tcp_connection, swork); +Emulating kthreads with workqueues?
Yes, I suppose. Depending on the workload the work items may be short. Would there not be a latency impact from using a kthread here?
Quoted 7 lines
drivers/nvme/target/tcp.c line 606
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+ +if (count++ >10 ) { +cond_resched (); + count =0 ; + }Please re-queue yourself and don't do this
cond_reched()tricks.
Makes sense, thanks.
Quoted 6 lines
drivers/nvme/target/tcp.c line 796
Quoted 3 lines
drivers/nvme/target/tcp.c new file +3
-0+ connection->workqueue =alloc_ordered_workqueue ("nvmet-tcp-l%04x-c%04x" , +0 , listener->wq_index, wq_index); +if (!connection->workqueue) {rc =
ERR_PTR(-ENOMEM);
gotoerr_free_connection;
Thanks.
Quoted 9 lines
drivers/nvme/target/tcp.c line 805
Quoted 7 lines
drivers/nvme/target/tcp.c new file +7
-0+ } + +listener_get (listener); + + rc =nvmet_sq_init (&connection->nvme_sq); +if (rc) + goto err;goto
err_destroy_wq?
Thanks again...
Quoted 4 lines
err_destroy_wq:Review
Quoted 1 line
(quoted diff) +1
-0+ destroy_workqueue(connection->workqueue);
err_free_connection:
... and again.
Quoted 9 lines
Quoted 9 lines
drivers/nvme/target/tcp.c new file +9
-0+static void listener_rwork (struct work_struct *work) +{ +struct nvmet_tcp_listener *listener; +int count =0 ; +int rc; + + listener = container_of(work, struct nvmet_tcp_listener, rwork); + +while (1 ) {
Quoted 8 lines
drivers/nvme/target/tcp.c line 889
Quoted 6 lines
drivers/nvme/target/tcp.c new file +6
-0+ rc =listener_accept (listener); +if (rc) +break ; + +if (count++ >10 ) { +cond_resched ();Why are you only calling
cond_resched()every 10th accept?
For bad, arbitrary reasons. Would you suggest re-queueing after each accept or continuing until there are no more connections to accept?
Quoted 5 lines
[...]
drivers/nvme/target/tcp.c line 936
Quoted 1 line
drivers/nvme/target/tcp.c new file +1
-0+ rc =kstrtou16 (disc_addr->trsvcid,0 , &port_in);Do you plan to allocate a well known port for NVMe over TCP?
It would make sense, yes, but that's something that would be better handled via a standardisation process.
Quoted 59 lines
Quoted 4 lines
drivers/nvme/target/tcp.c new file +4
-0+/* +* NVMET ops +*/ +static int nvmet_tcp_add_port (struct nvmet_port *nvmet_port)drivers/nvme/target/tcp.c line 1024
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+{ +struct nvmet_tcp_listener *listener; +static u16 wq_index; +int rc; + +if (!try_module_get (THIS_MODULE )) +return -EINVAL ; + + listener =kzalloc (sizeof (*listener),GFP_KERNEL ); +if (!listener) {goto
put_module;drivers/nvme/target/tcp.c line 1034
Quoted 10 lines
drivers/nvme/target/tcp.c new file +10
-0+module_put (THIS_MODULE ); +return -ENOMEM ; + } + + listener->wq_index = wq_index++; + listener->workqueue =alloc_ordered_workqueue ("nvmet-tcp-l%04x" ,0 , + listener->wq_index); +if (!listener->workqueue) { + rc = -ENOMEM ; + goto err;goto
free_listener;drivers/nvme/target/tcp.c line 1045
Quoted 11 lines
drivers/nvme/target/tcp.c new file +11
-0+ } + +INIT_WORK (&listener->rwork, listener_rwork); +INIT_WORK (&listener->relwork, listener_release_work); +kref_init (&listener->kref); + listener->nvmet_port = nvmet_port; + nvmet_port->priv = listener; + + rc =listener_sock_create (listener); +if (rc <0 ) + goto err;goto
destroy_listener_sock;drivers/nvme/target/tcp.c line 1053
Quoted 8 lines
drivers/nvme/target/tcp.c new file +8
-0+ +mutex_lock (&nvmet_tcp_listener_mutex); +list_add_tail (&listener->list, &nvmet_tcp_listener_list); +mutex_unlock (&nvmet_tcp_listener_mutex); + +return 0 ; + +err:
destroy_listener_sock:Review
Quoted 1 line
(quoted diff) +1
-0+ listener_sock_destroy(listener);
free_listener:
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ kfree(listener);
put_module:drivers/nvme/target/tcp.c line 1060
Quoted 5 lines
drivers/nvme/target/tcp.c new file +5
-0+module_put (THIS_MODULE ); + +return rc; +} +
Thanks for all the above.
Quoted 7 lines
Quoted 7 lines
drivers/nvme/target/tcp.c new file +7
-0+if (cmpxchg (&request->state,NVMET_TCP_REQ_AWAITING_RESPONSE , +NVMET_TCP_REQ_SENDING_RESPONSE ) == +NVMET_TCP_REQ_AWAITING_RESPONSE ) + goto valid; + + if (cmpxchg(&request->state, NVMET_TCP_REQ_AWAITING_CMD, + NVMET_TCP_REQ_SENDING_RESPONSE) ==
Quoted 8 lines
drivers/nvme/target/tcp.c line 1111
Quoted 6 lines
drivers/nvme/target/tcp.c new file +6
-0+ NVMET_TCP_REQ_AWAITING_CMD) + goto valid; + +pr_err ("Unexpected request state %d\n" , request->state); +return ; +Can you please invert the above logic and jump to an invalid label?
Ah yes. That's rather a lot nicer.
Quoted 7 lines
drivers/nvme/target/tcp.c line 1160
Quoted 4 lines
drivers/nvme/target/tcp.c new file +4
-0+static int __init nvmet_tcp_init (void ) +{ + request_cache =kmem_cache_create ("nvmet_tcp_request" , +sizeof (struct nvmet_tcp_request ),0 ,0 ,NULL );if (!request_cache) return -ENOMEM;
Thanks.
Bert.
Christoph Hellwig
maintainerRFC PATCH 2/4 nvme-fabrics: add target support for TCP transport
In reply to Bert KenwardQuoted 3 lines · Bert Kenward
Memory is currently allocated as zero-order pages. It's possible that using higher order pages may be desirable, trading potential allocation latency for less fragmented SKBs on transmit.
FYI, that would be a useful optimization for RDMA as well. I know SCST had some sophisticated code for doing nice SGL allocations and caching, maybe it's time to get that upstream. Bart, is that something you'd be interested in looking into?
RFC PATCH 2/4 nvme-fabrics: add target support for TCP transport
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Bert Kenward
Memory is currently allocated as zero-order pages. It's possible that using higher order pages may be desirable, trading potential allocation latency for less fragmented SKBs on transmit.
FYI, that would be a useful optimization for RDMA as well. I know SCST had some sophisticated code for doing nice SGL allocations and caching, maybe it's time to get that upstream. Bart, is that something you'd be interested in looking into?
This driver looks indeed like yet another driver that could use an SG-list + page list cache. I will have a look.
Bart.
RFC PATCH 3/4 nvme-fabrics: add host support for TCP transport
In reply to Bert KenwardFrom: Ripduman Sohan <[email protected]>
Introduce the initiator for an experimental proof of concept NVMe Fabrics transport running over TCP. This is intended to provide good performance over any interface.
Most work is deferred to workqueues for simplicity, but it may be possible to avoid this in some situations and thus reduce latency.
Much of the queue setup and teardown is derived from the RDMA code, and could probably be simplified given the simpler nature of the transport.
--- drivers/nvme/host/Kconfig | 14 + drivers/nvme/host/Makefile | 3 + drivers/nvme/host/tcp.c | 1616 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1633 insertions(+) create mode 100644 drivers/nvme/host/tcp.c
drivers/nvme/host/Kconfig +14 -0
@@ -60,3 +60,17 @@ config NVME_FC from https://github.com/linux-nvme/nvme-cli. If unsure, say N. + +config NVME_TCP + tristate "NVMe over Fabrics TCP host support" + select NVME_CORE + select NVME_FABRICS + select SG_POOL + help + This enables the NVMe TCP initiator support. This allows you to + use remote block devices exported over TCP using the NVMe protocol. + + To configure a NVMe over Fabrics controller use the nvme-cli tool + from https://github.com/linux-nvme/nvme-cli. + + If unsure, say N.
drivers/nvme/host/Makefile +3 -0
@@ -3,6 +3,7 @@ obj-$(CONFIG_BLK_DEV_NVME) += nvme.o obj-$(CONFIG_NVME_FABRICS) += nvme-fabrics.o obj-$(CONFIG_NVME_RDMA) += nvme-rdma.o obj-$(CONFIG_NVME_FC) += nvme-fc.o +obj-$(CONFIG_NVME_TCP) += nvme-tcp.o nvme-core-y := core.o nvme-core-$(CONFIG_BLK_DEV_NVME_SCSI) += scsi.o
@@ -15,3 +16,5 @@ nvme-fabrics-y += fabrics.o nvme-rdma-y += rdma.o nvme-fc-y += fc.o + +nvme-tcp-y += tcp.o
drivers/nvme/host/tcp.c new file +1616 -0
1,616 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 4/4 nvme-fabrics: add tcp to trtype list for configfs
In reply to Bert Kenward--- drivers/nvme/target/configfs.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
drivers/nvme/target/configfs.c +10 -0
@@ -215,6 +215,8 @@ static ssize_t nvmet_addr_trtype_show(struct config_item *item,return sprintf (page,"loop\n" );case NVMF_TRTYPE_FC :return sprintf (page,"fc\n" ); +case NVMF_TRTYPE_TCP : +return sprintf (page,"tcp\n" );default :return sprintf (page,"\n" ); }
@@ -241,6 +243,12 @@ static void nvmet_port_init_tsas_fc(struct nvmet_port *port)memset (&port->disc_addr.tsas,0 ,NVMF_TSAS_SIZE ); } +static void nvmet_port_init_tsas_tcp (struct nvmet_port *port) +{ + port->disc_addr.trtype =NVMF_TRTYPE_TCP ; +memset (&port->disc_addr.tsas,0 ,NVMF_TSAS_SIZE ); +} +static ssize_t nvmet_addr_trtype_store (struct config_item *item,const char *page,size_t count) {
@@ -258,6 +266,8 @@ static ssize_t nvmet_addr_trtype_store(struct config_item *item,nvmet_port_init_tsas_loop (port); }else if (sysfs_streq (page,"fc" )) {nvmet_port_init_tsas_fc (port); + }else if (sysfs_streq (page,"tcp" )) { +nvmet_port_init_tsas_tcp (port); }else {pr_err ("Invalid value '%s' for trtype\n" , page);return -EINVAL ;
Signature · 1 line
2.7.4
Sagi Grimberg
maintainercover nvme-fabrics: TCP transport
In reply to Bert KenwardHey Bert,
Quoted 8 lines
Experimental proof of concept NVMe Fabrics transport over TCP, intended to provide good performance over any network.
Initial performance tests using both ramdisks and NVMe SSDs show latency and throughput broadly comparable to a RoCE transport. Further optimisation is needed, but we welcome input and feedback on our approach and goal of adding a TCP transport to the NVMe Fabrics specification.
Quoted 5 lines
Using this requires a modified version of the nvme-cli tool, available at
https://github.com/solarflarecommunications/nvme-cli/IPv6 support is not yet implemented, but will be added using Sagi's new
inet_pton_with_scope.
I'd recommend that you join the fabrics 1.1 work that has started on the NVMe TWG. I can't really provide feedback on an non-standardized implementation, and we're obviously not going to accept it.
If you have any intentions for pursuing this, you should go through the NVMe TWG.
Cheers,
Sagi.
cover nvme-fabrics: TCP transport
Hello,
Quoted 6 lines · Sagi Grimberg
I'd recommend that you join the fabrics 1.1 work that has started on the NVMe TWG. I can't really provide feedback on an non-standardized implementation, and we're obviously not going to accept it.
If you have any intentions for pursuing this, you should go through the NVMe TWG.
Thanks for the feedback Sagi.
Bert.