nvmet-tcp: support specifying the congestion-control
Subsystems (2)
PATCH v2 1/2 nvmet-tcp: support specifying the congestion-control
From: Mingbao Sun <[email protected]>
congestion-control could have a noticeable impaction on the performance of TCP-based communications. This is of course true to NVMe_over_TCP.
Different congestion-controls (e.g., cubic, dctcp) are suitable for different scenarios. Proper adoption of congestion control would benefit the performance. On the contrary, the performance could be destroyed.
Though we can specify the congestion-control of NVMe_over_TCP via writing '/proc/sys/net/ipv4/tcp_congestion_control', but this also changes the congestion-control of all the future TCP sockets that have not been explicitly assigned the congestion-control, thus bringing potential impaction on their performance.
So it makes sense to make NVMe_over_TCP support specifying the congestion-control. And this commit addresses the target side.
Implementation approach:
the following new file entry was created for user to specify the congestion-control of each nvmet port. '/sys/kernel/config/nvmet/ports/X/tcp_congestion'
Then later in nvmet_tcp_add_port, the specified congestion-control would be applied to the listening socket of the nvmet port.
--- drivers/nvme/target/configfs.c | 37 ++++++++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 1 + drivers/nvme/target/tcp.c | 27 +++++++++++++++++++++++++ 3 files changed, 65 insertions(+)
drivers/nvme/target/configfs.c +37 -0
@@ -222,6 +222,41 @@ static ssize_t nvmet_addr_trsvcid_store(struct config_item *item,CONFIGFS_ATTR (nvmet_, addr_trsvcid); +static ssize_t nvmet_tcp_congestion_show (struct config_item *item, +char *page) +{ +struct nvmet_port *port =to_nvmet_port (item); + +return snprintf (page,PAGE_SIZE ,"%s\n" , + port->tcp_congestion ? port->tcp_congestion :"" ); +} + +static ssize_t nvmet_tcp_congestion_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +int len; +char *buf; + + len =strcspn (page,"\n" ); +if (!len) +return -EINVAL ; + +if (nvmet_is_port_enabled (port, __func__)) +return -EACCES ; + + buf =kmemdup_nul (page, len,GFP_KERNEL ); +if (!buf) +return -ENOMEM ; + +kfree (port->tcp_congestion); + port->tcp_congestion = buf; + +return count; +} + +CONFIGFS_ATTR (nvmet_, tcp_congestion); +static ssize_t nvmet_param_inline_data_size_show (struct config_item *item,char *page) {
@@ -1597,6 +1632,7 @@ static void nvmet_port_release(struct config_item *item)list_del (&port->global_entry);kfree (port->ana_state); +kfree (port->tcp_congestion);kfree (port); }
@@ -1605,6 +1641,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_treq, &nvmet_attr_addr_traddr, &nvmet_attr_addr_trsvcid, + &nvmet_attr_tcp_congestion, &nvmet_attr_addr_trtype, &nvmet_attr_param_inline_data_size, #ifdef CONFIG_BLK_DEV_INTEGRITY
drivers/nvme/target/nvmet.h +1 -0
@@ -145,6 +145,7 @@ struct nvmet_port {struct config_group ana_groups_group;struct nvmet_ana_group ana_default_group;enum nvme_ana_state *ana_state; +const char *tcp_congestion;void *priv;bool enabled;int inline_data_size;
drivers/nvme/target/tcp.c +27 -0
@@ -1657,8 +1657,10 @@ static void nvmet_tcp_accept_work(struct work_struct *w)struct nvmet_tcp_port *port = container_of(w, struct nvmet_tcp_port, accept_work);struct socket *newsock; +struct inet_connection_sock *icsk, *icsk_new;int ret; + icsk =inet_csk (port->sock->sk);while (true) { ret =kernel_accept (port->sock, &newsock,O_NONBLOCK );if (ret <0 ) {
@@ -1666,6 +1668,16 @@ static void nvmet_tcp_accept_work(struct work_struct *w)pr_warn ("failed to accept err=%d\n" , ret);return ; } + +if (port->nport->tcp_congestion) { + icsk_new =inet_csk (newsock->sk); +if (icsk_new->icsk_ca_ops != icsk->icsk_ca_ops) { +pr_warn ("congestion abnormal: expected %s, actual %s.\n" , + icsk->icsk_ca_ops->name, + icsk_new->icsk_ca_ops->name); + } + } + ret =nvmet_tcp_alloc_queue (port, newsock);if (ret) {pr_err ("failed to allocate queue\n" );
@@ -1693,6 +1705,8 @@ static int nvmet_tcp_add_port(struct nvmet_port *nport) {struct nvmet_tcp_port *port;__kernel_sa_family_t af; +char ca_name[TCP_CA_NAME_MAX ]; +sockptr_t optval;int ret; port =kzalloc (sizeof (*port),GFP_KERNEL );
@@ -1741,6 +1755,19 @@ static int nvmet_tcp_add_port(struct nvmet_port *nport)if (so_priority >0 )sock_set_priority (port->sock->sk, so_priority); +if (nport->tcp_congestion) { +strncpy (ca_name, nport->tcp_congestion,TCP_CA_NAME_MAX -1 ); + optval =KERNEL_SOCKPTR (ca_name); + ret =sock_common_setsockopt (port->sock,IPPROTO_TCP , +TCP_CONGESTION , optval, +strlen (ca_name)); +if (ret) { +pr_err ("failed to set port socket's congestion to %s: %d\n" , + ca_name, ret); + goto err_sock; + } + } + ret =kernel_bind (port->sock, (struct sockaddr *)&port->addr,sizeof (port->addr));if (ret) {
Signature · 1 line
2.26.2
The patch v1 for target side also inserted some calls to networking APIs in the generic part (configfs.c) for validating the congestion-control specified by the user.
Per the comments from Christoph Hellwig on
‘[PATCH 2/2] nvme-tcp: support specifying the congestion-control’ (the patch for the host side), here delete these calls.
Since the tcp_congestion specified by the user could also get checked later within sock_common_setsockopt in nvmet_tcp_add_port. So this deletion brings little downside.
Christoph Hellwig
maintainerdrivers/nvme/target/tcp.c static void nvmet_tcp_accept_work(struct work_struct *w) · line 1679
Quoted 8 lines · Mingbao Sun
drivers/nvme/target/tcp.c +8
-0+if (port->nport->tcp_congestion) { + icsk_new =inet_csk (newsock->sk); +if (icsk_new->icsk_ca_ops != icsk->icsk_ca_ops) { +pr_warn ("congestion abnormal: expected %s, actual %s.\n" , + icsk->icsk_ca_ops->name, + icsk_new->icsk_ca_ops->name); + } + }
What is the point of having this code?
drivers/nvme/target/tcp.c static int nvmet_tcp_add_port(struct nvmet_port *nport) · line 1769
Quoted 12 lines
drivers/nvme/target/tcp.c +12
-0+if (nport->tcp_congestion) { +strncpy (ca_name, nport->tcp_congestion,TCP_CA_NAME_MAX -1 ); + optval =KERNEL_SOCKPTR (ca_name); + ret =sock_common_setsockopt (port->sock,IPPROTO_TCP , +TCP_CONGESTION , optval, +strlen (ca_name)); +if (ret) { +pr_err ("failed to set port socket's congestion to %s: %d\n" , + ca_name, ret); + goto err_sock; + } + }
Same comment as for the host side.
Quoted 11 lines · On Wed, 9 Mar 2022 07:15:41 +0100 Christoph Hellwig <[email protected]> wrote:
drivers/nvme/target/tcp.c static void nvmet_tcp_accept_work(struct work_struct *w) · line 1679
Quoted 8 lines · Mingbao Sun
drivers/nvme/target/tcp.c +8
-0+if (port->nport->tcp_congestion) { + icsk_new =inet_csk (newsock->sk); +if (icsk_new->icsk_ca_ops != icsk->icsk_ca_ops) { +pr_warn ("congestion abnormal: expected %s, actual %s.\n" , + icsk->icsk_ca_ops->name, + icsk_new->icsk_ca_ops->name); + } + }What is the point of having this code?
Well, this could happen in certain circumstances.
Take the result from my test as an example:
- The congestion of the listening socket of the target was set to ‘dctcp’.
- But the congestion of the socket of the host side was set to ‘cubic’.
- Then the congestion of the socket of the new connection at the target side would automatically be altered to ‘dctcp-reno’.
In case tcp_congestion was explicitly set for the target, it can be supposed that the user attaches great importance to performance. So we’d better make the users aware that the system is not working in the way they expect.
Thus the checking and warning was added here.
Quoted 15 lines
drivers/nvme/target/tcp.c static int nvmet_tcp_add_port(struct nvmet_port *nport) · line 1769
Quoted 13 lines
drivers/nvme/target/tcp.c +12
-0+if (nport->tcp_congestion) { +strncpy (ca_name, nport->tcp_congestion,TCP_CA_NAME_MAX -1 ); + optval =KERNEL_SOCKPTR (ca_name); + ret =sock_common_setsockopt (port->sock,IPPROTO_TCP , +TCP_CONGESTION , optval, +strlen (ca_name)); +if (ret) { +pr_err ("failed to set port socket's congestion to %s: %d\n" , + ca_name, ret); + goto err_sock; + } + }Same comment as for the host side.
This will be handled as the host side in the next version.
Christoph Hellwig
maintainerQuoted 24 lines · Mingbao Sun
Quoted 11 lines · On Wed, 9 Mar 2022 07:15:41 +0100 Christoph Hellwig <[email protected]> wrote:
drivers/nvme/target/tcp.c static void nvmet_tcp_accept_work(struct work_struct *w) · line 1679
Quoted 8 lines · Mingbao Sun
drivers/nvme/target/tcp.c +8
-0+if (port->nport->tcp_congestion) { + icsk_new =inet_csk (newsock->sk); +if (icsk_new->icsk_ca_ops != icsk->icsk_ca_ops) { +pr_warn ("congestion abnormal: expected %s, actual %s.\n" , + icsk->icsk_ca_ops->name, + icsk_new->icsk_ca_ops->name); + } + }What is the point of having this code?
Well, this could happen in certain circumstances.
Take the result from my test as an example:- The congestion of the listening socket of the target was set to ‘dctcp’.
- But the congestion of the socket of the host side was set to ‘cubic’.
Quoted 7 lines
- Then the congestion of the socket of the new connection at the target side would automatically be altered to ‘dctcp-reno’.
In case
tcp_congestionwas explicitly set for the target, it can be supposed that the user attaches great importance to performance. So we’d better make the users aware that the system is not working in the way they expect.
A warning message really seems very severe for a condition like this. Maybe the better interface is a way to figure out which congestion control algorithm is in use by reading a sysfs file.
Quoted 28 lines · On Thu, 10 Mar 2022 09:38:11 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 27 lines · Mingbao Sun
Quoted 11 lines · On Wed, 9 Mar 2022 07:15:41 +0100 Christoph Hellwig <[email protected]> wrote:
drivers/nvme/target/tcp.c static void nvmet_tcp_accept_work(struct work_struct *w) · line 1679
Quoted 8 lines · Mingbao Sun
drivers/nvme/target/tcp.c +8
-0+if (port->nport->tcp_congestion) { + icsk_new =inet_csk (newsock->sk); +if (icsk_new->icsk_ca_ops != icsk->icsk_ca_ops) { +pr_warn ("congestion abnormal: expected %s, actual %s.\n" , + icsk->icsk_ca_ops->name, + icsk_new->icsk_ca_ops->name); + } + }What is the point of having this code?
Well, this could happen in certain circumstances.
Take the result from my test as an example:- The congestion of the listening socket of the target was set to ‘dctcp’.
- But the congestion of the socket of the host side was set to ‘cubic’.
- Then the congestion of the socket of the new connection at the target side would automatically be altered to ‘dctcp-reno’.
Quoted 8 lines
Quoted 4 lines
In case
tcp_congestionwas explicitly set for the target, it can be supposed that the user attaches great importance to performance. So we’d better make the users aware that the system is not working in the way they expect.A warning message really seems very severe for a condition like this. Maybe the better interface is a way to figure out which congestion control algorithm is in use by reading a sysfs file.
Well, a target could have a great number of TCP sockets.
I feel it’s not proper to create a sysfs entry for each socket. And for those sockets that do not have the exception of congestion-control, it’s merely a waste of resources.
Also, since these sockets generate and die dynamically, the info exported via fs may even do not have the opportunity to be seen by the user.
Anyway, if you insist that the checking and warning here is not proper, I can remove it.
Quoted 46 lines · On Thu, 10 Mar 2022 19:06:36 +0800 Mingbao Sun <[email protected]> wrote:
Quoted 28 lines · On Thu, 10 Mar 2022 09:38:11 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 27 lines · Mingbao Sun
Quoted 11 lines · On Wed, 9 Mar 2022 07:15:41 +0100 Christoph Hellwig <[email protected]> wrote:
drivers/nvme/target/tcp.c static void nvmet_tcp_accept_work(struct work_struct *w) · line 1679
Quoted 8 lines · Mingbao Sun
drivers/nvme/target/tcp.c +8
-0+if (port->nport->tcp_congestion) { + icsk_new =inet_csk (newsock->sk); +if (icsk_new->icsk_ca_ops != icsk->icsk_ca_ops) { +pr_warn ("congestion abnormal: expected %s, actual %s.\n" , + icsk->icsk_ca_ops->name, + icsk_new->icsk_ca_ops->name); + } + }What is the point of having this code?
Well, this could happen in certain circumstances.
Take the result from my test as an example:- The congestion of the listening socket of the target was set to ‘dctcp’.
- But the congestion of the socket of the host side was set to ‘cubic’.
- Then the congestion of the socket of the new connection at the target side would automatically be altered to ‘dctcp-reno’.
Quoted 8 lines
Quoted 4 lines
In case
tcp_congestionwas explicitly set for the target, it can be supposed that the user attaches great importance to performance. So we’d better make the users aware that the system is not working in the way they expect.A warning message really seems very severe for a condition like this. Maybe the better interface is a way to figure out which congestion control algorithm is in use by reading a sysfs file.
Well, a target could have a great number of TCP sockets.
I feel it’s not proper to create a sysfs entry for each socket. And for those sockets that do not have the exception of congestion-control, it’s merely a waste of resources.
Quoted 6 lines
Also, since these sockets generate and die dynamically, the info exported via fs may even do not have the opportunity to be seen by the user.
Anyway, if you insist that the checking and warning here is not proper, I can remove it.
How about replacing pr_warn with pr_warn_once?
Christoph Hellwig
maintainerQuoted 10 lines · Mingbao Sun
I feel it’s not proper to create a sysfs entry for each socket. And for those sockets that do not have the exception of congestion-control, it’s merely a waste of resources.
Also, since these sockets generate and die dynamically, the info exported via fs may even do not have the opportunity to be seen by the user.
Anyway, if you insist that the checking and warning here is not proper, I can remove it.
Something that can happen during normal operation is per definition no something that should be warned about.
Quoted 9 lines · On Thu, 10 Mar 2022 15:20:34 +0100 Christoph Hellwig <[email protected]> wrote:
Quoted 8 lines · Mingbao Sun
I feel it’s not proper to create a sysfs entry for each socket. And for those sockets that do not have the exception of congestion-control, it’s merely a waste of resources.
Also, since these sockets generate and die dynamically, the info exported via fs may even do not have the opportunity to be seen by the user.
Quoted 5 lines
Quoted 2 lines
Anyway, if you insist that the checking and warning here is not proper, I can remove it.
Something that can happen during normal operation is per definition no something that should be warned about.
Got.
Will remove this checking and warning in the next version.