nvme-tcp: send quota for nvme_tcp_send_all()
PATCH v2 1/1 nvme-tcp: send quota for nvme_tcp_send_all()
Add a send quota in nvme_tcp_send_all() to avoid stalls when sending large amounts of requests.
---
IMO, this patch might still be a good idea to add. At least in my test setup where I only have one ethernet port it makes a big difference when accessing the system via ssh. When nvme-tcp is pushing a lot of data via the network, the ssh session is completely blocked by the storage traffic. With it, the ssh session stays responsive.
I suspect a proper storage setup would use more than one ethernet port.
Daniel
v1:
https://lore.kernel.org/linux-nvme/[email protected]/
drivers/nvme/host/tcp.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
drivers/nvme/host/tcp.c +14 -3
@@ -308,12 +308,23 @@ static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,static inline void nvme_tcp_send_all (struct nvme_tcp_queue *queue) { -int ret; +unsigned long deadline = jiffies +msecs_to_jiffies (1 );/* drain the send queue as much as we can... */ do { - ret =nvme_tcp_try_send (queue); - }while (ret >0 ); +bool pending = false; +int result; + + result =nvme_tcp_try_send (queue); +if (result >0 ) + pending = true; +else if (unlikely (result <0 )) +return ; + +if (!pending) +return ; + + }while (!time_after (jiffies, deadline));/* quota is exhausted */ }static inline bool nvme_tcp_queue_more (struct nvme_tcp_queue *queue)
Signature · 1 line
2.38.0
Quoted 5 lines · Daniel Wagner
Add a send quota in
nvme_tcp_send_all()to avoid stalls when sending large amounts of requests.Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>
Quoted 7 lines
---
IMO, this patch might still be a good idea to add. At least in my test setup where I only have one ethernet port it makes a big difference when accessing the system via ssh. When nvme-tcp is pushing a lot of data via the network, the ssh session is completely blocked by the storage traffic. With it, the ssh session stays responsive. >
I'm not sure whether it is possible but is there a way to gather some form of quantitative data and present it here to we all know exactly which aspect is improving by this patch in context of "ssd session is completely blocked" ?
Quoted 16 lines
I suspect a proper storage setup would use more than one ethernet port.
Daniel
v1:
https://lore.kernel.org/linux-nvme/[email protected]/drivers/nvme/host/tcp.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req, · line 311
Quoted 5 lines
drivers/nvme/host/tcp.c +1
-1@@ -308,12 +308,23 @@ static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,static inline void nvme_tcp_send_all (struct nvme_tcp_queue *queue) { -int ret; +unsigned long deadline = jiffies +msecs_to_jiffies (1 );
We need to provide some flexibility to set this value as one specified value may not work with all the setups and H/W.
Can we make it tunable and not statically coded ?
Quoted 11 lines
drivers/nvme/host/tcp.c +6
-2/* drain the send queue as much as we can... */ do { - ret =nvme_tcp_try_send (queue); - }while (ret >0 ); + bool pending = false; + int result; + + result = nvme_tcp_try_send(queue); + if (result > 0) + pending = true;
drivers/nvme/host/tcp.c static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req, · line 330
Quoted 10 lines
drivers/nvme/host/tcp.c +7
-0+else if (unlikely (result <0 )) +return ; + +if (!pending) +return ; + + }while (!time_after (jiffies, deadline));/* quota is exhausted */ }static inline bool nvme_tcp_queue_more (struct nvme_tcp_queue *queue)
-ck
Quoted 14 lines · Chaitanya Kulkarni
Quoted 5 lines · Daniel Wagner
Add a send quota in
nvme_tcp_send_all()to avoid stalls when sending large amounts of requests.Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>Quoted 7 lines
---
IMO, this patch might still be a good idea to add. At least in my test setup where I only have one ethernet port it makes a big difference when accessing the system via ssh. When nvme-tcp is pushing a lot of data via the network, the ssh session is completely blocked by the storage traffic. With it, the ssh session stays responsive. >
Quoted 4 lines
I'm not sure whether it is possible but is there a way to gather some form of quantitative data and present it here to we all know exactly which aspect is improving by this patch in context of "ssd session is completely blocked" ?
Before starting a fio test run, the remote shell works fine and when fio runs, no keystroke gets echoed until the fio stops again. Obviously, this is heavily depending on the workload. So my observation is that limiting an unbound send loop prevents a 'fair' usage of the bandwidth.
Quoted 4 lines
drivers/nvme/host/tcp.c static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req, · line 311
Quoted 1 line
drivers/nvme/host/tcp.c +1
-0+unsigned long deadline = jiffies +msecs_to_jiffies (1 );We need to provide some flexibility to set this value as one specified value may not work with all the setups and H/W.
Note, this is the identically approach we have in nvme_tcp_io_work() already. So nothing new. Though I was wondering too why 1 jiffy.
Quoted 1 line
Can we make it tunable and not statically coded ?
I would really like to avoid having a tunable knob like a sysfs entry. None will get it right. That means this would need to be algorithm which auto adapts, though I don't have any good idea how this algorithm should operate. Any ideas?
Quoted 14 lines · Chaitanya Kulkarni
Quoted 5 lines · Daniel Wagner
Add a send quota in
nvme_tcp_send_all()to avoid stalls when sending large amounts of requests.Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>Quoted 7 lines
---
IMO, this patch might still be a good idea to add. At least in my test setup where I only have one ethernet port it makes a big difference when accessing the system via ssh. When nvme-tcp is pushing a lot of data via the network, the ssh session is completely blocked by the storage traffic. With it, the ssh session stays responsive. >
Quoted 5 lines
I'm not sure whether it is possible but is there a way to gather some form of quantitative data and present it here to we all know exactly which aspect is improving by this patch in context of "ssd session is completely blocked" ?
Doubt that we can do it. Point is, the send code will run in a tight loop, making scheduling of other processes / packets really hard.
So if you have several processes on the same interface (as here with the ssh connection) nvme-tcp will eat up the entire bandwidth sending its data, and everyone else on the line will suffer.
I guess the same effect could be had by adding a 'schedule()' after each nvme_tcp_try_send() call.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Sagi Grimberg
maintainerQuoted 19 lines
Quoted 14 lines
Quoted 6 lines
Add a send quota in
nvme_tcp_send_all()to avoid stalls when sending large amounts of requests.Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>Quoted 7 lines
---
IMO, this patch might still be a good idea to add. At least in my test setup where I only have one ethernet port it makes a big difference when accessing the system via ssh. When nvme-tcp is pushing a lot of data via the network, the ssh session is completely blocked by the storage traffic. With it, the ssh session stays responsive. >
Quoted 5 lines
I'm not sure whether it is possible but is there a way to gather some form of quantitative data and present it here to we all know exactly which aspect is improving by this patch in context of "ssd session is completely blocked" ?
Quoted 4 lines
Doubt that we can do it. Point is, the send code will run in a tight loop, making scheduling of other processes / packets really hard.
So if you have several processes on the same interface (as here with the ssh connection) nvme-tcp will eat up the entire bandwidth sending its data, and everyone else on the line will suffer.I guess the same effect could be had by adding a '
schedule()' after eachnvme_tcp_try_send()call.
Daniel, does adding cond_resched() make the system responsive again?
Quoted 1 line · Sagi Grimberg
Daniel, does adding
cond_resched()make the system responsive again?
As it turns out I can't reproduce it anymore. Our test lab got restructured and the cabling between machine changed (maybe even different network switches, idk). Also the target got a firmware update.
It looks more like the real problem was caused by the network infrastructure than the host itself. And adding the additional delay in the send path was just reducing the load which made the ssh session working.
Given this, I don't think we currently need to touch this code. Though the unbounded loop makes me a bit uneasy.
Sagi Grimberg
maintainerQuoted 8 lines
Quoted 2 lines
Daniel, does adding
cond_resched()make the system responsive again?As it turns out I can't reproduce it anymore. Our test lab got restructured and the cabling between machine changed (maybe even different network switches, idk). Also the target got a firmware update.
Quoted 4 lines
It looks more like the real problem was caused by the network infrastructure than the host itself. And adding the additional delay in the send path was just reducing the load which made the ssh session working.
Let's table it for now.
Quoted 2 lines
Given this, I don't think we currently need to touch this code. Though the unbounded loop makes me a bit uneasy.
This flow only enters when there is a single request queued, so it should not be something that takes long. Unless there is a real problem, lets not add code that may produce one.
Quoted 6 lines · Sagi Grimberg
Quoted 2 lines
Given this, I don't think we currently need to touch this code. Though the unbounded loop makes me a bit uneasy.
This flow only enters when there is a single request queued, so it should not be something that takes long. Unless there is a real problem, lets not add code that may produce one.
Okay, make sense. I just wonder how to diagnose this scenario.
Quoted 45 lines · Daniel Wagner
Add a send quota in
nvme_tcp_send_all()to avoid stalls when sending large amounts of requests.Cc Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>---
IMO, this patch might still be a good idea to add. At least in my test setup where I only have one ethernet port it makes a big difference when accessing the system via ssh. When nvme-tcp is pushing a lot of data via the network, the ssh session is completely blocked by the storage traffic. With it, the ssh session stays responsive.
I suspect a proper storage setup would use more than one ethernet port.
Daniel
v1:
https://lore.kernel.org/linux-nvme/[email protected]/drivers/nvme/host/tcp.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)drivers/nvme/host/tcp.c +7
-3@@ -308,12 +308,23 @@ static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,static inline void nvme_tcp_send_all (struct nvme_tcp_queue *queue) { -int ret; +unsigned long deadline = jiffies +msecs_to_jiffies (1 );/* drain the send queue as much as we can... */ do { - ret =nvme_tcp_try_send (queue); - }while (ret >0 ); + bool pending = false; + int result; + + result = nvme_tcp_try_send(queue); + if (result > 0) + pending = true;
drivers/nvme/host/tcp.c static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req, · line 330
Quoted 10 lines
drivers/nvme/host/tcp.c +7
-0+else if (unlikely (result <0 )) +return ; + +if (!pending) +return ; + + }while (!time_after (jiffies, deadline));/* quota is exhausted */ }static inline bool nvme_tcp_queue_more (struct nvme_tcp_queue *queue)
Looks like a good idea indeed.
One might want to make the deadline configurable, though. Other than that:
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman