RFC nvme-tcp: remove the direct send path
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest2 patches 1 replyIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/2 | nvme-tcp: remove the direct send pathdrivers/ | +2 | 1 reply | |
| 2/2 | nvme-tcp: remove send_mutexdrivers/ | +7 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC PATCH 0/2] nvme-tcp: remove the direct send path
Hannes suggested removing the direct send path in nvme_tcp_queue_request() while reviewing commit 5db358f3a7f9 ("nvme-tcp:
fix io_work priority inversion when send_mutex is held"). This series does that, on top of nvme-7.4.
Patch 1 removes the direct send path, so io_work is the only sender. Patch 2 removes send_mutex, which has nothing left to protect after patch 1.
The direct send path was added to cut latency when the queue is idle, so I measured it on two machines with 1Gb NICs. The target is nvmet-tcp backed by null_blk (irqmode=0). The host runs fio like this, 60 seconds per run and 3 runs per case:
fio --name=qd1 --filename=/dev/nvme0n1 --direct=1 --ioengine=libaio \
--bs=4k --rw=randread --iodepth=1 --numjobs=1 --cpus_allowed=2 \
--time_based --runtime=60 --ramp_time=5Average of the 3 runs at queue depth 1, latency in usec, "patched" is this series:
IOPS slat clat lat clat p99
randread base 4153 7.1 232.3 239.3 277
patched 4154 4.7 234.7 239.4 280
randwrite base 2472 24.5 375.6 400.1 553
patched 2460 14.3 387.6 402.0 570The send time moves from slat to clat. With the direct send path, io_submit() itself sends the command on the socket, so that time is counted in slat. Without it, io_submit() only queues the request and returns, and io_work sends it, so the same time is counted in clat. lat is the sum of the two, and it changes by 0.1 usec for randread and 1.9 usec (+0.5%) for randwrite.
At queue depth 32 with 4 jobs, both kernels reach the 1Gb link limit (28.2k IOPS), so there is no difference there.
On the same two machines, I also ran 5 controller resets during a 120 second randrw fio run on the patched kernel, and fio finished with no errors. Separately, blktests nvme over tcp on a VM with an nvmet tcp loopback target gives the same results with and without this series.
On a 1Gb network one I/O takes about 240 us, so a difference of a few microseconds is easy to miss. Numbers from faster networks would show whether it matters there. I am posting this as RFC so the performance side can be discussed on the list.
Seiji Nishikawa (2):
nvme-tcp: remove the direct send path
nvme-tcp: remove send_mutex
drivers/nvme/host/tcp.c | 60 +++++++---------------------------------- 1 file changed, 9 insertions(+), 51 deletions(-)
base-commit: 68099cf27253a8c89d703a282d3d811446d9453b
Signature · 1 line
2.56.1