nvme-tcp request timeouts
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)
This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Thanks,
Seth
Hi Seth,
Quoted 26 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Quoted 7 lines
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Thanks,
Seth
Can you bisect it ? that will help to understand the commit causing issue.
-ck
Quoted 33 lines · Chaitanya Kulkarni
Hi Seth,
Quoted 30 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Quoted 6 lines
Quoted 3 lines
Thanks,
SethCan you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007 nvme-tcp: rerun io_work if req_list is not empty 70f437fb4395 nvme-tcp: fix io_work priority inversion3770a42bb8ce nvme-tcp: fix regression that causes sporadic requests to time out
5.10 still has timeouts with these backported, so whatever the problem is it has existed at least that long. I suppose I could go back to older kernels with these backported if that's going to be the best path forward here.
Thanks,
Seth
Quoted 49 lines · Seth Forshee
Quoted 33 lines · Chaitanya Kulkarni
Hi Seth,
Quoted 30 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Quoted 6 lines
Quoted 3 lines
Thanks,
SethCan you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007nvme-tcp: rerunio_workifreq_listis not empty70f437fb4395nvme-tcp: fixio_workpriority inversion3770a42bb8cenvme-tcp: fix regression that causes sporadic requests to time out
Quoted 7 lines
5.10 still has timeouts with these backported, so whatever the problem is it has existed at least that long. I suppose I could go back to older kernels with these backported if that's going to be the best path forward here.
Thanks,
Seth
Can you please share the fio config you are using ?
-ck
Quoted 55 lines · Chaitanya Kulkarni
Quoted 54 lines · Seth Forshee
Quoted 33 lines · Chaitanya Kulkarni
Hi Seth,
Quoted 30 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Quoted 6 lines
Quoted 3 lines
Thanks,
SethCan you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007nvme-tcp: rerunio_workifreq_listis not empty70f437fb4395nvme-tcp: fixio_workpriority inversion3770a42bb8cenvme-tcp: fix regression that causes sporadic requests to time out5.10 still has timeouts with these backported, so whatever the problem is it has existed at least that long. I suppose I could go back to older kernels with these backported if that's going to be the best path forward here.
Quoted 4 lines
Quoted 2 lines
Thanks,
SethCan you please share the fio config you are using ?
Sure. Note that I can reproduce it with a lower number of numjobs, but higher numbers make it easier, so I set it to the number of CPUs present on the system I'm using to test.
[global]
name=fio-rand-RW
filename=fio-rand-RW
rw=randrw
rwmixread=60
rwmixwrite=40
bs=4K
direct=0
numjobs=128time_based
runtime=900
[file1]
size=10G
ioengine=libaio
iodepth=16
Sagi Grimberg
maintainerHey Seth, thanks for reporting.
Quoted 63 lines
Quoted 53 lines
Quoted 53 lines
Quoted 33 lines
Hi Seth,
Quoted 30 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Quoted 6 lines
Quoted 3 lines
Thanks,
SethCan you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007nvme-tcp: rerunio_workifreq_listis not empty70f437fb4395nvme-tcp: fixio_workpriority inversion3770a42bb8cenvme-tcp: fix regression that causes sporadic requests to time out5.10 still has timeouts with these backported, so whatever the problem is it has existed at least that long. I suppose I could go back to older kernels with these backported if that's going to be the best path forward here.
Quoted 4 lines
Quoted 2 lines
Thanks,
SethCan you please share the fio config you are using ?
Sure. Note that I can reproduce it with a lower number of numjobs, but higher numbers make it easier, so I set it to the number of CPUs present on the system I'm using to test.
Quoted 8 lines
[global]
name=fio-rand-RW
filename=fio-rand-RW
rw=randrw
rwmixread=60
rwmixwrite=40
bs=4K
direct=0
Does this happen with direct=1?
Quoted 9 lines
numjobs=128
time_based
runtime=900[file1]
size=10G
ioengine=libaio
iodepth=16
Is it possible that the backend nvmet device is not fast enough? Are your backend devices nvme? Can you share nvmetcli ls output?
Also, just to understand if things are slow or stuck, perhaps you can artificially increase the io_timeout to say 60s or 120s.
Quoted 74 lines · Sagi Grimberg
Hey Seth, thanks for reporting.
Quoted 63 lines
Quoted 53 lines
Quoted 53 lines
Quoted 33 lines
Hi Seth,
Quoted 30 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Quoted 6 lines
Quoted 3 lines
Thanks,
SethCan you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007nvme-tcp: rerunio_workifreq_listis not empty70f437fb4395nvme-tcp: fixio_workpriority inversion3770a42bb8cenvme-tcp: fix regression that causes sporadic requests to time out5.10 still has timeouts with these backported, so whatever the problem is it has existed at least that long. I suppose I could go back to older kernels with these backported if that's going to be the best path forward here.
Quoted 4 lines
Quoted 2 lines
Thanks,
SethCan you please share the fio config you are using ?
Sure. Note that I can reproduce it with a lower number of numjobs, but higher numbers make it easier, so I set it to the number of CPUs present on the system I'm using to test.
Quoted 8 lines
[global]
name=fio-rand-RW
filename=fio-rand-RW
rw=randrw
rwmixread=60
rwmixwrite=40
bs=4K
direct=0
Quoted 1 line
Does this happen with direct=1?
So far I haven't been able to reproduce it so far with direct=1.
Quoted 4 lines
Quoted 4 lines
numjobs=128
time_based
runtime=900
Quoted 8 lines
Quoted 5 lines
[file1]
size=10G
ioengine=libaio
iodepth=16Is it possible that the backend nvmet device is not fast enough? Are your backend devices nvme? Can you share nvmetcli ls output?
Currently I have a quick-and-dirty setup. The systems I have to test with don't have a spare nvme device right now but do have a lot of RAM, so the backend is a loop device backed by a file in tmpfs. But it should be plenty fast. I'm working on getting a setup with an nvme backend device.
o- / ......................................................................................................................... [...]
o- hosts ................................................................................................................... [...] | o- hostnqn ............................................................................................................... [...] o- ports ................................................................................................................... [...]
| o- 2 ................................................... [trtype=tcp, traddr=..., trsvcid=4420, inline_data_size=16384]
| o- ana_groups .......................................................................................................... [...]
| | o- 1 ..................................................................................................... [state=optimized]
| o- referrals ........................................................................................................... [...]
| o- subsystems .......................................................................................................... [...]
| o- testnqn ........................................................................................................... [...]
o- subsystems .............................................................................................................. [...]
o- testnqn ............................................................. [version=1.3, allow_any=1, serial=2c2e39e2a551f7febf33]
o- allowed_hosts ....................................................................................................... [...]
o- namespaces .......................................................................................................... [...]
o- 1 [path=/dev/loop0, uuid=8a1561fb-82c3-4e9d-96b9-11c7b590d047, nguid=ef90689c-6c46-d44c-89c1-4067801309a8, grpid=1, enabled]Quoted 2 lines
Also, just to understand if things are slow or stuck, perhaps you can artificially increase the
io_timeoutto say 60s or 120s.
I could still reproduce it with io_timeout set to 120s.
Thanks,
Seth
Sagi Grimberg
maintainerQuoted 75 lines · Seth Forshee
Quoted 74 lines · Sagi Grimberg
Hey Seth, thanks for reporting.
Quoted 71 lines
Quoted 53 lines
Quoted 53 lines
Quoted 33 lines
Hi Seth,
Quoted 30 lines · Seth Forshee
Hi,
I'm seeing timeouts like the following from nvme-tcp:
[ 6369.513269] nvme nvme5: queue 102: timeout request 0x73 type 4 [ 6369.513283] nvme nvme5: starting error recovery [ 6369.514379] block nvme5n1: no usable path - requeuing I/O [ 6369.514385] block nvme5n1: no usable path - requeuing I/O [ 6369.514392] block nvme5n1: no usable path - requeuing I/O [ 6369.514393] block nvme5n1: no usable path - requeuing I/O [ 6369.514401] block nvme5n1: no usable path - requeuing I/O [ 6369.514414] block nvme5n1: no usable path - requeuing I/O [ 6369.514420] block nvme5n1: no usable path - requeuing I/O [ 6369.514427] block nvme5n1: no usable path - requeuing I/O [ 6369.514430] block nvme5n1: no usable path - requeuing I/O [ 6369.514432] block nvme5n1: no usable path - requeuing I/O [ 6369.514926] nvme nvme5: Reconnecting in 10 seconds... [ 6379.761015] nvme nvme5: creating 128 I/O queues. [ 6379.944389] nvme nvme5: mapped 128/0/0 default/read/poll queues. [ 6379.947922] nvme nvme5: Successfully reconnected (1 attempt)This is with 6.0, using nvmet-tcp on a different machine as the target. I've seen this sporadically with several test cases. The fio fio-rand-RW example test is a pretty good reproducer when numjobs in increased (I'm setting it equal to the number of CPUs in the system).
Let me know what I can do to help debug this. I'm currently adding some tracing to the driver to see if I can get an idea of the sequence of events that leads to this problem.
Quoted 6 lines
Quoted 3 lines
Thanks,
SethCan you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007nvme-tcp: rerunio_workifreq_listis not empty70f437fb4395nvme-tcp: fixio_workpriority inversion3770a42bb8cenvme-tcp: fix regression that causes sporadic requests to time out5.10 still has timeouts with these backported, so whatever the problem is it has existed at least that long. I suppose I could go back to older kernels with these backported if that's going to be the best path forward here.
Quoted 4 lines
Quoted 2 lines
Thanks,
SethCan you please share the fio config you are using ?
Sure. Note that I can reproduce it with a lower number of numjobs, but higher numbers make it easier, so I set it to the number of CPUs present on the system I'm using to test.
[global]
name=fio-rand-RW
filename=fio-rand-RW
rw=randrw
rwmixread=60
rwmixwrite=40
bs=4K
direct=0
Quoted 3 lines
Quoted 1 line
Does this happen with direct=1?
So far I haven't been able to reproduce it so far with direct=1.
It is probably because the ios are actually going through as 4k and blk-mq tags depth is throttling you. on direct=0 writes are batched and go down to the disk in larger blocks, and in bursts, which probably is creating longer latencies...
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
numjobs=128
time_based
runtime=900[file1]
size=10G
ioengine=libaio
iodepth=16
Quoted 8 lines
Quoted 2 lines
Is it possible that the backend nvmet device is not fast enough? Are your backend devices nvme? Can you share nvmetcli ls output?
Currently I have a quick-and-dirty setup. The systems I have to test with don't have a spare nvme device right now but do have a lot of RAM, so the backend is a loop device backed by a file in tmpfs. But it should be plenty fast. I'm working on getting a setup with an nvme backend device.
I see, note that ram drives are fast, but stall the cpu for memory copy, which will prevent nvme io threads from servicing new requests, this can be magnified with large heavy bursts...
Quoted 5 lines
o- / ......................................................................................................................... [...]
o- hosts ................................................................................................................... [...] | o- hostnqn ............................................................................................................... [...] o- ports ................................................................................................................... [...]
Quoted 11 lines
| o- 2 ................................................... [trtype=tcp, traddr=..., trsvcid=4420, inline_data_size=16384]
| o- ana_groups .......................................................................................................... [...]
| | o- 1 ..................................................................................................... [state=optimized]
| o- referrals ........................................................................................................... [...]
| o- subsystems .......................................................................................................... [...]
| o- testnqn ........................................................................................................... [...]
o- subsystems .............................................................................................................. [...]
o- testnqn ............................................................. [version=1.3, allow_any=1, serial=2c2e39e2a551f7febf33]
o- allowed_hosts ....................................................................................................... [...]
o- namespaces .......................................................................................................... [...]
o- 1 [path=/dev/loop0, uuid=8a1561fb-82c3-4e9d-96b9-11c7b590d047, nguid=ef90689c-6c46-d44c-89c1-4067801309a8, grpid=1, enabled]Ohh, I'd say that would be the culprit...
the loop driver uses only a single queue to access the disk. This means that all your 100+ nvme-tcp queues are all serializing access on the single loop disk queue. This heavy back-pressure bubbles all the way back to the host and manifests in IO timeouts when large bursts hit...
I can say that loop is not the best way to benchmark performance, and I'd expect to see such phenomenons when attempting to drive high loads to a loop device...
Maybe you can possibly use a tmpfs file directly instead (nvmet supports file backends as well).
Or maybe you can try to use null_blk with memory_backed=Y modparam (may need to define cache_size modparam as well, never tried it with memory backing...)? That would be more efficient.
Quoted 22 lines · Sagi Grimberg
Quoted 4 lines
o- / ......................................................................................................................... [...]
o- hosts ................................................................................................................... [...] | o- hostnqn ............................................................................................................... [...] o- ports ................................................................................................................... [...]Quoted 11 lines
| o- 2 ................................................... [trtype=tcp, traddr=..., trsvcid=4420, inline_data_size=16384] | o- ana_groups .......................................................................................................... [...] | | o- 1 ..................................................................................................... [state=optimized] | o- referrals ........................................................................................................... [...] | o- subsystems .......................................................................................................... [...] | o- testnqn ........................................................................................................... [...] o- subsystems .............................................................................................................. [...] o- testnqn ............................................................. [version=1.3, allow_any=1, serial=2c2e39e2a551f7febf33] o- allowed_hosts ....................................................................................................... [...] o- namespaces .......................................................................................................... [...] o- 1 [path=/dev/loop0, uuid=8a1561fb-82c3-4e9d-96b9-11c7b590d047, nguid=ef90689c-6c46-d44c-89c1-4067801309a8, grpid=1, enabled]Ohh, I'd say that would be the culprit...
the loop driver uses only a single queue to access the disk. This means that all your 100+ nvme-tcp queues are all serializing access on the single loop disk queue. This heavy back-pressure bubbles all the way
back to the host and manifests in IO timeouts when large bursts hit...
Quoted 3 lines
I can say that loop is not the best way to benchmark performance, and I'd expect to see such phenomenons when attempting to drive high loads to a loop device...
The goal wasn't to benchmark performance with this setup, just to start getting familiar.
Quoted 6 lines
Maybe you can possibly use a tmpfs file directly instead (nvmet supports file backends as well).
Or maybe you can try to use
null_blkwith memory_backed=Y modparam (may need to definecache_sizemodparam as well, never tried it with memory backing...)? That would be more efficient.
I've got this set up now with an nvme drive as the backend for the target, and as you predicted the timeouts went away. So it does seem the problem was with using a loop device. Thanks for the help!
Seth
Sagi Grimberg
maintainerQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Can you bisect it ? that will help to understand the commit causing issue.
I don't know of any "good" version right now. I started with a 5.10 kernel and saw this, and tested 6.0 and still see it. I found several commits since 5.10 which fix some kind of timeouts:
a0fdd1418007nvme-tcp: rerunio_workifreq_listis not empty70f437fb4395nvme-tcp: fixio_workpriority inversion3770a42bb8cenvme-tcp: fix regression that causes sporadic requests to time out
btw Seth,
I've just looked at stable v5.10.147 and all of these fixes were taken to 5.10 stable branch.
If you happen to use a 5.10 stable kernel (or a modified version of it), let us know if you spot any missing fixes so we'll make sure they end up there as well.