iscsi issue of scale with MNoT
Subsystems (2)
[RESEND -- apologies if you see this more than once]
The iSCSI protocol continues to be used in Linux, but some of the users push the system past its normal limits. And using multipath just exacerbates that problem (usually doubling the number of sessions).
I'd like to gather some numbers for open-iscsi (the standard Linux iSCSI initiator) and the kernel target code (i.e. LIO/targetcli) on what happens when there are MNoT -- massive numbers of targets.
"Massive" in my case, will be relative, since I don't have access to a supercomputer, but I believe it will not be too hard to start pushing the system too far. For example, a recent user problem found that even at 2000 sessions using multipath, the system takes about 80 seconds to switch paths. Each switch takes 80ms (and they are currently serialized), but when you multiply that by 1000 it adds up.
For the initiator, I've long suspected some parts of the code were not designed for scale, so this might give me a chance to find and possibly address some of these issues.
--
Lee Duncan
Quoted 18 lines · On Thu, Feb 17, 2022 at 10:19 AM <[email protected]> wrote:
[RESEND -- apologies if you see this more than once]
The iSCSI protocol continues to be used in Linux, but some of the users push the system past its normal limits. And using multipath just exacerbates that problem (usually doubling the number of sessions).
I'd like to gather some numbers for open-iscsi (the standard Linux iSCSI initiator) and the kernel target code (i.e. LIO/targetcli) on what happens when there are MNoT -- massive numbers of targets.
"Massive" in my case, will be relative, since I don't have access to a supercomputer, but I believe it will not be too hard to start pushing the system too far. For example, a recent user problem found that even at 2000 sessions using multipath, the system takes about 80 seconds to switch paths. Each switch takes 80ms (and they are currently serialized), but when you multiply that by 1000 it adds up.
Quoted 3 lines
For the initiator, I've long suspected some parts of the code were not designed for scale, so this might give me a chance to find and possibly address some of these issues.
There are some linear lookups (sess_list, conn_list) in the iSCSI netlink which may be low hanging fruit, and do show up in heatmaps when creating/destroying sessions/connections in the presence of 10k+ sessions in a machine. (Though this doesn't manifest 80s+ stalls, thankfully)
Quoted 3 lines
--
Lee Duncan