Skip to contentSkip to conversation

NVMe-TCP performance improvements

LSF/MM/BPF TOPIC Hannes Reinecke 1 participant 1 message Mar 21, 2025 NVMe

lore
Share & tools
lore
#0 lore raw

Hi all,

a partner of ours reported a severe performance and scalability issue with TLS on NVMe-TCP, where spurious I/O timeouts have been occurring once the system scales up to more controllers. With enough controllers he even ran into timeouts during the initial connect command, rendering
the system unusable.
There have been several attempts to fix this (my attempts [1], spreading
load to CPUs by Sagi [2]), but they didn't really fix the issue.
I have since then done quite some performance analysis, but found that
'classical' statistical methods can't easily used as we have a really
high divergence in performance; the standard deviation vastly exceeds
the mean value, rendering the calculation meaningless.
Eventually I nailed down the problem with [3], but the performance analysis problem remains.
In the discussion I would like to present the performance analysis,
and discuss ways on how the analyze the performance in the light of
high noise in the measurements.

[1] https://lore.kernel.org/linux-nvme/[email protected]/#r
[2] https://lore.kernel.org/linux-nvme/[email protected]/
[3] https://lore.kernel.org/linux-nvme/[email protected]/

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich