Skip to contentSkip to conversation

NVMe over MPTCP: Multi-Fold Acceleration for NVMe over TCP in Multi-NIC Environments

LSF/MM/BPF TOPIC Geliang Tang 4 participants 10 messages Jan 29 – May 28

Subsystems (2)
lore
Share & tools
lore

Geliang Tang

author reviewer
#0 lore raw

As one of the MPTCP upstream developers, I'm recently working on adding MPTCP support to 'NVMe over TCP'. This approach achieves a multi-fold performance improvement over using standard TCP. The implementation and testing phases are largely complete. The code is currently in the RFC stage and has undergone several rounds of discussion and iteration on the MPTCP mailing list [1]. It will be sent to the NVMe mailing list shortly.

1. Introduction to MPTCP

Multipath TCP (MPTCP), standardized in RFC 8684, represents a major evolution of the TCP protocol. It enables a single transport connection to utilize multiple network paths simultaneously, providing benefits in redundancy, resilience, and bandwidth aggregation. Since its introduction in Linux kernel v5.6, it has become a key technology for modern networking, particularly in multi-NIC environments.

On a supported system such as Linux, an MPTCP socket is created by specifying the IPPROTO_MPTCP protocol in the socket() system call:

	int fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

This creates a socket that appears as a standard TCP socket to the application but uses the MPTCP protocol stack underneath.

For more details, please visit the project website: https://mptcp.dev.

2. Implementation

'NVMe over TCP' establishes multiple TCP connections between the target and host for data transfer. This includes one admin queue connection for management traffic and multiple I/O queue connections for data traffic, with the number typically scaling with available CPU cores. While these multiple TCP connections (using the same IP address but different ports) help distribute computational load across CPUs, all data traffic still flows through a single network interface card (NIC), even in multi-NIC environments.

The 'NVMe over MPTCP' solution enhances 'NVMe over TCP' by replacing the multiple TCP connections with multiple MPTCP connections, leaving other mechanisms unchanged. Internally, each MPTCP connection can establish multiple subflows based on the number of configured NICs. This distributes data traffic across all available NICs, thereby increasing aggregate transmission speed.

Therefore, the primary change required is to modify the protocol parameter from IPPROTO_TCP to IPPROTO_MPTCP when creating sockets on both the target and host sides:

	Target side:

	sock_create(port->addr.ss_family, SOCK_STREAM,
			IPPROTO_TCP, &port->sock);

	Host side:

	sock_create_kern(current->nsproxy->net_ns,
			ctrl->addr.ss_family, SOCK_STREAM,
			IPPROTO_TCP, &queue->sock);

A new NVMe transport type, named NVMF_TRTYPE_MPTCP (suggested by Hannes Reinecke), has been introduced to determine whether to create a TCP or MPTCP socket:

	Target side:

	if (nport->disc_addr.trtype == NVMF_TRTYPE_MPTCP)
		proto = IPPROTO_MPTCP;

	Host side:

	if (!strcmp(ctrl->ctrl.opts->transport, "mptcp"))
		proto = IPPROTO_MPTCP;

3. Performance Benefits

This new feature has been evaluated in different environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped with two Gigabit NICs and directly connected via Ethernet cables. Using 'NVMe over TCP', the fio benchmark showed a speed of approximately 100 MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s with fio, doubling the throughput.

In a virtual machine test environment simulating four NICs on both sides, 'NVMe over MPTCP' delivered bandwidth up to four times that of standard TCP.

4. Configuration

To achieve the described multi-fold acceleration benefits, both the target and host sides must be deployed in multi-NIC environments with properly configured MPTCP endpoints. The target side should use the 'signal' flag for its endpoints, while the host side should use the 'subflow' flag.

Target side:

# ip mptcp endpoint add 192.168.1.2 id 2 dev enp3s0f1 signal # echo mptcp > /sys/kernel/config/nvmet/ports/1234/addr_trtype

Host side:

# ip mptcp endpoint add 192.168.1.4 id 2 dev enp1s0f1 subflow # nvme discover -t mptcp ...
# nvme connect -t mptcp ...

5. Dependencies

The modifications in the NVMe subsystem are minimal. Most of the code change is on the MPTCP side, to implement interfaces to use MPTCP from the kernel space, similar to what is done today with TCP.

'NVMe over TCP' uses the read_sock interface for data receiving. Consequently, the read_sock interface for MPTCP has been implemented ('implement mptcp read_sock' series [2]), which is under review on the MPTCP mailing list.

As 'NVMe over TCP' can utilize TLS for encryption, KTLS support for MPTCP has also been added ('MPTCP KTLS support' series [3]), which is currently in the RFC stage. TLS mode for 'NVMe over MPTCP' has been successfully validated.

Corresponding updates are also required in user-space libraries and tools, including libnvme [4], nvme-cli [5], and ktls-utils [6], to add MPTCP support.

6. Discussion

The current approach is to define a new transport type, NVMF_TRTYPE_MPTCP, but I understand it will need to register a new protocol number from NVMexpress.org. Hannes Reinecke also suggested declaring MPTCP as a TCP 'variant', but I found some drawbacks. I would like to discuss them and find possible solutions.

I also seek guidance on how to incorporate MPTCP support into the NVMe protocol specifications. I lack experience in modifying NVMe protocol specifications and would appreciate guidance and assistance from the NVMe community.

Thanks,
-Geliang

[1]
NVME over MPTCP
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
[2]
implement mptcp read_sock
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
[3]
MPTCP KTLS support
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/
[4]
libnvme: add mptcp trtype
https://patchwork.kernel.org/project/mptcp/patch/99f6e63b5c9677f29a9bc8cdd87b2064b258435f.1764206766.git.tanggeliang@kylinos.cn/
[5]
fabrics: add mptcp support
https://github.com/linux-nvme/nvme-cli/commit/f468531d0592ad22b71760d883409363b1f8a9d6
[6]
add mptcp support
https://github.com/oracle/ktls-utils/commit/4a45e486c65be986ef349ed10b0fc9bd5dbf107d

#1 lore raw

Hi Geliang,

Looks one interesting topic!

Quoted 38 lines · Geliang Tang

As one of the MPTCP upstream developers, I'm recently working on adding MPTCP support to 'NVMe over TCP'. This approach achieves a multi-fold performance improvement over using standard TCP. The implementation and testing phases are largely complete. The code is currently in the RFC stage and has undergone several rounds of discussion and iteration on the MPTCP mailing list [1]. It will be sent to the NVMe mailing list shortly.

1. Introduction to MPTCP

Multipath TCP (MPTCP), standardized in RFC 8684, represents a major evolution of the TCP protocol. It enables a single transport connection to utilize multiple network paths simultaneously, providing benefits in redundancy, resilience, and bandwidth aggregation. Since its introduction in Linux kernel v5.6, it has become a key technology for modern networking, particularly in multi-NIC environments.

On a supported system such as Linux, an MPTCP socket is created by specifying the IPPROTO_MPTCP protocol in the socket() system call:

	int fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

This creates a socket that appears as a standard TCP socket to the application but uses the MPTCP protocol stack underneath.

For more details, please visit the project website: https://mptcp.dev.

2. Implementation

'NVMe over TCP' establishes multiple TCP connections between the target and host for data transfer. This includes one admin queue connection for management traffic and multiple I/O queue connections for data traffic, with the number typically scaling with available CPU cores. While these multiple TCP connections (using the same IP address but different ports) help distribute computational load across CPUs, all data traffic still flows through a single network interface card (NIC), even in multi-NIC environments.

Quoted 6 lines

The 'NVMe over MPTCP' solution enhances 'NVMe over TCP' by replacing the multiple TCP connections with multiple MPTCP connections, leaving other mechanisms unchanged. Internally, each MPTCP connection can establish multiple subflows based on the number of configured NICs. This distributes data traffic across all available NICs, thereby increasing aggregate transmission speed.

NVMe supports multipath, which can apply load balance or sort of algorithm to maximize network link/bandwidth too.

Maybe you can compare mptcp with multipath in this viewpoint.

Thanks,
Ming

author reviewer
#2 lore raw

Hi Ming,

Quoted 3 lines · Ming Lei

Hi Geliang,

Looks one interesting topic!

Thanks for your reply.

Quoted 64 lines · Geliang Tang
Quoted 53 lines

As one of the MPTCP upstream developers, I'm recently working on adding
MPTCP support to 'NVMe over TCP'. This approach achieves a multi- fold
performance improvement over using standard TCP. The implementation and
testing phases are largely complete. The code is currently in the RFC
stage and has undergone several rounds of discussion and iteration on
the MPTCP mailing list [1]. It will be sent to the NVMe mailing list
shortly.

1. Introduction to MPTCP

Multipath TCP (MPTCP), standardized in RFC 8684, represents a major evolution of the TCP protocol. It enables a single transport connection
to utilize multiple network paths simultaneously, providing benefits in
redundancy, resilience, and bandwidth aggregation. Since its introduction in Linux kernel v5.6, it has become a key technology for
modern networking, particularly in multi-NIC environments.

On a supported system such as Linux, an MPTCP socket is created by specifying the IPPROTO_MPTCP protocol in the socket() system call:

	int fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

This creates a socket that appears as a standard TCP socket to the application but uses the MPTCP protocol stack underneath.

For more details, please visit the project website:
https://mptcp.dev.

2. Implementation

'NVMe over TCP' establishes multiple TCP connections between the target
and host for data transfer. This includes one admin queue connection
for management traffic and multiple I/O queue connections for data traffic, with the number typically scaling with available CPU cores.
While these multiple TCP connections (using the same IP address but different ports) help distribute computational load across CPUs, all
data traffic still flows through a single network interface card (NIC),
even in multi-NIC environments.

Quoted 8 lines

The 'NVMe over MPTCP' solution enhances 'NVMe over TCP' by replacing
the multiple TCP connections with multiple MPTCP connections, leaving
other mechanisms unchanged. Internally, each MPTCP connection can establish multiple subflows based on the number of configured NICs. This distributes data traffic across all available NICs, thereby increasing aggregate transmission speed.

Quoted 5 lines

NVMe supports multipath, which can apply load balance or sort of algorithm
to maximize network link/bandwidth too.

Maybe you can compare mptcp with multipath in this viewpoint.

Indeed worth comparing. Although they work at different layers, their goals share similarities. I'll compare them in a follow-up and get back to you.

Thanks,
-Geliang

Quoted 6 lines

Thanks,
Ming

#3 lore raw
In reply to Geliang Tang
Quoted 10 lines · Geliang Tang

3. Performance Benefits

This new feature has been evaluated in different environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped with two Gigabit NICs and directly connected via Ethernet cables. Using 'NVMe over TCP', the fio benchmark showed a speed of approximately 100 MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s with fio, doubling the throughput.

Quoted 3 lines

In a virtual machine test environment simulating four NICs on both sides, 'NVMe over MPTCP' delivered bandwidth up to four times that of standard TCP.

This is interesting. Did you try using an NVMe multipath iopolicy other than the default numa policy? Assuming both the host and target are multihomed, configuring round-robin or queue-depth may provide performance comparable to what you are seeing with MPTCP.

I think MPTCP shall distribute traffic using transport-level metrics such as RTT, cwnd, and packet loss, whereas the NVMe multipath layer makes decisions based on ANA state, queue depth, and NUMA locality. In a setup with multiple active paths, switching the iopolicy from numa to round-robin or queue-depth could improve load distribution across controllers and thus improve performance.

IMO, it would be useful to test with those policies and compare the results against the MPTCP setup.

Thanks,
--Nilay

author reviewer
#4 lore raw

Hi Nilay,

Thanks for your reply.

Quoted 39 lines · Nilay Shroff
Quoted 12 lines · Geliang Tang

3. Performance Benefits

This new feature has been evaluated in different environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped with
two Gigabit NICs and directly connected via Ethernet cables. Using 'NVMe over TCP', the fio benchmark showed a speed of approximately 100
MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s with fio, doubling the throughput.

Quoted 4 lines

In a virtual machine test environment simulating four NICs on both sides, 'NVMe over MPTCP' delivered bandwidth up to four times that of
standard TCP.

This is interesting. Did you try using an NVMe multipath iopolicy other
than the default numa policy? Assuming both the host and target are multihomed,
configuring round-robin or queue-depth may provide performance comparable
to what you are seeing with MPTCP.

I think MPTCP shall distribute traffic using transport-level metrics such as
RTT, cwnd, and packet loss, whereas the NVMe multipath layer makes decisions
based on ANA state, queue depth, and NUMA locality. In a setup with multiple
active paths, switching the iopolicy from numa to round-robin or queue-depth
could improve load distribution across controllers and thus improve performance.

Quoted 3 lines

IMO, it would be useful to test with those policies and compare the results
against the MPTCP setup.

Ming Lei also made a similar comment. In my experiments, I didn't set the multipath iopolicy, so I was using the default numa policy. In the follow-up, I'll adjust it to round-robin or queue-depth and rerun the experiments. I'll share the results in this email thread.

Thanks,
-Geliang

Quoted 3 lines

Thanks,
--Nilay

author reviewer
#5 lore raw

Hi Nilay, Ming,

Thank you again for your interest in NVMe over MPTCP.

Quoted 55 lines · Geliang Tang

Hi Nilay,

Thanks for your reply.

Quoted 46 lines · Nilay Shroff
Quoted 16 lines · Geliang Tang

3. Performance Benefits

This new feature has been evaluated in different environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped
with
two Gigabit NICs and directly connected via Ethernet cables. Using
'NVMe over TCP', the fio benchmark showed a speed of approximately
100
MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s with
fio, doubling the throughput.

Quoted 6 lines

In a virtual machine test environment simulating four NICs on both
sides, 'NVMe over MPTCP' delivered bandwidth up to four times that
of
standard TCP.

This is interesting. Did you try using an NVMe multipath iopolicy other
than the default numa policy? Assuming both the host and target are multihomed,
configuring round-robin or queue-depth may provide performance comparable
to what you are seeing with MPTCP.

I think MPTCP shall distribute traffic using transport-level metrics
such as
RTT, cwnd, and packet loss, whereas the NVMe multipath layer makes decisions
based on ANA state, queue depth, and NUMA locality. In a setup with multiple
active paths, switching the iopolicy from numa to round-robin or queue-depth
could improve load distribution across controllers and thus improve performance.

Quoted 3 lines

IMO, it would be useful to test with those policies and compare the results
against the MPTCP setup.

Quoted 5 lines

Ming Lei also made a similar comment. In my experiments, I didn't set the multipath iopolicy, so I was using the default numa policy. In the
follow-up, I'll adjust it to round-robin or queue-depth and rerun the experiments. I'll share the results in this email thread.

Based on your feedback, I have added iopolicy support to the NVMe over MPTCP selftest script (see patch 8 in [1]). We can set the iopolicy to round-robin like this:

 # ./mptcp_nvme.sh mptcp round-robin

This demonstrates that "NVMe over MPTCP" and "NVMe multipath" can work simultaneously without conflict.

Using this test script, I compared three I/O policies: numa, round- robin, and queue-depth. The results for fio were very similar. It's possible that this test environment doesn't fully reflect the differences in I/O policies. I will continue to follow up with further tests.

Thanks,
-Geliang

[1]
NVME over MPTCP, v4
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Quoted 7 lines

Thanks,
-Geliang

Quoted 3 lines

Thanks,
--Nilay

author reviewer
#6 lore raw

Hello everyone,

Thank you for your interest in NVMe over MPTCP. I have attached the slides from the presentation to this email.

Please note that the demo in the slides only configured a single NVMe multipath. Subsequently, I will post the MPTCP performance test results under several NVMe multipaths here.

Thanks,
-Geliang

Quoted 106 lines · Geliang Tang

Hi Nilay, Ming,

Thank you again for your interest in NVMe over MPTCP.

Quoted 60 lines · Geliang Tang

Hi Nilay,

Thanks for your reply.

Quoted 50 lines · Nilay Shroff
Quoted 16 lines · Geliang Tang

3. Performance Benefits

This new feature has been evaluated in different environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped
with
two Gigabit NICs and directly connected via Ethernet cables. Using
'NVMe over TCP', the fio benchmark showed a speed of approximately
100
MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s with
fio, doubling the throughput.

Quoted 6 lines

In a virtual machine test environment simulating four NICs on both
sides, 'NVMe over MPTCP' delivered bandwidth up to four times that
of
standard TCP.

This is interesting. Did you try using an NVMe multipath iopolicy other
than the default numa policy? Assuming both the host and target are
multihomed,
configuring round-robin or queue-depth may provide performance comparable
to what you are seeing with MPTCP.

I think MPTCP shall distribute traffic using transport-level metrics
such as
RTT, cwnd, and packet loss, whereas the NVMe multipath layer makes
decisions
based on ANA state, queue depth, and NUMA locality. In a setup with
multiple
active paths, switching the iopolicy from numa to round-robin or queue-depth
could improve load distribution across controllers and thus improve
performance.

Quoted 4 lines

IMO, it would be useful to test with those policies and compare the
results
against the MPTCP setup.

Quoted 7 lines

Ming Lei also made a similar comment. In my experiments, I didn't set
the multipath iopolicy, so I was using the default numa policy. In the
follow-up, I'll adjust it to round-robin or queue-depth and rerun the
experiments. I'll share the results in this email thread.

Based on your feedback, I have added iopolicy support to the NVMe over
MPTCP selftest script (see patch 8 in [1]). We can set the iopolicy to
round-robin like this:

 # ./mptcp_nvme.sh mptcp round-robin

This demonstrates that "NVMe over MPTCP" and "NVMe multipath" can work
simultaneously without conflict.

Using this test script, I compared three I/O policies: numa, round- robin, and queue-depth. The results for fio were very similar. It's possible that this test environment doesn't fully reflect the differences in I/O policies. I will continue to follow up with further
tests.

Thanks,
-Geliang

[1]
NVME over MPTCP, v4
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Quoted 7 lines

Thanks,
-Geliang

Quoted 3 lines

Thanks,
--Nilay

author reviewer
#7 lore raw

Hi,

The performance test results of MPTCP under several NVMe multipath settings are now ready.

Quoted 9 lines · Geliang Tang

Hello everyone,

Thank you for your interest in NVMe over MPTCP. I have attached the slides from the presentation to this email.

Please note that the demo in the slides only configured a single NVMe multipath. Subsequently, I will post the MPTCP performance test results
under several NVMe multipaths here.

To test the performance of TCP and MPTCP under NVMe multipath, I added two more arguments, "path" and "loss", to the original NVMe MPTCP self test script. The latest code is available at [1].

The script now accepts the following four arguments:

mptcp_nvme.sh [trtype] [path] [iopolicy] [loss]

trtype Transport type (tcp|mptcp) - default: mptcp
path Number of multipath (1-4) - default: 1
iopolicy I/O policy (numa|round-robin|queue-depth) - default: numa
loss Enable packet loss (0|1) - default: 0

The first argument is the transport type. The second argument, "path", specifies how many NVMe multipaths to create. The third argument is the I/O policy. The fourth argument controls whether the network environment is lossy. When set to 0, each NIC is rate-limited to 125 MB/s (tc arguments: rate 1000mbit). When set to 1, in addition to the same rate limit of 125 MB/s, each NIC also experiences a 5 ms delay and 0.5% packet loss (tc arguments: rate 1000mbit delay 5ms loss 0.5%).

First set of tests: lossless network, path=4, loss=0. The tc output is as follows:

qdisc netem 8031: root refcnt 25 limit 1000 rate 1Gbit

			seed 1626193586047356330

Lossless network, comparison between TCP and MPTCP using the "numa" policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 numa 0
   READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s),
			io=1200MiB (1259MB), run=10533-10533msec
  WRITE: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s),
			io=1203MiB (1261MB), run=10570-10570msec
# ./mptcp_nvme.sh mptcp 4 numa 0
   READ: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
			io=4512MiB (4731MB), run=10130-10130msec
  WRITE: bw=443MiB/s (465MB/s), 443MiB/s-443MiB/s (465MB/s-465MB/s),
			io=4504MiB (4723MB), run=10158-10158msec

Lossless network, comparison between TCP and MPTCP using the "round- robin" policy - MPTCP and TCP show similar performance:

# ./mptcp_nvme.sh tcp 4 round-robin 0
   READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s),
			io=4683MiB (4910MB), run=10278-10278msec
  WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s),
			io=4660MiB (4887MB), run=10239-10239msec
# ./mptcp_nvme.sh mptcp 4 round-robin 0
   READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s),
			io=4565MiB (4786MB), run=10239-10239msec
  WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
			io=4575MiB (4797MB), run=10280-10280msec

Lossless network, comparison between TCP and MPTCP using the "queue- depth" policy - MPTCP and TCP show similar performance:

# ./mptcp_nvme.sh tcp 4 queue-depth 0
   READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s),
			io=4632MiB (4857MB), run=10169-10169msec
  WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s),
			io=4666MiB (4893MB), run=10250-10250msec
# ./mptcp_nvme.sh mptcp 4 queue-depth 0
   READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s),
			io=4568MiB (4790MB), run=10249-10249msec
  WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
			io=4563MiB (4784MB), run=10245-10245msec

Second set of tests: lossy network, path=4, loss=1. The tc output is as follows:

qdisc netem 8051: root refcnt 25 limit 1000 delay 5ms loss 0.5%

			rate 1Gbit seed 14946049878654165618

Lossy network, comparison between TCP and MPTCP using the "round-robin" policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 round-robin 1
   READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s),
			io=1574MiB (1650MB), run=14906-14906msec
  WRITE: bw=98.5MiB/s (103MB/s), 98.5MiB/s-98.5MiB/s (103MB/s-103MB/s),
			io=1455MiB (1526MB), run=14770-14770msec
# ./mptcp_nvme.sh mptcp 4 round-robin 1
   READ: bw=426MiB/s (447MB/s), 426MiB/s-426MiB/s (447MB/s-447MB/s),
			io=4533MiB (4753MB), run=10637-10637msec
  WRITE: bw=428MiB/s (449MB/s), 428MiB/s-428MiB/s (449MB/s-449MB/s),
			io=4507MiB (4725MB), run=10522-10522msec

Lossy network, comparison between TCP and MPTCP using the "queue-depth" policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 queue-depth 1
   READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s),
			io=2179MiB (2285MB), run=12965-12965msec
  WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s),
			io=1590MiB (1667MB), run=12418-12418msec
# ./mptcp_nvme.sh mptcp 4 queue-depth 1
   READ: bw=425MiB/s (445MB/s), 425MiB/s-425MiB/s (445MB/s-445MB/s),
			io=4536MiB (4756MB), run=10677-10677msec
  WRITE: bw=414MiB/s (434MB/s), 414MiB/s-414MiB/s (434MB/s-434MB/s),
			io=4447MiB (4663MB), run=10733-10733msec

Conclusion: MPTCP achieves bandwidth aggregation comparable to that of NVMe multipath while offering better resilience against network interference.

The full test results are in the attachment.

Thanks,
-Geliang

[1]
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Quoted 116 lines

Thanks,
-Geliang

Quoted 111 lines · Geliang Tang

Hi Nilay, Ming,

Thank you again for your interest in NVMe over MPTCP.

Quoted 64 lines · Geliang Tang

Hi Nilay,

Thanks for your reply.

Quoted 54 lines · Nilay Shroff
Quoted 18 lines · Geliang Tang

3. Performance Benefits

This new feature has been evaluated in different
environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped
with
two Gigabit NICs and directly connected via Ethernet cables. Using
'NVMe over TCP', the fio benchmark showed a speed of approximately
100
MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s
with
fio, doubling the throughput.

Quoted 6 lines

In a virtual machine test environment simulating four NICs on both
sides, 'NVMe over MPTCP' delivered bandwidth up to four times that
of
standard TCP.

This is interesting. Did you try using an NVMe multipath iopolicy
other
than the default numa policy? Assuming both the host and target are
multihomed,
configuring round-robin or queue-depth may provide performance comparable
to what you are seeing with MPTCP.

I think MPTCP shall distribute traffic using transport-level metrics
such as
RTT, cwnd, and packet loss, whereas the NVMe multipath layer makes
decisions
based on ANA state, queue depth, and NUMA locality. In a setup with
multiple
active paths, switching the iopolicy from numa to round-robin or
queue-depth
could improve load distribution across controllers and thus improve
performance.

Quoted 4 lines

IMO, it would be useful to test with those policies and compare the
results
against the MPTCP setup.

Quoted 8 lines

Ming Lei also made a similar comment. In my experiments, I didn't set
the multipath iopolicy, so I was using the default numa policy. In
the
follow-up, I'll adjust it to round-robin or queue-depth and rerun the
experiments. I'll share the results in this email thread.

Based on your feedback, I have added iopolicy support to the NVMe over
MPTCP selftest script (see patch 8 in [1]). We can set the iopolicy to
round-robin like this:

 # ./mptcp_nvme.sh mptcp round-robin

This demonstrates that "NVMe over MPTCP" and "NVMe multipath" can work
simultaneously without conflict.

Using this test script, I compared three I/O policies: numa, round- robin, and queue-depth. The results for fio were very similar. It's possible that this test environment doesn't fully reflect the differences in I/O policies. I will continue to follow up with further
tests.

Thanks,
-Geliang

[1]
NVME over MPTCP, v4
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Quoted 7 lines

Thanks,
-Geliang

Quoted 3 lines

Thanks,
--Nilay

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh tcp 4 numa 0
0+0 records in
0+0 records out
0 bytes copied, 0.000033553 s, 0.0 B/s
qdisc netem 8031: root refcnt 25 limit 1000 rate 1Gbit seed 1626193586047356330
qdisc netem 8032: root refcnt 25 limit 1000 rate 1Gbit seed 6042711589015947724
qdisc netem 8033: root refcnt 25 limit 1000 rate 1Gbit seed 17633132008389812116
qdisc netem 8034: root refcnt 25 limit 1000 rate 1Gbit seed 7224188079500556986
qdisc netem 8035: root refcnt 25 limit 1000 rate 1Gbit seed 9443293930724710079
qdisc netem 8036: root refcnt 25 limit 1000 rate 1Gbit seed 16373570367022413035
qdisc netem 8037: root refcnt 25 limit 1000 rate 1Gbit seed 13985178702777805862
qdisc netem 8038: root refcnt 25 limit 1000 rate 1Gbit seed 17161966827319017756
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 47 =====Discovery Log Entry 0======

trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  20316
trsvcid: 23502
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  20316
trsvcid: 23502
subnqn:  nqn.2014-08.org.nvmexpress.tcpdev.6825.11970
traddr:  10.1.1.1
eflags:  none
sectype: none

Connecting to 10.1.1.1:23502
connecting to device: nvme1
Connecting to 10.1.2.1:23502
connecting to device: nvme2
Connecting to 10.1.3.1:23502
connecting to device: nvme3
Connecting to 10.1.4.1:23502
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            10da529f68f6534c815e Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=114MiB/s][r=913 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=6996: Tue May 19 14:28:47 2026
read: IOPS=911, BW=114MiB/s (119MB/s)(1200MiB/10533msec)

    slat (usec): min=2, max=962951, avg=4180.26, stdev=24042.85
    clat (msec): min=88, max=3807, avg=1090.51, stdev=670.77
     lat (msec): min=88, max=3807, avg=1094.69, stdev=672.21
    clat percentiles (msec):
     |  1.00th=[  384],  5.00th=[  447], 10.00th=[  481], 20.00th=[  600],
     | 30.00th=[  718], 40.00th=[  835], 50.00th=[  894], 60.00th=[  969],
     | 70.00th=[ 1099], 80.00th=[ 1368], 90.00th=[ 2106], 95.00th=[ 2769],
     | 99.00th=[ 3306], 99.50th=[ 3473], 99.90th=[ 3742], 99.95th=[ 3809],
     | 99.99th=[ 3809]
   bw (  KiB/s): min=11264, max=253440, per=99.26%, avg=115826.65, stdev=17233.15, samples=76
   iops        : min=   88, max= 1980, avg=904.90, stdev=134.63, samples=76
  lat (msec)   : 100=0.06%, 250=0.33%, 500=11.35%, 750=20.27%, 1000=31.71%
  lat (msec)   : 2000=24.87%, >=2000=11.40%
  cpu          : usr=0.01%, sys=0.20%, ctx=10293, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.7%, 32=1.3%, >=64=97.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=9602,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=1200MiB (1259MB), run=10533-10533msec

Disk stats (read/write):
nvme1n1: ios=9467/0, sectors=2423552/0, merge=0/0, ticks=5060639/0, in_queue=5060639, util=99.12%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=109MiB/s][w=874 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=7029: Tue May 19 14:28:59 2026
write: IOPS=910, BW=114MiB/s (119MB/s)(1203MiB/10570msec); 0 zone resets

    slat (usec): min=2, max=1095.4k, avg=4161.24, stdev=28251.90
    clat (msec): min=176, max=9210, avg=1088.71, stdev=1244.47
     lat (msec): min=178, max=9210, avg=1092.87, stdev=1247.38
    clat percentiles (msec):
     |  1.00th=[  259],  5.00th=[  363], 10.00th=[  405], 20.00th=[  481],
     | 30.00th=[  531], 40.00th=[  609], 50.00th=[  701], 60.00th=[  835],
     | 70.00th=[ 1045], 80.00th=[ 1301], 90.00th=[ 2072], 95.00th=[ 2903],
     | 99.00th=[ 8087], 99.50th=[ 9194], 99.90th=[ 9194], 99.95th=[ 9194],
     | 99.99th=[ 9194]
   bw (  KiB/s): min=11008, max=305152, per=100.00%, avg=119677.01, stdev=22187.64, samples=74
   iops        : min=   86, max= 2384, avg=934.98, stdev=173.34, samples=74
  lat (msec)   : 250=0.85%, 500=23.29%, 750=30.67%, 1000=13.58%, 2000=20.92%
  lat (msec)   : >=2000=10.69%
  cpu          : usr=0.08%, sys=0.11%, ctx=6954, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.7%, 32=1.3%, >=64=97.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,9623,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=1203MiB (1261MB), run=10570-10570msec

Disk stats (read/write):
nvme1n1: ios=56/9456, sectors=1984/2420736, merge=0/0, ticks=76/4946676, in_queue=4946752, util=100.00%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over tcp test # time=25025ms NQN:nqn.2014-08.org.nvmexpress.tcpdev.6825.11970 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh mptcp 4 numa 0
0+0 records in
0+0 records out
0 bytes copied, 0.000035046 s, 0.0 B/s
qdisc netem 8039: root refcnt 25 limit 1000 rate 1Gbit seed 6085134754699925076
qdisc netem 803a: root refcnt 25 limit 1000 rate 1Gbit seed 450455490517112707
qdisc netem 803b: root refcnt 25 limit 1000 rate 1Gbit seed 264120141577159822
qdisc netem 803c: root refcnt 25 limit 1000 rate 1Gbit seed 14027700267729747694
qdisc netem 803d: root refcnt 25 limit 1000 rate 1Gbit seed 1021893091432102865
qdisc netem 803e: root refcnt 25 limit 1000 rate 1Gbit seed 9767710148348395904
qdisc netem 803f: root refcnt 25 limit 1000 rate 1Gbit seed 10286576419486170728
qdisc netem 8040: root refcnt 25 limit 1000 rate 1Gbit seed 12482312460578339362
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 56 =====Discovery Log Entry 0======

trtype:  unrecognized
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  24021
trsvcid: 5862
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
=====Discovery Log Entry 1======
trtype:  unrecognized
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  24021
trsvcid: 5862
subnqn:  nqn.2014-08.org.nvmexpress.mptcpdev.7084.21368
traddr:  10.1.1.1
eflags:  none

Connecting to 10.1.1.1:5862
connecting to device: nvme1
Connecting to 10.1.2.1:5862
connecting to device: nvme2
Connecting to 10.1.3.1:5862
connecting to device: nvme3
Connecting to 10.1.4.1:5862
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            2641199e93532d5f5106 Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=442MiB/s][r=3532 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=7255: Tue May 19 14:29:20 2026
read: IOPS=3563, BW=445MiB/s (467MB/s)(4512MiB/10130msec)

    slat (usec): min=2, max=385056, avg=1097.92, stdev=8410.04
    clat (msec): min=11, max=1294, avg=285.22, stdev=164.51
     lat (msec): min=38, max=1298, avg=286.32, stdev=164.88
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[  104], 10.00th=[  125], 20.00th=[  153],
     | 30.00th=[  180], 40.00th=[  213], 50.00th=[  247], 60.00th=[  296],
     | 70.00th=[  342], 80.00th=[  393], 90.00th=[  485], 95.00th=[  584],
     | 99.00th=[  827], 99.50th=[ 1167], 99.90th=[ 1267], 99.95th=[ 1267],
     | 99.99th=[ 1301]
   bw (  KiB/s): min=119040, max=852992, per=98.43%, avg=448986.20, stdev=44471.69, samples=80
   iops        : min=  930, max= 6664, avg=3507.65, stdev=347.44, samples=80
  lat (msec)   : 20=0.01%, 50=0.04%, 100=4.16%, 250=46.61%, 500=40.23%
  lat (msec)   : 750=7.57%, 1000=0.65%, 2000=0.73%
  cpu          : usr=0.07%, sys=0.70%, ctx=4877, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=36098,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s), io=4512MiB (4731MB), run=10130-10130msec

Disk stats (read/write):
nvme1n1: ios=35140/0, sectors=8995840/0, merge=0/0, ticks=5536502/0, in_queue=5536502, util=99.04%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=443MiB/s][w=3541 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=7290: Tue May 19 14:29:31 2026
write: IOPS=3547, BW=443MiB/s (465MB/s)(4504MiB/10158msec); 0 zone resets

    slat (usec): min=2, max=202481, avg=1024.27, stdev=6657.70
    clat (msec): min=31, max=1350, avg=286.57, stdev=153.23
     lat (msec): min=31, max=1350, avg=287.60, stdev=153.56
    clat percentiles (msec):
     |  1.00th=[   66],  5.00th=[  104], 10.00th=[  127], 20.00th=[  163],
     | 30.00th=[  192], 40.00th=[  224], 50.00th=[  253], 60.00th=[  292],
     | 70.00th=[  338], 80.00th=[  401], 90.00th=[  485], 95.00th=[  575],
     | 99.00th=[  776], 99.50th=[  877], 99.90th=[ 1099], 99.95th=[ 1099],
     | 99.99th=[ 1351]
   bw (  KiB/s): min=189952, max=765952, per=98.70%, avg=448125.70, stdev=39877.96, samples=80
   iops        : min= 1484, max= 5984, avg=3500.95, stdev=311.54, samples=80
  lat (msec)   : 50=0.27%, 100=4.29%, 250=44.62%, 500=41.75%, 750=7.88%
  lat (msec)   : 1000=0.85%, 2000=0.34%
  cpu          : usr=0.41%, sys=0.44%, ctx=9102, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,36032,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=443MiB/s (465MB/s), 443MiB/s-443MiB/s (465MB/s-465MB/s), io=4504MiB (4723MB), run=10158-10158msec

Disk stats (read/write):
nvme1n1: ios=48/35948, sectors=1672/9202688, merge=0/0, ticks=7/6644720, in_queue=6644727, util=99.02%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over mptcp test # time=24516ms NQN:nqn.2014-08.org.nvmexpress.mptcpdev.7084.21368 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh tcp 4 round-robin 0
0+0 records in
0+0 records out
0 bytes copied, 0.000031583 s, 0.0 B/s
qdisc netem 8041: root refcnt 25 limit 1000 rate 1Gbit seed 17645609376687379696
qdisc netem 8042: root refcnt 25 limit 1000 rate 1Gbit seed 17131351233718588600
qdisc netem 8043: root refcnt 25 limit 1000 rate 1Gbit seed 12336840797390620536
qdisc netem 8044: root refcnt 25 limit 1000 rate 1Gbit seed 3442025196063469443
qdisc netem 8045: root refcnt 25 limit 1000 rate 1Gbit seed 1220869316842653191
qdisc netem 8046: root refcnt 25 limit 1000 rate 1Gbit seed 5375117655844217989
qdisc netem 8047: root refcnt 25 limit 1000 rate 1Gbit seed 7113140317580801833
qdisc netem 8048: root refcnt 25 limit 1000 rate 1Gbit seed 8218613337378753343
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 65 =====Discovery Log Entry 0======

trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  26201
trsvcid: 22625
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  26201
trsvcid: 22625
subnqn:  nqn.2014-08.org.nvmexpress.tcpdev.7486.4297
traddr:  10.1.1.1
eflags:  none
sectype: none

Connecting to 10.1.1.1:22625
connecting to device: nvme1
Connecting to 10.1.2.1:22625
connecting to device: nvme2
Connecting to 10.1.3.1:22625
connecting to device: nvme3
Connecting to 10.1.4.1:22625
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            10989b90c05a143e4117 Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=456MiB/s][r=3644 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=7655: Tue May 19 14:29:54 2026
read: IOPS=3645, BW=456MiB/s (478MB/s)(4683MiB/10278msec)

    slat (usec): min=2, max=1076, avg= 9.75, stdev=17.01
    clat (msec): min=7, max=2007, avg=280.55, stdev=214.94
     lat (msec): min=7, max=2007, avg=280.56, stdev=214.94
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   52], 10.00th=[   63], 20.00th=[   93],
     | 30.00th=[  136], 40.00th=[  186], 50.00th=[  236], 60.00th=[  288],
     | 70.00th=[  355], 80.00th=[  430], 90.00th=[  542], 95.00th=[  667],
     | 99.00th=[  995], 99.50th=[ 1167], 99.90th=[ 1670], 99.95th=[ 1770],
     | 99.99th=[ 1921]
   bw (  KiB/s): min=243200, max=716800, per=99.98%, avg=466483.20, stdev=34411.30, samples=80
   iops        : min= 1900, max= 5600, avg=3644.40, stdev=268.84, samples=80
  lat (msec)   : 10=0.03%, 20=0.26%, 50=4.24%, 100=17.38%, 250=30.84%
  lat (msec)   : 500=34.50%, 750=9.36%, 1000=2.45%, 2000=0.94%, >=2000=0.01%
  cpu          : usr=0.18%, sys=0.95%, ctx=38408, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=37464,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s), io=4683MiB (4910MB), run=10278-10278msec

Disk stats (read/write):
nvme1n1: ios=36957/0, sectors=9460992/0, merge=0/0, ticks=10056816/0, in_queue=10056816, util=100.00%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=481MiB/s][w=3844 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=7695: Tue May 19 14:30:05 2026
write: IOPS=3641, BW=455MiB/s (477MB/s)(4660MiB/10239msec); 0 zone resets

    slat (usec): min=2, max=189874, avg=241.45, stdev=4981.17
    clat (msec): min=3, max=3202, avg=280.63, stdev=340.07
     lat (msec): min=4, max=3202, avg=280.87, stdev=340.69
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   43], 10.00th=[   57], 20.00th=[   79],
     | 30.00th=[  101], 40.00th=[  131], 50.00th=[  171], 60.00th=[  226],
     | 70.00th=[  292], 80.00th=[  401], 90.00th=[  609], 95.00th=[  860],
     | 99.00th=[ 1972], 99.50th=[ 2433], 99.90th=[ 2869], 99.95th=[ 2937],
     | 99.99th=[ 3171]
   bw (  KiB/s): min=112640, max=868096, per=99.59%, avg=464153.60, stdev=50584.91, samples=80
   iops        : min=  880, max= 6782, avg=3626.20, stdev=395.19, samples=80
  lat (msec)   : 4=0.01%, 10=0.05%, 20=0.26%, 50=7.31%, 100=22.09%
  lat (msec)   : 250=34.13%, 500=22.17%, 750=7.29%, 1000=3.68%, 2000=2.05%
  lat (msec)   : >=2000=0.96%
  cpu          : usr=0.38%, sys=0.74%, ctx=37915, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,37283,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=4660MiB (4887MB), run=10239-10239msec

Disk stats (read/write):
nvme1n1: ios=52/36889, sectors=1952/9443584, merge=0/0, ticks=7/9065684, in_queue=9065691, util=99.03%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over tcp test # time=24541ms NQN:nqn.2014-08.org.nvmexpress.tcpdev.7486.4297 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh mptcp 4 round-robin 0
0+0 records in
0+0 records out
0 bytes copied, 0.000034286 s, 0.0 B/s
qdisc netem 8049: root refcnt 25 limit 1000 rate 1Gbit seed 1506589290665984997
qdisc netem 804a: root refcnt 25 limit 1000 rate 1Gbit seed 16222507226790466665
qdisc netem 804b: root refcnt 25 limit 1000 rate 1Gbit seed 1183797428335127788
qdisc netem 804c: root refcnt 25 limit 1000 rate 1Gbit seed 1369710343512163978
qdisc netem 804d: root refcnt 25 limit 1000 rate 1Gbit seed 14215765399017529043
qdisc netem 804e: root refcnt 25 limit 1000 rate 1Gbit seed 16863051369914697256
qdisc netem 804f: root refcnt 25 limit 1000 rate 1Gbit seed 1086698515664012853
qdisc netem 8050: root refcnt 25 limit 1000 rate 1Gbit seed 9133601558953778418
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 74 =====Discovery Log Entry 0======

trtype:  unrecognized
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  25587
trsvcid: 20822
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
=====Discovery Log Entry 1======
trtype:  unrecognized
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  25587
trsvcid: 20822
subnqn:  nqn.2014-08.org.nvmexpress.mptcpdev.7759.18799
traddr:  10.1.1.1
eflags:  none

Connecting to 10.1.1.1:20822
connecting to device: nvme1
Connecting to 10.1.2.1:20822
connecting to device: nvme2
Connecting to 10.1.3.1:20822
connecting to device: nvme3
Connecting to 10.1.4.1:20822
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            c8903574af001e2d8144 Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=443MiB/s][r=3547 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=7932: Tue May 19 14:30:23 2026
read: IOPS=3566, BW=446MiB/s (467MB/s)(4565MiB/10239msec)

    slat (usec): min=2, max=230767, avg=107.20, stdev=3298.93
    clat (msec): min=12, max=1820, avg=286.67, stdev=234.19
     lat (msec): min=12, max=1952, avg=286.78, stdev=234.60
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   47], 10.00th=[   64], 20.00th=[   99],
     | 30.00th=[  136], 40.00th=[  176], 50.00th=[  226], 60.00th=[  288],
     | 70.00th=[  359], 80.00th=[  439], 90.00th=[  567], 95.00th=[  718],
     | 99.00th=[ 1133], 99.50th=[ 1469], 99.90th=[ 1703], 99.95th=[ 1720],
     | 99.99th=[ 1754]
   bw (  KiB/s): min=258816, max=738816, per=99.53%, avg=454371.10, stdev=35123.19, samples=80
   iops        : min= 2022, max= 5772, avg=3549.70, stdev=274.40, samples=80
  lat (msec)   : 20=0.24%, 50=5.56%, 100=14.60%, 250=33.79%, 500=31.58%
  lat (msec)   : 750=9.98%, 1000=2.77%, 2000=1.49%
  cpu          : usr=0.11%, sys=1.11%, ctx=28917, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=36516,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s), io=4565MiB (4786MB), run=10239-10239msec

Disk stats (read/write):
nvme1n1: ios=36065/0, sectors=9232640/0, merge=0/0, ticks=9849440/0, in_queue=9849440, util=99.08%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=449MiB/s][w=3591 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=7964: Tue May 19 14:30:34 2026
write: IOPS=3560, BW=445MiB/s (467MB/s)(4575MiB/10280msec); 0 zone resets

    slat (usec): min=3, max=106224, avg=36.02, stdev=1317.16
    clat (msec): min=3, max=1458, avg=287.19, stdev=208.34
     lat (msec): min=3, max=1458, avg=287.23, stdev=208.38
    clat percentiles (msec):
     |  1.00th=[   56],  5.00th=[   74], 10.00th=[   90], 20.00th=[  117],
     | 30.00th=[  148], 40.00th=[  182], 50.00th=[  224], 60.00th=[  279],
     | 70.00th=[  347], 80.00th=[  426], 90.00th=[  584], 95.00th=[  718],
     | 99.00th=[  961], 99.50th=[ 1083], 99.90th=[ 1368], 99.95th=[ 1385],
     | 99.99th=[ 1435]
   bw (  KiB/s): min=273664, max=696320, per=99.94%, avg=455459.95, stdev=27313.01, samples=80
   iops        : min= 2138, max= 5440, avg=3558.20, stdev=213.39, samples=80
  lat (msec)   : 4=0.01%, 10=0.05%, 20=0.10%, 50=0.39%, 100=13.15%
  lat (msec)   : 250=41.14%, 500=30.83%, 750=10.33%, 1000=3.22%, 2000=0.80%
  cpu          : usr=0.48%, sys=0.81%, ctx=26520, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,36601,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s), io=4575MiB (4797MB), run=10280-10280msec

Disk stats (read/write):
nvme1n1: ios=73/36055, sectors=2520/9230080, merge=0/0, ticks=23/10146459, in_queue=10146482, util=100.00%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over mptcp test # time=24539ms NQN:nqn.2014-08.org.nvmexpress.mptcpdev.7759.18799 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

$ sudo ./mptcp_nvme.sh tcp 4 queue-depth 0
[sudo: authenticate] Password:        
0+0 records in
0+0 records out
0 bytes copied, 0.000034443 s, 0.0 B/s
qdisc netem 8061: root refcnt 25 limit 1000 rate 1Gbit seed 3295449951790515148
qdisc netem 8062: root refcnt 25 limit 1000 rate 1Gbit seed 3138537503814513313
qdisc netem 8063: root refcnt 25 limit 1000 rate 1Gbit seed 10184262077164245255
qdisc netem 8064: root refcnt 25 limit 1000 rate 1Gbit seed 8196776831254145068
qdisc netem 8065: root refcnt 25 limit 1000 rate 1Gbit seed 3794130921615195350
qdisc netem 8066: root refcnt 25 limit 1000 rate 1Gbit seed 3768240293358984722
qdisc netem 8067: root refcnt 25 limit 1000 rate 1Gbit seed 11220714754356808479
qdisc netem 8068: root refcnt 25 limit 1000 rate 1Gbit seed 12767913258139064713
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 101 =====Discovery Log Entry 0======

trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  29891
trsvcid: 7688
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  29891
trsvcid: 7688
subnqn:  nqn.2014-08.org.nvmexpress.tcpdev.8843.27030
traddr:  10.1.1.1
eflags:  none
sectype: none

Connecting to 10.1.1.1:7688
connecting to device: nvme1
Connecting to 10.1.2.1:7688
connecting to device: nvme2
Connecting to 10.1.3.1:7688
connecting to device: nvme3
Connecting to 10.1.4.1:7688
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            9f68b5437684bbcdddd1 Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=454MiB/s][r=3635 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=9014: Tue May 19 14:37:44 2026
read: IOPS=3644, BW=456MiB/s (478MB/s)(4632MiB/10169msec)

    slat (usec): min=2, max=830059, avg=160.63, stdev=6754.87
    clat (usec): min=1666, max=3091.4k, avg=280481.59, stdev=335991.37
     lat (msec): min=2, max=3091, avg=280.64, stdev=336.28
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   31], 10.00th=[   44], 20.00th=[   65],
     | 30.00th=[   89], 40.00th=[  126], 50.00th=[  174], 60.00th=[  239],
     | 70.00th=[  313], 80.00th=[  418], 90.00th=[  609], 95.00th=[  835],
     | 99.00th=[ 1838], 99.50th=[ 2366], 99.90th=[ 2769], 99.95th=[ 2802],
     | 99.99th=[ 2937]
   bw (  KiB/s): min=101888, max=941312, per=98.88%, avg=461260.80, stdev=54054.15, samples=80
   iops        : min=  796, max= 7354, avg=3603.60, stdev=422.30, samples=80
  lat (msec)   : 2=0.01%, 4=0.04%, 10=0.35%, 20=1.41%, 50=11.51%
  lat (msec)   : 100=19.90%, 250=28.85%, 500=23.31%, 750=7.81%, 1000=3.63%
  lat (msec)   : 2000=2.44%, >=2000=0.75%
  cpu          : usr=0.11%, sys=0.98%, ctx=37347, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=37059,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s), io=4632MiB (4857MB), run=10169-10169msec

Disk stats (read/write):
nvme1n1: ios=36964/0, sectors=9462784/0, merge=0/0, ticks=9407103/0, in_queue=9407103, util=100.00%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=457MiB/s][w=3656 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=9056: Tue May 19 14:37:56 2026
write: IOPS=3641, BW=455MiB/s (477MB/s)(4666MiB/10250msec); 0 zone resets

    slat (usec): min=2, max=281750, avg=150.14, stdev=4550.86
    clat (msec): min=4, max=3125, avg=280.88, stdev=311.14
     lat (msec): min=4, max=3125, avg=281.03, stdev=311.59
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   57], 10.00th=[   71], 20.00th=[   95],
     | 30.00th=[  121], 40.00th=[  150], 50.00th=[  184], 60.00th=[  236],
     | 70.00th=[  296], 80.00th=[  384], 90.00th=[  550], 95.00th=[  802],
     | 99.00th=[ 1787], 99.50th=[ 2072], 99.90th=[ 2769], 99.95th=[ 2937],
     | 99.99th=[ 3104]
   bw (  KiB/s): min=218880, max=769280, per=99.70%, avg=464743.95, stdev=37729.75, samples=80
   iops        : min= 1710, max= 6010, avg=3630.75, stdev=294.76, samples=80
  lat (msec)   : 10=0.05%, 20=0.18%, 50=3.43%, 100=18.35%, 250=40.40%
  lat (msec)   : 500=25.52%, 750=6.32%, 1000=2.32%, 2000=2.79%, >=2000=0.62%
  cpu          : usr=0.38%, sys=0.82%, ctx=39052, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,37327,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=4666MiB (4893MB), run=10250-10250msec

Disk stats (read/write):
nvme1n1: ios=52/36902, sectors=1952/9446912, merge=0/0, ticks=9/9443482, in_queue=9443491, util=99.05%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over tcp test # time=24526ms NQN:nqn.2014-08.org.nvmexpress.tcpdev.8843.27030 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh mptcp 4 queue-depth 0
0+0 records in
0+0 records out
0 bytes copied, 0.000033222 s, 0.0 B/s
qdisc netem 8069: root refcnt 25 limit 1000 rate 1Gbit seed 10161739475702683234
qdisc netem 806a: root refcnt 25 limit 1000 rate 1Gbit seed 12159989797435071587
qdisc netem 806b: root refcnt 25 limit 1000 rate 1Gbit seed 14498810451485745943
qdisc netem 806c: root refcnt 25 limit 1000 rate 1Gbit seed 18442905267492529790
qdisc netem 806d: root refcnt 25 limit 1000 rate 1Gbit seed 2985926051413726176
qdisc netem 806e: root refcnt 25 limit 1000 rate 1Gbit seed 4435161703691035974
qdisc netem 806f: root refcnt 25 limit 1000 rate 1Gbit seed 16832561787704440488
qdisc netem 8070: root refcnt 25 limit 1000 rate 1Gbit seed 8416006940920284114
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 110 =====Discovery Log Entry 0======

trtype:  unrecognized
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  28051
trsvcid: 19031
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
=====Discovery Log Entry 1======
trtype:  unrecognized
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  28051
trsvcid: 19031
subnqn:  nqn.2014-08.org.nvmexpress.mptcpdev.9156.18561
traddr:  10.1.1.1
eflags:  none

Connecting to 10.1.1.1:19031
connecting to device: nvme1
Connecting to 10.1.2.1:19031
connecting to device: nvme2
Connecting to 10.1.3.1:19031
connecting to device: nvme3
Connecting to 10.1.4.1:19031
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            261e91ea8603f223e6fe Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=458MiB/s][r=3665 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=9331: Tue May 19 14:38:16 2026
read: IOPS=3565, BW=446MiB/s (467MB/s)(4568MiB/10249msec)

    slat (usec): min=2, max=621, avg=11.66, stdev=15.54
    clat (msec): min=15, max=1787, avg=286.66, stdev=218.14
     lat (msec): min=15, max=1787, avg=286.67, stdev=218.14
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   49], 10.00th=[   70], 20.00th=[  108],
     | 30.00th=[  142], 40.00th=[  184], 50.00th=[  230], 60.00th=[  284],
     | 70.00th=[  347], 80.00th=[  439], 90.00th=[  584], 95.00th=[  726],
     | 99.00th=[  995], 99.50th=[ 1150], 99.90th=[ 1368], 99.95th=[ 1569],
     | 99.99th=[ 1737]
   bw (  KiB/s): min=265728, max=655872, per=99.62%, avg=454694.40, stdev=25269.02, samples=80
   iops        : min= 2076, max= 5124, avg=3552.30, stdev=197.41, samples=80
  lat (msec)   : 20=0.16%, 50=5.19%, 100=12.82%, 250=35.67%, 500=30.99%
  lat (msec)   : 750=10.73%, 1000=3.47%, 2000=0.98%
  cpu          : usr=0.11%, sys=1.25%, ctx=29785, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=36545,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s), io=4568MiB (4790MB), run=10249-10249msec

Disk stats (read/write):
nvme1n1: ios=36005/0, sectors=9217280/0, merge=0/0, ticks=10137035/0, in_queue=10137035, util=99.06%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=448MiB/s][w=3582 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=9365: Tue May 19 14:38:27 2026
write: IOPS=3562, BW=445MiB/s (467MB/s)(4563MiB/10245msec); 0 zone resets

    slat (usec): min=2, max=2021, avg=11.72, stdev=20.90
    clat (msec): min=4, max=1463, avg=287.22, stdev=202.83
     lat (msec): min=4, max=1463, avg=287.24, stdev=202.83
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   75], 10.00th=[   92], 20.00th=[  122],
     | 30.00th=[  153], 40.00th=[  190], 50.00th=[  232], 60.00th=[  279],
     | 70.00th=[  342], 80.00th=[  430], 90.00th=[  558], 95.00th=[  684],
     | 99.00th=[  969], 99.50th=[ 1133], 99.90th=[ 1368], 99.95th=[ 1418],
     | 99.99th=[ 1418]
   bw (  KiB/s): min=287744, max=602624, per=99.59%, avg=454153.50, stdev=21675.66, samples=80
   iops        : min= 2248, max= 4708, avg=3548.05, stdev=169.34, samples=80
  lat (msec)   : 10=0.06%, 20=0.07%, 50=0.31%, 100=12.62%, 250=41.04%
  lat (msec)   : 500=32.01%, 750=10.08%, 1000=2.96%, 2000=0.86%
  cpu          : usr=0.50%, sys=0.83%, ctx=26931, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,36500,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s), io=4563MiB (4784MB), run=10245-10245msec

Disk stats (read/write):
nvme1n1: ios=45/36009, sectors=1328/9218304, merge=0/0, ticks=9/10121003, in_queue=10121012, util=99.06%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over mptcp test # time=24554ms NQN:nqn.2014-08.org.nvmexpress.mptcpdev.9156.18561 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh tcp 4 round-robin 1
0+0 records in
0+0 records out
0 bytes copied, 0.000036756 s, 0.0 B/s
qdisc netem 8051: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 14946049878654165618
qdisc netem 8052: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 11230459861705135323
qdisc netem 8053: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 9641862848092404184
qdisc netem 8054: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 6252522120019877633
qdisc netem 8055: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 15159846717589848988
qdisc netem 8056: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 2882562256880768222
qdisc netem 8057: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 9830209649295320810
qdisc netem 8058: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 11605172172603852230
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 83 =====Discovery Log Entry 0======

trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  25757
trsvcid: 8795
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  25757
trsvcid: 8795
subnqn:  nqn.2014-08.org.nvmexpress.tcpdev.8020.19160
traddr:  10.1.1.1
eflags:  none
sectype: none

Connecting to 10.1.1.1:8795
connecting to device: nvme1
Connecting to 10.1.2.1:8795
connecting to device: nvme2
Connecting to 10.1.3.1:8795
connecting to device: nvme3
Connecting to 10.1.4.1:8795
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            d713f8a662af6c1145b6 Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 1 (f=1): [_(3),r(1)][62.5%][r=63.8MiB/s][r=510 IOPS][eta 00m:09s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=8190: Tue May 19 14:31:01 2026
read: IOPS=844, BW=106MiB/s (111MB/s)(1574MiB/14906msec)

    slat (usec): min=2, max=593278, avg=701.94, stdev=14737.94
    clat (msec): min=11, max=9029, avg=1158.52, stdev=1632.75
     lat (msec): min=11, max=9029, avg=1159.22, stdev=1633.95
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   12], 10.00th=[   18], 20.00th=[   32],
     | 30.00th=[   66], 40.00th=[  169], 50.00th=[  351], 60.00th=[  718],
     | 70.00th=[ 1284], 80.00th=[ 2106], 90.00th=[ 3574], 95.00th=[ 5134],
     | 99.00th=[ 6544], 99.50th=[ 7550], 99.90th=[ 8792], 99.95th=[ 8792],
     | 99.99th=[ 9060]
   bw (  KiB/s): min=32000, max=442796, per=100.00%, avg=149431.05, stdev=30858.28, samples=79
   iops        : min=  250, max= 3459, avg=1167.37, stdev=241.06, samples=79
  lat (msec)   : 20=11.95%, 50=14.53%, 100=7.70%, 250=11.45%, 500=8.76%
  lat (msec)   : 750=6.30%, 1000=4.42%, 2000=14.27%, >=2000=20.62%
  cpu          : usr=0.03%, sys=0.20%, ctx=10272, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=1.0%, >=64=98.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=12590,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=1574MiB (1650MB), run=14906-14906msec

Disk stats (read/write):
nvme1n1: ios=12589/0, sectors=3222784/0, merge=0/0, ticks=10598011/0, in_queue=10598011, util=99.39%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 3 (f=3): [w(1),f(1),w(1),_(1)][70.0%][w=31.9MiB/s][w=255 IOPS][eta 00m:06s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=8222: Tue May 19 14:31:17 2026
write: IOPS=788, BW=98.5MiB/s (103MB/s)(1455MiB/14770msec); 0 zone resets

    slat (usec): min=2, max=831622, avg=358.60, stdev=12909.46
    clat (msec): min=21, max=8504, avg=1214.70, stdev=1453.79
     lat (msec): min=21, max=8504, avg=1215.06, stdev=1453.93
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   36], 10.00th=[   53], 20.00th=[  112],
     | 30.00th=[  213], 40.00th=[  401], 50.00th=[  659], 60.00th=[ 1020],
     | 70.00th=[ 1418], 80.00th=[ 1989], 90.00th=[ 3339], 95.00th=[ 4799],
     | 99.00th=[ 6141], 99.50th=[ 6342], 99.90th=[ 7819], 99.95th=[ 7819],
     | 99.99th=[ 7886]
   bw (  KiB/s): min=39936, max=271265, per=100.00%, avg=139243.92, stdev=14876.84, samples=80
   iops        : min=  312, max= 2119, avg=1087.77, stdev=116.20, samples=80
  lat (msec)   : 50=8.18%, 100=10.16%, 250=13.95%, 500=12.16%, 750=8.50%
  lat (msec)   : 1000=6.50%, 2000=20.70%, >=2000=19.85%
  cpu          : usr=0.12%, sys=0.21%, ctx=13938, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=1.1%, >=64=97.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,11643,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=98.5MiB/s (103MB/s), 98.5MiB/s-98.5MiB/s (103MB/s-103MB/s), io=1455MiB (1526MB), run=14770-14770msec

Disk stats (read/write):
nvme1n1: ios=90/11639, sectors=2976/2979584, merge=0/0, ticks=16935/10792314, in_queue=10809249, util=100.00%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over tcp test # time=37684ms NQN:nqn.2014-08.org.nvmexpress.tcpdev.8020.19160 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh mptcp 4 round-robin 1
0+0 records in
0+0 records out
0 bytes copied, 0.000034975 s, 0.0 B/s
qdisc netem 8059: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 16648645294543629576
qdisc netem 805a: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 12381805247586694531
qdisc netem 805b: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 16335976436197243438
qdisc netem 805c: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 14431250154863119196
qdisc netem 805d: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 16886676323171265152
qdisc netem 805e: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 8248230441049121426
qdisc netem 805f: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 3846556862478236863
qdisc netem 8060: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 18342167217463336805
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 92 =====Discovery Log Entry 0======

trtype:  unrecognized
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  23106
trsvcid: 23601
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
=====Discovery Log Entry 1======
trtype:  unrecognized
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  23106
trsvcid: 23601
subnqn:  nqn.2014-08.org.nvmexpress.mptcpdev.8281.3834
traddr:  10.1.1.1
eflags:  none

Connecting to 10.1.1.1:23601
connecting to device: nvme1
Connecting to 10.1.2.1:23601
connecting to device: nvme2
Connecting to 10.1.3.1:23601
connecting to device: nvme3
Connecting to 10.1.4.1:23601
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            e40fc3718d44d3bc547c Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=442MiB/s][r=3534 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=8455: Tue May 19 14:31:41 2026
read: IOPS=3409, BW=426MiB/s (447MB/s)(4533MiB/10637msec)

    slat (usec): min=2, max=8926, avg= 9.82, stdev=49.04
    clat (msec): min=13, max=1821, avg=294.51, stdev=256.51
     lat (msec): min=13, max=1821, avg=294.52, stdev=256.52
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   39], 10.00th=[   51], 20.00th=[   80],
     | 30.00th=[  120], 40.00th=[  167], 50.00th=[  228], 60.00th=[  288],
     | 70.00th=[  363], 80.00th=[  464], 90.00th=[  634], 95.00th=[  818],
     | 99.00th=[ 1167], 99.50th=[ 1284], 99.90th=[ 1603], 99.95th=[ 1636],
     | 99.99th=[ 1754]
   bw (  KiB/s): min=252779, max=695040, per=100.00%, avg=448147.72, stdev=26919.12, samples=81
   iops        : min= 1974, max= 5430, avg=3501.11, stdev=210.33, samples=81
  lat (msec)   : 20=0.64%, 50=9.10%, 100=15.72%, 250=28.46%, 500=28.97%
  lat (msec)   : 750=11.02%, 1000=3.52%, 2000=2.56%
  cpu          : usr=0.10%, sys=0.99%, ctx=29544, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=36266,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=426MiB/s (447MB/s), 426MiB/s-426MiB/s (447MB/s-447MB/s), io=4533MiB (4753MB), run=10637-10637msec

Disk stats (read/write):
nvme1n1: ios=36139/0, sectors=9251584/0, merge=0/0, ticks=10343665/0, in_queue=10343665, util=100.00%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=444MiB/s][w=3555 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=8499: Tue May 19 14:31:53 2026
write: IOPS=3426, BW=428MiB/s (449MB/s)(4507MiB/10522msec); 0 zone resets

    slat (usec): min=3, max=45871, avg=14.63, stdev=249.32
    clat (msec): min=20, max=1490, avg=297.39, stdev=222.83
     lat (msec): min=20, max=1490, avg=297.40, stdev=222.83
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   51], 10.00th=[   66], 20.00th=[   99],
     | 30.00th=[  144], 40.00th=[  194], 50.00th=[  247], 60.00th=[  305],
     | 70.00th=[  368], 80.00th=[  464], 90.00th=[  617], 95.00th=[  743],
     | 99.00th=[  986], 99.50th=[ 1083], 99.90th=[ 1301], 99.95th=[ 1368],
     | 99.99th=[ 1485]
   bw (  KiB/s): min=286976, max=641024, per=100.00%, avg=446654.17, stdev=24140.84, samples=81
   iops        : min= 2242, max= 5008, avg=3489.49, stdev=188.60, samples=81
  lat (msec)   : 50=4.77%, 100=15.95%, 250=30.00%, 500=32.46%, 750=12.07%
  lat (msec)   : 1000=3.86%, 2000=0.89%
  cpu          : usr=0.45%, sys=0.96%, ctx=25181, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,36052,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=428MiB/s (449MB/s), 428MiB/s-428MiB/s (449MB/s-449MB/s), io=4507MiB (4725MB), run=10522-10522msec

Disk stats (read/write):
nvme1n1: ios=85/35927, sectors=2736/9197312, merge=0/0, ticks=2753/10368824, in_queue=10371577, util=100.00%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over mptcp test # time=29910ms NQN:nqn.2014-08.org.nvmexpress.mptcpdev.8281.3834 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

$ sudo ./mptcp_nvme.sh tcp 4 queue-depth 1
0+0 records in
0+0 records out
0 bytes copied, 0.000030985 s, 0.0 B/s
qdisc netem 8071: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 13101495999964109604
qdisc netem 8072: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 11517974220235216241
qdisc netem 8073: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 12492982716523034098
qdisc netem 8074: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 10279723881541909769
qdisc netem 8075: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 2346606414154275258
qdisc netem 8076: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 7578393669647277521
qdisc netem 8077: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 17269306750778379857
qdisc netem 8078: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 908546285485823085
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 119 =====Discovery Log Entry 0======

trtype:  tcp
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  24646
trsvcid: 14447
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  24646
trsvcid: 14447
subnqn:  nqn.2014-08.org.nvmexpress.tcpdev.10012.27992
traddr:  10.1.1.1
eflags:  none
sectype: none

Connecting to 10.1.1.1:14447
connecting to device: nvme1
Connecting to 10.1.2.1:14447
connecting to device: nvme2
Connecting to 10.1.3.1:14447
connecting to device: nvme3
Connecting to 10.1.4.1:14447
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            e9540cf45de68e42c683 Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 1 (f=1): [_(2),r(1),_(1)][100.0%][r=63.8MiB/s][r=510 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=10182: Tue May 19 14:39:25 2026
read: IOPS=1344, BW=168MiB/s (176MB/s)(2179MiB/12965msec)

    slat (usec): min=2, max=503567, avg=115.68, stdev=6521.43
    clat (msec): min=11, max=5731, avg=713.51, stdev=1011.82
     lat (msec): min=11, max=5731, avg=713.63, stdev=1012.11
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   17], 10.00th=[   23], 20.00th=[   41],
     | 30.00th=[   71], 40.00th=[  142], 50.00th=[  232], 60.00th=[  384],
     | 70.00th=[  667], 80.00th=[ 1267], 90.00th=[ 2333], 95.00th=[ 3171],
     | 99.00th=[ 4010], 99.50th=[ 4329], 99.90th=[ 5604], 99.95th=[ 5671],
     | 99.99th=[ 5738]
   bw (  KiB/s): min=48128, max=585984, per=100.00%, avg=210060.80, stdev=34447.60, samples=80
   iops        : min=  376, max= 4578, avg=1641.10, stdev=269.12, samples=80
  lat (msec)   : 20=6.61%, 50=17.33%, 100=10.93%, 250=17.03%, 500=12.87%
  lat (msec)   : 750=7.16%, 1000=4.85%, 2000=9.51%, >=2000=13.72%
  cpu          : usr=0.04%, sys=0.36%, ctx=16206, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=0.7%, >=64=98.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=17431,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=2179MiB (2285MB), run=12965-12965msec

Disk stats (read/write):
nvme1n1: ios=17428/0, sectors=4461568/0, merge=0/0, ticks=10855250/0, in_queue=10855250, util=100.00%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 3 (f=3): [w(3),_(1)][66.7%][w=31.9MiB/s][w=255 IOPS][eta 00m:06s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=10218: Tue May 19 14:39:39 2026
write: IOPS=1024, BW=128MiB/s (134MB/s)(1590MiB/12418msec); 0 zone resets

    slat (usec): min=2, max=889, avg=12.18, stdev=10.43
    clat (msec): min=21, max=4787, avg=973.45, stdev=979.48
     lat (msec): min=21, max=4787, avg=973.47, stdev=979.48
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   37], 10.00th=[   62], 20.00th=[  150],
     | 30.00th=[  271], 40.00th=[  426], 50.00th=[  617], 60.00th=[  860],
     | 70.00th=[ 1217], 80.00th=[ 1787], 90.00th=[ 2534], 95.00th=[ 3071],
     | 99.00th=[ 3876], 99.50th=[ 4044], 99.90th=[ 4329], 99.95th=[ 4665],
     | 99.99th=[ 4799]
   bw (  KiB/s): min=49664, max=286720, per=100.00%, avg=149785.60, stdev=17248.23, samples=80
   iops        : min=  388, max= 2240, avg=1170.20, stdev=134.75, samples=80
  lat (msec)   : 50=7.35%, 100=7.59%, 250=13.54%, 500=15.93%, 750=11.41%
  lat (msec)   : 1000=8.88%, 2000=18.64%, >=2000=16.65%
  cpu          : usr=0.13%, sys=0.27%, ctx=16346, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=1.0%, >=64=98.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,12722,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=1590MiB (1667MB), run=12418-12418msec

Disk stats (read/write):
nvme1n1: ios=131/12721, sectors=3784/3256576, merge=0/0, ticks=1945/10940196, in_queue=10942141, util=100.00%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over tcp test # time=35027ms NQN:nqn.2014-08.org.nvmexpress.tcpdev.10012.27992 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

tgl@ThinkBook:~/mptcp_net-next/mptcp-selftests$ sudo ./mptcp_nvme.sh mptcp 4 queue-depth 1
0+0 records in
0+0 records out
0 bytes copied, 0.000034865 s, 0.0 B/s
qdisc netem 8079: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 11684875286331491500
qdisc netem 807a: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 15405903518535477173
qdisc netem 807b: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 11645838655906894761
qdisc netem 807c: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 1760572176314863578
qdisc netem 807d: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 15635004858252533828
qdisc netem 807e: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 933733748701051554
qdisc netem 807f: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 13388314664729634730
qdisc netem 8080: root refcnt 25 limit 1000 delay 5ms loss 0.5% rate 1Gbit seed 2611167444991410732
nvme discover -a 10.1.1.1

Discovery Log Number of Records 2, Generation counter 128 =====Discovery Log Entry 0======

trtype:  unrecognized
adrfam:  ipv4
subtype: current discovery subsystem
treq:    not specified, sq flow control disable supported
portid:  24481
trsvcid: 1192
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.1.1.1
eflags:  none
=====Discovery Log Entry 1======
trtype:  unrecognized
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  24481
trsvcid: 1192
subnqn:  nqn.2014-08.org.nvmexpress.mptcpdev.10360.8450
traddr:  10.1.1.1
eflags:  none

Connecting to 10.1.1.1:1192
connecting to device: nvme1
Connecting to 10.1.2.1:1192
connecting to device: nvme2
Connecting to 10.1.3.1:1192
connecting to device: nvme3
Connecting to 10.1.4.1:1192
connecting to device: nvme4
nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------

/dev/nvme0n1          /dev/ng0n1            SS1Q24313Z2CD54R50B8 UMIS RPJYJ1T24MML1AWY                    0x1          1.02  TB /   1.02  TB    512   B +  0 B   1.0L06C1
/dev/nvme1n1          /dev/ng1n1            ec42151c9f27c53e5e5b Linux                                    0x1        536.87  MB / 536.87  MB    512   B +  0 B   7.1.0-rc

fio randread /dev/nvme1n1
libaio_4_256_128k_randread: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [r(4)][100.0%][r=444MiB/s][r=3548 IOPS][eta 00m:00s]
libaio_4_256_128k_randread: (groupid=0, jobs=4): err= 0: pid=10530: Tue May 19 14:40:12 2026
read: IOPS=3398, BW=425MiB/s (445MB/s)(4536MiB/10677msec)

    slat (usec): min=2, max=1225, avg= 9.51, stdev=11.40
    clat (msec): min=11, max=1802, avg=295.38, stdev=251.61
     lat (msec): min=11, max=1802, avg=295.39, stdev=251.61
    clat percentiles (msec):
     |  1.00th=[   22],  5.00th=[   41], 10.00th=[   56], 20.00th=[   90],
     | 30.00th=[  130], 40.00th=[  178], 50.00th=[  230], 60.00th=[  288],
     | 70.00th=[  359], 80.00th=[  456], 90.00th=[  625], 95.00th=[  802],
     | 99.00th=[ 1200], 99.50th=[ 1368], 99.90th=[ 1636], 99.95th=[ 1720],
     | 99.99th=[ 1754]
   bw (  KiB/s): min=254208, max=672512, per=100.00%, avg=451392.00, stdev=26485.91, samples=80
   iops        : min= 1986, max= 5254, avg=3526.50, stdev=206.92, samples=80
  lat (msec)   : 20=0.65%, 50=7.55%, 100=14.81%, 250=30.73%, 500=29.98%
  lat (msec)   : 750=10.05%, 1000=4.12%, 2000=2.11%
  cpu          : usr=0.12%, sys=0.99%, ctx=29081, majf=0, minf=32772
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=36285,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=425MiB/s (445MB/s), 425MiB/s-425MiB/s (445MB/s-445MB/s), io=4536MiB (4756MB), run=10677-10677msec

Disk stats (read/write):
nvme1n1: ios=36271/0, sectors=9285376/0, merge=0/0, ticks=10433719/0, in_queue=10433719, util=100.00%
fio randwrite /dev/nvme1n1
libaio_4_256_128k_randwrite: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=256
...
fio-3.41
Starting 4 threads
Jobs: 4 (f=4): [w(4)][100.0%][w=435MiB/s][w=3480 IOPS][eta 00m:00s]
libaio_4_256_128k_randwrite: (groupid=0, jobs=4): err= 0: pid=10574: Tue May 19 14:40:24 2026
write: IOPS=3314, BW=414MiB/s (434MB/s)(4447MiB/10733msec); 0 zone resets

    slat (usec): min=3, max=391916, avg=64.18, stdev=2661.10
    clat (msec): min=20, max=1286, avg=303.98, stdev=245.38
     lat (msec): min=20, max=1286, avg=304.04, stdev=245.45
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   45], 10.00th=[   59], 20.00th=[   92],
     | 30.00th=[  134], 40.00th=[  184], 50.00th=[  243], 60.00th=[  305],
     | 70.00th=[  376], 80.00th=[  477], 90.00th=[  634], 95.00th=[  827],
     | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1250],
     | 99.99th=[ 1284]
   bw (  KiB/s): min=209664, max=781056, per=100.00%, avg=439949.26, stdev=34944.39, samples=81
   iops        : min= 1638, max= 6102, avg=3437.04, stdev=273.01, samples=81
  lat (msec)   : 50=6.96%, 100=15.29%, 250=29.00%, 500=30.76%, 750=11.29%
  lat (msec)   : 1000=4.32%, 2000=2.38%
  cpu          : usr=0.46%, sys=0.94%, ctx=23523, majf=0, minf=4
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,35573,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
WRITE: bw=414MiB/s (434MB/s), 414MiB/s-414MiB/s (434MB/s-434MB/s), io=4447MiB (4663MB), run=10733-10733msec

Disk stats (read/write):
nvme1n1: ios=117/35561, sectors=3008/9103616, merge=0/0, ticks=4265/10379195, in_queue=10383460, util=100.00%
NVMe Flush: success

TAP version 13
1..1
ok 1 - mptcp_nvme: nvme over mptcp test # time=30332ms NQN:nqn.2014-08.org.nvmexpress.mptcpdev.10360.8450 disconnected 4 controller(s) /sys/kernel/config/nvmet /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp /home/tgl/mptcp_net-next/tools/testing/selftests/net/mptcp

author reviewer
#8 lore raw

Hi Nilay,

Off-list.

Quoted 138 lines · Geliang Tang

Hi,

The performance test results of MPTCP under several NVMe multipath settings are now ready.

Quoted 10 lines · Geliang Tang

Hello everyone,

Thank you for your interest in NVMe over MPTCP. I have attached the slides from the presentation to this email.

Please note that the demo in the slides only configured a single NVMe
multipath. Subsequently, I will post the MPTCP performance test results
under several NVMe multipaths here.

To test the performance of TCP and MPTCP under NVMe multipath, I added
two more arguments, "path" and "loss", to the original NVMe MPTCP self
test script. The latest code is available at [1].

The script now accepts the following four arguments:

  mptcp_nvme.sh [trtype] [path] [iopolicy] [loss]

  trtype   Transport type (tcp|mptcp) - default: mptcp path     Number of multipath (1-4) - default: 1 iopolicy I/O policy (numa|round-robin|queue-depth) - default: numa loss     Enable packet loss (0|1) - default: 0

The first argument is the transport type. The second argument, "path",
specifies how many NVMe multipaths to create. The third argument is the
I/O policy. The fourth argument controls whether the network environment is lossy. When set to 0, each NIC is rate-limited to 125 MB/s (tc arguments: rate 1000mbit). When set to 1, in addition to the same rate limit of 125 MB/s, each NIC also experiences a 5 ms delay and
0.5% packet loss (tc arguments: rate 1000mbit delay 5ms loss 0.5%).

First set of tests: lossless network, path=4, loss=0. The tc output is
as follows:

  qdisc netem 8031: root refcnt 25 limit 1000 rate 1Gbit

			seed 1626193586047356330

Lossless network, comparison between TCP and MPTCP using the "numa" policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 numa 0
   READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s),
			io=1200MiB (1259MB), run=10533-10533msec
  WRITE: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s),
			io=1203MiB (1261MB), run=10570-10570msec
# ./mptcp_nvme.sh mptcp 4 numa 0
   READ: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
			io=4512MiB (4731MB), run=10130-10130msec
  WRITE: bw=443MiB/s (465MB/s), 443MiB/s-443MiB/s (465MB/s-465MB/s),
			io=4504MiB (4723MB), run=10158-10158msec

Lossless network, comparison between TCP and MPTCP using the "round- robin" policy - MPTCP and TCP show similar performance:

# ./mptcp_nvme.sh tcp 4 round-robin 0
   READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s),
			io=4683MiB (4910MB), run=10278-10278msec
  WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s),
			io=4660MiB (4887MB), run=10239-10239msec
# ./mptcp_nvme.sh mptcp 4 round-robin 0
   READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s),
			io=4565MiB (4786MB), run=10239-10239msec
  WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
			io=4575MiB (4797MB), run=10280-10280msec

Lossless network, comparison between TCP and MPTCP using the "queue- depth" policy - MPTCP and TCP show similar performance:

# ./mptcp_nvme.sh tcp 4 queue-depth 0
   READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s),
			io=4632MiB (4857MB), run=10169-10169msec
  WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s),
			io=4666MiB (4893MB), run=10250-10250msec
# ./mptcp_nvme.sh mptcp 4 queue-depth 0
   READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s),
			io=4568MiB (4790MB), run=10249-10249msec
  WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
			io=4563MiB (4784MB), run=10245-10245msec

Second set of tests: lossy network, path=4, loss=1. The tc output is as
follows:

  qdisc netem 8051: root refcnt 25 limit 1000 delay 5ms loss 0.5%

			rate 1Gbit seed 14946049878654165618

Lossy network, comparison between TCP and MPTCP using the "round- robin"
policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 round-robin 1
   READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s),
			io=1574MiB (1650MB), run=14906-14906msec
  WRITE: bw=98.5MiB/s (103MB/s), 98.5MiB/s-98.5MiB/s (103MB/s-
103MB/s),
			io=1455MiB (1526MB), run=14770-14770msec
# ./mptcp_nvme.sh mptcp 4 round-robin 1
   READ: bw=426MiB/s (447MB/s), 426MiB/s-426MiB/s (447MB/s-447MB/s),
			io=4533MiB (4753MB), run=10637-10637msec
  WRITE: bw=428MiB/s (449MB/s), 428MiB/s-428MiB/s (449MB/s-449MB/s),
			io=4507MiB (4725MB), run=10522-10522msec

Lossy network, comparison between TCP and MPTCP using the "queue- depth"
policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 queue-depth 1
   READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s),
			io=2179MiB (2285MB), run=12965-12965msec
  WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s),
			io=1590MiB (1667MB), run=12418-12418msec
# ./mptcp_nvme.sh mptcp 4 queue-depth 1
   READ: bw=425MiB/s (445MB/s), 425MiB/s-425MiB/s (445MB/s-445MB/s),
			io=4536MiB (4756MB), run=10677-10677msec
  WRITE: bw=414MiB/s (434MB/s), 414MiB/s-414MiB/s (434MB/s-434MB/s),
			io=4447MiB (4663MB), run=10733-10733msec
Quoted 6 lines

Conclusion: MPTCP achieves bandwidth aggregation comparable to that of
NVMe multipath while offering better resilience against network interference.

The full test results are in the attachment.

Thank you very much for your interest in NVMe MPTCP. During my presentation, I mentioned that I would show you the performance test results of MPTCP when configuring several NVMe multipath paths. I spent some time updating the scripts, and here are the latest results. What do you think of these results? Please give me some suggestions.

Thanks,
-Geliang

Quoted 135 lines

Thanks,
-Geliang

[1]
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Quoted 128 lines

Thanks,
-Geliang

Quoted 123 lines · Geliang Tang

Hi Nilay, Ming,

Thank you again for your interest in NVMe over MPTCP.

Quoted 71 lines · Geliang Tang

Hi Nilay,

Thanks for your reply.

Quoted 60 lines · Nilay Shroff
Quoted 19 lines · Geliang Tang

3. Performance Benefits

This new feature has been evaluated in different
environments:

I conducted 'NVMe over MPTCP' tests between two PCs, each equipped
with
two Gigabit NICs and directly connected via Ethernet cables.
Using
'NVMe over TCP', the fio benchmark showed a speed of approximately
100
MiB/s. In contrast, 'NVMe over MPTCP' achieved about 200 MiB/s
with
fio, doubling the throughput.

Quoted 8 lines

In a virtual machine test environment simulating four NICs on
both
sides, 'NVMe over MPTCP' delivered bandwidth up to four times
that
of
standard TCP.

This is interesting. Did you try using an NVMe multipath iopolicy
other
than the default numa policy? Assuming both the host and target
are
multihomed,
configuring round-robin or queue-depth may provide performance
comparable
to what you are seeing with MPTCP.

I think MPTCP shall distribute traffic using transport-level metrics
such as
RTT, cwnd, and packet loss, whereas the NVMe multipath layer makes
decisions
based on ANA state, queue depth, and NUMA locality. In a setup
with
multiple
active paths, switching the iopolicy from numa to round-robin or
queue-depth
could improve load distribution across controllers and thus improve
performance.

Quoted 5 lines

IMO, it would be useful to test with those policies and compare
the
results
against the MPTCP setup.

Quoted 10 lines

Ming Lei also made a similar comment. In my experiments, I didn't
set
the multipath iopolicy, so I was using the default numa policy. In
the
follow-up, I'll adjust it to round-robin or queue-depth and rerun
the
experiments. I'll share the results in this email thread.

Based on your feedback, I have added iopolicy support to the NVMe over
MPTCP selftest script (see patch 8 in [1]). We can set the iopolicy
to
round-robin like this:

 # ./mptcp_nvme.sh mptcp round-robin

This demonstrates that "NVMe over MPTCP" and "NVMe multipath" can work
simultaneously without conflict.

Using this test script, I compared three I/O policies: numa, round-
robin, and queue-depth. The results for fio were very similar. It's
possible that this test environment doesn't fully reflect the differences in I/O policies. I will continue to follow up with further
tests.

Thanks,
-Geliang

[1]
NVME over MPTCP, v4
https://patchwork.kernel.org/project/mptcp/cover/[email protected]/

Quoted 7 lines

Thanks,
-Geliang

Quoted 3 lines

Thanks,
--Nilay

#9 lore raw
In reply to Geliang Tang
Quoted 9 lines · Geliang Tang

Lossless network, comparison between TCP and MPTCP using the "queue- depth" policy - MPTCP and TCP show similar performance:

# ./mptcp_nvme.sh tcp 4 queue-depth 0
   READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s),
                        io=4632MiB (4857MB), run=10169-10169msec
  WRITE: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s),
                        io=4666MiB (4893MB), run=10250-10250msec
Quoted 5 lines
# ./mptcp_nvme.sh mptcp 4 queue-depth 0
   READ: bw=446MiB/s (467MB/s), 446MiB/s-446MiB/s (467MB/s-467MB/s),
                        io=4568MiB (4790MB), run=10249-10249msec
  WRITE: bw=445MiB/s (467MB/s), 445MiB/s-445MiB/s (467MB/s-467MB/s),
                        io=4563MiB (4784MB), run=10245-10245msec

This makes much more sense to me. Have you tested where one path is _not_ flaky but is slower? (3 100GBps, 1 50GBps or something like that)

Quoted 25 lines

Second set of tests: lossy network, path=4, loss=1. The tc output is as follows:

qdisc netem 8051: root refcnt 25 limit 1000 delay 5ms loss 0.5%

                        rate 1Gbit seed 14946049878654165618

Lossy network, comparison between TCP and MPTCP using the "queue-depth" policy - MPTCP is four times faster than TCP:

# ./mptcp_nvme.sh tcp 4 queue-depth 1
   READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s),
                        io=2179MiB (2285MB), run=12965-12965msec
  WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s),
                        io=1590MiB (1667MB), run=12418-12418msec
# ./mptcp_nvme.sh mptcp 4 queue-depth 1
   READ: bw=425MiB/s (445MB/s), 425MiB/s-425MiB/s (445MB/s-445MB/s),
                        io=4536MiB (4756MB), run=10677-10677msec
  WRITE: bw=414MiB/s (434MB/s), 414MiB/s-414MiB/s (434MB/s-434MB/s),
                        io=4447MiB (4663MB), run=10733-10733msec
Quoted 3 lines

Conclusion: MPTCP achieves bandwidth aggregation comparable to that of NVMe multipath while offering better resilience against network interference.

This is interesting. So, one path of 4 flaky reduces bandwidth to 1/4 bandwidth, effectively a penalty of 2 paths (I was expecting more of a penalty), while tcpmp can shake it off. Do you have a hypothosis/understanding of why?

I have a guess that selective retransmission might be kicking in (which would be good), but how is that different than expected behavior for IP/NIC bonding? (which, I think, could be implemented without an
NVMe driver/protocol change?) We generally point people away from IP/NIC bonding; although, I am (personally) not sure why.

Sincerely,
Randy Jennings