net/smc: Add autocork support
Subsystems (5)
Dust Li
maintainerPATCH v2 1/1 net/smc: Add autocork support
This patch adds autocork support for SMC which could improve throughput for small message by x2 ~ x4.
The main idea is borrowed from TCP autocork with some RDMA specific modification:
1. The first message should never cork to make sure we won't bring extra latency
2. If we have posted any Tx WRs to the NIC that have not completed, cork the new messages until:
a) Receive CQE for the last Tx WR
b) We have corked enough message on the connection
3. Try to push the corked data out when we receive CQE of the last Tx WR to prevent the corked messages hang in the send queue.
Both SMC autocork and TCP autocork check the TX completion to decide whether we should cork or not. The difference is when we got a SMC Tx WR completion, the data have been confirmed by the RNIC while TCP TX completion just tells us the data have been sent out by the local NIC.
Add an atomic variable tx_pushing in smc_connection to make sure only one can send to let it cork more and save CDC slot.
SMC autocork should not bring extra latency since the first message will always been sent out immediately.
The qperf tcp_bw test shows more than x4 increase under small message size with Mellanox connectX4-Lx, same result with other throughput benchmarks like sockperf/netperf.
The qperf tcp_lat test shows SMC autocork has not increase any ping-pong latency.
BW test:
client: smc_run taskset -c 1 qperf smc-server -oo msg_size:1:64K:*2 \ -t 30 -vu tcp_bw
server: smc_run taskset -c 1 qperf
MsgSize(Bytes) TCP SMC-NoCork SMC-AutoCork
1 2.57 MB/s 698 KB/s(-73.5%) 2.98 MB/s(16.0% )
2 5.1 MB/s 1.41 MB/s(-72.4%) 5.82 MB/s(14.1% )
4 10.2 MB/s 2.83 MB/s(-72.3%) 11.7 MB/s(14.7% )
8 20.8 MB/s 5.62 MB/s(-73.0%) 22.9 MB/s(10.1% )
16 42.5 MB/s 11.5 MB/s(-72.9%) 45.5 MB/s(7.1% )
32 80.7 MB/s 22.3 MB/s(-72.4%) 86.7 MB/s(7.4% )
64 155 MB/s 45.6 MB/s(-70.6%) 160 MB/s(3.2% )
128 295 MB/s 90.1 MB/s(-69.5%) 273 MB/s(-7.5% )
256 539 MB/s 179 MB/s(-66.8%) 610 MB/s(13.2% )
512 943 MB/s 360 MB/s(-61.8%) 1.02 GB/s(10.8% )
1024 1.58 GB/s 710 MB/s(-56.1%) 1.91 GB/s(20.9% )
2048 2.47 GB/s 1.34 GB/s(-45.7%) 2.92 GB/s(18.2% )
4096 2.86 GB/s 2.5 GB/s(-12.6%) 2.4 GB/s(-16.1%)
8192 3.89 GB/s 3.14 GB/s(-19.3%) 4.05 GB/s(4.1% )
16384 3.29 GB/s 4.67 GB/s(41.9% ) 5.09 GB/s(54.7% )
32768 2.73 GB/s 5.48 GB/s(100.7%) 5.49 GB/s(101.1%)65536 3 GB/s 4.85 GB/s(61.7% ) 5.24 GB/s(74.7% )
Latency test:
client: smc_run taskset -c 1 qperf smc-server -oo msg_size:1:64K:*2 \ -t 30 -vu tcp_lat
server: smc_run taskset -c 1 qperf
MsgSize SMC-NoCork SMC-AutoCork
1 9.7 us 9.6 us( -1.03%)
2 9.43 us 9.39 us( -0.42%)
4 9.6 us 9.35 us( -2.60%)
8 9.42 us 9.2 us( -2.34%)
16 9.13 us 9.43 us( 3.29%)
32 9.19 us 9.5 us( 3.37%)
64 9.38 us 9.5 us( 1.28%)
128 9.9 us 9.29 us( -6.16%)
256 9.42 us 9.26 us( -1.70%)
512 10 us 9.45 us( -5.50%)
1024 10.4 us 9.6 us( -7.69%)
2048 10.4 us 10.2 us( -1.92%)
4096 11 us 10.5 us( -4.55%)
8192 11.7 us 11.8 us( 0.85%)
16384 14.5 us 14.2 us( -2.07%)
32768 19.4 us 19.3 us( -0.52%)
65536 28.1 us 28.8 us( 2.49%)With SMC autocork support, we can archive better throughput than TCP in most message sizes without any latency tradeoff.
---
v2: 1. Remove empty line in smc_connection
2. use Reverse Christmas tree style for local variable.
3. remove redundant container_of
--- net/smc/smc.h | 1 + net/smc/smc_cdc.c | 11 +++-- net/smc/smc_tx.c | 120 +++++++++++++++++++++++++++++++++++++++------- 3 files changed, 113 insertions(+), 19 deletions(-)
net/smc/smc.h +1 -0
@@ -192,6 +192,7 @@ struct smc_connection {* - dec on polled tx cqe */ wait_queue_head_t cdc_pend_tx_wq;/* wakeup on no cdc_pend_tx_wr*/ +atomic_t tx_pushing;/* nr_threads trying tx push */ struct delayed_work tx_work;/* retry of smc_cdc_msg_send */ u32 tx_off;/* base offset in peer rmb */
net/smc/smc_cdc.c +8 -3
@@ -48,9 +48,14 @@ static void smc_cdc_tx_handler(struct smc_wr_tx_pend_priv *pnd_snd, conn->tx_cdc_seq_fin = cdcpend->ctrl_seq; } -if (atomic_dec_and_test (&conn->cdc_pend_tx_wr) && -unlikely (wq_has_sleeper (&conn->cdc_pend_tx_wq))) -wake_up (&conn->cdc_pend_tx_wq); +if (atomic_dec_and_test (&conn->cdc_pend_tx_wr)) { +/* If this is the last pending WR complete, we must push to +* prevent hang when autocork enabled. +*/ +smc_tx_sndbuf_nonempty (conn); +if (unlikely (wq_has_sleeper (&conn->cdc_pend_tx_wq))) +wake_up (&conn->cdc_pend_tx_wq); + }WARN_ON (atomic_read (&conn->cdc_pend_tx_wr) <0 );smc_tx_sndbuf_nonfull (smc);
net/smc/smc_tx.c +104 -16
164 lines not on this page · raw message
Signature · 1 line
2.19.1.3.ge56e4f7
Quoted 72 lines · Dust Li
This patch adds autocork support for SMC which could improve throughput for small message by x2 ~ x4.
The main idea is borrowed from TCP autocork with some RDMA specific modification:
1. The first message should never cork to make sure we won't bring extra latency
2. If we have posted any Tx WRs to the NIC that have not completed, cork the new messages until:
a) Receive CQE for the last Tx WR
b) We have corked enough message on the connection
3. Try to push the corked data out when we receive CQE of the last Tx WR to prevent the corked messages hang in the send queue.Both SMC autocork and TCP autocork check the TX completion to decide whether we should cork or not. The difference is when we got a SMC Tx WR completion, the data have been confirmed by the RNIC while TCP TX completion just tells us the data have been sent out by the local NIC.
Add an atomic variable
tx_pushinginsmc_connectionto make sure only one can send to let it cork more and save CDC slot.SMC autocork should not bring extra latency since the first message will always been sent out immediately.
The qperf
tcp_bwtest shows more than x4 increase under small message size with Mellanox connectX4-Lx, same result with other throughput benchmarks like sockperf/netperf.
The qperftcp_lattest shows SMC autocork has not increase any ping-pong latency.BW test:
client: smc_run taskset -c 1 qperf smc-server -oo msg_size:1:64K:*2 \ -t 30 -vu tcp_bwserver:
smc_runtaskset -c 1 qperfMsgSize(Bytes) TCP SMC-NoCork SMC-AutoCork
1 2.57 MB/s 698 KB/s(-73.5%) 2.98 MB/s(16.0% ) 2 5.1 MB/s 1.41 MB/s(-72.4%) 5.82 MB/s(14.1% ) 4 10.2 MB/s 2.83 MB/s(-72.3%) 11.7 MB/s(14.7% ) 8 20.8 MB/s 5.62 MB/s(-73.0%) 22.9 MB/s(10.1% ) 16 42.5 MB/s 11.5 MB/s(-72.9%) 45.5 MB/s(7.1% ) 32 80.7 MB/s 22.3 MB/s(-72.4%) 86.7 MB/s(7.4% ) 64 155 MB/s 45.6 MB/s(-70.6%) 160 MB/s(3.2% ) 128 295 MB/s 90.1 MB/s(-69.5%) 273 MB/s(-7.5% ) 256 539 MB/s 179 MB/s(-66.8%) 610 MB/s(13.2% ) 512 943 MB/s 360 MB/s(-61.8%) 1.02 GB/s(10.8% ) 1024 1.58 GB/s 710 MB/s(-56.1%) 1.91 GB/s(20.9% ) 2048 2.47 GB/s 1.34 GB/s(-45.7%) 2.92 GB/s(18.2% ) 4096 2.86 GB/s 2.5 GB/s(-12.6%) 2.4 GB/s(-16.1%) 8192 3.89 GB/s 3.14 GB/s(-19.3%) 4.05 GB/s(4.1% ) 16384 3.29 GB/s 4.67 GB/s(41.9% ) 5.09 GB/s(54.7% ) 32768 2.73 GB/s 5.48 GB/s(100.7%) 5.49 GB/s(101.1%)65536 3 GB/s 4.85 GB/s(61.7% ) 5.24 GB/s(74.7% )
Latency test:
client: smc_run taskset -c 1 qperf smc-server -oo msg_size:1:64K:*2 \ -t 30 -vu tcp_latserver:
smc_runtaskset -c 1 qperfMsgSize SMC-NoCork SMC-AutoCork
1 9.7 us 9.6 us( -1.03%) 2 9.43 us 9.39 us( -0.42%) 4 9.6 us 9.35 us( -2.60%) 8 9.42 us 9.2 us( -2.34%) 16 9.13 us 9.43 us( 3.29%) 32 9.19 us 9.5 us( 3.37%) 64 9.38 us 9.5 us( 1.28%) 128 9.9 us 9.29 us( -6.16%) 256 9.42 us 9.26 us( -1.70%)
Quoted 8 lines
512 10 us 9.45 us( -5.50%)
1024 10.4 us 9.6 us( -7.69%)
2048 10.4 us 10.2 us( -1.92%)
4096 11 us 10.5 us( -4.55%)
8192 11.7 us 11.8 us( 0.85%)
16384 14.5 us 14.2 us( -2.07%)
32768 19.4 us 19.3 us( -0.52%)
65536 28.1 us 28.8 us( 2.49%)This is quite an impressive improvement! Thanks for your effort!
Could you share a bit more about how you performed these tests to give a bit more context and allow us to reproduce them on s390. I'm assuming the ConnectX-4 Lx card you're using is a 50 Gb/s model? Are you doing these tests on two bare metal hosts, one host with client/server namespaces, or between VMs? If it's namespaces or VMs are you using VFs from the same card/port or different cards. If it is two cards/ports do you have a switch or a cross cable between them?
dust.li
maintainerQuoted 84 lines · Niklas Schnelle
Quoted 72 lines · Dust Li
This patch adds autocork support for SMC which could improve throughput for small message by x2 ~ x4.
The main idea is borrowed from TCP autocork with some RDMA specific modification:
1. The first message should never cork to make sure we won't bring extra latency
2. If we have posted any Tx WRs to the NIC that have not completed, cork the new messages until:
a) Receive CQE for the last Tx WR
b) We have corked enough message on the connection
3. Try to push the corked data out when we receive CQE of the last Tx WR to prevent the corked messages hang in the send queue.Both SMC autocork and TCP autocork check the TX completion to decide whether we should cork or not. The difference is when we got a SMC Tx WR completion, the data have been confirmed by the RNIC while TCP TX completion just tells us the data have been sent out by the local NIC.
Add an atomic variable
tx_pushinginsmc_connectionto make sure only one can send to let it cork more and save CDC slot.SMC autocork should not bring extra latency since the first message will always been sent out immediately.
The qperf
tcp_bwtest shows more than x4 increase under small message size with Mellanox connectX4-Lx, same result with other throughput benchmarks like sockperf/netperf.
The qperftcp_lattest shows SMC autocork has not increase any ping-pong latency.BW test:
client: smc_run taskset -c 1 qperf smc-server -oo msg_size:1:64K:*2 \ -t 30 -vu tcp_bwserver:
smc_runtaskset -c 1 qperfMsgSize(Bytes) TCP SMC-NoCork SMC-AutoCork
1 2.57 MB/s 698 KB/s(-73.5%) 2.98 MB/s(16.0% ) 2 5.1 MB/s 1.41 MB/s(-72.4%) 5.82 MB/s(14.1% ) 4 10.2 MB/s 2.83 MB/s(-72.3%) 11.7 MB/s(14.7% ) 8 20.8 MB/s 5.62 MB/s(-73.0%) 22.9 MB/s(10.1% ) 16 42.5 MB/s 11.5 MB/s(-72.9%) 45.5 MB/s(7.1% ) 32 80.7 MB/s 22.3 MB/s(-72.4%) 86.7 MB/s(7.4% ) 64 155 MB/s 45.6 MB/s(-70.6%) 160 MB/s(3.2% ) 128 295 MB/s 90.1 MB/s(-69.5%) 273 MB/s(-7.5% ) 256 539 MB/s 179 MB/s(-66.8%) 610 MB/s(13.2% ) 512 943 MB/s 360 MB/s(-61.8%) 1.02 GB/s(10.8% ) 1024 1.58 GB/s 710 MB/s(-56.1%) 1.91 GB/s(20.9% ) 2048 2.47 GB/s 1.34 GB/s(-45.7%) 2.92 GB/s(18.2% ) 4096 2.86 GB/s 2.5 GB/s(-12.6%) 2.4 GB/s(-16.1%) 8192 3.89 GB/s 3.14 GB/s(-19.3%) 4.05 GB/s(4.1% ) 16384 3.29 GB/s 4.67 GB/s(41.9% ) 5.09 GB/s(54.7% ) 32768 2.73 GB/s 5.48 GB/s(100.7%) 5.49 GB/s(101.1%)65536 3 GB/s 4.85 GB/s(61.7% ) 5.24 GB/s(74.7% )
Latency test:
client: smc_run taskset -c 1 qperf smc-server -oo msg_size:1:64K:*2 \ -t 30 -vu tcp_latserver:
smc_runtaskset -c 1 qperfMsgSize SMC-NoCork SMC-AutoCork
1 9.7 us 9.6 us( -1.03%) 2 9.43 us 9.39 us( -0.42%) 4 9.6 us 9.35 us( -2.60%) 8 9.42 us 9.2 us( -2.34%) 16 9.13 us 9.43 us( 3.29%) 32 9.19 us 9.5 us( 3.37%) 64 9.38 us 9.5 us( 1.28%) 128 9.9 us 9.29 us( -6.16%) 256 9.42 us 9.26 us( -1.70%)Quoted 8 lines
512 10 us 9.45 us( -5.50%) 1024 10.4 us 9.6 us( -7.69%) 2048 10.4 us 10.2 us( -1.92%) 4096 11 us 10.5 us( -4.55%) 8192 11.7 us 11.8 us( 0.85%) 16384 14.5 us 14.2 us( -2.07%) 32768 19.4 us 19.3 us( -0.52%) 65536 28.1 us 28.8 us( 2.49%)This is quite an impressive improvement! Thanks for your effort!
Quoted 7 lines
Could you share a bit more about how you performed these tests to give a bit more context and allow us to reproduce them on s390. I'm assuming the ConnectX-4 Lx card you're using is a 50 Gb/s model? Are you doing these tests on two bare metal hosts, one host with client/server namespaces, or between VMs? If it's namespaces or VMs are you using VFs from the same card/port or different cards. If it is two cards/ports do you have a switch or a cross cable between them?
Sure
I did the test in the VM environment. 2 VMs within a single physical host, using 2 VFs from the same single ConnectX-4 Lx card, passthrough to each VM. the card is dual-25Gbps so the internal chip should support 50Gbps. A rough graph of the test setup is like this:
-------------------------------------
| --------- --------- | | | | | | | | | VM1 | | VM2 | | | | | | | | | ---VF1--- ---VF2--- | | ^ ^ | | | | | | |----- CX-4 Lx -----| | | Host | |------------------------------------