RFC net/smc: Introduce SMC-D-based OS internal communication acceleration
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v4 posting 2/218 patches 30 repliesIn review
- v5 RFC Latest9 patches 7 repliesIn review
- v4 RFC posting 1/29 patches 7 repliesSuperseded
Earlier versions · 3
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v3 RFC | 9 patches | 3 replies | Superseded | |
| v2 RFC | 5 patches | 12 replies | Superseded | |
| v1 RFC | 5 patches | 0 replies | Superseded |
v5->v4 from the cover letter
1. The loopback device generates SEID in the same way as the ISM devices when coexisting with ISM devices and uses a default fixed SEID in other cases.
2. Ensure each DMB token of the same loopback device is unique.
3. Fixe a crash caused by setting smcd_ops->signal_event interface to NULL.
4. Fixe a compilation warning complained by kernel test rebot.
Patches in v5 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/9 | net/smc: Decouple ism_dev from SMC-D device dumpinclude/ net/ | +4 | 0 replies | |
| 2/9 | net/smc: Decouple ism_dev from SMC-D DMB registrationdrivers/ include/ net/ | +7 | 2 replies | |
| 3/9 | net/smc: Extract v2 check helper from SMC-D device registrationnet/ net/ | +18 | 0 replies | |
| 4/9 | net/smc: Introduce SMC-D loopback deviceinclude/ net/ net/ +5 more | +486 | 1 reply | |
| 5/9 | net/smc: Introduce an interface for getting DMB attributeinclude/ net/ net/ | +14 | 1 reply | |
| 6/9 | net/smc: Introudce interfaces for DMB attach and detachinclude/ net/ net/ | +35 | 0 replies | |
| 7/9 | net/smc: Avoid data copy from sndbuf to peer RMB in SMC-Dnet/ net/ net/ | +84 | 0 replies | |
| 8/9 | net/smc: Modify cursor update logic when using mappable DMBnet/ | +39 | 0 replies | |
| 9/9 | net/smc: Add interface implementation of loopback devicenet/ net/ | +98 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v5 yet.
Cover letter [RFC PATCH net-next v5 0/9] net/smc: Introduce SMC-D-based OS internal communication acceleration
Hi, all
# Background
The background and previous discussion can be referred from [1]~[3].
We found SMC-D can be used to accelerate OS internal communication, such as loopback or between two containers within the same OS instance. So this patch set provides a kind of SMC-D dummy device (we call it the SMC-D loopback device) to emulate an ISM device, so that SMC-D can also be used on architectures other than s390. The SMC-D loopback device are designed as a system global device, visible to all containers.
# Design
This patch set basically follows the design of the previous version.
Patch #1/9 ~ #3/9 attempt to decouple ISM-related structures from the SMC-D generalized code and extract some helpers to make SMC-D protocol compatible with devices other than s390 ISM device.
Patch #4/9 introduces a kind of loopback device, which is defined as SMC-Dv2 device and designed to provide communication between SMC sockets on the same OS instance.
+-------------------------------------------+ | +--------------+ +--------------+ | | | SMC socket A | | SMC socket B | | | +--------------+ +--------------+ | | ^ ^ | | | +----------------+ | | | | | SMC stack | | | | +--->| +------------+ |<--| | | | | dummy | | | | | | device | | | | +-+------------+-+ | | OS | +-------------------------------------------+
Patch #5/9 ~ #8/9 expand SMC-D protocol interface (smcd_ops) for scenarios where SMC-D is used to communicate within VM (loopback here) or between VMs on the same host (based on virtio-ism device, see [4]). What these scenarios have in common is that the local sndbuf and peer RMB can be mapped to same physical memory region, so the data copy between the local sndbuf and peer RMB can be omitted. Performance improvement brought by this extension can be found in # Benchmark Test.
+----------+ +----------+
| socket A | | socket B |
+----------+ +----------+
| ^
| +---------+ |
regard as | | ----------|
local sndbuf | B's | regard as
| | RMB | local RMB
|-------> | |
+---------+Patch #9/9 realizes the support of loopback device for the above-mentioned expanded SMC-D protocol interface.
# Benchmark Test
* Test environments:
- VM with Intel Xeon Platinum 8 core 2.50GHz, 16 GiB mem.
- SMC sndbuf/RMB size 1MB.
* Test object:
- TCP lo: run on TCP loopback.
- domain: run on UNIX domain.
- SMC lo: run on SMC loopback device with patch #1/9 ~ #4/9.
- SMC lo-nocpy: run on SMC loopback device with patch #1/9 ~ #9/9.
1. ipc-benchmark (see [5])
- ./<foo> -c 1000000 -s 100
TCP-lo domain SMC-lo SMC-lo-nocpy
Message
rate (msg/s) 79025 115736(+46.45%) 146760(+85.71%) 149800(+89.56%)
2. sockperf
- serv: <smc_run> taskset -c <cpu> sockperf sr --tcp
- clnt: <smc_run> taskset -c <cpu> sockperf { tp | pp } --tcp --msg-size={ 64000 for tp | 14 for pp } -i 127.0.0.1 -t 30
TCP-lo SMC-lo SMC-lo-nocpy
Bandwidth(MBps) 4822.388 4940.918(+2.56%) 8086.67(+67.69%)
Latency(us) 6.298 3.352(-46.78%) 3.35(-46.81%)
3. iperf3
- serv: <smc_run> taskset -c <cpu> iperf3 -s
- clnt: <smc_run> taskset -c <cpu> iperf3 -c 127.0.0.1 -t 15
TCP-lo SMC-lo SMC-lo-nocpy
Bitrate(Gb/s) 40.7 40.5(-0.49%) 72.4(+77.89%)
4. nginx/wrk
- serv: <smc_run> nginx
- clnt: <smc_run> wrk -t 8 -c 500 -d 30 http://127.0.0.1:80
TCP-lo SMC-lo SMC-lo-nocpy
Requests/s 155994.57 214544.79(+37.53%) 215538.55(+38.17%)
v5->v4
1. The loopback device generates SEID in the same way as the ISM devices when coexisting with ISM devices and uses a default fixed SEID in other cases.
2. Ensure each DMB token of the same loopback device is unique.
3. Fixe a crash caused by setting smcd_ops->signal_event interface to NULL.
4. Fixe a compilation warning complained by kernel test rebot.
v4->v3
1. Rebase to the latest net-next;
2. Introduce SEID helper. SMC-D loopback will return SMCD_DEFAULT_V2_SEID. And if it coexist with ISM device, the SEID of ISM device will overwrite SMCD_DEFAULT_V2_SEID as smc_ism_v2_system_eid.
3. Won't remove dmb_node from hashtable until no sndbuf attaching to it.
Something postponed in this version
1. Hierarchy perference of SMC-D devices when loopback and ISM devices coexist, which will be determinated after comparing the performance of loopback and ISM.
v3->v2
1. Adapt new generalized interface provided by [2];
2. Select loopback device through SMC-D v2 protocol;
3. Split the loopback-related implementation and generic implementation into different patches more reasonably.
v1->v2
1. Fix some build WARNINGs complained by kernel test rebot
2. Add iperf3 test data.
[1] https://lore.kernel.org/netdev/[email protected]/
[2] https://lore.kernel.org/netdev/[email protected]/
[3] https://lore.kernel.org/netdev/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://github.com/goldsborough/ipc-bench
Wen Gu (9):
net/smc: Decouple ism_dev from SMC-D device dump net/smc: Decouple ism_dev from SMC-D DMB registration net/smc: Extract v2 check helper from SMC-D device registration net/smc: Introduce SMC-D loopback device
net/smc: Introduce an interface for getting DMB attribute net/smc: Introudce interfaces for DMB attach and detach net/smc: Avoid data copy from sndbuf to peer RMB in SMC-D net/smc: Modify cursor update logic when using mappable DMB net/smc: Add interface implementation of loopback device
drivers/s390/net/ism_drv.c | 5 +- include/net/smc.h | 18 +- net/smc/Makefile | 2 +- net/smc/af_smc.c | 26 ++- net/smc/smc_cdc.c | 59 ++++-- net/smc/smc_cdc.h | 1 + net/smc/smc_core.c | 70 ++++++- net/smc/smc_core.h | 1 + net/smc/smc_ism.c | 79 ++++++-- net/smc/smc_ism.h | 6 + net/smc/smc_loopback.c | 491 +++++++++++++++++++++++++++++++++++++++++++++ net/smc/smc_loopback.h | 56 ++++++ 12 files changed, 777 insertions(+), 37 deletions(-) create mode 100644 net/smc/smc_loopback.c create mode 100644 net/smc/smc_loopback.h
Signature · 1 line
1.8.3.1