Rework retry algorithm used when sending MADs
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest9 patches 0 repliesNew
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/9 | IB/mad: Apply timeout modification (CM MRA) only oncedrivers/ | +0 | 0 replies | |
| 2/9 | IB/mad: Add deadline for send MADsdrivers/ drivers/ include/ | +80 | 0 replies | |
| 3/9 | RDMA/sa_query: Enforce min retry interval and deadlinedrivers/ | +22 | 0 replies | |
| 4/9 | RDMA/nldev: Add sa-min-timeout management attributedrivers/ drivers/ drivers/ +1 more | +170 | 0 replies | |
| 5/9 | IB/umad: Set deadline when sending non-RMPP MADsdrivers/ include/ | +18 | 0 replies | |
| 6/9 | IB/cm: Set deadline when sending MADsdrivers/ | +13 | 0 replies | |
| 7/9 | IB/mad: Exponential backoff when retrying sendsdrivers/ drivers/ | +53 | 0 replies | |
| 8/9 | RDMA/nldev: Add mad-linear-timeouts management attributedrivers/ drivers/ drivers/ +1 more | +60 | 0 replies | |
| 9/9 | IB/cma: Lower response timeout to roughly 1sdrivers/ | +1 | 0 replies |
Reviews trailers per patch
All 9 patches have a Reviewed-by or Acked-by.
| Patch | Sean Hefty |
|---|---|
| 1/9 | R-b |
| 2/9 | R-b |
| 3/9 | R-b |
| 4/9 | R-b |
| 5/9 | R-b |
| 6/9 | R-b |
| 7/9 | R-b |
| 8/9 | R-b |
| 9/9 | R-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH rdma-next 0/9] Rework retry algorithm used when sending MADs
From Vlad,
This series aims to improve behaviour of a MAD sender under congestion and/or receiver overload. We've seen significant drops in goodput when MAD receivers are overloaded. This typically happens with SA requests, which are served by a single node (SM), but can also happen with CM.
Patch 7 introduces the main change: exponential backoff. This new retry algorithm is applied to all MADs, except RMPP and OPA. To avoid reductions in recovery speed under transient failures, the exponential backoff algorithm only engages after a certain number of linear timeouts is experienced. The backoff algorithm resets to beginning after a CM MRA, assuming the remote is not longer overloaded.
Because a trade-off between speed of recovery under transient failure and reducing load from unnecessary retries under persistent failure must be made, and this trade-off depends on the network scale, patch 8 makes mad-linear-timeouts configurable.
Patch 1 makes CM MRA apply only once, to prevent entering an excessive delay condition, even when the receiver is likely no longer overloaded.
The exponential backoff algorithm (a) increases the time until a send MAD reaches the final timeout, and (b) makes it hard to predict by callers. Since certain callers appear to care about this, Patch 2 introduces a new option, deadline, which can be used to enforce when the final timeout is reached. SA, UMAD and CM are updated to use this new parameter (patches 3, 5, 6).
Patch 3 also solves a related issue in SA, which configures the MAD layer with extremely aggressive retry intervals, in certain cases. Because the current aggressive retry was introduced to solve another issue, patch 4 makes sa-min-timeout configurable.
Patch 9 resolves another related issue in CM, which uses a retry interval that is way too high for (low latency) RDMA networks.
In summary:
1) IB/mad: Apply timeout modification (CM MRA) only once
2) IB/mad: Add deadline for send MADs
3) RDMA/sa_query: Enforce min retry interval and deadline
4) RDMA/nldev: Add sa-min-timeout management attribute
5) IB/umad: Set deadline when sending non-RMPP MADs
6) IB/cm: Set deadline when sending MADs
7) IB/mad: Exponential backoff when retrying sends
8) RDMA/nldev: Add mad-linear-timeouts management attribute
9) IB/cma: Lower response timeout to roughly 1s
Two tunables will be added to RDMA tool (iproute2), under the 'management' namespace as follow-up:
mad-linear-timeouts
sa-min-timeout
Thanks
Vlad Dumitrescu (9):
IB/mad: Apply timeout modification (CM MRA) only once IB/mad: Add deadline for send MADs RDMA/sa_query: Enforce min retry interval and deadline RDMA/nldev: Add sa-min-timeout management attribute IB/umad: Set deadline when sending non-RMPP MADs IB/cm: Set deadline when sending MADs
IB/mad: Exponential backoff when retrying sends RDMA/nldev: Add mad-linear-timeouts management attribute IB/cma: Lower response timeout to roughly 1s
drivers/infiniband/core/cm.c | 13 +++ drivers/infiniband/core/cma.c | 2 +- drivers/infiniband/core/core_priv.h | 4 + drivers/infiniband/core/mad.c | 141 ++++++++++++++++++++++++++-- drivers/infiniband/core/mad_priv.h | 8 ++ drivers/infiniband/core/nldev.c | 133 ++++++++++++++++++++++++++ drivers/infiniband/core/sa_query.c | 81 +++++++++++++--- drivers/infiniband/core/user_mad.c | 8 ++ include/rdma/ib_mad.h | 29 ++++++ include/uapi/rdma/ib_user_mad.h | 12 ++- include/uapi/rdma/rdma_netlink.h | 7 ++ 11 files changed, 416 insertions(+), 22 deletions(-)
Signature · 1 line
2.47.0