tcp: avoid struct socket cache line miss in tcp_check_space()
Status from this message: This series was applied to netdev/net-next.git (main)
Get series with b4
b4 am [email protected]Versions newest first
Changes in v3 from the cover letter
- Rebase and CC subsystem maintainers (Jakub)
- Link to v2: https://lore.kernel.org/netdev/[email protected]/
Patches in v3 All 9 applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/9 | dlm: fix send buffer backpressure handlingfs/ | +2 | 2 replies | Applied |
| 2/9 | net: add sk_set_nospace() and sk_clear_nospace()include/ net/ net/ +7 more | +52 | 0 replies | Applied |
| 3/9 | sunrpc: use sk_set_nospace() and sk_clear_nospace()net/ net/ | +4 | 1 reply | Applied |
| 4/9 | rds: use sk_set_nospace()net/ | +2 | 1 reply | Applied |
| 5/9 | dlm: use sk_set_nospace() and sk_clear_nospace()fs/ | +2 | 0 replies | Applied |
| 6/9 | drbd: use sk_set_nospace()drivers/ | +1 | 1 reply | Applied |
| 7/9 | nvme-tcp: use sk_clear_nospace()drivers/ drivers/ | +2 | 0 replies | Applied |
| 8/9 | libceph: use sk_clear_nospace()net/ | +1 | 0 replies | Applied |
| 9/9 | tcp: add tp->tcp_nospaceDocumentation/ include/ include/ +4 more | +59 | 0 replies | Applied |
Reviews trailers per patch
8 of 9 patches have a Reviewed-by or Acked-by.
| Patch | Alexander Aring | Kuniyuki Iwashima | Christoph Böhmwalder | Chuck Lever | Allison Henderson |
|---|---|---|---|---|---|
| 1/9 | A-b | R-b | |||
| 2/9 | R-b | ||||
| 3/9 | R-b | A-b | |||
| 4/9 | R-b | A-b | |||
| 5/9 | R-b | ||||
| 6/9 | R-b | A-b | |||
| 7/9 | R-b | ||||
| 8/9 | R-b | ||||
| 9/9 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 5 reviewers.
Cover letter [PATCH v3 net-next 0/9] tcp: avoid struct socket cache line miss in tcp_check_space()
tcp_check_space() is called on every incoming ACK and every transmitted packet, and tests SOCK_NOSPACE in sk->sk_socket->flags. Because struct socket lives in its own cache line and the TCP fast paths do not touch it for anything else, that test pulls in an extra cache line that is cold when the working set of active sockets is large.
This series mirrors SOCK_NOSPACE into a new u8 field, tp->tcp_nospace, placed right after tp->chrono_type in the tcp_sock_write_tx cache line group (fitting in an existing 3-byte hole before chrono_start, so no other field moves and sizeof(struct tcp_sock) is unchanged). Both the transmit and ACK fast paths already touch that cache line, making the fast-path test in tcp_check_space() free of extra cache misses while keeping __tcp_check_space() authoritative on SOCK_NOSPACE.
To maintain the invariant (SOCK_NOSPACE set => tp->tcp_nospace set) from a single choke point:
- Patch 1 fixes a long-standing bug in dlm where SOCKWQ_ASYNC_NOSPACE was tested and cleared on con->sock->flags instead of SOCK_NOSPACE.
- Patch 2 introduces sk_set_nospace() and sk_clear_nospace() and converts the core networking setters and clearers.
- Patches 3-8 convert the remaining in-kernel callers (sunrpc, rds, dlm, drbd, nvme-tcp, libceph) so that no open-coded set_bit() or clear_bit() of SOCK_NOSPACE remains in the tree.
- Patch 9 adds tp->tcp_nospace, wires it into sk_set_nospace() and sk_clear_nospace(), and switches tcp_check_space() to test it.
v3:
- Rebase and CC subsystem maintainers (Jakub)
- Link to v2: https://lore.kernel.org/netdev/[email protected]/
v2:
- Order set_bit(SOCK_NOSPACE) before tp->tcp_nospace = 1 in sk_set_nospace() and tp->tcp_nospace = 0 before
clear_bit(SOCK_NOSPACE) in sk_clear_nospace() so a concurrent
lockless tcp_poll() cannot leave SOCK_NOSPACE set with
tp->tcp_nospace cleared (Sashiko)
- Move tp->tcp_nospace to the 3-byte hole after tp->chrono_type so no
field in struct tcp_sock shifts after the AccECN bitfield additions (Sashiko)
- Clarify comment and Patch 2 changelog wording (Sashiko)
- Link to v1: https://lore.kernel.org/netdev/[email protected]/
Eric Dumazet (9):
dlm: fix send buffer backpressure handling
net: add sk_set_nospace() and sk_clear_nospace()
sunrpc: use sk_set_nospace() and sk_clear_nospace()
rds: use sk_set_nospace()
dlm: use sk_set_nospace() and sk_clear_nospace()
drbd: use sk_set_nospace()
nvme-tcp: use sk_clear_nospace()
libceph: use sk_clear_nospace()
tcp: add tp->tcp_nospace
.../networking/net_cachelines/tcp_sock.rst | 1 + drivers/block/drbd/drbd_worker.c | 3 +- drivers/nvme/host/tcp.c | 2 +- drivers/nvme/target/tcp.c | 2 +- fs/dlm/lowcomms.c | 10 ++-- include/linux/tcp.h | 3 ++ include/net/sock.h | 2 + include/net/tcp.h | 30 +++++++++++- net/ceph/messenger.c | 2 +- net/core/sock.c | 46 ++++++++++++++++++- net/core/stream.c | 6 +-- net/ipv4/tcp.c | 5 +- net/ipv4/tcp_bpf.c | 2 +- net/ipv4/tcp_input.c | 8 +++- net/kcm/kcmsock.c | 4 +- net/mptcp/protocol.c | 4 +- net/mptcp/subflow.c | 5 ++ net/rds/tcp_send.c | 5 +- net/smc/af_smc.c | 2 +- net/smc/smc_tx.c | 6 +-- net/sunrpc/svcsock.c | 4 +- net/sunrpc/xprtsock.c | 4 +- net/tls/tls_sw.c | 2 +- 23 files changed, 121 insertions(+), 37 deletions(-)
Signature · 1 line
2.56.0.rc1.315.gc6ed9934b7-goog