tcp: clamp route advmss to TCP_MIN_MSS

From: Yong Wang <[email protected]>

Hi Linux kernel maintainers,

This is v2 of the fix for the divide-by-zero reachable from tcp_select_initial_window() via undersized route-derived advmss.

The previous version only rejected undersized non-zero RTAX_ADVMSS values at the route metric input point. As pointed out in review, that does not fully fix the bug because the panic is still reachable through the "default advmss" path when RTAX_ADVMSS is 0 and the effective advmss is later driven down by route MTU and min_adv_mss.

This version fixes the bug in TCP by introducing a helper that clamps route-derived advmss to TCP_MIN_MSS before it is consumed by the TCP paths that derive advmss from dst metrics.

I tested this version in QEMU and confirmed that the previously reproducible divide-by-zero no longer occurs on the advmss=0 + small-MTU/min_adv_mss path.

Thanks,
Yong Wang

---
Changes in v2:
- drop the RTAX_ADVMSS input validation from previous version
- clamp route-derived advmss in TCP instead
- switch the relevant TCP IPv4/IPv6 call sites to the new helper
- drop the long reproducer script and crash log from the cover letter
v1 link: https://lore.kernel.org/all/2c3901162c65a1d85cc1756a83a458db834d70c1.1786610865.git.edragain@163.com/

Yong Wang (1):
tcp: clamp route advmss to TCP_MIN_MSS

 include/net/tcp.h        | 5 +++++
 net/ipv4/tcp_ipv4.c      | 2 +-
 net/ipv4/tcp_minisocks.c | 2 +-
 net/ipv4/tcp_output.c    | 6 +++---
 net/ipv6/tcp_ipv6.c      | 2 +-
 5 files changed, 11 insertions(+), 6 deletions(-)

2.53.0