implement .splice_eof
Subsystems (4)
Geliang Tang
reviewercover implement .splice_eof
From: Geliang Tang <[email protected]>
This set implements .splice_eof for MPTCP and tests it.
Geliang Tang (3):
tcp: export do_tcp_splice_eof
mptcp: implement .splice_eof
selftests: mptcp: connect: trigger splice_eof
include/net/tcp.h | 1 + net/ipv4/tcp.c | 8 ++++++-- net/mptcp/protocol.c | 16 ++++++++++++++++ .../testing/selftests/net/mptcp/mptcp_connect.c | 2 +- 4 files changed, 24 insertions(+), 3 deletions(-)
Signature · 1 line
2.51.0
Geliang Tang
reviewerPATCH 1/3 tcp: export do_tcp_splice_eof
From: Geliang Tang <[email protected]>
Extract a do_tcp_splice_eof() helper from tcp_splice_eof() and export it to net/tcp.h, so that it can be used in MPTCP.
--- include/net/tcp.h | 1 + net/ipv4/tcp.c | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-)
include/net/tcp.h +1 -0
@@ -370,6 +370,7 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);int tcp_sendmsg_locked (struct sock *sk,struct msghdr *msg,size_t size);int tcp_sendmsg_fastopen (struct sock *sk,struct msghdr *msg,int *copied,size_t size,struct ubuf_info *uarg); +void do_tcp_splice_eof (struct sock *sk);void tcp_splice_eof (struct socket *sock);int tcp_send_mss (struct sock *sk,int *size_goal,int flags);int tcp_wmem_schedule (struct sock *sk,int copy);
net/ipv4/tcp.c +6 -2
@@ -1468,9 +1468,8 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) }EXPORT_SYMBOL (tcp_sendmsg); -void tcp_splice_eof (struct socket *sock) +void do_tcp_splice_eof (struct sock *sk) { -struct sock *sk = sock->sk;struct tcp_sock *tp =tcp_sk (sk);int mss_now, size_goal;
@@ -1482,6 +1481,11 @@ void tcp_splice_eof(struct socket *sock)tcp_push (sk,0 , mss_now, tp->nonagle, size_goal);release_sock (sk); } + +void tcp_splice_eof (struct socket *sock) +{ +do_tcp_splice_eof (sock->sk); +}EXPORT_IPV6_MOD_GPL (tcp_splice_eof);/*
Signature · 1 line
2.51.0
Geliang Tang
reviewerPATCH 2/3 mptcp: implement .splice_eof
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
This patch implements the .splice_eof interface for MPTCP, namely mptcp_splice_eof(), which sequentially calls do_tcp_splice_eof() for each subflow.
--- net/mptcp/protocol.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
net/mptcp/protocol.c +16 -0
@@ -4018,6 +4018,20 @@ static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr,return 0 ; } +static void mptcp_splice_eof (struct socket *sock) +{ +struct mptcp_subflow_context *subflow; +struct sock *sk = sock->sk, *ssk; + +lock_sock (sk); +mptcp_for_each_subflow (mptcp_sk (sk), subflow) { + ssk =mptcp_subflow_tcp_sock (subflow); + +do_tcp_splice_eof (ssk); + } +release_sock (sk); +} +static struct proto mptcp_prot = { .name ="MPTCP" , .owner =THIS_MODULE ,
@@ -4049,6 +4063,7 @@ static struct proto mptcp_prot = { .obj_size = sizeof(struct mptcp_sock), .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, + .splice_eof = mptcp_splice_eof, };static int mptcp_bind (struct socket *sock,struct sockaddr_unsized *uaddr,int addr_len)
@@ -4540,6 +4555,7 @@ static const struct proto_ops mptcp_stream_ops = { .set_rcvlowat = mptcp_set_rcvlowat, .read_sock = mptcp_read_sock, .splice_read = mptcp_splice_read, + .splice_eof = inet_splice_eof, };static struct inet_protosw mptcp_protosw = {
Signature · 1 line
2.51.0
Matthieu Baerts
maintainerRe: [PATCH mptcp-next 2/3] mptcp: implement .splice_eof
Hi Geliang,
Thank you for looking at that!
Quoted 5 lines · Geliang Tang
From: Geliang Tang <[email protected]>
This patch implements the .splice_eof interface for MPTCP, namely
mptcp_splice_eof(), which sequentially callsdo_tcp_splice_eof()for each subflow.
Can you please explain what this hook is supposed to do / used for please? And also why the solution is to call do_tcp_splice_eof() on each subflow?
Also, I'm a bit confused: why is this needed? Does it fix something or is it a new feature or an optimisation?
Quoted 14 lines
Suggested-by Matthieu Baerts <[email protected]>Signed-off-by Geliang Tang <[email protected]>--- net/mptcp/protocol.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)net/mptcp/protocol.c +0
-0@@ -4018,6 +4018,20 @@ static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr,return 0 ; }
net/mptcp/protocol.c static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr, · line 4030
Quoted 10 lines
net/mptcp/protocol.c +10
-0+static void mptcp_splice_eof (struct socket *sock) +{ +struct mptcp_subflow_context *subflow; +struct sock *sk = sock->sk, *ssk; + +lock_sock (sk); +mptcp_for_each_subflow (mptcp_sk (sk), subflow) { + ssk =mptcp_subflow_tcp_sock (subflow); + +do_tcp_splice_eof (ssk);
Is it fine to call this on closed subflows? e.g. if the initial subflow has been closed. (I didn't check, maybe that's OK)
Quoted 16 lines
net/mptcp/protocol.c +5
-0+ } +release_sock (sk); +} +static struct proto mptcp_prot = { .name ="MPTCP" , .owner =THIS_MODULE ,@@ -4049,6 +4063,7 @@ static struct proto mptcp_prot = { .obj_size = sizeof(struct mptcp_sock), .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, + .splice_eof = mptcp_splice_eof, };static int mptcp_bind (struct socket *sock,struct sockaddr_unsized *uaddr,int addr_len)
net/mptcp/protocol.c static const struct proto_ops mptcp_stream_ops = { · line 4558
Quoted 4 lines
net/mptcp/protocol.c +1
-0@@ -4540,6 +4555,7 @@ static const struct proto_ops mptcp_stream_ops = { .set_rcvlowat = mptcp_set_rcvlowat, .read_sock = mptcp_read_sock, .splice_read = mptcp_splice_read, + .splice_eof = inet_splice_eof,
Is this line required? Will it not call inet_splice_eof() by default? (I didn't check)
net/mptcp/protocol.c static const struct proto_ops mptcp_stream_ops = { · line 4561
Quoted 3 lines
net/mptcp/protocol.c +0
-0};static struct inet_protosw mptcp_protosw = {
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Geliang Tang
reviewerRe: [PATCH mptcp-next 2/3] mptcp: implement .splice_eof
Hi Matt,
Quoted 7 lines · Matthieu Baerts
Hi Geliang,
Thank you for looking at that!
Quoted 2 lines · Geliang Tang
From: Geliang Tang <[email protected]>
Quoted 7 lines
Quoted 4 lines
This patch implements the .splice_eof interface for MPTCP, namely
mptcp_splice_eof(), which sequentially callsdo_tcp_splice_eof()for
each subflow.Can you please explain what this hook is supposed to do / used for please?
do_tcp_splice_eof() ensures that any remaining data in the TCP send queue is flushed immediately when a sendfile() operation reaches end- of-file (EOF).
Quoted 2 lines
And also why the solution is to call
do_tcp_splice_eof()on each subflow?
MPTCP operates over multiple TCP subflows. When splicing data through an MPTCP socket, each subflow may have pending data in its send buffer that needs to be properly finalized. So here calls do_tcp_splice_eof() on each subflow.
Quoted 4 lines
Also, I'm a bit confused: why is this needed? Does it fix something or
is it a new feature or an optimisation?
It is not a fix, but a new feature, to keep consistent with TCP. Since TCP handles the splice EOF notification but MPTCP didn't.
Quoted 22 lines
Quoted 22 lines
Suggested-by Matthieu Baerts <[email protected]>Signed-off-by Geliang Tang <[email protected]>---
net/mptcp/protocol.c | 16 ++++++++++++++++1 file changed, 16 insertions(+)net/mptcp/protocol.c +0
-0@@ -4018,6 +4018,20 @@ static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr, return 0; }net/mptcp/protocol.c +6
-0+static void mptcp_splice_eof (struct socket *sock) +{ +struct mptcp_subflow_context *subflow; +struct sock *sk = sock->sk, *ssk; + +lock_sock (sk);
Quoted 8 lines
net/mptcp/protocol.c static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr, · line 4030
Quoted 4 lines
net/mptcp/protocol.c +4
-0+mptcp_for_each_subflow (mptcp_sk (sk), subflow) { + ssk =mptcp_subflow_tcp_sock (subflow); + +do_tcp_splice_eof (ssk);Is it fine to call this on closed subflows? e.g. if the initial subflow
has been closed. (I didn't check, maybe that's OK)
Good point. I will add this check in v2:
if (ssk->sk_state == TCP_CLOSE)
continue;Quoted 19 lines
Quoted 19 lines
net/mptcp/protocol.c +4
-0+ } +release_sock (sk); +} +static struct proto
mptcp_prot= {
.name = "MPTCP",
.owner =THIS_MODULE,net/mptcp/protocol.c +1
-0@@ -4049,6 +4063,7 @@ static struct proto mptcp_prot = { .obj_size = sizeof(struct mptcp_sock), .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, + .splice_eof = mptcp_splice_eof, };
static intmptcp_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)net/mptcp/protocol.c +0
-0@@ -4540,6 +4555,7 @@ static const struct proto_ops mptcp_stream_ops = {
Quoted 8 lines
net/mptcp/protocol.c static const struct proto_ops mptcp_stream_ops = { · line 4558
Quoted 4 lines
net/mptcp/protocol.c +1
-0.set_rcvlowat = mptcp_set_rcvlowat, .read_sock = mptcp_read_sock, .splice_read = mptcp_splice_read, + .splice_eof = inet_splice_eof,Is this line required? Will it not call
inet_splice_eof()by default? (I
didn't check)
Yes, this is required.
sock_splice_eof() needs to call the .splice_eof interface from struct proto_ops. To maintain consistency with regular TCP behavior, the .splice_eof interface of mptcp_stream_ops is set to inet_splice_eof too. inet_splice_eof() will switch to the protocol-specific implementation (sk->sk_prot->splice_eof), which for MPTCP is mptcp_splice_eof().
Thanks,
-Geliang
Quoted 7 lines
Quoted 4 lines
net/mptcp/protocol.c +0
-0};
static structinet_protoswmptcp_protosw= {Cheers,
Matt
Matthieu Baerts
maintainerRe: [PATCH mptcp-next 2/3] mptcp: implement .splice_eof
Hi Geliang,
Thank you for your reply!
Quoted 15 lines · Geliang Tang
Hi Matt,
Quoted 12 lines · Matthieu Baerts
Hi Geliang,
Thank you for looking at that!
Quoted 6 lines · Geliang Tang
From: Geliang Tang <[email protected]>
This patch implements the .splice_eof interface for MPTCP, namely
mptcp_splice_eof(), which sequentially callsdo_tcp_splice_eof()for
each subflow.
Quoted 6 lines
Quoted 2 lines
Can you please explain what this hook is supposed to do / used for please?
do_tcp_splice_eof()ensures that any remaining data in the TCP send queue is flushed immediately when asendfile()operation reaches end- of-file (EOF).
OK, so if I understand correctly, it means that without .splice_eof() support, the queue is not flushed immediately when a sendfile() operation reaches end-of-file (EOF). But that's OK, it will be flushed, eventually with a small delay but the most important is that all data will be transferred. Is that correct?
If it is, can you please reflect that in the commit message? I think it is essential to mention it is not linked to the 'splice()' syscall, it is an improvement, and nothing was broken before.
Quoted 7 lines
Quoted 2 lines
And also why the solution is to call
do_tcp_splice_eof()on each subflow?MPTCP operates over multiple TCP subflows. When splicing data through an MPTCP socket, each subflow may have pending data in its send buffer that needs to be properly finalized. So here calls
do_tcp_splice_eof()on each subflow.
Can you also please add a note about that in the commit message?
Quoted 71 lines
Quoted 3 lines
Also, I'm a bit confused: why is this needed? Does it fix something or
is it a new feature or an optimisation?It is not a fix, but a new feature, to keep consistent with TCP. Since TCP handles the splice EOF notification but MPTCP didn't.
Quoted 22 lines
Quoted 22 lines
Suggested-by Matthieu Baerts <[email protected]>Signed-off-by Geliang Tang <[email protected]>---
net/mptcp/protocol.c | 16 ++++++++++++++++1 file changed, 16 insertions(+)net/mptcp/protocol.c +0
-0@@ -4018,6 +4018,20 @@ static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr, return 0; }net/mptcp/protocol.c +6
-0+static void mptcp_splice_eof (struct socket *sock) +{ +struct mptcp_subflow_context *subflow; +struct sock *sk = sock->sk, *ssk; + +lock_sock (sk);Quoted 8 lines
net/mptcp/protocol.c static int mptcp_connect(struct sock *sk, struct sockaddr_unsized *uaddr, · line 4030
Quoted 4 lines
net/mptcp/protocol.c +4
-0+mptcp_for_each_subflow (mptcp_sk (sk), subflow) { + ssk =mptcp_subflow_tcp_sock (subflow); + +do_tcp_splice_eof (ssk);Is it fine to call this on closed subflows? e.g. if the initial subflow
has been closed. (I didn't check, maybe that's OK)Good point. I will add this check in v2:
if (ssk->sk_state == TCP_CLOSE) continue;Quoted 19 lines
Quoted 19 lines
net/mptcp/protocol.c +4
-0+ } +release_sock (sk); +} +static struct proto
mptcp_prot= {
.name = "MPTCP",
.owner =THIS_MODULE,net/mptcp/protocol.c +1
-0@@ -4049,6 +4063,7 @@ static struct proto mptcp_prot = { .obj_size = sizeof(struct mptcp_sock), .slab_flags = SLAB_TYPESAFE_BY_RCU, .no_autobind = true, + .splice_eof = mptcp_splice_eof, };
static intmptcp_bind(struct socket *sock, struct sockaddr_unsized *uaddr, int addr_len)net/mptcp/protocol.c +0
-0@@ -4540,6 +4555,7 @@ static const struct proto_ops mptcp_stream_ops = {Quoted 8 lines
net/mptcp/protocol.c static const struct proto_ops mptcp_stream_ops = { · line 4558
Quoted 4 lines
net/mptcp/protocol.c +1
-0.set_rcvlowat = mptcp_set_rcvlowat, .read_sock = mptcp_read_sock, .splice_read = mptcp_splice_read, + .splice_eof = inet_splice_eof,Is this line required? Will it not call
inet_splice_eof()by default? (I
didn't check)
Quoted 8 lines
Yes, this is required.
sock_splice_eof()needs to call the .splice_eof interface from structproto_ops. To maintain consistency with regular TCP behavior, the .splice_eof interface ofmptcp_stream_opsis set toinet_splice_eoftoo.inet_splice_eof()will switch to the protocol-specific implementation (sk->sk_prot->splice_eof), which for MPTCP ismptcp_splice_eof().
OK. Then be careful that inet_splice_eof() will call inet_send_prepare() which will call sock_rps_record_flow(sk). mptcp_rps_record_subflows() should be called on each subflow, probably from mptcp_splice_eof().
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Geliang Tang
reviewerPATCH 3/3 selftests: mptcp: connect: trigger splice_eof
In reply to Geliang TangFrom: Geliang Tang <[email protected]>
Increase the sendfile count by one to ensure the transmission size exceeds the actual data length. This triggers the splice_eof path in the kernel, allowing the newly implemented MPTCP splice_eof interface to be exercised during testing.
The change from 'count' to 'count + 1' forces the sendfile operation to attempt sending one more byte than available, which activates the end-of-file handling in the splicing logic and ensures coverage of the related MPTCP code paths.
--- tools/testing/selftests/net/mptcp/mptcp_connect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +1 -1
@@ -870,7 +870,7 @@ static int do_sendfile(int infd, int outfd, unsigned int count,while (count >0 ) {ssize_t r; - r =sendfile (outfd, infd,NULL , count); + r =sendfile (outfd, infd,NULL , count +1 );if (r <0 ) {perror ("sendfile" );return 3 ;
Signature · 1 line
2.51.0
Matthieu Baerts
maintainerRe: [PATCH mptcp-next 3/3] selftests: mptcp: connect: trigger splice_eof
Hi Geliang,
Quoted 11 lines · Geliang Tang
From: Geliang Tang <[email protected]>
Increase the sendfile count by one to ensure the transmission size exceeds the actual data length. This triggers the
splice_eofpath in the kernel, allowing the newly implemented MPTCPsplice_eofinterface to be exercised during testing.The change from 'count' to 'count + 1' forces the sendfile operation to attempt sending one more byte than available, which activates the end-of-file handling in the splicing logic and ensures coverage of the related MPTCP code paths.
I'm a bit confused: is this splice_eof interface not linked to "splice"? Why is it used with sendfile()?
Also, what's the behaviour without the implementation of "splice_eof()"? Was it a wrong behaviour or is it the same? What's the differences between the situation before and after this series?
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
Geliang Tang
reviewerRe: [PATCH mptcp-next 3/3] selftests: mptcp: connect: trigger splice_eof
Hi Matt,
Quoted 17 lines · Matthieu Baerts
Hi Geliang,
Quoted 7 lines · Geliang Tang
From: Geliang Tang <[email protected]>
Increase the sendfile count by one to ensure the transmission size exceeds the actual data length. This triggers the
splice_eofpath in the kernel, allowing the newly implemented MPTCPsplice_eofinterface to be exercised during testing.Quoted 6 lines
The change from 'count' to 'count + 1' forces the sendfile operation
to attempt sending one more byte than available, which activates the
end-of-file handling in the splicing logic and ensures coverage of the related MPTCP code paths.
Quoted 3 lines
I'm a bit confused: is this
splice_eofinterface not linked to "splice"?
Why is it used withsendfile()?
.splice_eof hook is indeed triggered by sendfile(), not by splice().
sendfile -> do_sendfile -> do_splice_direct -> do_splice_direct_actor -> splice_direct_to_actor -> do_splice_eof.
Quoted 5 lines
Also, what's the behaviour without the implementation of "
splice_eof()"?
Was it a wrong behaviour or is it the same? What's the differences between the situation before and after this series?
Without 'splice_eof', it was no a wrong behaviour, but MPTCP didn't previously handle the splice EOF notification. TCP handled it.
Quoted 3 lines
Cheers,
Matt
Hi Geliang,
Thank you for your modifications, that's great!
Our CI did some validations and here is its report:
- KVM Validation: normal (except selftest_mptcp_join): Success! ✅
- KVM Validation: normal (only selftest_mptcp_join): Success! ✅
- KVM Validation: debug (except selftest_mptcp_join): Unstable: 1 failed test(s): packetdrill_dss 🔴
- KVM Validation: debug (only selftest_mptcp_join): Success! ✅
- KVM Validation: btf-normal (only bpftest_all): Success! ✅
- KVM Validation: btf-debug (only bpftest_all): Success! ✅
- Task: https://github.com/multipath-tcp/mptcp_net-next/actions/runs/21584982091
Initiator: Patchew Applier
Commits: https://github.com/multipath-tcp/mptcp_net-next/commits/b605c7ce18aa
Patchwork: https://patchwork.kernel.org/project/mptcp/list/?series=1049653
If there are some issues, you can reproduce them using the same environment as the one used by the CI thanks to a docker image, e.g.:
$ cd [kernel source code]
$ docker run -v "${PWD}:${PWD}:rw" -w "${PWD}" --privileged --rm -it \
--pull always mptcp/mptcp-upstream-virtme-docker:latest \
auto-normalFor more details:
https://github.com/multipath-tcp/mptcp-upstream-virtme-docker
Please note that despite all the efforts that have been already done to have a stable tests suite when executed on a public CI like here, it is possible some reported issues are not due to your modifications. Still, do not hesitate to help us improve that ;-)
Cheers,
MPTCP GH Action bot
Bot operated by Matthieu Baerts (NGI0 Core)