Skip to contentSkip to conversation

MPTCP architecture proposal for discussion

RFC PATCH Superseded Mat Martineau 4 participants 56 messages Oct 5, 2018 – Oct 11, 2018 Page 1 of 2

Subsystems (3)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Mat Martineau

author
#0 lore raw

v3 cover MPTCP architecture proposal for discussion

Hello everyone,

Peter and I have been working on this patch set to show how how MPTCP can fit in to the Linux networking stack using these design ideas:

* Applications opt-in to MPTCP using IPPROTO_MPTCP, regular TCP sockets are still the default. A socket created with socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP) will attempt to form a MPTCP connection. IPPROTO_MPTCP == 99 as a placeholder.

* Subflows exist within the kernel as separate sockets, owned by a MPTCP connection-level socket that is visible to userspace.

* Adds private pointers to struct sk_buff to store MPTCP metadata.

* Adds the CONFIG_MPTCP option to Kconfig.

The following patches can form an MPTCP connection with the multipath-tcp.org kernel (tested with v0.94), and send DSS mappings that are accepted for the initial data packet. It is an early implementation, and I don't represent it as being upstreamable as-is or being everyone's idea of what an eventual upstream implementation will necessarily look like. It has significant limitations:

* Only one subflow is supported, no joins, and only ipv4.

* Does not support DSS checksums. Checksums must be disabled on the remote stack (for multipath-tcp.org, 'sudo sysctl -w net.mptcp.mptcp_checksum=0')

* Lots of debug statements (although they use dynamic debug and are disabled by default) and TODOs.

* It's only been tested sending small amounts of data for each send

Hopefully there are are some interesting concepts to discuss, and this code helps us assess how workable the above design principles are. Thanks in advance for your feedback on the benefits or drawbacks of this code, how it might be improved, or how other approaches might compare.

The patch set applies to net-next (as of commit 2ad0d5269970). I have also pushed it to:

https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git
(mptcp-proposal branch)

v3 changes: Change skb extension technique, change rx path to use error queue, add foundational code for multiple subflows, and many bug fixes.

v2 changes: Added receive path implementation (last two patches). Reworked TCP option writing. Miscellaneous bug fixes including header dependency cleanup.

Mat Martineau (6):
tcp: Add MPTCP option number
tcp: Define IPPROTO_MPTCP
skbuff: Add private data pointer
tcp: Export low-level TCP functions
mptcp: Write MPTCP DSS headers to outgoing data packets
mptcp: Implement MPTCP receive path

Peter Krystad (10):
mptcp: Add MPTCP socket stubs
mptcp: Handle MPTCP TCP options
tcp: Add IPPROTO_SUBFLOW
tcp: expose tcp routines and structs for MPTCP
mptcp: Create SUBFLOW socket for outgoing connections
mptcp: Create SUBFLOW socket for incoming connections
mptcp: Add key generation and token tree
mptcp: Add shutdown() socket operation
mptcp: Add setsockopt()/getsockopt() socket operations
mptcp: Make connection_list a real list of subflows

 include/linux/skbuff.h        |  16 +-
 include/linux/tcp.h           |  26 +
 include/net/inet_common.h     |   3 +
 include/net/mptcp.h           | 205 ++++++++
 include/net/tcp.h             |   8 +
 include/uapi/linux/errqueue.h |   1 +
 include/uapi/linux/in.h       |   4 +
 net/Kconfig                   |   1 +
 net/Makefile                  |   1 +
 net/core/skbuff.c             |   5 +
 net/ipv4/af_inet.c            |   2 +-
 net/ipv4/tcp.c                |  12 +-
 net/ipv4/tcp_input.c          |  18 +
 net/ipv4/tcp_ipv4.c           |   4 +-
 net/ipv4/tcp_output.c         | 239 ++++++++-
 net/mptcp/Kconfig             |  10 +
 net/mptcp/Makefile            |   3 +
 net/mptcp/crypto.c            | 215 ++++++++
 net/mptcp/options.c           | 263 ++++++++++
 net/mptcp/protocol.c          | 891 ++++++++++++++++++++++++++++++++++
 net/mptcp/subflow.c           | 377 ++++++++++++++
 net/mptcp/token.c             | 265 ++++++++++
 22 files changed, 2544 insertions(+), 25 deletions(-)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/crypto.c
 create mode 100644 net/mptcp/options.c
 create mode 100644 net/mptcp/protocol.c
 create mode 100644 net/mptcp/subflow.c
 create mode 100644 net/mptcp/token.c
Signature · 1 line
2.19.1
author
#1 lore raw

RFC PATCH v3 1/16 tcp: Add MPTCP option number

TCP option 30 is allocated for MPTCP by the IANA.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/tcp.h | 1 +
 1 file changed, 1 insertion(+)
include/net/tcp.h +1 -0
index 770917d0caa7..9a7e4ac9892a 100644
@@ -183,6 +183,7 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOPT_SACK             5       /* SACK Block */
 #define TCPOPT_TIMESTAMP	8	/* Better RTT estimations/PAWS */
 #define TCPOPT_MD5SIG		19	/* MD5 Signature (RFC2385) */
+#define TCPOPT_MPTCP		30	/* Multipath TCP (RFC6824) */
 #define TCPOPT_FASTOPEN		34	/* Fast open (RFC7413) */
 #define TCPOPT_EXP		254	/* Experimental */
 /* Magic number to be after the option value for sharing TCP
Signature · 1 line
2.19.1
author
#2 lore raw

RFC PATCH v3 2/16 tcp: Define IPPROTO_MPTCP

In reply to Mat Martineau

To open a MPTCP socket with socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP), IPPROTO_MPTCP needs a value that differs from IPPROTO_TCP. The existing IPPROTO numbers mostly map directly to IANA-specified protocol numbers. MPTCP does not have a protocol number allocated because MPTCP packets use the TCP protocol number. Use private number not used OTA.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -0
index 48e8a225b985..f65975c801e5 100644
@@ -64,6 +64,8 @@ enum {
 #define IPPROTO_BEETPH		IPPROTO_BEETPH
   IPPROTO_ENCAP = 98,		/* Encapsulation Header			*/
 #define IPPROTO_ENCAP		IPPROTO_ENCAP
+  IPPROTO_MPTCP = 99,		/* Multipath TCP connection 		*/
+#define IPPROTO_MPTCP		IPPROTO_MPTCP
   IPPROTO_PIM = 103,		/* Protocol Independent Multicast	*/
 #define IPPROTO_PIM		IPPROTO_PIM
   IPPROTO_COMP = 108,		/* Compression Header Protocol		*/
Signature · 1 line
2.19.1
author
#3 lore raw

RFC PATCH v3 3/16 mptcp: Add MPTCP socket stubs

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Implements the infrastructure for MPTCP sockets.

MPTCP sockets open one in-kernel TCP socket per subflow. These subflow sockets are only managed by the MPTCP socket that owns them and are not visible from userspace. This commit allows a userspace program to open an MPTCP socket with:

  sock = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

The resulting socket is simply a wrapper around a single regular TCP socket, without any of the MPTCP protocol implemented over the wire.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h  |  33 ++++++++++
 net/Kconfig          |   1 +
 net/Makefile         |   1 +
 net/mptcp/Kconfig    |  10 ++++
 net/mptcp/Makefile   |   3 +
 net/mptcp/protocol.c | 139 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 187 insertions(+)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/protocol.c
include/net/mptcp.h new file +33 -0
new file mode 100644
index 000000000000..7f7b18b000fe
@@ -0,0 +1,33 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __NET_MPTCP_H
+#define __NET_MPTCP_H
+
+#include <linux/tcp.h>
+
+/* MPTCP connection sock */
+struct mptcp_sock {
+	/* inet_connection_sock must be the first member */
+	struct	inet_connection_sock sk;
+	struct	socket *subflow;
+};
+
+static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
+{
+	return (struct mptcp_sock *)sk;
+}
+
+#endif /* __NET_MPTCP_H */
net/Kconfig +1 -0
index 228dfa382eec..274282e9b742 100644
@@ -89,6 +89,7 @@ if INET
 source "net/ipv4/Kconfig"
 source "net/ipv6/Kconfig"
 source "net/netlabel/Kconfig"
+source "net/mptcp/Kconfig"
 
 endif # if INET
 
net/Makefile +1 -0
index bdaf53925acd..1673aab222d8 100644
@@ -87,3 +87,4 @@ endif
 obj-$(CONFIG_QRTR)		+= qrtr/
 obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
+obj-$(CONFIG_MPTCP)		+= mptcp/
net/mptcp/Kconfig new file +10 -0
new file mode 100644
index 000000000000..8e48190e5fed
@@ -0,0 +1,10 @@
+
+config MPTCP
+	bool "Multipath TCP"
+	depends on INET
+	---help---
+	  Multipath TCP (MPTCP) connections send and receive data over multiple
+	  subflows in order to utilize multiple network paths. Each subflow
+	  uses the TCP protocol, and TCP options carry header information for
+	  MPTCP.
+
net/mptcp/Makefile new file +3 -0
new file mode 100644
index 000000000000..5624e7d51d48
@@ -0,0 +1,3 @@
+obj-$(CONFIG_MPTCP) += mptcp.o
+
+mptcp-y := protocol.o
net/mptcp/protocol.c new file +139 -0

139 lines not on this page · raw message

Signature · 1 line
2.19.1
#4 lore raw

Re: [MPTCP] [RFC PATCH v3 03/16] mptcp: Add MPTCP socket stubs

Quoted 10 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Implements the infrastructure for MPTCP sockets.

MPTCP sockets open one in-kernel TCP socket per subflow. These subflow sockets are only managed by the MPTCP socket that owns them and are not visible from userspace. This commit allows a userspace program to open an MPTCP socket with:

  sock = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

Have you considered AF_MULTIPATH instead?

Because, MPTCP does not really care about IP-address family. Thus, it might make sense here.

Christoph

Quoted 255 lines

The resulting socket is simply a wrapper around a single regular TCP socket, without any of the MPTCP protocol implemented over the wire.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h  |  33 ++++++++++
 net/Kconfig          |   1 +
 net/Makefile         |   1 +
 net/mptcp/Kconfig    |  10 ++++
 net/mptcp/Makefile   |   3 +
 net/mptcp/protocol.c | 139 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 187 insertions(+)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/protocol.c
include/net/mptcp.h new file +33 -0
new file mode 100644
index 000000000000..7f7b18b000fe
@@ -0,0 +1,33 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __NET_MPTCP_H
+#define __NET_MPTCP_H
+
+#include <linux/tcp.h>
+
+/* MPTCP connection sock */
+struct mptcp_sock {
+	/* inet_connection_sock must be the first member */
+	struct	inet_connection_sock sk;
+	struct	socket *subflow;
+};
+
+static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
+{
+	return (struct mptcp_sock *)sk;
+}
+
+#endif /* __NET_MPTCP_H */
net/Kconfig +1 -0
index 228dfa382eec..274282e9b742 100644
@@ -89,6 +89,7 @@ if INET
 source "net/ipv4/Kconfig"
 source "net/ipv6/Kconfig"
 source "net/netlabel/Kconfig"
+source "net/mptcp/Kconfig"
 
 endif # if INET
 
net/Makefile +1 -0
index bdaf53925acd..1673aab222d8 100644
@@ -87,3 +87,4 @@ endif
 obj-$(CONFIG_QRTR)		+= qrtr/
 obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
+obj-$(CONFIG_MPTCP)		+= mptcp/
net/mptcp/Kconfig new file +10 -0
new file mode 100644
index 000000000000..8e48190e5fed
@@ -0,0 +1,10 @@
+
+config MPTCP
+	bool "Multipath TCP"
+	depends on INET
+	---help---
+	  Multipath TCP (MPTCP) connections send and receive data over multiple
+	  subflows in order to utilize multiple network paths. Each subflow
+	  uses the TCP protocol, and TCP options carry header information for
+	  MPTCP.
+
net/mptcp/Makefile new file +3 -0
new file mode 100644
index 000000000000..5624e7d51d48
@@ -0,0 +1,3 @@
+obj-$(CONFIG_MPTCP) += mptcp.o
+
+mptcp-y := protocol.o
net/mptcp/protocol.c new file +139 -0
new file mode 100644
index 000000000000..c1eb4afc3ca4
@@ -0,0 +1,139 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_sendmsg(subflow, msg);
+}
+
+static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			 int nonblock, int flags, int *addr_len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_recvmsg(subflow, msg, flags);
+}
+
+static int mptcp_init_sock(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *sf;
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			       &sf);
+	if (!err) {
+		pr_debug("subflow=%p", sf->sk);
+		msk->subflow = sf;
+	}
+
+	return err;
+}
+
+static void mptcp_close(struct sock *sk, long timeout)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		sock_release(msk->subflow);
+	}
+}
+
+static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int err;
+
+	saddr->sa_family = AF_INET;
+
+	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+
+	err = kernel_connect(msk->subflow, saddr, len, 0);
+
+	sk->sk_state = TCP_ESTABLISHED;
+
+	return err;
+}
+
+static struct proto mptcp_prot = {
+	.name		= "MPTCP",
+	.owner		= THIS_MODULE,
+	.init		= mptcp_init_sock,
+	.close		= mptcp_close,
+	.accept		= inet_csk_accept,
+	.connect	= mptcp_connect,
+	.shutdown	= tcp_shutdown,
+	.sendmsg	= mptcp_sendmsg,
+	.recvmsg	= mptcp_recvmsg,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.obj_size	= sizeof(struct mptcp_sock),
+	.no_autobind	= 1,
+};
+
+static struct inet_protosw mptcp_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_MPTCP,
+	.prot		= &mptcp_prot,
+	.ops		= &inet_stream_ops,
+};
+
+static int __init mptcp_init(void)
+{
+	int err;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err)
+		return err;
+
+	inet_register_protosw(&mptcp_protosw);
+
+	return 0;
+}
+
+static void __exit mptcp_exit(void)
+{
+	inet_unregister_protosw(&mptcp_protosw);
+	proto_unregister(&mptcp_prot);
+}
+
+module_init(mptcp_init);
+module_exit(mptcp_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NET_PF_PROTO(PF_INET, IPPROTO_MPTCP);
+MODULE_ALIAS_NET_PF_PROTO(PF_INET6, IPPROTO_MPTCP);
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp
author
#5 lore raw

Re: [MPTCP] [RFC PATCH v3 03/16] mptcp: Add MPTCP socket stubs

Quoted 10 lines · Christoph Paasch
Quoted 9 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Implements the infrastructure for MPTCP sockets.

MPTCP sockets open one in-kernel TCP socket per subflow. These subflow sockets are only managed by the MPTCP socket that owns them and are not visible from userspace. This commit allows a userspace program to open an MPTCP socket with:

Quoted 6 lines
Quoted 1 line
  sock = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

Have you considered AF_MULTIPATH instead?

Because, MPTCP does not really care about IP-address family. Thus, it might make sense here.

Yes - but it's been a while since we discussed it.

My first attempt at implementing the socket infrastructure did take the AF_MULTIPATH approach, and I quickly learned a few things:

1. When I tried to write a test program using python's socket module, I found that a lot of TCP-like behavior depended on using AF_INET/AF_INET6, so nothing worked with a non-standard address family. It did, however, handle a non-standard proto well. As I considered what this meant (just a python quirk, or something bigger?), it did seem that the IPPROTO_MPTCP approach would be easier to adapt to existing programs in general.

2. The generic functionality provided by the kernel's af_inet infrastructure is useful for MPTCP

3. Various combinations of AF_INET, AF_INET6, and the IPV6_V6ONLY socket option provide a way to control v4/v6 selection for the initial subflow and whether later subflows may be mixed v4/v6.

4. MPTCP is a layer above IP and does have a natural place grouped with TCP/UDP/UDP-Lite/SCTP/etc. under AF_INET/AF_INET6.

Some time later I found this post with a paragraph about AF_SMC (vs AF_INET plus an option) that made some good points:

https://lwn.net/Articles/723123/

I can update the wiki design page if we're in agreement about this aspect of the API. If we're not in agreement yet, let's work on that!

Mat

Quoted 252 lines
Quoted 252 lines

The resulting socket is simply a wrapper around a single regular TCP socket, without any of the MPTCP protocol implemented over the wire.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h  |  33 ++++++++++
 net/Kconfig          |   1 +
 net/Makefile         |   1 +
 net/mptcp/Kconfig    |  10 ++++
 net/mptcp/Makefile   |   3 +
 net/mptcp/protocol.c | 139 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 187 insertions(+)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/protocol.c
include/net/mptcp.h new file +33 -0
new file mode 100644
index 000000000000..7f7b18b000fe
@@ -0,0 +1,33 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __NET_MPTCP_H
+#define __NET_MPTCP_H
+
+#include <linux/tcp.h>
+
+/* MPTCP connection sock */
+struct mptcp_sock {
+	/* inet_connection_sock must be the first member */
+	struct	inet_connection_sock sk;
+	struct	socket *subflow;
+};
+
+static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
+{
+	return (struct mptcp_sock *)sk;
+}
+
+#endif /* __NET_MPTCP_H */
net/Kconfig +1 -0
index 228dfa382eec..274282e9b742 100644
@@ -89,6 +89,7 @@ if INET
 source "net/ipv4/Kconfig"
 source "net/ipv6/Kconfig"
 source "net/netlabel/Kconfig"
+source "net/mptcp/Kconfig"
 
 endif # if INET
 
net/Makefile +1 -0
index bdaf53925acd..1673aab222d8 100644
@@ -87,3 +87,4 @@ endif
 obj-$(CONFIG_QRTR)		+= qrtr/
 obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
+obj-$(CONFIG_MPTCP)		+= mptcp/
net/mptcp/Kconfig new file +10 -0
new file mode 100644
index 000000000000..8e48190e5fed
@@ -0,0 +1,10 @@
+
+config MPTCP
+	bool "Multipath TCP"
+	depends on INET
+	---help---
+	  Multipath TCP (MPTCP) connections send and receive data over multiple
+	  subflows in order to utilize multiple network paths. Each subflow
+	  uses the TCP protocol, and TCP options carry header information for
+	  MPTCP.
+
net/mptcp/Makefile new file +3 -0
new file mode 100644
index 000000000000..5624e7d51d48
@@ -0,0 +1,3 @@
+obj-$(CONFIG_MPTCP) += mptcp.o
+
+mptcp-y := protocol.o
net/mptcp/protocol.c new file +139 -1
new file mode 100644
index 000000000000..c1eb4afc3ca4
@@ -0,0 +1,139 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_sendmsg(subflow, msg);
+}
+
+static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			 int nonblock, int flags, int *addr_len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_recvmsg(subflow, msg, flags);
+}
+
+static int mptcp_init_sock(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *sf;
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			       &sf);
+	if (!err) {
+		pr_debug("subflow=%p", sf->sk);
+		msk->subflow = sf;
+	}
+
+	return err;
+}
+
+static void mptcp_close(struct sock *sk, long timeout)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		sock_release(msk->subflow);
+	}
+}
+
+static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int err;
+
+	saddr->sa_family = AF_INET;
+
+	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+
+	err = kernel_connect(msk->subflow, saddr, len, 0);
+
+	sk->sk_state = TCP_ESTABLISHED;
+
+	return err;
+}
+
+static struct proto mptcp_prot = {
+	.name		= "MPTCP",
+	.owner		= THIS_MODULE,
+	.init		= mptcp_init_sock,
+	.close		= mptcp_close,
+	.accept		= inet_csk_accept,
+	.connect	= mptcp_connect,
+	.shutdown	= tcp_shutdown,
+	.sendmsg	= mptcp_sendmsg,
+	.recvmsg	= mptcp_recvmsg,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.obj_size	= sizeof(struct mptcp_sock),
+	.no_autobind	= 1,
+};
+
+static struct inet_protosw mptcp_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_MPTCP,
+	.prot		= &mptcp_prot,
+	.ops		= &inet_stream_ops,
+};
+
+static int __init mptcp_init(void)
+{
+	int err;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err)
+		return err;
+
+	inet_register_protosw(&mptcp_protosw);
+
+	return 0;
+}
+
+static void __exit mptcp_exit(void)
+{
+	inet_unregister_protosw(&mptcp_protosw);
+	proto_unregister(&mptcp_prot);
+}
+
+module_init(mptcp_init);
+module_exit(mptcp_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NET_PF_PROTO(PF_INET, IPPROTO_MPTCP);
+MODULE_ALIAS_NET_PF_PROTO(PF_INET6, IPPROTO_MPTCP);
--

2.19.1

Quoted 5 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

#6 lore raw

Re: [MPTCP] [RFC PATCH v3 03/16] mptcp: Add MPTCP socket stubs

Quoted 25 lines · Mat Martineau
Quoted 10 lines · Christoph Paasch
Quoted 9 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Implements the infrastructure for MPTCP sockets.

MPTCP sockets open one in-kernel TCP socket per subflow. These subflow sockets are only managed by the MPTCP socket that owns them and are not visible from userspace. This commit allows a userspace program to open an MPTCP socket with:

Quoted 6 lines
Quoted 1 line
  sock = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

Have you considered AF_MULTIPATH instead?

Because, MPTCP does not really care about IP-address family. Thus, it might make sense here.

Yes - but it's been a while since we discussed it.

My first attempt at implementing the socket infrastructure did take the AF_MULTIPATH approach, and I quickly learned a few things:

Quoted 6 lines

1. When I tried to write a test program using python's socket module, I found that a lot of TCP-like behavior depended on using AF_INET/AF_INET6, so nothing worked with a non-standard address family. It did, however, handle a non-standard proto well. As I considered what this meant (just a python quirk, or something bigger?), it did seem that the IPPROTO_MPTCP approach would be easier to adapt to existing programs in general.

That might be. We need to look into how a few programs are structured.

The only one I really deeply looked at is ATS, and there I don't think it would be a big deal.

Quoted 2 lines

2. The generic functionality provided by the kernel's af_inet infrastructure is useful for MPTCP

What do you mean by that? I mean, having a separate AF_* does not prevent us from using a separate struct proto_ops.

Having AF_MULTIPATH even allows to have our own struct net_proto_family which could be intersting if there are bits in inet_create that we don't need.

Quoted 2 lines

3. Various combinations of AF_INET, AF_INET6, and the IPV6_V6ONLY socket option provide a way to control v4/v6 selection for the initial subflow.

v4/v6 selection is already driven through the connect() address-family and/or bind() (on the listener side).

Quoted 4 lines

and whether later subflows may be mixed v4/v6.

4. MPTCP is a layer above IP and does have a natural place grouped with TCP/UDP/UDP-Lite/SCTP/etc. under AF_INET/AF_INET6.

Fair enough.

Quoted 4 lines

Some time later I found this post with a paragraph about AF_SMC (vs AF_INET plus an option) that made some good points:

https://lwn.net/Articles/723123/

Yeah - seems like the biggest pain-point is that AF_SMC did not have IPv6-support baked in and it's more an afterthought.

In the end, I think transitioning from AF_MULTIPATH to IPPROTO_MPTCP or the other way around won't be a big deal and it's a thing we can leave fairly open-ended. We should document the advantages/disadvantages of each.

Christoph

Quoted 271 lines

I can update the wiki design page if we're in agreement about this aspect of the API. If we're not in agreement yet, let's work on that!

Mat

Quoted 252 lines
Quoted 252 lines

The resulting socket is simply a wrapper around a single regular TCP socket, without any of the MPTCP protocol implemented over the wire.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h  |  33 ++++++++++
 net/Kconfig          |   1 +
 net/Makefile         |   1 +
 net/mptcp/Kconfig    |  10 ++++
 net/mptcp/Makefile   |   3 +
 net/mptcp/protocol.c | 139 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 187 insertions(+)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/protocol.c
include/net/mptcp.h new file +33 -0
new file mode 100644
index 000000000000..7f7b18b000fe
@@ -0,0 +1,33 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __NET_MPTCP_H
+#define __NET_MPTCP_H
+
+#include <linux/tcp.h>
+
+/* MPTCP connection sock */
+struct mptcp_sock {
+	/* inet_connection_sock must be the first member */
+	struct	inet_connection_sock sk;
+	struct	socket *subflow;
+};
+
+static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
+{
+	return (struct mptcp_sock *)sk;
+}
+
+#endif /* __NET_MPTCP_H */
net/Kconfig +1 -0
index 228dfa382eec..274282e9b742 100644
@@ -89,6 +89,7 @@ if INET
 source "net/ipv4/Kconfig"
 source "net/ipv6/Kconfig"
 source "net/netlabel/Kconfig"
+source "net/mptcp/Kconfig"
 
 endif # if INET
 
net/Makefile +1 -0
index bdaf53925acd..1673aab222d8 100644
@@ -87,3 +87,4 @@ endif
 obj-$(CONFIG_QRTR)		+= qrtr/
 obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
+obj-$(CONFIG_MPTCP)		+= mptcp/
net/mptcp/Kconfig new file +10 -0
new file mode 100644
index 000000000000..8e48190e5fed
@@ -0,0 +1,10 @@
+
+config MPTCP
+	bool "Multipath TCP"
+	depends on INET
+	---help---
+	  Multipath TCP (MPTCP) connections send and receive data over multiple
+	  subflows in order to utilize multiple network paths. Each subflow
+	  uses the TCP protocol, and TCP options carry header information for
+	  MPTCP.
+
net/mptcp/Makefile new file +3 -0
new file mode 100644
index 000000000000..5624e7d51d48
@@ -0,0 +1,3 @@
+obj-$(CONFIG_MPTCP) += mptcp.o
+
+mptcp-y := protocol.o
net/mptcp/protocol.c new file +139 -1
new file mode 100644
index 000000000000..c1eb4afc3ca4
@@ -0,0 +1,139 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_sendmsg(subflow, msg);
+}
+
+static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			 int nonblock, int flags, int *addr_len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_recvmsg(subflow, msg, flags);
+}
+
+static int mptcp_init_sock(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *sf;
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			       &sf);
+	if (!err) {
+		pr_debug("subflow=%p", sf->sk);
+		msk->subflow = sf;
+	}
+
+	return err;
+}
+
+static void mptcp_close(struct sock *sk, long timeout)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		sock_release(msk->subflow);
+	}
+}
+
+static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int err;
+
+	saddr->sa_family = AF_INET;
+
+	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+
+	err = kernel_connect(msk->subflow, saddr, len, 0);
+
+	sk->sk_state = TCP_ESTABLISHED;
+
+	return err;
+}
+
+static struct proto mptcp_prot = {
+	.name		= "MPTCP",
+	.owner		= THIS_MODULE,
+	.init		= mptcp_init_sock,
+	.close		= mptcp_close,
+	.accept		= inet_csk_accept,
+	.connect	= mptcp_connect,
+	.shutdown	= tcp_shutdown,
+	.sendmsg	= mptcp_sendmsg,
+	.recvmsg	= mptcp_recvmsg,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.obj_size	= sizeof(struct mptcp_sock),
+	.no_autobind	= 1,
+};
+
+static struct inet_protosw mptcp_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_MPTCP,
+	.prot		= &mptcp_prot,
+	.ops		= &inet_stream_ops,
+};
+
+static int __init mptcp_init(void)
+{
+	int err;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err)
+		return err;
+
+	inet_register_protosw(&mptcp_protosw);
+
+	return 0;
+}
+
+static void __exit mptcp_exit(void)
+{
+	inet_unregister_protosw(&mptcp_protosw);
+	proto_unregister(&mptcp_prot);
+}
+
+module_init(mptcp_init);
+module_exit(mptcp_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NET_PF_PROTO(PF_INET, IPPROTO_MPTCP);
+MODULE_ALIAS_NET_PF_PROTO(PF_INET6, IPPROTO_MPTCP);
--

2.19.1

Quoted 5 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

author
#7 lore raw

Re: [MPTCP] [RFC PATCH v3 03/16] mptcp: Add MPTCP socket stubs

Quoted 41 lines · Christoph Paasch
Quoted 25 lines · Mat Martineau
Quoted 10 lines · Christoph Paasch
Quoted 9 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Implements the infrastructure for MPTCP sockets.

MPTCP sockets open one in-kernel TCP socket per subflow. These subflow sockets are only managed by the MPTCP socket that owns them and are not visible from userspace. This commit allows a userspace program to open an MPTCP socket with:

Quoted 6 lines
Quoted 1 line
  sock = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

Have you considered AF_MULTIPATH instead?

Because, MPTCP does not really care about IP-address family. Thus, it might make sense here.

Yes - but it's been a while since we discussed it.

My first attempt at implementing the socket infrastructure did take the AF_MULTIPATH approach, and I quickly learned a few things:

Quoted 6 lines

1. When I tried to write a test program using python's socket module, I found that a lot of TCP-like behavior depended on using AF_INET/AF_INET6, so nothing worked with a non-standard address family. It did, however, handle a non-standard proto well. As I considered what this meant (just a python quirk, or something bigger?), it did seem that the IPPROTO_MPTCP approach would be easier to adapt to existing programs in general.

That might be. We need to look into how a few programs are structured.

The only one I really deeply looked at is ATS, and there I don't think it would be a big deal.

Quoted 2 lines

2. The generic functionality provided by the kernel's af_inet infrastructure is useful for MPTCP

Quoted 6 lines

What do you mean by that? I mean, having a separate AF_* does not prevent us from using a separate struct proto_ops.

Having AF_MULTIPATH even allows to have our own struct net_proto_family which could be intersting if there are bits in inet_create that we don't need.

After digging through some code and old email, I think there was some quirk in the early implementations that allowed us to use IPPROTO_MPTCP for both the subflows and the MPTCP-level socket. As the code looks now, AF_INET doesn't buy us much any more. Please disregard :)

Thinking of similarities between AF_MULTIPATH and AF_KCM, they can both be thought of as socket types that are built on top of TCP sockets, but aren't directly layered on IP.

I did remember something else that works nicely with IPPROTO_MPTCP: that idea from the NetDev conference to add a BPF hook to make TCP connections in a given cgroup default to MPTCP.

Quoted 19 lines
Quoted 3 lines

3. Various combinations of AF_INET, AF_INET6, and the IPV6_V6ONLY socket option provide a way to control v4/v6 selection for the initial subflow.

v4/v6 selection is already driven through the connect() address-family and/or bind() (on the listener side).

Quoted 4 lines

and whether later subflows may be mixed v4/v6.

4. MPTCP is a layer above IP and does have a natural place grouped with TCP/UDP/UDP-Lite/SCTP/etc. under AF_INET/AF_INET6.

Fair enough.

Quoted 4 lines

Some time later I found this post with a paragraph about AF_SMC (vs AF_INET plus an option) that made some good points:

https://lwn.net/Articles/723123/

Quoted 7 lines

Yeah - seems like the biggest pain-point is that AF_SMC did not have IPv6-support baked in and it's more an afterthought.

In the end, I think transitioning from AF_MULTIPATH to IPPROTO_MPTCP or the other way around won't be a big deal and it's a thing we can leave fairly open-ended. We should document the advantages/disadvantages of each.

I agree.

Thanks,

Mat

Quoted 269 lines
Quoted 269 lines

I can update the wiki design page if we're in agreement about this aspect of the API. If we're not in agreement yet, let's work on that!

Mat

Quoted 257 lines
Quoted 256 lines

The resulting socket is simply a wrapper around a single regular TCP socket, without any of the MPTCP protocol implemented over the wire.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h  |  33 ++++++++++
 net/Kconfig          |   1 +
 net/Makefile         |   1 +
 net/mptcp/Kconfig    |  10 ++++
 net/mptcp/Makefile   |   3 +
 net/mptcp/protocol.c | 139 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 187 insertions(+)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/protocol.c
include/net/mptcp.h new file +33 -0
new file mode 100644
index 000000000000..7f7b18b000fe
@@ -0,0 +1,33 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __NET_MPTCP_H
+#define __NET_MPTCP_H
+
+#include <linux/tcp.h>
+
+/* MPTCP connection sock */
+struct mptcp_sock {
+	/* inet_connection_sock must be the first member */
+	struct	inet_connection_sock sk;
+	struct	socket *subflow;
+};
+
+static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
+{
+	return (struct mptcp_sock *)sk;
+}
+
+#endif /* __NET_MPTCP_H */
net/Kconfig +1 -0
index 228dfa382eec..274282e9b742 100644
@@ -89,6 +89,7 @@ if INET
 source "net/ipv4/Kconfig"
 source "net/ipv6/Kconfig"
 source "net/netlabel/Kconfig"
+source "net/mptcp/Kconfig"
 
 endif # if INET
 
net/Makefile +1 -0
index bdaf53925acd..1673aab222d8 100644
@@ -87,3 +87,4 @@ endif
 obj-$(CONFIG_QRTR)		+= qrtr/
 obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
+obj-$(CONFIG_MPTCP)		+= mptcp/
net/mptcp/Kconfig new file +10 -0
new file mode 100644
index 000000000000..8e48190e5fed
@@ -0,0 +1,10 @@
+
+config MPTCP
+	bool "Multipath TCP"
+	depends on INET
+	---help---
+	  Multipath TCP (MPTCP) connections send and receive data over multiple
+	  subflows in order to utilize multiple network paths. Each subflow
+	  uses the TCP protocol, and TCP options carry header information for
+	  MPTCP.
+
net/mptcp/Makefile new file +3 -0
new file mode 100644
index 000000000000..5624e7d51d48
@@ -0,0 +1,3 @@
+obj-$(CONFIG_MPTCP) += mptcp.o
+
+mptcp-y := protocol.o
net/mptcp/protocol.c new file +139 -1
new file mode 100644
index 000000000000..c1eb4afc3ca4
@@ -0,0 +1,139 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_sendmsg(subflow, msg);
+}
+
+static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			 int nonblock, int flags, int *addr_len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_recvmsg(subflow, msg, flags);
+}
+
+static int mptcp_init_sock(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *sf;
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			       &sf);
+	if (!err) {
+		pr_debug("subflow=%p", sf->sk);
+		msk->subflow = sf;
+	}
+
+	return err;
+}
+
+static void mptcp_close(struct sock *sk, long timeout)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		sock_release(msk->subflow);
+	}
+}
+
+static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int err;
+
+	saddr->sa_family = AF_INET;
+
+	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+
+	err = kernel_connect(msk->subflow, saddr, len, 0);
+
+	sk->sk_state = TCP_ESTABLISHED;
+
+	return err;
+}
+
+static struct proto mptcp_prot = {
+	.name		= "MPTCP",
+	.owner		= THIS_MODULE,
+	.init		= mptcp_init_sock,
+	.close		= mptcp_close,
+	.accept		= inet_csk_accept,
+	.connect	= mptcp_connect,
+	.shutdown	= tcp_shutdown,
+	.sendmsg	= mptcp_sendmsg,
+	.recvmsg	= mptcp_recvmsg,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.obj_size	= sizeof(struct mptcp_sock),
+	.no_autobind	= 1,
+};
+
+static struct inet_protosw mptcp_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_MPTCP,
+	.prot		= &mptcp_prot,
+	.ops		= &inet_stream_ops,
+};
+
+static int __init mptcp_init(void)
+{
+	int err;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err)
+		return err;
+
+	inet_register_protosw(&mptcp_protosw);
+
+	return 0;
+}
+
+static void __exit mptcp_exit(void)
+{
+	inet_unregister_protosw(&mptcp_protosw);
+	proto_unregister(&mptcp_prot);
+}
+
+module_init(mptcp_init);
+module_exit(mptcp_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NET_PF_PROTO(PF_INET, IPPROTO_MPTCP);
+MODULE_ALIAS_NET_PF_PROTO(PF_INET6, IPPROTO_MPTCP);
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Quoted 4 lines
Quoted 3 lines

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

#8 lore raw

Re: [MPTCP] [RFC PATCH v3 03/16] mptcp: Add MPTCP socket stubs

In reply to Christoph Paasch
Quoted 5 lines · Christoph Paasch
Quoted 4 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Implements the infrastructure for MPTCP sockets.

Quoted 8 lines
Quoted 6 lines

MPTCP sockets open one in-kernel TCP socket per subflow. These subflow sockets are only managed by the MPTCP socket that owns them and are not visible from userspace. This commit allows a userspace program to open an MPTCP socket with:

  sock = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

Have you considered AF_MULTIPATH instead?

No. But that is a good suggestion.

Peter.

Quoted 266 lines

Because, MPTCP does not really care about IP-address family. Thus, it might make sense here.

Christoph

Quoted 255 lines

The resulting socket is simply a wrapper around a single regular TCP socket, without any of the MPTCP protocol implemented over the wire.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h  |  33 ++++++++++
 net/Kconfig          |   1 +
 net/Makefile         |   1 +
 net/mptcp/Kconfig    |  10 ++++
 net/mptcp/Makefile   |   3 +
 net/mptcp/protocol.c | 139 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 187 insertions(+)
 create mode 100644 include/net/mptcp.h
 create mode 100644 net/mptcp/Kconfig
 create mode 100644 net/mptcp/Makefile
 create mode 100644 net/mptcp/protocol.c
include/net/mptcp.h new file +33 -0
new file mode 100644
index 000000000000..7f7b18b000fe
@@ -0,0 +1,33 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __NET_MPTCP_H
+#define __NET_MPTCP_H
+
+#include <linux/tcp.h>
+
+/* MPTCP connection sock */
+struct mptcp_sock {
+	/* inet_connection_sock must be the first member */
+	struct	inet_connection_sock sk;
+	struct	socket *subflow;
+};
+
+static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
+{
+	return (struct mptcp_sock *)sk;
+}
+
+#endif /* __NET_MPTCP_H */
net/Kconfig +1 -0
index 228dfa382eec..274282e9b742 100644
@@ -89,6 +89,7 @@ if INET
 source "net/ipv4/Kconfig"
 source "net/ipv6/Kconfig"
 source "net/netlabel/Kconfig"
+source "net/mptcp/Kconfig"
 
 endif # if INET
 
net/Makefile +1 -0
index bdaf53925acd..1673aab222d8 100644
@@ -87,3 +87,4 @@ endif
 obj-$(CONFIG_QRTR)		+= qrtr/
 obj-$(CONFIG_NET_NCSI)		+= ncsi/
 obj-$(CONFIG_XDP_SOCKETS)	+= xdp/
+obj-$(CONFIG_MPTCP)		+= mptcp/
net/mptcp/Kconfig new file +10 -0
new file mode 100644
index 000000000000..8e48190e5fed
@@ -0,0 +1,10 @@
+
+config MPTCP
+	bool "Multipath TCP"
+	depends on INET
+	---help---
+	  Multipath TCP (MPTCP) connections send and receive data over multiple
+	  subflows in order to utilize multiple network paths. Each subflow
+	  uses the TCP protocol, and TCP options carry header information for
+	  MPTCP.
+
net/mptcp/Makefile new file +3 -0
new file mode 100644
index 000000000000..5624e7d51d48
@@ -0,0 +1,3 @@
+obj-$(CONFIG_MPTCP) += mptcp.o
+
+mptcp-y := protocol.o
net/mptcp/protocol.c new file +139 -0
new file mode 100644
index 000000000000..c1eb4afc3ca4
@@ -0,0 +1,139 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_sendmsg(subflow, msg);
+}
+
+static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			 int nonblock, int flags, int *addr_len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("subflow=%p", subflow->sk);
+
+	return sock_recvmsg(subflow, msg, flags);
+}
+
+static int mptcp_init_sock(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *sf;
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
+			       &sf);
+	if (!err) {
+		pr_debug("subflow=%p", sf->sk);
+		msk->subflow = sf;
+	}
+
+	return err;
+}
+
+static void mptcp_close(struct sock *sk, long timeout)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		sock_release(msk->subflow);
+	}
+}
+
+static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	int err;
+
+	saddr->sa_family = AF_INET;
+
+	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+
+	err = kernel_connect(msk->subflow, saddr, len, 0);
+
+	sk->sk_state = TCP_ESTABLISHED;
+
+	return err;
+}
+
+static struct proto mptcp_prot = {
+	.name		= "MPTCP",
+	.owner		= THIS_MODULE,
+	.init		= mptcp_init_sock,
+	.close		= mptcp_close,
+	.accept		= inet_csk_accept,
+	.connect	= mptcp_connect,
+	.shutdown	= tcp_shutdown,
+	.sendmsg	= mptcp_sendmsg,
+	.recvmsg	= mptcp_recvmsg,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.obj_size	= sizeof(struct mptcp_sock),
+	.no_autobind	= 1,
+};
+
+static struct inet_protosw mptcp_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_MPTCP,
+	.prot		= &mptcp_prot,
+	.ops		= &inet_stream_ops,
+};
+
+static int __init mptcp_init(void)
+{
+	int err;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err)
+		return err;
+
+	inet_register_protosw(&mptcp_protosw);
+
+	return 0;
+}
+
+static void __exit mptcp_exit(void)
+{
+	inet_unregister_protosw(&mptcp_protosw);
+	proto_unregister(&mptcp_prot);
+}
+
+module_init(mptcp_init);
+module_exit(mptcp_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_ALIAS_NET_PF_PROTO(PF_INET, IPPROTO_MPTCP);
+MODULE_ALIAS_NET_PF_PROTO(PF_INET6, IPPROTO_MPTCP);
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

author
#9 lore raw

RFC PATCH v3 4/16 mptcp: Handle MPTCP TCP options

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |  11 +++
 include/net/mptcp.h   |  35 ++++++++++
 include/net/tcp.h     |   2 +
 net/ipv4/tcp_input.c  |   5 ++
 net/ipv4/tcp_output.c |  67 +++++++++++++++----
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   | 152 ++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 261 insertions(+), 13 deletions(-)
 create mode 100644 net/mptcp/options.c
include/linux/tcp.h +11 -0
index 263e37271afd..cb40dcb43f95 100644
@@ -104,6 +104,16 @@ struct tcp_options_received {
 	u8	num_sacks;	/* Number of SACK blocks		*/
 	u16	user_mss;	/* mss requested by user in ioctl	*/
 	u16	mss_clamp;	/* Maximal mss, negotiated at connection setup */
+	struct {
+		u8      mp_capable : 1,
+			mp_join : 1,
+			dss : 1,
+			version : 4;
+		u8      flags;
+		u64     sndr_key;
+		u64     rcvr_key;
+	} mptcp;
+
 };
 
 static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
@@ -113,6 +123,7 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #if IS_ENABLED(CONFIG_SMC)
 	rx_opt->smc_ok = 0;
 #endif
+	rx_opt->mptcp.mp_capable = rx_opt->mptcp.mp_join = rx_opt->mptcp.dss = 0;
 }
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
include/net/mptcp.h +35 -0
index 7f7b18b000fe..f88213e31fbe 100644
@@ -18,6 +18,23 @@
 
 #include <linux/tcp.h>
 
+/* MPTCP option subtypes */
+
+#define MPTCPOPT_MP_CAPABLE	0
+#define MPTCPOPT_MP_JOIN	1
+#define MPTCPOPT_DSS		2
+#define MPTCPOPT_ADD_ADDR	3
+#define MPTCPOPT_REMOVE_ADDR	4
+#define MPTCPOPT_MP_PRIO	5
+#define MPTCPOPT_MP_FAIL	6
+#define MPTCPOPT_MP_FASTCLOSE	7
+
+/* MPTCP handshake flags */
+
+#define MPTCP_CAP_CHECKSUM_REQD	(1 << 7)
+#define MPTCP_CAP_EXTENSIBILITY	(1 << 6)
+#define MPTCP_CAP_HMAC_SHA1	(1 << 0)
+
 /* MPTCP connection sock */
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
@@ -30,4 +47,22 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 	return (struct mptcp_sock *)sk;
 }
 
+#ifdef CONFIG_MPTCP
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+			struct tcp_options_received *opt_rx);
+
+void mptcp_get_options(const struct sk_buff *skb,
+		       struct tcp_options_received *options);
+
+#else
+
+static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
+				      struct tcp_options_received *opt_rx)
+{
+}
+
+
+
+#endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
include/net/tcp.h +2 -0
index 9a7e4ac9892a..56a3455f7a50 100644
@@ -215,6 +215,8 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_MD5SIG_ALIGNED		20
 #define TCPOLEN_MSS_ALIGNED		4
 #define TCPOLEN_EXP_SMC_BASE_ALIGNED	8
+#define TCPOLEN_MPTCP_MPC_SYN		12
+#define TCPOLEN_MPTCP_MPC_SYNACK	20
 
 /* Flags in tp->nonagle */
 #define TCP_NAGLE_OFF		1	/* Nagle's algo is disabled */
net/ipv4/tcp_input.c +5 -0
index 4c2dd9f863f7..81fccf0c9120 100644
@@ -79,6 +79,7 @@
 #include <trace/events/tcp.h>
 #include <linux/static_key.h>
 #include <net/busy_poll.h>
+#include <net/mptcp.h>
 
 int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
 
@@ -3865,6 +3866,10 @@ void tcp_parse_options(const struct net *net,
 				 */
 				break;
 #endif
+                        case TCPOPT_MPTCP:
+				mptcp_parse_option(ptr, opsize, opt_rx);
+				break;
+
 			case TCPOPT_FASTOPEN:
 				tcp_parse_fastopen_option(
 					opsize - TCPOLEN_FASTOPEN_BASE,
net/ipv4/tcp_output.c +55 -12

97 lines not on this page · raw message

net/mptcp/Makefile +1 -1
index 5624e7d51d48..2bd18e3b9fda 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o
+mptcp-y := protocol.o options.o
net/mptcp/options.c new file +152 -0

152 lines not on this page · raw message

Signature · 1 line
2.19.1
#10 lore raw

Re: [MPTCP] [RFC PATCH v3 04/16] mptcp: Handle MPTCP TCP options

Hi Peter,

Nice addition!

Here are just a few questions and suggestions, mostly some details. I am not sure we want to speak about this level of details now.

Quoted 22 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |  11 +++
 include/net/mptcp.h   |  35 ++++++++++
 include/net/tcp.h     |   2 +
 net/ipv4/tcp_input.c  |   5 ++
 net/ipv4/tcp_output.c |  67 +++++++++++++++----
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   | 152 ++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 261 insertions(+), 13 deletions(-)
 create mode 100644 net/mptcp/options.c
include/linux/tcp.h +0 -0
index 263e37271afd..cb40dcb43f95 100644
@@ -104,6 +104,16 @@ struct tcp_options_received {
        u8      num_sacks;      /* Number of SACK blocks                */
        u16     user_mss;       /* mss requested by user in ioctl       */

include/linux/tcp.h struct tcp_options_received { · line 115

Quoted 10 lines
include/linux/tcp.h +9 -0
        u16     mss_clamp;      /* Maximal mss, negotiated at connection setup */
+       struct {
+               u8      mp_capable : 1,
+                       mp_join : 1,
+                       dss : 1,
+                       version : 4;
+               u8      flags;
+               u64     sndr_key;
+               u64     rcvr_key;
+       } mptcp;

For upstream, should we not surround this new structure with #ifdef CONFIG_MPTCP?

include/linux/tcp.h static inline void tcp_clear_options(struct tcp_options_received *rx_opt) · line 126

Quoted 9 lines
include/linux/tcp.h +2 -0
+
 };
 
 static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
@@ -113,6 +123,7 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #if IS_ENABLED(CONFIG_SMC)
        rx_opt->smc_ok = 0;
 #endif
+       rx_opt->mptcp.mp_capable = rx_opt->mptcp.mp_join = rx_opt->mptcp.dss = 0;

Same here for the CONFIG_MPTCP?

Quoted 115 lines
include/linux/tcp.h +0 -0
 }
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
include/net/mptcp.h +35 -0
index 7f7b18b000fe..f88213e31fbe 100644
@@ -18,6 +18,23 @@
 
 #include <linux/tcp.h>
 
+/* MPTCP option subtypes */
+
+#define MPTCPOPT_MP_CAPABLE    0
+#define MPTCPOPT_MP_JOIN       1
+#define MPTCPOPT_DSS           2
+#define MPTCPOPT_ADD_ADDR      3
+#define MPTCPOPT_REMOVE_ADDR   4
+#define MPTCPOPT_MP_PRIO       5
+#define MPTCPOPT_MP_FAIL       6
+#define MPTCPOPT_MP_FASTCLOSE  7
+
+/* MPTCP handshake flags */
+
+#define MPTCP_CAP_CHECKSUM_REQD        (1 << 7)
+#define MPTCP_CAP_EXTENSIBILITY        (1 << 6)
+#define MPTCP_CAP_HMAC_SHA1    (1 << 0)
+
 /* MPTCP connection sock */
 struct mptcp_sock {
        /* inet_connection_sock must be the first member */
@@ -30,4 +47,22 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
        return (struct mptcp_sock *)sk;
 }
 
+#ifdef CONFIG_MPTCP
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                       struct tcp_options_received *opt_rx);
+
+void mptcp_get_options(const struct sk_buff *skb,
+                      struct tcp_options_received *options);
+
+#else
+
+static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                                     struct tcp_options_received *opt_rx)
+{
+}
+
+
+
+#endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
include/net/tcp.h +2 -0
index 9a7e4ac9892a..56a3455f7a50 100644
@@ -215,6 +215,8 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_MD5SIG_ALIGNED         20
 #define TCPOLEN_MSS_ALIGNED            4
 #define TCPOLEN_EXP_SMC_BASE_ALIGNED   8
+#define TCPOLEN_MPTCP_MPC_SYN          12
+#define TCPOLEN_MPTCP_MPC_SYNACK       20
 
 /* Flags in tp->nonagle */
 #define TCP_NAGLE_OFF          1       /* Nagle's algo is disabled */
net/ipv4/tcp_input.c +5 -0
index 4c2dd9f863f7..81fccf0c9120 100644
@@ -79,6 +79,7 @@
 #include <trace/events/tcp.h>
 #include <linux/static_key.h>
 #include <net/busy_poll.h>
+#include <net/mptcp.h>
 
 int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
 
@@ -3865,6 +3866,10 @@ void tcp_parse_options(const struct net *net,
                                 */
                                break;
 #endif
+                        case TCPOPT_MPTCP:
+                               mptcp_parse_option(ptr, opsize, opt_rx);
+                               break;
+
                        case TCPOPT_FASTOPEN:
                                tcp_parse_fastopen_option(
                                        opsize - TCPOLEN_FASTOPEN_BASE,
net/ipv4/tcp_output.c +10 -0
index 597dbd749f05..a6ea8520e5d9 100644
@@ -37,6 +37,7 @@
 #define pr_fmt(fmt) "TCP: " fmt
 
 #include <net/tcp.h>
+#include <net/mptcp.h>
 
 #include <linux/compiler.h>
 #include <linux/gfp.h>
@@ -415,6 +416,58 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_WSCALE          (1 << 3)
 #define OPTION_FAST_OPEN_COOKIE        (1 << 8)
 #define OPTION_SMC             (1 << 9)
+#define OPTION_MPTCP           (1 << 10)
+
+/* MPTCP option subtypes */
+#define OPTION_MPTCP_MPC_SYN   (1 << 0)
+#define OPTION_MPTCP_MPC_SYNACK        (1 << 1)
+#define OPTION_MPTCP_MPC_ACK   (1 << 2)
+
+struct tcp_out_options {
+       u16 options;            /* bit field of OPTION_* */

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 437

Quoted 10 lines
net/ipv4/tcp_output.c +10 -0
+       u16 mss;                /* 0 to disable */
+       u8 ws;                  /* window scale, 0 to disable */
+       u8 num_sack_blocks;     /* number of SACK blocks to include */
+       u8 hash_size;           /* bytes in hash_location */
+       __u8 *hash_location;    /* temporary pointer, overloaded */
+       __u32 tsval, tsecr;     /* need to include OPTION_TS */
+       struct tcp_fastopen_cookie *fastopen_cookie;    /* Fast open cookie */
+       u16 suboptions;         /* MPTCP sub-options */
+       u64 sndr_key;
+       u64 rcvr_key;

(detail: should we prefix these new entries with "mptcp_"? Or an "mptcp_options" structure like what has been done for tcp_fastopen?)

Quoted 12 lines
net/ipv4/tcp_output.c +12 -0
+};
+
+static void mptcp_options_write(__be32 *ptr, struct tcp_out_options *opts)
+{
+       if (!(OPTION_MPTCP & opts->options))
+               return;
+
+       if ((OPTION_MPTCP_MPC_SYN |
+            OPTION_MPTCP_MPC_SYNACK |
+            OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
+               u8 len;
+               __be64 key;

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 459

Quoted 10 lines
net/ipv4/tcp_output.c +10 -0
+
+               if (OPTION_MPTCP_MPC_SYN & opts->suboptions)
+                       len = TCPOLEN_MPTCP_MPC_SYN;
+               else
+                       len = TCPOLEN_MPTCP_MPC_SYNACK;
+
+               *ptr++ = htonl((0x1e << 24) | // TCP option: Multipath TCP
+                              (len << 16)  | // length
+                              (0 <<  8)    | // subtype=MP_CAPABLE | version=0
+                              (0x1));        // flags=HMAC-SHA1

(detail: good idea to have comment here! I guess we don't need to spend time on the code style now but it was strange not to see comments made with /* */ :) )

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 462

Quoted 3 lines
net/ipv4/tcp_output.c +3 -0
+               key = cpu_to_be64(opts->sndr_key);
+               memcpy((u8 *) ptr, (u8 *) &key, 8);
+               ptr += 2;

(detail: should we use define/sizeof here, below and in mptcp_parse_option?)

Quoted 39 lines
net/ipv4/tcp_output.c +10 -11
+               if ((OPTION_MPTCP_MPC_SYNACK |
+                    OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
+                       key = cpu_to_be64(opts->rcvr_key);
+                       memcpy((u8 *) ptr, (u8 *) &key, 8);
+                       ptr += 2;
+               }
+       }
+}
 
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
@@ -431,17 +484,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-       u16 options;            /* bit field of OPTION_* */
-       u16 mss;                /* 0 to disable */
-       u8 ws;                  /* window scale, 0 to disable */
-       u8 num_sack_blocks;     /* number of SACK blocks to include */
-       u8 hash_size;           /* bytes in hash_location */
-       __u8 *hash_location;    /* temporary pointer, overloaded */
-       __u32 tsval, tsecr;     /* need to include OPTION_TS */
-       struct tcp_fastopen_cookie *fastopen_cookie;    /* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
  *
  * Beware: Something in the Internet is very sensitive to the ordering of
@@ -550,6 +592,8 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
        }
 
        smc_options_write(ptr, &options);
+
+       mptcp_options_write(ptr, opts);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,

net/ipv4/tcp_output.c static unsigned int tcp_synack_options(const struct sock *sk, · line 716

Quoted 4 lines
net/ipv4/tcp_output.c +0 -1
@@ -713,7 +757,6 @@ static unsigned int tcp_synack_options(const struct sock *sk,
                        remaining -= need;
                }
        }
-

(detail: unwanted change?)

Quoted 44 lines
net/ipv4/tcp_output.c +0 -0
        smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
        return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/Makefile +1 -1
index 5624e7d51d48..2bd18e3b9fda 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o
+mptcp-y := protocol.o options.o
net/mptcp/options.c new file +26 -0
new file mode 100644
index 000000000000..4fd64e0bdd2e
@@ -0,0 +1,152 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                       struct tcp_options_received *opt_rx)
+{
+       u8 subtype;
+       u64 *p;
+
+       opsize -= 2;

net/mptcp/options.c line 36

Quoted 10 lines
net/mptcp/options.c new file +10 -0
+       subtype = *ptr++;
+
+       /* MPTCPOPT_MP_CAPABLE
+        * 0: 4MSB=subtype, 4LSB=version
+        * 1: Handshake flags
+        * 2-9: Sender key
+        * 10-17: Receiver key (optional)
+        */
+       switch (subtype & 0xF0) {
+       case 0x00:

(detail: I guess we will use macro here later)

Matthieu

Quoted 123 lines
net/mptcp/options.c new file +116 -1
+               pr_debug("MP_CAPABLE");
+               pr_debug("flags=%02x", *ptr);
+               opt_rx->mptcp.mp_capable = 1;
+               opt_rx->mptcp.version = subtype & 0x0F;
+               opt_rx->mptcp.flags = *ptr++;
+               opt_rx->mptcp.sndr_key = get_unaligned_be64(ptr);
+               pr_debug("sndr_key=%llu", opt_rx->mptcp.sndr_key);
+               ptr += 8;
+               if (opsize > TCPOLEN_MPTCP_MPC_SYN) {
+                       opt_rx->mptcp.rcvr_key = get_unaligned_be64(ptr);
+                       pr_debug("rcvr_key=%llu", opt_rx->mptcp.rcvr_key);
+                       ptr += 8;
+               }
+               break;
+
+       /* MPTCPOPT_MP_JOIN
+        *
+        * Initial SYN
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID
+        * 2-5: Receiver token
+        * 6-9: Sender random number
+        *
+        * SYN/ACK response
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID
+        * 2-9: Sender truncated HMAC
+        * 10-13: Sender random number
+        *
+        * Third ACK
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-21: Sender HMAC
+        */
+
+       /* MPTCPOPT_DSS
+        * 0: 4MSB=subtype, 0000
+        * 1: 3MSB=0, F=Data FIN, m=DSN length, M=has DSN/SSN/DLL/checksum,
+        *    a=DACK length, A=has DACK
+        * 0, 4, or 8 bytes of DACK (depending on A/a)
+        * 0, 4, or 8 bytes of DSN (depending on M/m)
+        * 0 or 4 bytes of SSN (depending on M)
+        * 0 or 2 bytes of DLL (depending on M)
+        * 0 or 2 bytes of checksum (depending on M)
+        */
+       case 0x20:
+               pr_debug("DSS");
+               opt_rx->mptcp.dss = 1;
+               break;
+
+       /* MPTCPOPT_ADD_ADDR
+        * 0: 4MSB=subtype, 4LSB=IP version (4 or 6)
+        * 1: Address ID
+        * 4 or 16 bytes of address (depending on ip version)
+        * 0 or 2 bytes of port (depending on length)
+        */
+
+       /* MPTCPOPT_REMOVE_ADDR
+        * 0: 4MSB=subtype, 0000
+        * 1: Address ID
+        * Additional bytes: More address IDs (depending on length)
+        */
+
+       /* MPTCPOPT_MP_PRIO
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID (optional, current addr implied if not present)
+        */
+
+       /* MPTCPOPT_MP_FAIL
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-9: DSN
+        */
+
+       /* MPTCPOPT_MP_FASTCLOSE
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-9: Receiver key
+        */
+       default:
+               break;
+       }
+}
+
+void mptcp_get_options(const struct sk_buff *skb,
+                      struct tcp_options_received *opt_rx)
+{
+       const unsigned char *ptr;
+       const struct tcphdr *th = tcp_hdr(skb);
+       int length = (th->doff * 4) - sizeof(struct tcphdr);
+
+       ptr = (const unsigned char *)(th + 1);
+
+       while (length > 0) {
+               int opcode = *ptr++;
+               int opsize;
+
+               switch (opcode) {
+               case TCPOPT_EOL:
+                       return;
+               case TCPOPT_NOP:        /* Ref: RFC 793 section 3.1 */
+                       length--;
+                       continue;
+               default:
+                       opsize = *ptr++;
+                       if (opsize < 2) /* "silly options" */
+                               return;
+                       if (opsize > length)
+                               return; /* don't parse partial options */
+                       if (opcode == TCPOPT_MPTCP)
+                               mptcp_parse_option(ptr, opsize, opt_rx);
+                       ptr += opsize - 2;
+                       length -= opsize;
+               }
+       }
+}
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
#11 lore raw

Re: [MPTCP] [RFC PATCH v3 04/16] mptcp: Handle MPTCP TCP options

Quoted 6 lines · Matthieu Baerts

Hi Peter,

Nice addition!

Here are just a few questions and suggestions, mostly some details. I am not sure we want to speak about this level of details now.

Some of these details I can tidy up for our next release such as #defines for numbers, comment style, etc. I deliberately tried to leave out CONFIG_MPTCP as much as possible due to the maintainer advice we received to "just add the code so we can see what impact it has". Perhaps we will be advised to insert them during reviews...

Peter.

Quoted 196 lines
Quoted 22 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |  11 +++
 include/net/mptcp.h   |  35 ++++++++++
 include/net/tcp.h     |   2 +
 net/ipv4/tcp_input.c  |   5 ++
 net/ipv4/tcp_output.c |  67 +++++++++++++++----
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   | 152 ++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 261 insertions(+), 13 deletions(-)
 create mode 100644 net/mptcp/options.c
include/linux/tcp.h +0 -0
index 263e37271afd..cb40dcb43f95 100644
@@ -104,6 +104,16 @@ struct tcp_options_received {
        u8      num_sacks;      /* Number of SACK blocks                */
        u16     user_mss;       /* mss requested by user in ioctl       */

include/linux/tcp.h struct tcp_options_received { · line 115

Quoted 10 lines
include/linux/tcp.h +9 -0
        u16     mss_clamp;      /* Maximal mss, negotiated at connection setup */
+       struct {
+               u8      mp_capable : 1,
+                       mp_join : 1,
+                       dss : 1,
+                       version : 4;
+               u8      flags;
+               u64     sndr_key;
+               u64     rcvr_key;
+       } mptcp;

For upstream, should we not surround this new structure with #ifdef CONFIG_MPTCP?

include/linux/tcp.h static inline void tcp_clear_options(struct tcp_options_received *rx_opt) · line 126

Quoted 9 lines
include/linux/tcp.h +2 -0
+
 };
 
 static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
@@ -113,6 +123,7 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #if IS_ENABLED(CONFIG_SMC)
        rx_opt->smc_ok = 0;
 #endif
+       rx_opt->mptcp.mp_capable = rx_opt->mptcp.mp_join = rx_opt->mptcp.dss = 0;

Same here for the CONFIG_MPTCP?

Quoted 115 lines
include/linux/tcp.h +0 -0
 }
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
include/net/mptcp.h +35 -0
index 7f7b18b000fe..f88213e31fbe 100644
@@ -18,6 +18,23 @@
 
 #include <linux/tcp.h>
 
+/* MPTCP option subtypes */
+
+#define MPTCPOPT_MP_CAPABLE    0
+#define MPTCPOPT_MP_JOIN       1
+#define MPTCPOPT_DSS           2
+#define MPTCPOPT_ADD_ADDR      3
+#define MPTCPOPT_REMOVE_ADDR   4
+#define MPTCPOPT_MP_PRIO       5
+#define MPTCPOPT_MP_FAIL       6
+#define MPTCPOPT_MP_FASTCLOSE  7
+
+/* MPTCP handshake flags */
+
+#define MPTCP_CAP_CHECKSUM_REQD        (1 << 7)
+#define MPTCP_CAP_EXTENSIBILITY        (1 << 6)
+#define MPTCP_CAP_HMAC_SHA1    (1 << 0)
+
 /* MPTCP connection sock */
 struct mptcp_sock {
        /* inet_connection_sock must be the first member */
@@ -30,4 +47,22 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
        return (struct mptcp_sock *)sk;
 }
 
+#ifdef CONFIG_MPTCP
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                       struct tcp_options_received *opt_rx);
+
+void mptcp_get_options(const struct sk_buff *skb,
+                      struct tcp_options_received *options);
+
+#else
+
+static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                                     struct tcp_options_received *opt_rx)
+{
+}
+
+
+
+#endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
include/net/tcp.h +2 -0
index 9a7e4ac9892a..56a3455f7a50 100644
@@ -215,6 +215,8 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_MD5SIG_ALIGNED         20
 #define TCPOLEN_MSS_ALIGNED            4
 #define TCPOLEN_EXP_SMC_BASE_ALIGNED   8
+#define TCPOLEN_MPTCP_MPC_SYN          12
+#define TCPOLEN_MPTCP_MPC_SYNACK       20
 
 /* Flags in tp->nonagle */
 #define TCP_NAGLE_OFF          1       /* Nagle's algo is disabled */
net/ipv4/tcp_input.c +5 -0
index 4c2dd9f863f7..81fccf0c9120 100644
@@ -79,6 +79,7 @@
 #include <trace/events/tcp.h>
 #include <linux/static_key.h>
 #include <net/busy_poll.h>
+#include <net/mptcp.h>
 
 int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
 
@@ -3865,6 +3866,10 @@ void tcp_parse_options(const struct net *net,
                                 */
                                break;
 #endif
+                        case TCPOPT_MPTCP:
+                               mptcp_parse_option(ptr, opsize, opt_rx);
+                               break;
+
                        case TCPOPT_FASTOPEN:
                                tcp_parse_fastopen_option(
                                        opsize - TCPOLEN_FASTOPEN_BASE,
net/ipv4/tcp_output.c +10 -0
index 597dbd749f05..a6ea8520e5d9 100644
@@ -37,6 +37,7 @@
 #define pr_fmt(fmt) "TCP: " fmt
 
 #include <net/tcp.h>
+#include <net/mptcp.h>
 
 #include <linux/compiler.h>
 #include <linux/gfp.h>
@@ -415,6 +416,58 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_WSCALE          (1 << 3)
 #define OPTION_FAST_OPEN_COOKIE        (1 << 8)
 #define OPTION_SMC             (1 << 9)
+#define OPTION_MPTCP           (1 << 10)
+
+/* MPTCP option subtypes */
+#define OPTION_MPTCP_MPC_SYN   (1 << 0)
+#define OPTION_MPTCP_MPC_SYNACK        (1 << 1)
+#define OPTION_MPTCP_MPC_ACK   (1 << 2)
+
+struct tcp_out_options {
+       u16 options;            /* bit field of OPTION_* */

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 437

Quoted 10 lines
net/ipv4/tcp_output.c +10 -0
+       u16 mss;                /* 0 to disable */
+       u8 ws;                  /* window scale, 0 to disable */
+       u8 num_sack_blocks;     /* number of SACK blocks to include */
+       u8 hash_size;           /* bytes in hash_location */
+       __u8 *hash_location;    /* temporary pointer, overloaded */
+       __u32 tsval, tsecr;     /* need to include OPTION_TS */
+       struct tcp_fastopen_cookie *fastopen_cookie;    /* Fast open cookie */
+       u16 suboptions;         /* MPTCP sub-options */
+       u64 sndr_key;
+       u64 rcvr_key;

(detail: should we prefix these new entries with "mptcp_"? Or an "mptcp_options" structure like what has been done for tcp_fastopen?)

Quoted 18 lines
net/ipv4/tcp_output.c +18 -0
+};
+
+static void mptcp_options_write(__be32 *ptr, struct tcp_out_options *opts)
+{
+       if (!(OPTION_MPTCP & opts->options))
+               return;
+
+       if ((OPTION_MPTCP_MPC_SYN |
+            OPTION_MPTCP_MPC_SYNACK |
+            OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
+               u8 len;
+               __be64 key;
+
+               if (OPTION_MPTCP_MPC_SYN & opts->suboptions)
+                       len = TCPOLEN_MPTCP_MPC_SYN;
+               else
+                       len = TCPOLEN_MPTCP_MPC_SYNACK;
+
Quoted 8 lines

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 459

Quoted 4 lines
net/ipv4/tcp_output.c +4 -0
+               *ptr++ = htonl((0x1e << 24) | // TCP option: Multipath TCP
+                              (len << 16)  | // length
+                              (0 <<  8)    | // subtype=MP_CAPABLE | version=0
+                              (0x1));        // flags=HMAC-SHA1

(detail: good idea to have comment here! I guess we don't need to spend time on the code style now but it was strange not to see comments made with /* */ :) )

OK.

Quoted 6 lines

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 462

Quoted 4 lines
net/ipv4/tcp_output.c +3 -0
+               key = cpu_to_be64(opts->sndr_key);
+               memcpy((u8 *) ptr, (u8 *) &key, 8);
+               ptr += 2;

(detail: should we use define/sizeof here, below and in mptcp_parse_option?)

OK.

Quoted 232 lines
Quoted 40 lines
net/ipv4/tcp_output.c +10 -11
+               if ((OPTION_MPTCP_MPC_SYNACK |
+                    OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
+                       key = cpu_to_be64(opts->rcvr_key);
+                       memcpy((u8 *) ptr, (u8 *) &key, 8);
+                       ptr += 2;
+               }
+       }
+}
 
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
@@ -431,17 +484,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-       u16 options;            /* bit field of OPTION_* */
-       u16 mss;                /* 0 to disable */
-       u8 ws;                  /* window scale, 0 to disable */
-       u8 num_sack_blocks;     /* number of SACK blocks to include */
-       u8 hash_size;           /* bytes in hash_location */
-       __u8 *hash_location;    /* temporary pointer, overloaded */
-       __u32 tsval, tsecr;     /* need to include OPTION_TS */
-       struct tcp_fastopen_cookie *fastopen_cookie;    /* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
  *
  * Beware: Something in the Internet is very sensitive to the ordering of
@@ -550,6 +592,8 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
        }
 
        smc_options_write(ptr, &options);
+
+       mptcp_options_write(ptr, opts);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,

net/ipv4/tcp_output.c static unsigned int tcp_synack_options(const struct sock *sk, · line 716

Quoted 4 lines
net/ipv4/tcp_output.c +0 -1
@@ -713,7 +757,6 @@ static unsigned int tcp_synack_options(const struct sock *sk,
                        remaining -= need;
                }
        }
-

(detail: unwanted change?)

Quoted 44 lines
net/ipv4/tcp_output.c +0 -0
        smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
        return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/Makefile +1 -1
index 5624e7d51d48..2bd18e3b9fda 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o
+mptcp-y := protocol.o options.o
net/mptcp/options.c new file +26 -0
new file mode 100644
index 000000000000..4fd64e0bdd2e
@@ -0,0 +1,152 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                       struct tcp_options_received *opt_rx)
+{
+       u8 subtype;
+       u64 *p;
+
+       opsize -= 2;

net/mptcp/options.c line 36

Quoted 10 lines
net/mptcp/options.c new file +10 -0
+       subtype = *ptr++;
+
+       /* MPTCPOPT_MP_CAPABLE
+        * 0: 4MSB=subtype, 4LSB=version
+        * 1: Handshake flags
+        * 2-9: Sender key
+        * 10-17: Receiver key (optional)
+        */
+       switch (subtype & 0xF0) {
+       case 0x00:

(detail: I guess we will use macro here later)

Matthieu

Quoted 123 lines
net/mptcp/options.c new file +116 -1
+               pr_debug("MP_CAPABLE");
+               pr_debug("flags=%02x", *ptr);
+               opt_rx->mptcp.mp_capable = 1;
+               opt_rx->mptcp.version = subtype & 0x0F;
+               opt_rx->mptcp.flags = *ptr++;
+               opt_rx->mptcp.sndr_key = get_unaligned_be64(ptr);
+               pr_debug("sndr_key=%llu", opt_rx->mptcp.sndr_key);
+               ptr += 8;
+               if (opsize > TCPOLEN_MPTCP_MPC_SYN) {
+                       opt_rx->mptcp.rcvr_key = get_unaligned_be64(ptr);
+                       pr_debug("rcvr_key=%llu", opt_rx->mptcp.rcvr_key);
+                       ptr += 8;
+               }
+               break;
+
+       /* MPTCPOPT_MP_JOIN
+        *
+        * Initial SYN
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID
+        * 2-5: Receiver token
+        * 6-9: Sender random number
+        *
+        * SYN/ACK response
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID
+        * 2-9: Sender truncated HMAC
+        * 10-13: Sender random number
+        *
+        * Third ACK
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-21: Sender HMAC
+        */
+
+       /* MPTCPOPT_DSS
+        * 0: 4MSB=subtype, 0000
+        * 1: 3MSB=0, F=Data FIN, m=DSN length, M=has DSN/SSN/DLL/checksum,
+        *    a=DACK length, A=has DACK
+        * 0, 4, or 8 bytes of DACK (depending on A/a)
+        * 0, 4, or 8 bytes of DSN (depending on M/m)
+        * 0 or 4 bytes of SSN (depending on M)
+        * 0 or 2 bytes of DLL (depending on M)
+        * 0 or 2 bytes of checksum (depending on M)
+        */
+       case 0x20:
+               pr_debug("DSS");
+               opt_rx->mptcp.dss = 1;
+               break;
+
+       /* MPTCPOPT_ADD_ADDR
+        * 0: 4MSB=subtype, 4LSB=IP version (4 or 6)
+        * 1: Address ID
+        * 4 or 16 bytes of address (depending on ip version)
+        * 0 or 2 bytes of port (depending on length)
+        */
+
+       /* MPTCPOPT_REMOVE_ADDR
+        * 0: 4MSB=subtype, 0000
+        * 1: Address ID
+        * Additional bytes: More address IDs (depending on length)
+        */
+
+       /* MPTCPOPT_MP_PRIO
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID (optional, current addr implied if not present)
+        */
+
+       /* MPTCPOPT_MP_FAIL
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-9: DSN
+        */
+
+       /* MPTCPOPT_MP_FASTCLOSE
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-9: Receiver key
+        */
+       default:
+               break;
+       }
+}
+
+void mptcp_get_options(const struct sk_buff *skb,
+                      struct tcp_options_received *opt_rx)
+{
+       const unsigned char *ptr;
+       const struct tcphdr *th = tcp_hdr(skb);
+       int length = (th->doff * 4) - sizeof(struct tcphdr);
+
+       ptr = (const unsigned char *)(th + 1);
+
+       while (length > 0) {
+               int opcode = *ptr++;
+               int opsize;
+
+               switch (opcode) {
+               case TCPOPT_EOL:
+                       return;
+               case TCPOPT_NOP:        /* Ref: RFC 793 section 3.1 */
+                       length--;
+                       continue;
+               default:
+                       opsize = *ptr++;
+                       if (opsize < 2) /* "silly options" */
+                               return;
+                       if (opsize > length)
+                               return; /* don't parse partial options */
+                       if (opcode == TCPOPT_MPTCP)
+                               mptcp_parse_option(ptr, opsize, opt_rx);
+                       ptr += opsize - 2;
+                       length -= opsize;
+               }
+       }
+}
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

#12 lore raw

Re: [MPTCP] [RFC PATCH v3 04/16] mptcp: Handle MPTCP TCP options

Quoted 10 lines · Peter
Quoted 6 lines · Matthieu Baerts

Hi Peter,

Nice addition!

Here are just a few questions and suggestions, mostly some details. I am not sure we want to speak about this level of details now.

Quoted 5 lines

Some of these details I can tidy up for our next release such as #defines for numbers, comment style, etc. I deliberately tried to leave out CONFIG_MPTCP as much as possible due to the maintainer advice we received to "just add the code so we can see what impact it has". Perhaps we will be advised to insert them during reviews...

It makes sense. Thank you for your reply!

Matthieu

Quoted 448 lines

Peter.

Quoted 196 lines
Quoted 22 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |  11 +++
 include/net/mptcp.h   |  35 ++++++++++
 include/net/tcp.h     |   2 +
 net/ipv4/tcp_input.c  |   5 ++
 net/ipv4/tcp_output.c |  67 +++++++++++++++----
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   | 152 ++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 261 insertions(+), 13 deletions(-)
 create mode 100644 net/mptcp/options.c
include/linux/tcp.h +0 -0
index 263e37271afd..cb40dcb43f95 100644
@@ -104,6 +104,16 @@ struct tcp_options_received {
        u8      num_sacks;      /* Number of SACK blocks                */
        u16     user_mss;       /* mss requested by user in ioctl       */

include/linux/tcp.h struct tcp_options_received { · line 115

Quoted 10 lines
include/linux/tcp.h +9 -0
        u16     mss_clamp;      /* Maximal mss, negotiated at connection setup */
+       struct {
+               u8      mp_capable : 1,
+                       mp_join : 1,
+                       dss : 1,
+                       version : 4;
+               u8      flags;
+               u64     sndr_key;
+               u64     rcvr_key;
+       } mptcp;

For upstream, should we not surround this new structure with #ifdef CONFIG_MPTCP?

include/linux/tcp.h static inline void tcp_clear_options(struct tcp_options_received *rx_opt) · line 126

Quoted 9 lines
include/linux/tcp.h +2 -0
+
 };
 
 static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
@@ -113,6 +123,7 @@ static inline void tcp_clear_options(struct tcp_options_received *rx_opt)
 #if IS_ENABLED(CONFIG_SMC)
        rx_opt->smc_ok = 0;
 #endif
+       rx_opt->mptcp.mp_capable = rx_opt->mptcp.mp_join = rx_opt->mptcp.dss = 0;

Same here for the CONFIG_MPTCP?

Quoted 115 lines
include/linux/tcp.h +0 -0
 }
 
 /* This is the max number of SACKS that we'll generate and process. It's safe
include/net/mptcp.h +35 -0
index 7f7b18b000fe..f88213e31fbe 100644
@@ -18,6 +18,23 @@
 
 #include <linux/tcp.h>
 
+/* MPTCP option subtypes */
+
+#define MPTCPOPT_MP_CAPABLE    0
+#define MPTCPOPT_MP_JOIN       1
+#define MPTCPOPT_DSS           2
+#define MPTCPOPT_ADD_ADDR      3
+#define MPTCPOPT_REMOVE_ADDR   4
+#define MPTCPOPT_MP_PRIO       5
+#define MPTCPOPT_MP_FAIL       6
+#define MPTCPOPT_MP_FASTCLOSE  7
+
+/* MPTCP handshake flags */
+
+#define MPTCP_CAP_CHECKSUM_REQD        (1 << 7)
+#define MPTCP_CAP_EXTENSIBILITY        (1 << 6)
+#define MPTCP_CAP_HMAC_SHA1    (1 << 0)
+
 /* MPTCP connection sock */
 struct mptcp_sock {
        /* inet_connection_sock must be the first member */
@@ -30,4 +47,22 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
        return (struct mptcp_sock *)sk;
 }
 
+#ifdef CONFIG_MPTCP
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                       struct tcp_options_received *opt_rx);
+
+void mptcp_get_options(const struct sk_buff *skb,
+                      struct tcp_options_received *options);
+
+#else
+
+static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                                     struct tcp_options_received *opt_rx)
+{
+}
+
+
+
+#endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
include/net/tcp.h +2 -0
index 9a7e4ac9892a..56a3455f7a50 100644
@@ -215,6 +215,8 @@ void tcp_time_wait(struct sock *sk, int state, int timeo);
 #define TCPOLEN_MD5SIG_ALIGNED         20
 #define TCPOLEN_MSS_ALIGNED            4
 #define TCPOLEN_EXP_SMC_BASE_ALIGNED   8
+#define TCPOLEN_MPTCP_MPC_SYN          12
+#define TCPOLEN_MPTCP_MPC_SYNACK       20
 
 /* Flags in tp->nonagle */
 #define TCP_NAGLE_OFF          1       /* Nagle's algo is disabled */
net/ipv4/tcp_input.c +5 -0
index 4c2dd9f863f7..81fccf0c9120 100644
@@ -79,6 +79,7 @@
 #include <trace/events/tcp.h>
 #include <linux/static_key.h>
 #include <net/busy_poll.h>
+#include <net/mptcp.h>
 
 int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
 
@@ -3865,6 +3866,10 @@ void tcp_parse_options(const struct net *net,
                                 */
                                break;
 #endif
+                        case TCPOPT_MPTCP:
+                               mptcp_parse_option(ptr, opsize, opt_rx);
+                               break;
+
                        case TCPOPT_FASTOPEN:
                                tcp_parse_fastopen_option(
                                        opsize - TCPOLEN_FASTOPEN_BASE,
net/ipv4/tcp_output.c +10 -0
index 597dbd749f05..a6ea8520e5d9 100644
@@ -37,6 +37,7 @@
 #define pr_fmt(fmt) "TCP: " fmt
 
 #include <net/tcp.h>
+#include <net/mptcp.h>
 
 #include <linux/compiler.h>
 #include <linux/gfp.h>
@@ -415,6 +416,58 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_WSCALE          (1 << 3)
 #define OPTION_FAST_OPEN_COOKIE        (1 << 8)
 #define OPTION_SMC             (1 << 9)
+#define OPTION_MPTCP           (1 << 10)
+
+/* MPTCP option subtypes */
+#define OPTION_MPTCP_MPC_SYN   (1 << 0)
+#define OPTION_MPTCP_MPC_SYNACK        (1 << 1)
+#define OPTION_MPTCP_MPC_ACK   (1 << 2)
+
+struct tcp_out_options {
+       u16 options;            /* bit field of OPTION_* */

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 437

Quoted 10 lines
net/ipv4/tcp_output.c +10 -0
+       u16 mss;                /* 0 to disable */
+       u8 ws;                  /* window scale, 0 to disable */
+       u8 num_sack_blocks;     /* number of SACK blocks to include */
+       u8 hash_size;           /* bytes in hash_location */
+       __u8 *hash_location;    /* temporary pointer, overloaded */
+       __u32 tsval, tsecr;     /* need to include OPTION_TS */
+       struct tcp_fastopen_cookie *fastopen_cookie;    /* Fast open cookie */
+       u16 suboptions;         /* MPTCP sub-options */
+       u64 sndr_key;
+       u64 rcvr_key;

(detail: should we prefix these new entries with "mptcp_"? Or an "mptcp_options" structure like what has been done for tcp_fastopen?)

Quoted 18 lines
net/ipv4/tcp_output.c +18 -0
+};
+
+static void mptcp_options_write(__be32 *ptr, struct tcp_out_options *opts)
+{
+       if (!(OPTION_MPTCP & opts->options))
+               return;
+
+       if ((OPTION_MPTCP_MPC_SYN |
+            OPTION_MPTCP_MPC_SYNACK |
+            OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
+               u8 len;
+               __be64 key;
+
+               if (OPTION_MPTCP_MPC_SYN & opts->suboptions)
+                       len = TCPOLEN_MPTCP_MPC_SYN;
+               else
+                       len = TCPOLEN_MPTCP_MPC_SYNACK;
+
Quoted 8 lines

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 459

Quoted 4 lines
net/ipv4/tcp_output.c +4 -0
+               *ptr++ = htonl((0x1e << 24) | // TCP option: Multipath TCP
+                              (len << 16)  | // length
+                              (0 <<  8)    | // subtype=MP_CAPABLE | version=0
+                              (0x1));        // flags=HMAC-SHA1

(detail: good idea to have comment here! I guess we don't need to spend time on the code style now but it was strange not to see comments made with /* */ :) )

OK.

Quoted 6 lines

net/ipv4/tcp_output.c static inline bool tcp_urg_mode(const struct tcp_sock *tp) · line 462

Quoted 4 lines
net/ipv4/tcp_output.c +3 -0
+               key = cpu_to_be64(opts->sndr_key);
+               memcpy((u8 *) ptr, (u8 *) &key, 8);
+               ptr += 2;

(detail: should we use define/sizeof here, below and in mptcp_parse_option?)

OK.

Quoted 232 lines
Quoted 40 lines
net/ipv4/tcp_output.c +10 -11
+               if ((OPTION_MPTCP_MPC_SYNACK |
+                    OPTION_MPTCP_MPC_ACK) & opts->suboptions) {
+                       key = cpu_to_be64(opts->rcvr_key);
+                       memcpy((u8 *) ptr, (u8 *) &key, 8);
+                       ptr += 2;
+               }
+       }
+}
 
 static void smc_options_write(__be32 *ptr, u16 *options)
 {
@@ -431,17 +484,6 @@ static void smc_options_write(__be32 *ptr, u16 *options)
 #endif
 }
 
-struct tcp_out_options {
-       u16 options;            /* bit field of OPTION_* */
-       u16 mss;                /* 0 to disable */
-       u8 ws;                  /* window scale, 0 to disable */
-       u8 num_sack_blocks;     /* number of SACK blocks to include */
-       u8 hash_size;           /* bytes in hash_location */
-       __u8 *hash_location;    /* temporary pointer, overloaded */
-       __u32 tsval, tsecr;     /* need to include OPTION_TS */
-       struct tcp_fastopen_cookie *fastopen_cookie;    /* Fast open cookie */
-};
-
 /* Write previously computed TCP options to the packet.
  *
  * Beware: Something in the Internet is very sensitive to the ordering of
@@ -550,6 +592,8 @@ static void tcp_options_write(__be32 *ptr, struct tcp_sock *tp,
        }
 
        smc_options_write(ptr, &options);
+
+       mptcp_options_write(ptr, opts);
 }
 
 static void smc_set_option(const struct tcp_sock *tp,

net/ipv4/tcp_output.c static unsigned int tcp_synack_options(const struct sock *sk, · line 716

Quoted 4 lines
net/ipv4/tcp_output.c +0 -1
@@ -713,7 +757,6 @@ static unsigned int tcp_synack_options(const struct sock *sk,
                        remaining -= need;
                }
        }
-

(detail: unwanted change?)

Quoted 44 lines
net/ipv4/tcp_output.c +0 -0
        smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
        return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/Makefile +1 -1
index 5624e7d51d48..2bd18e3b9fda 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o
+mptcp-y := protocol.o options.o
net/mptcp/options.c new file +26 -0
new file mode 100644
index 000000000000..4fd64e0bdd2e
@@ -0,0 +1,152 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+void mptcp_parse_option(const unsigned char *ptr, int opsize,
+                       struct tcp_options_received *opt_rx)
+{
+       u8 subtype;
+       u64 *p;
+
+       opsize -= 2;

net/mptcp/options.c line 36

Quoted 10 lines
net/mptcp/options.c new file +10 -0
+       subtype = *ptr++;
+
+       /* MPTCPOPT_MP_CAPABLE
+        * 0: 4MSB=subtype, 4LSB=version
+        * 1: Handshake flags
+        * 2-9: Sender key
+        * 10-17: Receiver key (optional)
+        */
+       switch (subtype & 0xF0) {
+       case 0x00:

(detail: I guess we will use macro here later)

Matthieu

Quoted 123 lines
net/mptcp/options.c new file +116 -1
+               pr_debug("MP_CAPABLE");
+               pr_debug("flags=%02x", *ptr);
+               opt_rx->mptcp.mp_capable = 1;
+               opt_rx->mptcp.version = subtype & 0x0F;
+               opt_rx->mptcp.flags = *ptr++;
+               opt_rx->mptcp.sndr_key = get_unaligned_be64(ptr);
+               pr_debug("sndr_key=%llu", opt_rx->mptcp.sndr_key);
+               ptr += 8;
+               if (opsize > TCPOLEN_MPTCP_MPC_SYN) {
+                       opt_rx->mptcp.rcvr_key = get_unaligned_be64(ptr);
+                       pr_debug("rcvr_key=%llu", opt_rx->mptcp.rcvr_key);
+                       ptr += 8;
+               }
+               break;
+
+       /* MPTCPOPT_MP_JOIN
+        *
+        * Initial SYN
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID
+        * 2-5: Receiver token
+        * 6-9: Sender random number
+        *
+        * SYN/ACK response
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID
+        * 2-9: Sender truncated HMAC
+        * 10-13: Sender random number
+        *
+        * Third ACK
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-21: Sender HMAC
+        */
+
+       /* MPTCPOPT_DSS
+        * 0: 4MSB=subtype, 0000
+        * 1: 3MSB=0, F=Data FIN, m=DSN length, M=has DSN/SSN/DLL/checksum,
+        *    a=DACK length, A=has DACK
+        * 0, 4, or 8 bytes of DACK (depending on A/a)
+        * 0, 4, or 8 bytes of DSN (depending on M/m)
+        * 0 or 4 bytes of SSN (depending on M)
+        * 0 or 2 bytes of DLL (depending on M)
+        * 0 or 2 bytes of checksum (depending on M)
+        */
+       case 0x20:
+               pr_debug("DSS");
+               opt_rx->mptcp.dss = 1;
+               break;
+
+       /* MPTCPOPT_ADD_ADDR
+        * 0: 4MSB=subtype, 4LSB=IP version (4 or 6)
+        * 1: Address ID
+        * 4 or 16 bytes of address (depending on ip version)
+        * 0 or 2 bytes of port (depending on length)
+        */
+
+       /* MPTCPOPT_REMOVE_ADDR
+        * 0: 4MSB=subtype, 0000
+        * 1: Address ID
+        * Additional bytes: More address IDs (depending on length)
+        */
+
+       /* MPTCPOPT_MP_PRIO
+        * 0: 4MSB=subtype, 000, 1LSB=Backup
+        * 1: Address ID (optional, current addr implied if not present)
+        */
+
+       /* MPTCPOPT_MP_FAIL
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-9: DSN
+        */
+
+       /* MPTCPOPT_MP_FASTCLOSE
+        * 0: 4MSB=subtype, 0000
+        * 1: 0 (Reserved)
+        * 2-9: Receiver key
+        */
+       default:
+               break;
+       }
+}
+
+void mptcp_get_options(const struct sk_buff *skb,
+                      struct tcp_options_received *opt_rx)
+{
+       const unsigned char *ptr;
+       const struct tcphdr *th = tcp_hdr(skb);
+       int length = (th->doff * 4) - sizeof(struct tcphdr);
+
+       ptr = (const unsigned char *)(th + 1);
+
+       while (length > 0) {
+               int opcode = *ptr++;
+               int opsize;
+
+               switch (opcode) {
+               case TCPOPT_EOL:
+                       return;
+               case TCPOPT_NOP:        /* Ref: RFC 793 section 3.1 */
+                       length--;
+                       continue;
+               default:
+                       opsize = *ptr++;
+                       if (opsize < 2) /* "silly options" */
+                               return;
+                       if (opsize > length)
+                               return; /* don't parse partial options */
+                       if (opcode == TCPOPT_MPTCP)
+                               mptcp_parse_option(ptr, opsize, opt_rx);
+                       ptr += opsize - 2;
+                       length -= opsize;
+               }
+       }
+}
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/
author
#13 lore raw

RFC PATCH v3 5/16 tcp: Add IPPROTO_SUBFLOW

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -0
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS		IPPROTO_MPLS
   IPPROTO_RAW = 255,		/* Raw IP packets			*/
 #define IPPROTO_RAW		IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,	/* Multipath TCP (IPPROTO_TCP | 0x100)	*/
+#define IPPROTO_SUBFLOW		IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
Signature · 1 line
2.19.1
#14 lore raw

Re: [MPTCP] [RFC PATCH v3 05/16] tcp: Add IPPROTO_SUBFLOW

Quoted 8 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Should we then call it IPPROTO_TCP_SUBFLOW? :)

Cheers,
Matthieu

Quoted 26 lines
Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -1
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS           IPPROTO_MPLS
   IPPROTO_RAW = 255,           /* Raw IP packets                       */
 #define IPPROTO_RAW            IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,       /* Multipath TCP (IPPROTO_TCP | 0x100)  */
+#define IPPROTO_SUBFLOW                IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/
author
#15 lore raw

Re: [MPTCP] [RFC PATCH v3 05/16] tcp: Add IPPROTO_SUBFLOW

Quoted 5 lines · Matthieu Baerts
Quoted 3 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Quoted 8 lines
Quoted 5 lines

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Should we then call it IPPROTO_TCP_SUBFLOW? :)

Hopefully we won't need a special IPPROTO_ for the subflows in the long run. It could be defined internally for kernel code with some checks to prevent userspace-owned sockets of this type. Since the MPTCP kernel code has complete control over the subflow sockets, maybe we could create regular TCP sockets and modify them before using them (kind of like ULP does, but without using the ULP infrastructure between an MPTCP socket and its subflows).

Mat

Quoted 22 lines
Quoted 22 lines
Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -1
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS           IPPROTO_MPLS
   IPPROTO_RAW = 255,           /* Raw IP packets                       */
 #define IPPROTO_RAW            IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,       /* Multipath TCP (IPPROTO_TCP | 0x100)  */
+#define IPPROTO_SUBFLOW                IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
--

2.19.1

Quoted 16 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/

--
Mat Martineau
Intel OTC

#16 lore raw

Re: [MPTCP] [RFC PATCH v3 05/16] tcp: Add IPPROTO_SUBFLOW

Quoted 24 lines · Mat Martineau
Quoted 5 lines · Matthieu Baerts
Quoted 3 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Quoted 8 lines
Quoted 5 lines

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Should we then call it IPPROTO_TCP_SUBFLOW? :)

Hopefully we won't need a special IPPROTO_ for the subflows in the long run. It could be defined internally for kernel code with some checks to prevent userspace-owned sockets of this type. Since the MPTCP kernel code has complete control over the subflow sockets, maybe we could create regular TCP sockets and modify them before using them (kind of like ULP does, but without using the ULP infrastructure between an MPTCP socket and its subflows).

Quoted 2 lines

Mat

The purpose of my approach with IPPROTO_SUBFLOW was it reduced a lot of impacts to the TCP code by allowing overriding all the 'struct proto' routines, and defining the sock structure size as the size of subflow_sock. Tradeoff would be more "if (sk->is_mptcp)" hooks in TCP code.

Peter.

Quoted 47 lines
Quoted 23 lines
Quoted 23 lines
Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -1
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS           IPPROTO_MPLS
   IPPROTO_RAW = 255,           /* Raw IP packets                       */
 #define IPPROTO_RAW            IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,       /* Multipath TCP (IPPROTO_TCP | 0x100)  */
+#define IPPROTO_SUBFLOW                IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
--

2.19.1

Quoted 16 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/

--
Mat Martineau
Intel OTC
_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

#17 lore raw

Re: [MPTCP] [RFC PATCH v3 05/16] tcp: Add IPPROTO_SUBFLOW

Quoted 25 lines · Peter
Quoted 24 lines · Mat Martineau
Quoted 5 lines · Matthieu Baerts
Quoted 3 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Quoted 8 lines
Quoted 5 lines

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Should we then call it IPPROTO_TCP_SUBFLOW? :)

Hopefully we won't need a special IPPROTO_ for the subflows in the long run. It could be defined internally for kernel code with some checks to prevent userspace-owned sockets of this type. Since the MPTCP kernel code has complete control over the subflow sockets, maybe we could create regular TCP sockets and modify them before using them (kind of like ULP does, but without using the ULP infrastructure between an MPTCP socket and its subflows).

Quoted 7 lines
Quoted 2 lines

Mat

The purpose of my approach with IPPROTO_SUBFLOW was it reduced a lot of impacts to the TCP code by allowing overriding all the 'struct proto' routines, and defining the sock structure size as the size of subflow_sock. Tradeoff would be more "if (sk->is_mptcp)" hooks in TCP code.

Yes, I think it's a good approach to have a separate struct proto for subflows.

Christoph

Quoted 54 lines

Peter.

Quoted 40 lines
Quoted 23 lines
Quoted 23 lines
Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -1
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS           IPPROTO_MPLS
   IPPROTO_RAW = 255,           /* Raw IP packets                       */
 #define IPPROTO_RAW            IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,       /* Multipath TCP (IPPROTO_TCP | 0x100)  */
+#define IPPROTO_SUBFLOW                IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
--

2.19.1

Quoted 16 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/
Quoted 7 lines

--
Mat Martineau
Intel OTC
_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

author
#18 lore raw

Re: [MPTCP] [RFC PATCH v3 05/16] tcp: Add IPPROTO_SUBFLOW

Quoted 28 lines · Christoph Paasch
Quoted 27 lines · Peter
Quoted 26 lines · Mat Martineau
Quoted 5 lines · Matthieu Baerts
Quoted 3 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Quoted 8 lines
Quoted 5 lines

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Should we then call it IPPROTO_TCP_SUBFLOW? :)

Hopefully we won't need a special IPPROTO_ for the subflows in the long run. It could be defined internally for kernel code with some checks to prevent userspace-owned sockets of this type. Since the MPTCP kernel code has complete control over the subflow sockets, maybe we could create regular TCP sockets and modify them before using them (kind of like ULP does, but without using the ULP infrastructure between an MPTCP socket and its subflows).

Mat

Quoted 8 lines
Quoted 5 lines

The purpose of my approach with IPPROTO_SUBFLOW was it reduced a lot of impacts to the TCP code by allowing overriding all the 'struct proto' routines, and defining the sock structure size as the size of subflow_sock. Tradeoff would be more "if (sk->is_mptcp)" hooks in TCP code.

Yes, I think it's a good approach to have a separate struct proto for subflows.

To be more specific about what I meant by "kind of like ULP does":

* inet_connection_sock->icsk_ulp_data can store the extra information
that's in subflow_sock now (so allocating a larger sock structure is not required)

* all the struct proto routines can still be overridden immediately after
subflow TCP socket creation without defining IPPROTO_SUBFLOW

Mat

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 37 lines
Quoted 23 lines
Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -1
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS           IPPROTO_MPLS
   IPPROTO_RAW = 255,           /* Raw IP packets                       */
 #define IPPROTO_RAW            IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,       /* Multipath TCP (IPPROTO_TCP | 0x100)  */
+#define IPPROTO_SUBFLOW                IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
--

2.19.1

Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Quoted 2 lines

Disclaimer: https://www.tessares.net/mail-disclaimer/

--
Mat Martineau
Intel OTC
_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Quoted 5 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

#19 lore raw

Re: [MPTCP] [RFC PATCH v3 05/16] tcp: Add IPPROTO_SUBFLOW

Quoted 41 lines · Mat Martineau
Quoted 28 lines · Christoph Paasch
Quoted 27 lines · Peter
Quoted 26 lines · Mat Martineau
Quoted 5 lines · Matthieu Baerts
Quoted 3 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Quoted 8 lines
Quoted 5 lines

IANA protocol numbers fit in a single byte, however the protocol field of struct inet_protosw is an unsigned short so defining IPPROTO_SUBFLOW as a 16-bit value greater than 0xFF with the same LSB as IPPROTO_TCP is convenient for registering a unique protocol with inet_register_protosw() that still transmits packets with IPPROTO_TCP as the protocol number.

Should we then call it IPPROTO_TCP_SUBFLOW? :)

Hopefully we won't need a special IPPROTO_ for the subflows in the long run. It could be defined internally for kernel code with some checks to prevent userspace-owned sockets of this type. Since the MPTCP kernel code has complete control over the subflow sockets, maybe we could create regular TCP sockets and modify them before using them (kind of like ULP does, but without using the ULP infrastructure between an MPTCP socket and its subflows).

Mat

Quoted 8 lines
Quoted 5 lines

The purpose of my approach with IPPROTO_SUBFLOW was it reduced a lot of impacts to the TCP code by allowing overriding all the 'struct proto' routines, and defining the sock structure size as the size of subflow_sock. Tradeoff would be more "if (sk->is_mptcp)" hooks in TCP code.

Yes, I think it's a good approach to have a separate struct proto for subflows.

Quoted 8 lines

To be more specific about what I meant by "kind of like ULP does":

* inet_connection_sock->icsk_ulp_data can store the extra information
that's in subflow_sock now (so allocating a larger sock structure is not required)

* all the struct proto routines can still be overridden immediately after
subflow TCP socket creation without defining IPPROTO_SUBFLOW

Sorry, my comment was not clear: I also think it is a good idea to have a dedicated struct proto for subflows but my point was: because the description of the commit was saying we are so closed from IPPROTO_TCP, should we then call it IPPROTO_TCP_SUBFLOW instead of IPPROTO_SUBFLOW. Because the latter looks like a new proto, not linked to MPTCP nor TCP? :)

Matthieu

Quoted 60 lines

Mat

Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
Quoted 37 lines
Quoted 23 lines
Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/uapi/linux/in.h | 2 ++
 1 file changed, 2 insertions(+)
include/uapi/linux/in.h +2 -1
index f65975c801e5..6986aecd0f46 100644
@@ -78,6 +78,8 @@ enum {
 #define IPPROTO_MPLS           IPPROTO_MPLS
   IPPROTO_RAW = 255,           /* Raw IP packets                       */
 #define IPPROTO_RAW            IPPROTO_RAW
+  IPPROTO_SUBFLOW = 262,       /* Multipath TCP (IPPROTO_TCP | 0x100)  */
+#define IPPROTO_SUBFLOW                IPPROTO_SUBFLOW
   IPPROTO_MAX
 };
 #endif
--

2.19.1

Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Quoted 2 lines

Disclaimer: https://www.tessares.net/mail-disclaimer/

--
Mat Martineau
Intel OTC
_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Quoted 5 lines
Quoted 4 lines

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/
author
#20 lore raw

RFC PATCH v3 6/16 tcp: expose tcp routines and structs for MPTCP

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

tcp_v4_init_sock(), inet_compat_ioctl()
tcp_request_sock_ipv4_ops

This function is needed for MPTCP subflow initialization.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/net/inet_common.h | 3 +++
 include/net/mptcp.h       | 2 ++
 include/net/tcp.h         | 1 +
 net/ipv4/af_inet.c        | 2 +-
 net/ipv4/tcp_ipv4.c       | 4 ++--
 5 files changed, 9 insertions(+), 3 deletions(-)
include/net/inet_common.h +3 -0
index 3ca969cbd161..1782c0d2ec32 100644
@@ -37,6 +37,9 @@ int __inet_bind(struct sock *sk, struct sockaddr *uaddr, int addr_len,
 int inet_getname(struct socket *sock, struct sockaddr *uaddr,
 		 int peer);
 int inet_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg);
+#ifdef CONFIG_COMPAT
+int inet_compat_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg);
+#endif
 int inet_ctl_sock_create(struct sock **sk, unsigned short family,
 			 unsigned short type, unsigned char protocol,
 			 struct net *net);
include/net/mptcp.h +2 -0
index f88213e31fbe..79daad717275 100644
@@ -55,6 +55,8 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
 
+extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
+
 #else
 
 static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
include/net/tcp.h +1 -0
index 56a3455f7a50..48c59c0d8760 100644
@@ -436,6 +436,7 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
 int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb);
 int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len);
 int tcp_connect(struct sock *sk);
+int tcp_v4_init_sock(struct sock *sk);
 enum tcp_synack_type {
 	TCP_SYNACK_NORMAL,
 	TCP_SYNACK_FASTOPEN,
net/ipv4/af_inet.c +1 -1
index 20fda8fb8ffd..46a9ecd5c8f0 100644
@@ -965,7 +965,7 @@ int inet_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg)
 EXPORT_SYMBOL(inet_ioctl);
 
 #ifdef CONFIG_COMPAT
-static int inet_compat_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg)
+int inet_compat_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg)
 {
 	struct sock *sk = sock->sk;
 	int err = -ENOIOCTLCMD;
net/ipv4/tcp_ipv4.c +2 -2
index 9e041fa5c545..1b22b2f83719 100644
@@ -1357,7 +1357,7 @@ struct request_sock_ops tcp_request_sock_ops __read_mostly = {
 	.syn_ack_timeout =	tcp_syn_ack_timeout,
 };
 
-static const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops = {
+const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops = {
 	.mss_clamp	=	TCP_MSS_DEFAULT,
 #ifdef CONFIG_TCP_MD5SIG
 	.req_md5_lookup	=	tcp_v4_md5_lookup,
@@ -1950,7 +1950,7 @@ static const struct tcp_sock_af_ops tcp_sock_ipv4_specific = {
 /* NOTE: A lot of things set to zero explicitly by call to
  *       sk_alloc() so need not be done here.
  */
-static int tcp_v4_init_sock(struct sock *sk)
+int tcp_v4_init_sock(struct sock *sk)
 {
 	struct inet_connection_sock *icsk = inet_csk(sk);
 
Signature · 1 line
2.19.1
author
#21 lore raw

RFC PATCH v3 7/16 mptcp: Create SUBFLOW socket for outgoing connections

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_sock type that extends tcp_sock and add an is_mptcp flag to tcp_sock to distinguish them.

Override the bind() and connect() methods of the MPTCP socket proto_ops so they may act on the subflow socket instead, and use the .sk_rx_dst_set() handler in the subflow proto to capture when the responding SYN-ACK is received.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN request for a subflow_sock and to the final ACK of the three-way handshake.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   2 +
 include/net/mptcp.h   |  35 +++++++-
 net/ipv4/tcp_input.c  |   9 ++
 net/ipv4/tcp_output.c |  32 +++++++
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   |  11 +++
 net/mptcp/protocol.c  | 161 +++++++++++++++++++++++++++++------
 net/mptcp/subflow.c   | 192 ++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 415 insertions(+), 29 deletions(-)
 create mode 100644 net/mptcp/subflow.c
include/linux/tcp.h +2 -0
index cb40dcb43f95..7f0dd688376c 100644
@@ -408,6 +408,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	bool	is_mptcp;
 };
 
 enum tsq_enum {
include/net/mptcp.h +33 -2
index 79daad717275..4b08eb4ccc6f 100644
@@ -39,7 +39,10 @@
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
 	struct	inet_connection_sock sk;
-	struct	socket *subflow;
+	u64	local_key;
+	u64	remote_key;
+	struct	socket *connection_list; /* @@ needs to be a list */
+	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -47,10 +50,35 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 	return (struct mptcp_sock *)sk;
 }
 
+/* MPTCP subflow sock structure */
+struct subflow_sock {
+	/* tcp_sock must be the first member */
+	struct	tcp_sock sk;
+	u64	local_key;
+	u64	remote_key;
+	bool	request_mptcp;	// send MP_CAPABLE
+	bool	checksum;
+	bool	version;
+	bool	mp_capable;	// remote is MPTCP capable
+	bool	fourth_ack;	// send initial DSS
+	struct	sock *conn;	// parent mptcp_sock
+};
+
+static inline struct subflow_sock *subflow_sk(const struct sock *sk)
+{
+	return (struct subflow_sock *)sk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+
+void mptcp_finish_connect(struct sock *sk, int mp_capable);
+
+int mptcp_subflow_init(void);
+void mptcp_subflow_exit(void);
 
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
@@ -64,7 +92,10 @@ static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
 {
 }
 
-
+static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+	return 0;
+}
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +9 -0
index 81fccf0c9120..4cb38904bb5f 100644
@@ -5856,6 +5856,15 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
+		// @@ could use a callout here
+		if (tp->is_mptcp) {
+			struct subflow_sock *subflow = subflow_sk(sk);
+			if (subflow->request_mptcp && tp->rx_opt.mptcp.mp_capable) {
+				subflow->mp_capable = 1;
+				subflow->remote_key = tp->rx_opt.mptcp.sndr_key;
+			}
+		}
+
 		/* Remember, tcp_poll() does not lock socket!
 		 * Change state from SYN-SENT only after copied_seq
 		 * is initialized. */
net/ipv4/tcp_output.c +32 -0
index a6ea8520e5d9..9919793e293b 100644
@@ -697,6 +697,16 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (tp->is_mptcp) {
+		u64 local_key;
+		if (mptcp_syn_options(sk, &local_key)) {
+			opts->options |= OPTION_MPTCP;
+			opts->suboptions |= OPTION_MPTCP_MPC_SYN;
+			opts->sndr_key = local_key;
+			remaining -= TCPOLEN_MPTCP_MPC_SYN;
+		}
+	}
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -804,6 +814,28 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 			opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;
 	}
 
+	if (tp->is_mptcp) {
+		struct subflow_sock *subflow = subflow_sk(sk);
+
+		pr_debug("tcp_established_options: subflow=%p", subflow);
+		opts->suboptions = 0;
+		if (subflow->mp_capable) {
+			const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
+			pr_debug("tcp_established_options: remaining=%d", remaining);
+			if (!subflow->fourth_ack) {
+				pr_debug("tcp_established_options: OPTION_MPTCP_MPC_ACK");
+				if (remaining >= 20) {
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions |= OPTION_MPTCP_MPC_ACK;
+					opts->sndr_key = subflow->local_key;
+					opts->rcvr_key = subflow->remote_key;
+					size += 20;
+				}
+				subflow->fourth_ack = 1;
+				// @@ also this is where first DSS goes in?
+			}
+		}
+	}
 	return size;
 }
 
net/mptcp/Makefile +1 -1
index 2bd18e3b9fda..3f0e7163fe80 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o options.o
+mptcp-y := protocol.o subflow.o options.o
net/mptcp/options.c +11 -0
index 4fd64e0bdd2e..4b1cbc3b3efe 100644
@@ -150,3 +150,14 @@ void mptcp_get_options(const struct sk_buff *skb,
 		}
 	}
 }
+
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	if (subflow->request_mptcp) {
+		pr_debug("local_key=%llu", subflow->local_key);
+		*local_key = subflow->local_key;
+	}
+	return subflow->request_mptcp;
+}
net/mptcp/protocol.c +135 -26

237 lines not on this page · raw message

net/mptcp/subflow.c new file +192 -0

192 lines not on this page · raw message

Signature · 1 line
2.19.1
#22 lore raw

Re: [MPTCP] [RFC PATCH v3 07/16] mptcp: Create SUBFLOW socket for outgoing connections

Hi Peter,

Nice again!
I only have some details and questions on my side.

I guess we can look at the details later.

Quoted 32 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_sock type that extends tcp_sock and add an is_mptcp flag to tcp_sock to distinguish them.

Override the bind() and connect() methods of the MPTCP socket proto_ops so they may act on the subflow socket instead, and use the .sk_rx_dst_set() handler in the subflow proto to capture when the responding SYN-ACK is received.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN request for a subflow_sock and to the final ACK of the three-way handshake.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   2 +
 include/net/mptcp.h   |  35 +++++++-
 net/ipv4/tcp_input.c  |   9 ++
 net/ipv4/tcp_output.c |  32 +++++++
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   |  11 +++
 net/mptcp/protocol.c  | 161 +++++++++++++++++++++++++++++------
 net/mptcp/subflow.c   | 192 ++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 415 insertions(+), 29 deletions(-)
 create mode 100644 net/mptcp/subflow.c
include/linux/tcp.h +0 -0
index cb40dcb43f95..7f0dd688376c 100644

include/linux/tcp.h struct tcp_sock { · line 412

Quoted 5 lines
include/linux/tcp.h +2 -0
@@ -408,6 +408,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	bool	is_mptcp;

detail: can we use bool? Or one bit in a byte shared with other fields?

Quoted 20 lines
include/linux/tcp.h +0 -0
 };
 
 enum tsq_enum {
include/net/mptcp.h +4 -1
index 79daad717275..4b08eb4ccc6f 100644
@@ -39,7 +39,10 @@
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
 	struct	inet_connection_sock sk;
-	struct	socket *subflow;
+	u64	local_key;
+	u64	remote_key;
+	struct	socket *connection_list; /* @@ needs to be a list */
+	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)

include/net/mptcp.h static inline struct mptcp_sock *mptcp_sk(const struct sock *sk) · line 54

Quoted 5 lines
include/net/mptcp.h +2 -0
@@ -47,10 +50,35 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 	return (struct mptcp_sock *)sk;
 }
 
+/* MPTCP subflow sock structure */
+struct subflow_sock {

detail: should we not start prefixing all MPTCP specific functions and structures with "mptcp_"?

include/net/mptcp.h static inline struct mptcp_sock *mptcp_sk(const struct sock *sk) · line 63

Quoted 9 lines
include/net/mptcp.h +9 -0
+	/* tcp_sock must be the first member */
+	struct	tcp_sock sk;
+	u64	local_key;
+	u64	remote_key;
+	bool	request_mptcp;	// send MP_CAPABLE
+	bool	checksum;
+	bool	version;
+	bool	mp_capable;	// remote is MPTCP capable
+	bool	fourth_ack;	// send initial DSS

detail: we should maybe continue in the same idea as with is_mptcp:
is_fourth_ack, etc. especially for fields like checksum and version. And maybe share a byte but can be changed later. I should certainly not look at these details :-)

Quoted 38 lines
include/net/mptcp.h +18 -1
+	struct	sock *conn;	// parent mptcp_sock
+};
+
+static inline struct subflow_sock *subflow_sk(const struct sock *sk)
+{
+	return (struct subflow_sock *)sk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+
+void mptcp_finish_connect(struct sock *sk, int mp_capable);
+
+int mptcp_subflow_init(void);
+void mptcp_subflow_exit(void);
 
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
@@ -64,7 +92,10 @@ static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
 {
 }
 
-
+static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+	return 0;
+}
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +0 -0
index 81fccf0c9120..4cb38904bb5f 100644

net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5865

Quoted 10 lines
net/ipv4/tcp_input.c +7 -0
@@ -5856,6 +5856,15 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
+		// @@ could use a callout here
+		if (tp->is_mptcp) {
+			struct subflow_sock *subflow = subflow_sk(sk);
+			if (subflow->request_mptcp && tp->rx_opt.mptcp.mp_capable) {
+				subflow->mp_capable = 1;
+				subflow->remote_key = tp->rx_opt.mptcp.sndr_key;
+			}

detail: Should we do this kind of think in MPTCP specific files?

Quoted 176 lines
net/ipv4/tcp_input.c +2 -0
+		}
+
 		/* Remember, tcp_poll() does not lock socket!
 		 * Change state from SYN-SENT only after copied_seq
 		 * is initialized. */
net/ipv4/tcp_output.c +32 -0
index a6ea8520e5d9..9919793e293b 100644
@@ -697,6 +697,16 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (tp->is_mptcp) {
+		u64 local_key;
+		if (mptcp_syn_options(sk, &local_key)) {
+			opts->options |= OPTION_MPTCP;
+			opts->suboptions |= OPTION_MPTCP_MPC_SYN;
+			opts->sndr_key = local_key;
+			remaining -= TCPOLEN_MPTCP_MPC_SYN;
+		}
+	}
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -804,6 +814,28 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 			opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;
 	}
 
+	if (tp->is_mptcp) {
+		struct subflow_sock *subflow = subflow_sk(sk);
+
+		pr_debug("tcp_established_options: subflow=%p", subflow);
+		opts->suboptions = 0;
+		if (subflow->mp_capable) {
+			const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
+			pr_debug("tcp_established_options: remaining=%d", remaining);
+			if (!subflow->fourth_ack) {
+				pr_debug("tcp_established_options: OPTION_MPTCP_MPC_ACK");
+				if (remaining >= 20) {
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions |= OPTION_MPTCP_MPC_ACK;
+					opts->sndr_key = subflow->local_key;
+					opts->rcvr_key = subflow->remote_key;
+					size += 20;
+				}
+				subflow->fourth_ack = 1;
+				// @@ also this is where first DSS goes in?
+			}
+		}
+	}
 	return size;
 }
 
net/mptcp/Makefile +1 -1
index 2bd18e3b9fda..3f0e7163fe80 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o options.o
+mptcp-y := protocol.o subflow.o options.o
net/mptcp/options.c +11 -0
index 4fd64e0bdd2e..4b1cbc3b3efe 100644
@@ -150,3 +150,14 @@ void mptcp_get_options(const struct sk_buff *skb,
 		}
 	}
 }
+
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	if (subflow->request_mptcp) {
+		pr_debug("local_key=%llu", subflow->local_key);
+		*local_key = subflow->local_key;
+	}
+	return subflow->request_mptcp;
+}
net/mptcp/protocol.c +34 -20
index c1eb4afc3ca4..1a3412a742ea 100644
@@ -26,9 +26,15 @@
 static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow = msk->subflow;
-
-	pr_debug("subflow=%p", subflow->sk);
+	struct socket *subflow;
+
+	if (msk->connection_list) {
+		subflow = msk->connection_list;
+		pr_debug("conn_list->subflow=%p", subflow->sk);
+	} else {
+		subflow = msk->subflow;
+		pr_debug("subflow=%p", subflow->sk);
+	}
 
 	return sock_sendmsg(subflow, msg);
 }
@@ -37,9 +43,15 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow = msk->subflow;
-
-	pr_debug("subflow=%p", subflow->sk);
+	struct socket *subflow;
+
+	if (msk->connection_list) {
+		subflow = msk->connection_list;
+		pr_debug("conn_list->subflow=%p", subflow->sk);
+	} else {
+		subflow = msk->subflow;
+		pr_debug("subflow=%p", subflow->sk);
+	}
 
 	return sock_recvmsg(subflow, msg, flags);
 }
@@ -47,19 +59,10 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 static int mptcp_init_sock(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *sf;
-	int err;
 
 	pr_debug("msk=%p", msk);
 
-	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
-			       &sf);
-	if (!err) {
-		pr_debug("subflow=%p", sf->sk);
-		msk->subflow = sf;
-	}
-
-	return err;
+	return 0;
 }
 
 static void mptcp_close(struct sock *sk, long timeout)
@@ -70,65 +73,171 @@ static void mptcp_close(struct sock *sk, long timeout)
 		pr_debug("subflow=%p", msk->subflow->sk);
 		sock_release(msk->subflow);
 	}
+
+	if (msk->connection_list) {
+		pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+		sock_release(msk->connection_list);
+	}
 }
 
-static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	int err;
+	struct sock *subflow = msk->subflow->sk;
+
+	pr_debug("msk=%p, subflow=%p", sk, subflow);
 
-	saddr->sa_family = AF_INET;
+	return inet_csk_get_port(subflow, snum);
+}
 
-	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+void mptcp_finish_connect(struct sock *sk, int mp_capable)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct subflow_sock *subflow = subflow_sk(msk->subflow->sk);
 

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 106

Quoted 10 lines
net/mptcp/protocol.c +7 -1
-	err = kernel_connect(msk->subflow, saddr, len, 0);
+	pr_debug("msk=%p", msk);
 
+	if (mp_capable) {
+		msk->remote_key = subflow->remote_key;
+		msk->local_key = subflow->local_key;
+		msk->connection_list = msk->subflow;
+		msk->subflow = NULL;
+	}
 	sk->sk_state = TCP_ESTABLISHED;

(should we not call tcp_set_state here?)

Quoted 23 lines
net/mptcp/protocol.c +19 -0
+}
 
+static int subflow_create(struct sock *sock)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock);
+	struct socket *sf;
+	int err;
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_SUBFLOW,
+			       &sf);
+	if (!err) {
+		struct subflow_sock *subflow = subflow_sk(sf->sk);
+		pr_debug("subflow=%p", subflow);
+		msk->subflow = sf;
+		subflow->conn = sock;
+		subflow->request_mptcp = 1; // @@ if MPTCP enabled
+		subflow->checksum = 1; // @@ if checksum enabled
+		subflow->version = 0;
+	}
 	return err;
 }
 
+int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 139

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+
+	return inet_bind(subflow, uaddr, addr_len);
+}
+
+int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,

detail: I guess these two functions can be static

Quoted 228 lines
net/mptcp/protocol.c +65 -5
+			 int addr_len, int flags)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+
+	return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
 	.init		= mptcp_init_sock,
 	.close		= mptcp_close,
 	.accept		= inet_csk_accept,
-	.connect	= mptcp_connect,
 	.shutdown	= tcp_shutdown,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
 	.hash		= inet_hash,
 	.unhash		= inet_unhash,
-	.get_port	= inet_csk_get_port,
+	.get_port	= mptcp_get_port,
 	.obj_size	= sizeof(struct mptcp_sock),
 	.no_autobind	= 1,
 };
 
+const struct proto_ops mptcp_stream_ops = {
+	.family		   = PF_INET,
+	.owner		   = THIS_MODULE,
+	.release	   = inet_release,
+	.bind		   = mptcp_stream_bind,
+	.connect	   = mptcp_stream_connect,
+	.socketpair	   = sock_no_socketpair,
+	.accept		   = inet_accept,
+	.getname	   = inet_getname,
+	.poll		   = tcp_poll,
+	.ioctl		   = inet_ioctl,
+	.listen		   = inet_listen,
+	.shutdown	   = inet_shutdown,
+	.setsockopt	   = sock_common_setsockopt,
+	.getsockopt	   = sock_common_getsockopt,
+	.sendmsg	   = inet_sendmsg,
+	.recvmsg	   = inet_recvmsg,
+	.mmap		   = sock_no_mmap,
+	.sendpage	   = inet_sendpage,
+	.splice_read	   = tcp_splice_read,
+	.read_sock	   = tcp_read_sock,
+	.peek_len	   = tcp_peek_len,
+#ifdef CONFIG_COMPAT
+	.compat_setsockopt = compat_sock_common_setsockopt,
+	.compat_getsockopt = compat_sock_common_getsockopt,
+	.compat_ioctl	   = inet_compat_ioctl,
+#endif
+};
+
 static struct inet_protosw mptcp_protosw = {
 	.type		= SOCK_STREAM,
 	.protocol	= IPPROTO_MPTCP,
 	.prot		= &mptcp_prot,
-	.ops		= &inet_stream_ops,
+	.ops		= &mptcp_stream_ops,
+	.flags		= INET_PROTOSW_ICSK,
 };
 
 static int __init mptcp_init(void)
 {
 	int err;
 
-	err = proto_register(&mptcp_prot, 1);
+	mptcp_prot.h.hashinfo = tcp_prot.h.hashinfo;
+
+	err = mptcp_subflow_init();
 	if (err)
-		return err;
+		goto subflow_failed;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err) {
+		goto proto_failed;
+	}
 
 	inet_register_protosw(&mptcp_protosw);
 
 	return 0;
+
+proto_failed:
+	mptcp_subflow_exit();
+
+subflow_failed:
+	return err;
 }
 
 static void __exit mptcp_exit(void)
 {
 	inet_unregister_protosw(&mptcp_protosw);
 	proto_unregister(&mptcp_prot);
+
+	mptcp_subflow_exit();
 }
 
 module_init(mptcp_init);
net/mptcp/subflow.c new file +113 -0
new file mode 100644
index 000000000000..5e5fdcb3175f
@@ -0,0 +1,192 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int subflow_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	saddr->sa_family = AF_INET; // @@ presume IPv4 for now
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_v4_connect(sk, saddr, len);
+}
+
+static int subflow_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_sendmsg(sk, msg, len);
+}
+
+static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			   int nonblock, int flags, int *addr_len)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
+}
+
+static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	inet_sk_rx_dst_set(sk, skb);
+
+	pr_debug("subflow=%p", subflow);
+
+	if (subflow->conn) {
+		pr_debug("remote_key=%llu", subflow->remote_key);
+		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
+		subflow->conn = NULL;
+	}
+}
+
+const struct inet_connection_sock_af_ops subflow_specific = {
+	.queue_xmit	   = ip_queue_xmit,
+	.send_check	   = tcp_v4_send_check,
+	.rebuild_header	   = inet_sk_rebuild_header,
+	.sk_rx_dst_set	   = subflow_finish_connect,
+	.conn_request	   = tcp_v4_conn_request,
+	.syn_recv_sock	   = tcp_v4_syn_recv_sock,
+	.net_header_len	   = sizeof(struct iphdr),
+	.setsockopt	   = ip_setsockopt,
+	.getsockopt	   = ip_getsockopt,
+	.addr2sockaddr	   = inet_csk_addr2sockaddr,
+	.sockaddr_len	   = sizeof(struct sockaddr_in),
+#ifdef CONFIG_COMPAT
+	.compat_setsockopt = compat_ip_setsockopt,
+	.compat_getsockopt = compat_ip_getsockopt,
+#endif
+	.mtu_reduced	   = tcp_v4_mtu_reduced,
+};
+
+static int subflow_init_sock(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct tcp_sock *tsk = tcp_sk(sk);
+	struct inet_connection_sock *icsk = inet_csk(sk);
+	int err;
+
+	pr_debug("subflow=%p", subflow);
+
+	err = tcp_v4_init_sock(sk);
+	if (!err) { // @@ AND mptcp is enabled
+		tsk->is_mptcp = 1;
+		icsk->icsk_af_ops = &subflow_specific;
+	}
+
+	return err;
+}
+
+static void subflow_close(struct sock *sk, long timeout)
+{
+	pr_debug("subflow=%p", sk);
+
+	tcp_close(sk, timeout);
+}

net/mptcp/subflow.c line 123

Quoted 10 lines
net/mptcp/subflow.c new file +10 -0
+
+static void subflow_destroy(struct sock *sk)
+{
+	pr_debug("subflow=%p", sk);
+
+	tcp_v4_destroy_sock(sk);
+}
+
+static struct proto subflow_prot = {
+	.name		= "SUBFLOW",

(MPTCP_SUBFLOW?)

Cheers,
Matthieu

Quoted 70 lines
net/mptcp/subflow.c new file +69 -0
+	.owner		= THIS_MODULE,
+	.close		= subflow_close,
+	.connect	= subflow_connect,
+	.disconnect	= tcp_disconnect,
+	.accept		= inet_csk_accept,
+	.ioctl		= tcp_ioctl,
+	.init		= subflow_init_sock,
+	.destroy	= subflow_destroy,
+	.shutdown	= tcp_shutdown,
+	.keepalive	= tcp_set_keepalive,
+	.recvmsg	= subflow_recvmsg,
+	.sendmsg	= subflow_sendmsg,
+	.sendpage	= tcp_sendpage,
+	.backlog_rcv	= tcp_v4_do_rcv,
+	.release_cb	= tcp_release_cb,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.enter_memory_pressure	= tcp_enter_memory_pressure,
+	.stream_memory_free	= tcp_stream_memory_free,
+	.sockets_allocated	= &tcp_sockets_allocated,
+	.orphan_count		= &tcp_orphan_count,
+	.memory_allocated	= &tcp_memory_allocated,
+	.memory_pressure	= &tcp_memory_pressure,
+	.sysctl_mem		= sysctl_tcp_mem,
+	.sysctl_wmem_offset	= offsetof(struct net, ipv4.sysctl_tcp_wmem),
+	.sysctl_rmem_offset	= offsetof(struct net, ipv4.sysctl_tcp_rmem),
+	.max_header		= MAX_TCP_HEADER,
+	.obj_size		= sizeof(struct subflow_sock),
+	.slab_flags		= SLAB_TYPESAFE_BY_RCU,
+
+	.no_autobind		= true,
+};
+
+static struct inet_protosw subflow_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_SUBFLOW,
+	.prot		= &subflow_prot,
+	.ops		= &inet_stream_ops,
+	.flags		= INET_PROTOSW_ICSK,
+};
+
+int mptcp_subflow_init(void)
+{
+	int err = -ENOMEM;
+
+	/* TODO: Register path manager callbacks. */
+
+	subflow_prot.twsk_prot		= tcp_prot.twsk_prot;
+	subflow_prot.h.hashinfo		= tcp_prot.h.hashinfo;
+	err = proto_register(&subflow_prot, 1);
+	if (err)
+		goto fail;
+
+	inet_register_protosw(&subflow_protosw);
+
+	return 0;
+
+fail:
+	return err;
+}
+
+void mptcp_subflow_exit(void)
+{
+	inet_unregister_protosw(&subflow_protosw);
+	proto_unregister(&subflow_prot);
+}
+
+MODULE_LICENSE("GPL");
Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
#23 lore raw

Re: [MPTCP] [RFC PATCH v3 07/16] mptcp: Create SUBFLOW socket for outgoing connections

Quoted 35 lines · Matthieu Baerts

Hi Peter,

Nice again!
I only have some details and questions on my side.

I guess we can look at the details later.

Quoted 27 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_sock type that extends tcp_sock and add an is_mptcp flag to tcp_sock to distinguish them.

Override the bind() and connect() methods of the MPTCP socket proto_ops so they may act on the subflow socket instead, and use the .sk_rx_dst_set() handler in the subflow proto to capture when the responding SYN-ACK is received.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN request for a subflow_sock and to the final ACK of the three-way handshake.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   2 +
 include/net/mptcp.h   |  35 +++++++-
 net/ipv4/tcp_input.c  |   9 ++
 net/ipv4/tcp_output.c |  32 +++++++
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   |  11 +++
 net/mptcp/protocol.c  | 161 +++++++++++++++++++++++++++++------
 net/mptcp/subflow.c   | 192 ++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 415 insertions(+), 29 deletions(-)
 create mode 100644 net/mptcp/subflow.c
Quoted 12 lines

include/linux/tcp.h struct tcp_sock { · line 412

Quoted 10 lines
include/linux/tcp.h +2 -0
index cb40dcb43f95..7f0dd688376c 100644
@@ -408,6 +408,8 @@ struct tcp_sock {
 	 */
 	struct request_sock *fastopen_rsk;
 	u32	*saved_syn;
+
+	bool	is_mptcp;

detail: can we use bool? Or one bit in a byte shared with other fields?

I will look for a place.

Quoted 24 lines
Quoted 24 lines
include/linux/tcp.h +0 -0
 };
 
 enum tsq_enum {
include/net/mptcp.h +4 -1
index 79daad717275..4b08eb4ccc6f 100644
@@ -39,7 +39,10 @@
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
 	struct	inet_connection_sock sk;
-	struct	socket *subflow;
+	u64	local_key;
+	u64	remote_key;
+	struct	socket *connection_list; /* @@ needs to be a list */
+	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -47,10 +50,35 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 	return (struct mptcp_sock *)sk;
 }
 
Quoted 5 lines

include/net/mptcp.h static inline struct mptcp_sock *mptcp_sk(const struct sock *sk) · line 54

Quoted 2 lines
include/net/mptcp.h +2 -0
+/* MPTCP subflow sock structure */
+struct subflow_sock {

detail: should we not start prefixing all MPTCP specific functions and structures with "mptcp_"?

My thinking is that we will try to only use this structure within MPTCP code (passed as 'struct sock' everywhere else) so the meaning of subflow within mptcp is clear. Also might be confusing with mptcp_sock and mptcp_subflow_sock.

Quoted 15 lines

include/net/mptcp.h static inline struct mptcp_sock *mptcp_sk(const struct sock *sk) · line 63

Quoted 10 lines
include/net/mptcp.h +9 -0
+	/* tcp_sock must be the first member */
+	struct	tcp_sock sk;
+	u64	local_key;
+	u64	remote_key;
+	bool	request_mptcp;	// send MP_CAPABLE
+	bool	checksum;
+	bool	version;
+	bool	mp_capable;	// remote is MPTCP capable
+	bool	fourth_ack;	// send initial DSS

detail: we should maybe continue in the same idea as with is_mptcp:
is_fourth_ack, etc. especially for fields like checksum and version. And maybe share a byte but can be changed later. I should certainly not look at these details :-)

Definitely should share a byte for these flags.

Quoted 41 lines
Quoted 41 lines
include/net/mptcp.h +18 -1
+	struct	sock *conn;	// parent mptcp_sock
+};
+
+static inline struct subflow_sock *subflow_sk(const struct sock *sk)
+{
+	return (struct subflow_sock *)sk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+
+void mptcp_finish_connect(struct sock *sk, int mp_capable);
+
+int mptcp_subflow_init(void);
+void mptcp_subflow_exit(void);
 
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
@@ -64,7 +92,10 @@ static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
 {
 }
 
-
+static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+	return 0;
+}
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +0 -0
index 81fccf0c9120..4cb38904bb5f 100644
@@ -5856,6 +5856,15 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
 		tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
 		tcp_initialize_rcv_mss(sk);
 
Quoted 9 lines

net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5865

Quoted 7 lines
net/ipv4/tcp_input.c +7 -0
+		// @@ could use a callout here
+		if (tp->is_mptcp) {
+			struct subflow_sock *subflow = subflow_sk(sk);
+			if (subflow->request_mptcp && tp->rx_opt.mptcp.mp_capable) {
+				subflow->mp_capable = 1;
+				subflow->remote_key = tp->rx_opt.mptcp.sndr_key;
+			}

detail: Should we do this kind of think in MPTCP specific files?

Agree, I should have followed the comment. This is removing subflow_sock from tcp code too.

Quoted 180 lines
Quoted 180 lines
net/ipv4/tcp_input.c +2 -0
+		}
+
 		/* Remember, tcp_poll() does not lock socket!
 		 * Change state from SYN-SENT only after copied_seq
 		 * is initialized. */
net/ipv4/tcp_output.c +32 -0
index a6ea8520e5d9..9919793e293b 100644
@@ -697,6 +697,16 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
 	smc_set_option(tp, opts, &remaining);
 
+	if (tp->is_mptcp) {
+		u64 local_key;
+		if (mptcp_syn_options(sk, &local_key)) {
+			opts->options |= OPTION_MPTCP;
+			opts->suboptions |= OPTION_MPTCP_MPC_SYN;
+			opts->sndr_key = local_key;
+			remaining -= TCPOLEN_MPTCP_MPC_SYN;
+		}
+	}
+
 	return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -804,6 +814,28 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 			opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;
 	}
 
+	if (tp->is_mptcp) {
+		struct subflow_sock *subflow = subflow_sk(sk);
+
+		pr_debug("tcp_established_options: subflow=%p", subflow);
+		opts->suboptions = 0;
+		if (subflow->mp_capable) {
+			const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
+			pr_debug("tcp_established_options: remaining=%d", remaining);
+			if (!subflow->fourth_ack) {
+				pr_debug("tcp_established_options: OPTION_MPTCP_MPC_ACK");
+				if (remaining >= 20) {
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions |= OPTION_MPTCP_MPC_ACK;
+					opts->sndr_key = subflow->local_key;
+					opts->rcvr_key = subflow->remote_key;
+					size += 20;
+				}
+				subflow->fourth_ack = 1;
+				// @@ also this is where first DSS goes in?
+			}
+		}
+	}
 	return size;
 }
 
net/mptcp/Makefile +1 -1
index 2bd18e3b9fda..3f0e7163fe80 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o options.o
+mptcp-y := protocol.o subflow.o options.o
net/mptcp/options.c +11 -0
index 4fd64e0bdd2e..4b1cbc3b3efe 100644
@@ -150,3 +150,14 @@ void mptcp_get_options(const struct sk_buff *skb,
 		}
 	}
 }
+
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	if (subflow->request_mptcp) {
+		pr_debug("local_key=%llu", subflow->local_key);
+		*local_key = subflow->local_key;
+	}
+	return subflow->request_mptcp;
+}
net/mptcp/protocol.c +35 -21
index c1eb4afc3ca4..1a3412a742ea 100644
@@ -26,9 +26,15 @@
 static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow = msk->subflow;
-
-	pr_debug("subflow=%p", subflow->sk);
+	struct socket *subflow;
+
+	if (msk->connection_list) {
+		subflow = msk->connection_list;
+		pr_debug("conn_list->subflow=%p", subflow->sk);
+	} else {
+		subflow = msk->subflow;
+		pr_debug("subflow=%p", subflow->sk);
+	}
 
 	return sock_sendmsg(subflow, msg);
 }
@@ -37,9 +43,15 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow = msk->subflow;
-
-	pr_debug("subflow=%p", subflow->sk);
+	struct socket *subflow;
+
+	if (msk->connection_list) {
+		subflow = msk->connection_list;
+		pr_debug("conn_list->subflow=%p", subflow->sk);
+	} else {
+		subflow = msk->subflow;
+		pr_debug("subflow=%p", subflow->sk);
+	}
 
 	return sock_recvmsg(subflow, msg, flags);
 }
@@ -47,19 +59,10 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 static int mptcp_init_sock(struct sock *sk)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *sf;
-	int err;
 
 	pr_debug("msk=%p", msk);
 
-	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
-			       &sf);
-	if (!err) {
-		pr_debug("subflow=%p", sf->sk);
-		msk->subflow = sf;
-	}
-
-	return err;
+	return 0;
 }
 
 static void mptcp_close(struct sock *sk, long timeout)
@@ -70,65 +73,171 @@ static void mptcp_close(struct sock *sk, long timeout)
 		pr_debug("subflow=%p", msk->subflow->sk);
 		sock_release(msk->subflow);
 	}
+
+	if (msk->connection_list) {
+		pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+		sock_release(msk->connection_list);
+	}
 }
 
-static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	int err;
+	struct sock *subflow = msk->subflow->sk;
+
+	pr_debug("msk=%p, subflow=%p", sk, subflow);
 
-	saddr->sa_family = AF_INET;
+	return inet_csk_get_port(subflow, snum);
+}
 
-	pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+void mptcp_finish_connect(struct sock *sk, int mp_capable)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct subflow_sock *subflow = subflow_sk(msk->subflow->sk);
 
-	err = kernel_connect(msk->subflow, saddr, len, 0);
+	pr_debug("msk=%p", msk);
 
Quoted 9 lines

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 106

Quoted 7 lines
net/mptcp/protocol.c +6 -0
+	if (mp_capable) {
+		msk->remote_key = subflow->remote_key;
+		msk->local_key = subflow->local_key;
+		msk->connection_list = msk->subflow;
+		msk->subflow = NULL;
+	}
 	sk->sk_state = TCP_ESTABLISHED;

(should we not call tcp_set_state here?)

Yes.

Quoted 23 lines
Quoted 23 lines
net/mptcp/protocol.c +18 -0
+}
 
+static int subflow_create(struct sock *sock)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock);
+	struct socket *sf;
+	int err;
+
+	err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_SUBFLOW,
+			       &sf);
+	if (!err) {
+		struct subflow_sock *subflow = subflow_sk(sf->sk);
+		pr_debug("subflow=%p", subflow);
+		msk->subflow = sf;
+		subflow->conn = sock;
+		subflow->request_mptcp = 1; // @@ if MPTCP enabled
+		subflow->checksum = 1; // @@ if checksum enabled
+		subflow->version = 0;
+	}
 	return err;
 }
 
Quoted 14 lines

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 139

Quoted 11 lines
net/mptcp/protocol.c +11 -0
+int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	struct socket *subflow = msk->subflow;
+
+	pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+
+	return inet_bind(subflow, uaddr, addr_len);
+}
+
+int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,

detail: I guess these two functions can be static

OK

Quoted 232 lines
Quoted 232 lines
net/mptcp/protocol.c +65 -5
+			 int addr_len, int flags)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+
+	return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
 	.init		= mptcp_init_sock,
 	.close		= mptcp_close,
 	.accept		= inet_csk_accept,
-	.connect	= mptcp_connect,
 	.shutdown	= tcp_shutdown,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
 	.hash		= inet_hash,
 	.unhash		= inet_unhash,
-	.get_port	= inet_csk_get_port,
+	.get_port	= mptcp_get_port,
 	.obj_size	= sizeof(struct mptcp_sock),
 	.no_autobind	= 1,
 };
 
+const struct proto_ops mptcp_stream_ops = {
+	.family		   = PF_INET,
+	.owner		   = THIS_MODULE,
+	.release	   = inet_release,
+	.bind		   = mptcp_stream_bind,
+	.connect	   = mptcp_stream_connect,
+	.socketpair	   = sock_no_socketpair,
+	.accept		   = inet_accept,
+	.getname	   = inet_getname,
+	.poll		   = tcp_poll,
+	.ioctl		   = inet_ioctl,
+	.listen		   = inet_listen,
+	.shutdown	   = inet_shutdown,
+	.setsockopt	   = sock_common_setsockopt,
+	.getsockopt	   = sock_common_getsockopt,
+	.sendmsg	   = inet_sendmsg,
+	.recvmsg	   = inet_recvmsg,
+	.mmap		   = sock_no_mmap,
+	.sendpage	   = inet_sendpage,
+	.splice_read	   = tcp_splice_read,
+	.read_sock	   = tcp_read_sock,
+	.peek_len	   = tcp_peek_len,
+#ifdef CONFIG_COMPAT
+	.compat_setsockopt = compat_sock_common_setsockopt,
+	.compat_getsockopt = compat_sock_common_getsockopt,
+	.compat_ioctl	   = inet_compat_ioctl,
+#endif
+};
+
 static struct inet_protosw mptcp_protosw = {
 	.type		= SOCK_STREAM,
 	.protocol	= IPPROTO_MPTCP,
 	.prot		= &mptcp_prot,
-	.ops		= &inet_stream_ops,
+	.ops		= &mptcp_stream_ops,
+	.flags		= INET_PROTOSW_ICSK,
 };
 
 static int __init mptcp_init(void)
 {
 	int err;
 
-	err = proto_register(&mptcp_prot, 1);
+	mptcp_prot.h.hashinfo = tcp_prot.h.hashinfo;
+
+	err = mptcp_subflow_init();
 	if (err)
-		return err;
+		goto subflow_failed;
+
+	err = proto_register(&mptcp_prot, 1);
+	if (err) {
+		goto proto_failed;
+	}
 
 	inet_register_protosw(&mptcp_protosw);
 
 	return 0;
+
+proto_failed:
+	mptcp_subflow_exit();
+
+subflow_failed:
+	return err;
 }
 
 static void __exit mptcp_exit(void)
 {
 	inet_unregister_protosw(&mptcp_protosw);
 	proto_unregister(&mptcp_prot);
+
+	mptcp_subflow_exit();
 }
 
 module_init(mptcp_init);
net/mptcp/subflow.c new file +117 -0
new file mode 100644
index 000000000000..5e5fdcb3175f
@@ -0,0 +1,192 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int subflow_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	saddr->sa_family = AF_INET; // @@ presume IPv4 for now
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_v4_connect(sk, saddr, len);
+}
+
+static int subflow_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_sendmsg(sk, msg, len);
+}
+
+static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+			   int nonblock, int flags, int *addr_len)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
+}
+
+static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	inet_sk_rx_dst_set(sk, skb);
+
+	pr_debug("subflow=%p", subflow);
+
+	if (subflow->conn) {
+		pr_debug("remote_key=%llu", subflow->remote_key);
+		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
+		subflow->conn = NULL;
+	}
+}
+
+const struct inet_connection_sock_af_ops subflow_specific = {
+	.queue_xmit	   = ip_queue_xmit,
+	.send_check	   = tcp_v4_send_check,
+	.rebuild_header	   = inet_sk_rebuild_header,
+	.sk_rx_dst_set	   = subflow_finish_connect,
+	.conn_request	   = tcp_v4_conn_request,
+	.syn_recv_sock	   = tcp_v4_syn_recv_sock,
+	.net_header_len	   = sizeof(struct iphdr),
+	.setsockopt	   = ip_setsockopt,
+	.getsockopt	   = ip_getsockopt,
+	.addr2sockaddr	   = inet_csk_addr2sockaddr,
+	.sockaddr_len	   = sizeof(struct sockaddr_in),
+#ifdef CONFIG_COMPAT
+	.compat_setsockopt = compat_ip_setsockopt,
+	.compat_getsockopt = compat_ip_getsockopt,
+#endif
+	.mtu_reduced	   = tcp_v4_mtu_reduced,
+};
+
+static int subflow_init_sock(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct tcp_sock *tsk = tcp_sk(sk);
+	struct inet_connection_sock *icsk = inet_csk(sk);
+	int err;
+
+	pr_debug("subflow=%p", subflow);
+
+	err = tcp_v4_init_sock(sk);
+	if (!err) { // @@ AND mptcp is enabled
+		tsk->is_mptcp = 1;
+		icsk->icsk_af_ops = &subflow_specific;
+	}
+
+	return err;
+}
+
+static void subflow_close(struct sock *sk, long timeout)
+{
+	pr_debug("subflow=%p", sk);
+
+	tcp_close(sk, timeout);
+}
+
+static void subflow_destroy(struct sock *sk)
+{
+	pr_debug("subflow=%p", sk);
Quoted 8 lines

net/mptcp/subflow.c line 123

Quoted 6 lines
net/mptcp/subflow.c new file +6 -0
+
+	tcp_v4_destroy_sock(sk);
+}
+
+static struct proto subflow_prot = {
+	.name		= "SUBFLOW",

(MPTCP_SUBFLOW?)

OK.

Thanks,

Peter.

Quoted 75 lines

Cheers,
Matthieu

Quoted 70 lines
net/mptcp/subflow.c new file +69 -0
+	.owner		= THIS_MODULE,
+	.close		= subflow_close,
+	.connect	= subflow_connect,
+	.disconnect	= tcp_disconnect,
+	.accept		= inet_csk_accept,
+	.ioctl		= tcp_ioctl,
+	.init		= subflow_init_sock,
+	.destroy	= subflow_destroy,
+	.shutdown	= tcp_shutdown,
+	.keepalive	= tcp_set_keepalive,
+	.recvmsg	= subflow_recvmsg,
+	.sendmsg	= subflow_sendmsg,
+	.sendpage	= tcp_sendpage,
+	.backlog_rcv	= tcp_v4_do_rcv,
+	.release_cb	= tcp_release_cb,
+	.hash		= inet_hash,
+	.unhash		= inet_unhash,
+	.get_port	= inet_csk_get_port,
+	.enter_memory_pressure	= tcp_enter_memory_pressure,
+	.stream_memory_free	= tcp_stream_memory_free,
+	.sockets_allocated	= &tcp_sockets_allocated,
+	.orphan_count		= &tcp_orphan_count,
+	.memory_allocated	= &tcp_memory_allocated,
+	.memory_pressure	= &tcp_memory_pressure,
+	.sysctl_mem		= sysctl_tcp_mem,
+	.sysctl_wmem_offset	= offsetof(struct net, ipv4.sysctl_tcp_wmem),
+	.sysctl_rmem_offset	= offsetof(struct net, ipv4.sysctl_tcp_rmem),
+	.max_header		= MAX_TCP_HEADER,
+	.obj_size		= sizeof(struct subflow_sock),
+	.slab_flags		= SLAB_TYPESAFE_BY_RCU,
+
+	.no_autobind		= true,
+};
+
+static struct inet_protosw subflow_protosw = {
+	.type		= SOCK_STREAM,
+	.protocol	= IPPROTO_SUBFLOW,
+	.prot		= &subflow_prot,
+	.ops		= &inet_stream_ops,
+	.flags		= INET_PROTOSW_ICSK,
+};
+
+int mptcp_subflow_init(void)
+{
+	int err = -ENOMEM;
+
+	/* TODO: Register path manager callbacks. */
+
+	subflow_prot.twsk_prot		= tcp_prot.twsk_prot;
+	subflow_prot.h.hashinfo		= tcp_prot.h.hashinfo;
+	err = proto_register(&subflow_prot, 1);
+	if (err)
+		goto fail;
+
+	inet_register_protosw(&subflow_protosw);
+
+	return 0;
+
+fail:
+	return err;
+}
+
+void mptcp_subflow_exit(void)
+{
+	inet_unregister_protosw(&subflow_protosw);
+	proto_unregister(&subflow_prot);
+}
+
+MODULE_LICENSE("GPL");
#24 lore raw

Re: [MPTCP] [RFC PATCH v3 07/16] mptcp: Create SUBFLOW socket for outgoing connections

Quoted 82 lines · Peter
Quoted 35 lines · Matthieu Baerts

Hi Peter,

Nice again!
I only have some details and questions on my side.

I guess we can look at the details later.

Quoted 27 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_sock type that extends tcp_sock and add an is_mptcp flag to tcp_sock to distinguish them.

Override the bind() and connect() methods of the MPTCP socket proto_ops so they may act on the subflow socket instead, and use the .sk_rx_dst_set() handler in the subflow proto to capture when the responding SYN-ACK is received.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN request for a subflow_sock and to the final ACK of the three-way handshake.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   2 +
 include/net/mptcp.h   |  35 +++++++-
 net/ipv4/tcp_input.c  |   9 ++
 net/ipv4/tcp_output.c |  32 +++++++
 net/mptcp/Makefile    |   2 +-
 net/mptcp/options.c   |  11 +++
 net/mptcp/protocol.c  | 161 +++++++++++++++++++++++++++++------
 net/mptcp/subflow.c   | 192 ++++++++++++++++++++++++++++++++++++++++++
 8 files changed, 415 insertions(+), 29 deletions(-)
 create mode 100644 net/mptcp/subflow.c
Quoted 12 lines

include/linux/tcp.h struct tcp_sock { · line 412

Quoted 10 lines
include/linux/tcp.h +2 -0
index cb40dcb43f95..7f0dd688376c 100644
@@ -408,6 +408,8 @@ struct tcp_sock {
     */
    struct request_sock *fastopen_rsk;
    u32     *saved_syn;
+
+   bool    is_mptcp;

detail: can we use bool? Or one bit in a byte shared with other fields?

I will look for a place.

Quoted 24 lines
Quoted 24 lines
include/linux/tcp.h +0 -0
 };
 
 enum tsq_enum {
include/net/mptcp.h +4 -1
index 79daad717275..4b08eb4ccc6f 100644
@@ -39,7 +39,10 @@
 struct mptcp_sock {
    /* inet_connection_sock must be the first member */
    struct  inet_connection_sock sk;
-   struct  socket *subflow;
+   u64     local_key;
+   u64     remote_key;
+   struct  socket *connection_list; /* @@ needs to be a list */
+   struct  socket *subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -47,10 +50,35 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
    return (struct mptcp_sock *)sk;
 }
 
Quoted 5 lines

include/net/mptcp.h static inline struct mptcp_sock *mptcp_sk(const struct sock *sk) · line 54

Quoted 2 lines
include/net/mptcp.h +2 -0
+/* MPTCP subflow sock structure */
+struct subflow_sock {

detail: should we not start prefixing all MPTCP specific functions and structures with "mptcp_"?

Quoted 4 lines

My thinking is that we will try to only use this structure within MPTCP code (passed as 'struct sock' everywhere else) so the meaning of subflow within mptcp is clear. Also might be confusing with mptcp_sock and mptcp_subflow_sock.

I was thinking that because it is exposed (and used below), we should maybe indicate that it is an MPTCP specific structure. Or should these structures be "private" and only defined in subflow.c?
But it might even be clearer to have this "mptcp_" within the MPTCP code and avoid generic subflow/token/crypto structures in the future. We can talk about that this evening if you want to.

Quoted 69 lines
Quoted 11 lines

include/net/mptcp.h static inline struct mptcp_sock *mptcp_sk(const struct sock *sk) · line 63

Quoted 10 lines
include/net/mptcp.h +9 -0
+   /* tcp_sock must be the first member */
+   struct  tcp_sock sk;
+   u64     local_key;
+   u64     remote_key;
+   bool    request_mptcp;  // send MP_CAPABLE
+   bool    checksum;
+   bool    version;
+   bool    mp_capable;     // remote is MPTCP capable
+   bool    fourth_ack;     // send initial DSS
Quoted 4 lines

detail: we should maybe continue in the same idea as with is_mptcp:
is_fourth_ack, etc. especially for fields like checksum and version. And maybe share a byte but can be changed later. I should certainly not look at these details :-)

Definitely should share a byte for these flags.

Quoted 41 lines
Quoted 41 lines
include/net/mptcp.h +18 -1
+   struct  sock *conn;     // parent mptcp_sock
+};
+
+static inline struct subflow_sock *subflow_sk(const struct sock *sk)
+{
+   return (struct subflow_sock *)sk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
                    struct tcp_options_received *opt_rx);
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+
+void mptcp_finish_connect(struct sock *sk, int mp_capable);
+
+int mptcp_subflow_init(void);
+void mptcp_subflow_exit(void);
 
 void mptcp_get_options(const struct sk_buff *skb,
                   struct tcp_options_received *options);
@@ -64,7 +92,10 @@ static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
 {
 }
 
-
+static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+   return 0;
+}
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +0 -0
index 81fccf0c9120..4cb38904bb5f 100644
@@ -5856,6 +5856,15 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
            tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
            tcp_initialize_rcv_mss(sk);
 
Quoted 9 lines

net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5865

Quoted 7 lines
net/ipv4/tcp_input.c +7 -0
+           // @@ could use a callout here
+           if (tp->is_mptcp) {
+                   struct subflow_sock *subflow = subflow_sk(sk);
+                   if (subflow->request_mptcp && tp->rx_opt.mptcp.mp_capable) {
+                           subflow->mp_capable = 1;
+                           subflow->remote_key = tp->rx_opt.mptcp.sndr_key;
+                   }

detail: Should we do this kind of think in MPTCP specific files?

Quoted 2 lines

Agree, I should have followed the comment. This is removing subflow_sock from tcp code too.

:-)

Matthieu

Quoted 554 lines
Quoted 180 lines
Quoted 180 lines
net/ipv4/tcp_input.c +2 -0
+           }
+
            /* Remember, tcp_poll() does not lock socket!
             * Change state from SYN-SENT only after copied_seq
             * is initialized. */
net/ipv4/tcp_output.c +32 -0
index a6ea8520e5d9..9919793e293b 100644
@@ -697,6 +697,16 @@ static unsigned int tcp_syn_options(struct sock *sk, struct sk_buff *skb,
 
    smc_set_option(tp, opts, &remaining);
 
+   if (tp->is_mptcp) {
+           u64 local_key;
+           if (mptcp_syn_options(sk, &local_key)) {
+                   opts->options |= OPTION_MPTCP;
+                   opts->suboptions |= OPTION_MPTCP_MPC_SYN;
+                   opts->sndr_key = local_key;
+                   remaining -= TCPOLEN_MPTCP_MPC_SYN;
+           }
+   }
+
    return MAX_TCP_OPTION_SPACE - remaining;
 }
 
@@ -804,6 +814,28 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
                    opts->num_sack_blocks * TCPOLEN_SACK_PERBLOCK;
    }
 
+   if (tp->is_mptcp) {
+           struct subflow_sock *subflow = subflow_sk(sk);
+
+           pr_debug("tcp_established_options: subflow=%p", subflow);
+           opts->suboptions = 0;
+           if (subflow->mp_capable) {
+                   const unsigned int remaining = MAX_TCP_OPTION_SPACE - size;
+                   pr_debug("tcp_established_options: remaining=%d", remaining);
+                   if (!subflow->fourth_ack) {
+                           pr_debug("tcp_established_options: OPTION_MPTCP_MPC_ACK");
+                           if (remaining >= 20) {
+                                   opts->options |= OPTION_MPTCP;
+                                   opts->suboptions |= OPTION_MPTCP_MPC_ACK;
+                                   opts->sndr_key = subflow->local_key;
+                                   opts->rcvr_key = subflow->remote_key;
+                                   size += 20;
+                           }
+                           subflow->fourth_ack = 1;
+                           // @@ also this is where first DSS goes in?
+                   }
+           }
+   }
    return size;
 }
 
net/mptcp/Makefile +1 -1
index 2bd18e3b9fda..3f0e7163fe80 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o options.o
+mptcp-y := protocol.o subflow.o options.o
net/mptcp/options.c +11 -0
index 4fd64e0bdd2e..4b1cbc3b3efe 100644
@@ -150,3 +150,14 @@ void mptcp_get_options(const struct sk_buff *skb,
            }
    }
 }
+
+unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
+{
+   struct subflow_sock *subflow = subflow_sk(sk);
+
+   if (subflow->request_mptcp) {
+           pr_debug("local_key=%llu", subflow->local_key);
+           *local_key = subflow->local_key;
+   }
+   return subflow->request_mptcp;
+}
net/mptcp/protocol.c +35 -21
index c1eb4afc3ca4..1a3412a742ea 100644
@@ -26,9 +26,15 @@
 static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 {
    struct mptcp_sock *msk = mptcp_sk(sk);
-   struct socket *subflow = msk->subflow;
-
-   pr_debug("subflow=%p", subflow->sk);
+   struct socket *subflow;
+
+   if (msk->connection_list) {
+           subflow = msk->connection_list;
+           pr_debug("conn_list->subflow=%p", subflow->sk);
+   } else {
+           subflow = msk->subflow;
+           pr_debug("subflow=%p", subflow->sk);
+   }
 
    return sock_sendmsg(subflow, msg);
 }
@@ -37,9 +43,15 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
                     int nonblock, int flags, int *addr_len)
 {
    struct mptcp_sock *msk = mptcp_sk(sk);
-   struct socket *subflow = msk->subflow;
-
-   pr_debug("subflow=%p", subflow->sk);
+   struct socket *subflow;
+
+   if (msk->connection_list) {
+           subflow = msk->connection_list;
+           pr_debug("conn_list->subflow=%p", subflow->sk);
+   } else {
+           subflow = msk->subflow;
+           pr_debug("subflow=%p", subflow->sk);
+   }
 
    return sock_recvmsg(subflow, msg, flags);
 }
@@ -47,19 +59,10 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 static int mptcp_init_sock(struct sock *sk)
 {
    struct mptcp_sock *msk = mptcp_sk(sk);
-   struct socket *sf;
-   int err;
 
    pr_debug("msk=%p", msk);
 
-   err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_TCP,
-                          &sf);
-   if (!err) {
-           pr_debug("subflow=%p", sf->sk);
-           msk->subflow = sf;
-   }
-
-   return err;
+   return 0;
 }
 
 static void mptcp_close(struct sock *sk, long timeout)
@@ -70,65 +73,171 @@ static void mptcp_close(struct sock *sk, long timeout)
            pr_debug("subflow=%p", msk->subflow->sk);
            sock_release(msk->subflow);
    }
+
+   if (msk->connection_list) {
+           pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+           sock_release(msk->connection_list);
+   }
 }
 
-static int mptcp_connect(struct sock *sk, struct sockaddr *saddr, int len)
+static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
    struct mptcp_sock *msk = mptcp_sk(sk);
-   int err;
+   struct sock *subflow = msk->subflow->sk;
+
+   pr_debug("msk=%p, subflow=%p", sk, subflow);
 
-   saddr->sa_family = AF_INET;
+   return inet_csk_get_port(subflow, snum);
+}
 
-   pr_debug("msk=%p, subflow=%p", msk, msk->subflow->sk);
+void mptcp_finish_connect(struct sock *sk, int mp_capable)
+{
+   struct mptcp_sock *msk = mptcp_sk(sk);
+   struct subflow_sock *subflow = subflow_sk(msk->subflow->sk);
 
-   err = kernel_connect(msk->subflow, saddr, len, 0);
+   pr_debug("msk=%p", msk);
 
Quoted 9 lines

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 106

Quoted 7 lines
net/mptcp/protocol.c +6 -0
+   if (mp_capable) {
+           msk->remote_key = subflow->remote_key;
+           msk->local_key = subflow->local_key;
+           msk->connection_list = msk->subflow;
+           msk->subflow = NULL;
+   }
    sk->sk_state = TCP_ESTABLISHED;

(should we not call tcp_set_state here?)

Yes.

Quoted 23 lines
Quoted 23 lines
net/mptcp/protocol.c +18 -0
+}
 
+static int subflow_create(struct sock *sock)
+{
+   struct mptcp_sock *msk = mptcp_sk(sock);
+   struct socket *sf;
+   int err;
+
+   err = sock_create_kern(&init_net, PF_INET, SOCK_STREAM, IPPROTO_SUBFLOW,
+                          &sf);
+   if (!err) {
+           struct subflow_sock *subflow = subflow_sk(sf->sk);
+           pr_debug("subflow=%p", subflow);
+           msk->subflow = sf;
+           subflow->conn = sock;
+           subflow->request_mptcp = 1; // @@ if MPTCP enabled
+           subflow->checksum = 1; // @@ if checksum enabled
+           subflow->version = 0;
+   }
    return err;
 }
 
Quoted 14 lines

net/mptcp/protocol.c static void mptcp_close(struct sock *sk, long timeout) · line 139

Quoted 11 lines
net/mptcp/protocol.c +11 -0
+int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
+{
+   struct mptcp_sock *msk = mptcp_sk(sock->sk);
+   struct socket *subflow = msk->subflow;
+
+   pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+
+   return inet_bind(subflow, uaddr, addr_len);
+}
+
+int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,

detail: I guess these two functions can be static

OK

Quoted 232 lines
Quoted 232 lines
net/mptcp/protocol.c +65 -5
+                    int addr_len, int flags)
+{
+   struct mptcp_sock *msk = mptcp_sk(sock->sk);
+   int err;
+
+   pr_debug("msk=%p", msk);
+
+   if (msk->subflow == NULL) {
+           err = subflow_create(sock->sk);
+           if (err)
+                   return err;
+   }
+
+   return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
+}
+
 static struct proto mptcp_prot = {
    .name           = "MPTCP",
    .owner          = THIS_MODULE,
    .init           = mptcp_init_sock,
    .close          = mptcp_close,
    .accept         = inet_csk_accept,
-   .connect        = mptcp_connect,
    .shutdown       = tcp_shutdown,
    .sendmsg        = mptcp_sendmsg,
    .recvmsg        = mptcp_recvmsg,
    .hash           = inet_hash,
    .unhash         = inet_unhash,
-   .get_port       = inet_csk_get_port,
+   .get_port       = mptcp_get_port,
    .obj_size       = sizeof(struct mptcp_sock),
    .no_autobind    = 1,
 };
 
+const struct proto_ops mptcp_stream_ops = {
+   .family            = PF_INET,
+   .owner             = THIS_MODULE,
+   .release           = inet_release,
+   .bind              = mptcp_stream_bind,
+   .connect           = mptcp_stream_connect,
+   .socketpair        = sock_no_socketpair,
+   .accept            = inet_accept,
+   .getname           = inet_getname,
+   .poll              = tcp_poll,
+   .ioctl             = inet_ioctl,
+   .listen            = inet_listen,
+   .shutdown          = inet_shutdown,
+   .setsockopt        = sock_common_setsockopt,
+   .getsockopt        = sock_common_getsockopt,
+   .sendmsg           = inet_sendmsg,
+   .recvmsg           = inet_recvmsg,
+   .mmap              = sock_no_mmap,
+   .sendpage          = inet_sendpage,
+   .splice_read       = tcp_splice_read,
+   .read_sock         = tcp_read_sock,
+   .peek_len          = tcp_peek_len,
+#ifdef CONFIG_COMPAT
+   .compat_setsockopt = compat_sock_common_setsockopt,
+   .compat_getsockopt = compat_sock_common_getsockopt,
+   .compat_ioctl      = inet_compat_ioctl,
+#endif
+};
+
 static struct inet_protosw mptcp_protosw = {
    .type           = SOCK_STREAM,
    .protocol       = IPPROTO_MPTCP,
    .prot           = &mptcp_prot,
-   .ops            = &inet_stream_ops,
+   .ops            = &mptcp_stream_ops,
+   .flags          = INET_PROTOSW_ICSK,
 };
 
 static int __init mptcp_init(void)
 {
    int err;
 
-   err = proto_register(&mptcp_prot, 1);
+   mptcp_prot.h.hashinfo = tcp_prot.h.hashinfo;
+
+   err = mptcp_subflow_init();
    if (err)
-           return err;
+           goto subflow_failed;
+
+   err = proto_register(&mptcp_prot, 1);
+   if (err) {
+           goto proto_failed;
+   }
 
    inet_register_protosw(&mptcp_protosw);
 
    return 0;
+
+proto_failed:
+   mptcp_subflow_exit();
+
+subflow_failed:
+   return err;
 }
 
 static void __exit mptcp_exit(void)
 {
    inet_unregister_protosw(&mptcp_protosw);
    proto_unregister(&mptcp_prot);
+
+   mptcp_subflow_exit();
 }
 
 module_init(mptcp_init);
net/mptcp/subflow.c new file +117 -0
new file mode 100644
index 000000000000..5e5fdcb3175f
@@ -0,0 +1,192 @@
+/*
+ * Multipath TCP
+ *
+ * Copyright (c) 2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/netdevice.h>
+#include <net/sock.h>
+#include <net/inet_common.h>
+#include <net/inet_hashtables.h>
+#include <net/protocol.h>
+#include <net/tcp.h>
+#include <net/mptcp.h>
+
+static int subflow_connect(struct sock *sk, struct sockaddr *saddr, int len)
+{
+   struct subflow_sock *subflow = subflow_sk(sk);
+
+   saddr->sa_family = AF_INET; // @@ presume IPv4 for now
+
+   pr_debug("subflow=%p", subflow);
+
+   return tcp_v4_connect(sk, saddr, len);
+}
+
+static int subflow_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
+{
+   struct subflow_sock *subflow = subflow_sk(sk);
+
+   pr_debug("subflow=%p", subflow);
+
+   return tcp_sendmsg(sk, msg, len);
+}
+
+static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
+                      int nonblock, int flags, int *addr_len)
+{
+   struct subflow_sock *subflow = subflow_sk(sk);
+
+   pr_debug("subflow=%p", subflow);
+
+   return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
+}
+
+static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
+{
+   struct subflow_sock *subflow = subflow_sk(sk);
+
+   inet_sk_rx_dst_set(sk, skb);
+
+   pr_debug("subflow=%p", subflow);
+
+   if (subflow->conn) {
+           pr_debug("remote_key=%llu", subflow->remote_key);
+           mptcp_finish_connect(subflow->conn, subflow->mp_capable);
+           subflow->conn = NULL;
+   }
+}
+
+const struct inet_connection_sock_af_ops subflow_specific = {
+   .queue_xmit        = ip_queue_xmit,
+   .send_check        = tcp_v4_send_check,
+   .rebuild_header    = inet_sk_rebuild_header,
+   .sk_rx_dst_set     = subflow_finish_connect,
+   .conn_request      = tcp_v4_conn_request,
+   .syn_recv_sock     = tcp_v4_syn_recv_sock,
+   .net_header_len    = sizeof(struct iphdr),
+   .setsockopt        = ip_setsockopt,
+   .getsockopt        = ip_getsockopt,
+   .addr2sockaddr     = inet_csk_addr2sockaddr,
+   .sockaddr_len      = sizeof(struct sockaddr_in),
+#ifdef CONFIG_COMPAT
+   .compat_setsockopt = compat_ip_setsockopt,
+   .compat_getsockopt = compat_ip_getsockopt,
+#endif
+   .mtu_reduced       = tcp_v4_mtu_reduced,
+};
+
+static int subflow_init_sock(struct sock *sk)
+{
+   struct subflow_sock *subflow = subflow_sk(sk);
+   struct tcp_sock *tsk = tcp_sk(sk);
+   struct inet_connection_sock *icsk = inet_csk(sk);
+   int err;
+
+   pr_debug("subflow=%p", subflow);
+
+   err = tcp_v4_init_sock(sk);
+   if (!err) { // @@ AND mptcp is enabled
+           tsk->is_mptcp = 1;
+           icsk->icsk_af_ops = &subflow_specific;
+   }
+
+   return err;
+}
+
+static void subflow_close(struct sock *sk, long timeout)
+{
+   pr_debug("subflow=%p", sk);
+
+   tcp_close(sk, timeout);
+}
+
+static void subflow_destroy(struct sock *sk)
+{
+   pr_debug("subflow=%p", sk);
Quoted 8 lines

net/mptcp/subflow.c line 123

Quoted 6 lines
net/mptcp/subflow.c new file +6 -0
+
+   tcp_v4_destroy_sock(sk);
+}
+
+static struct proto subflow_prot = {
+   .name           = "SUBFLOW",

(MPTCP_SUBFLOW?)

OK.

Thanks,

Peter.

Quoted 75 lines

Cheers,
Matthieu

Quoted 70 lines
net/mptcp/subflow.c new file +69 -0
+   .owner          = THIS_MODULE,
+   .close          = subflow_close,
+   .connect        = subflow_connect,
+   .disconnect     = tcp_disconnect,
+   .accept         = inet_csk_accept,
+   .ioctl          = tcp_ioctl,
+   .init           = subflow_init_sock,
+   .destroy        = subflow_destroy,
+   .shutdown       = tcp_shutdown,
+   .keepalive      = tcp_set_keepalive,
+   .recvmsg        = subflow_recvmsg,
+   .sendmsg        = subflow_sendmsg,
+   .sendpage       = tcp_sendpage,
+   .backlog_rcv    = tcp_v4_do_rcv,
+   .release_cb     = tcp_release_cb,
+   .hash           = inet_hash,
+   .unhash         = inet_unhash,
+   .get_port       = inet_csk_get_port,
+   .enter_memory_pressure  = tcp_enter_memory_pressure,
+   .stream_memory_free     = tcp_stream_memory_free,
+   .sockets_allocated      = &tcp_sockets_allocated,
+   .orphan_count           = &tcp_orphan_count,
+   .memory_allocated       = &tcp_memory_allocated,
+   .memory_pressure        = &tcp_memory_pressure,
+   .sysctl_mem             = sysctl_tcp_mem,
+   .sysctl_wmem_offset     = offsetof(struct net, ipv4.sysctl_tcp_wmem),
+   .sysctl_rmem_offset     = offsetof(struct net, ipv4.sysctl_tcp_rmem),
+   .max_header             = MAX_TCP_HEADER,
+   .obj_size               = sizeof(struct subflow_sock),
+   .slab_flags             = SLAB_TYPESAFE_BY_RCU,
+
+   .no_autobind            = true,
+};
+
+static struct inet_protosw subflow_protosw = {
+   .type           = SOCK_STREAM,
+   .protocol       = IPPROTO_SUBFLOW,
+   .prot           = &subflow_prot,
+   .ops            = &inet_stream_ops,
+   .flags          = INET_PROTOSW_ICSK,
+};
+
+int mptcp_subflow_init(void)
+{
+   int err = -ENOMEM;
+
+   /* TODO: Register path manager callbacks. */
+
+   subflow_prot.twsk_prot          = tcp_prot.twsk_prot;
+   subflow_prot.h.hashinfo         = tcp_prot.h.hashinfo;
+   err = proto_register(&subflow_prot, 1);
+   if (err)
+           goto fail;
+
+   inet_register_protosw(&subflow_protosw);
+
+   return 0;
+
+fail:
+   return err;
+}
+
+void mptcp_subflow_exit(void)
+{
+   inet_unregister_protosw(&subflow_protosw);
+   proto_unregister(&subflow_prot);
+}
+
+MODULE_LICENSE("GPL");
Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/
author
#25 lore raw

RFC PATCH v3 8/16 mptcp: Create SUBFLOW socket for incoming connections

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_request_sock type that extends tcp_request_sock and add an is_mptcp flag to tcp_request_sock distinguish them.

Override the listen() and accept() methods of the MPTCP socket proto_ops so they may act on the subflow socket.

Override the conn_request() and syn_recv_sock() handlers in the inet_connection_sock to handle incoming MPTCP SYNs and the ACK to the response SYN.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN-ACK response for a subflow_request_sock.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   1 +
 include/net/mptcp.h   |  27 ++++++++++
 net/ipv4/tcp_input.c  |   1 +
 net/ipv4/tcp_output.c |  14 +++++
 net/mptcp/options.c   |  14 +++++
 net/mptcp/protocol.c  | 102 ++++++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c   | 115 ++++++++++++++++++++++++++++++++++++++++--
 7 files changed, 264 insertions(+), 10 deletions(-)
include/linux/tcp.h +1 -0
index 7f0dd688376c..b109798482d3 100644
@@ -148,6 +148,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	bool				is_mptcp;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
include/net/mptcp.h +27 -0
index 4b08eb4ccc6f..56883a1ee2fe 100644
@@ -69,11 +69,30 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+struct subflow_request_sock {
+	struct	tcp_request_sock sk;
+	u8	mp_capable : 1,
+		mp_join : 1,
+		checksum : 1,
+		backup : 1,
+		version : 4;
+	u64	local_key;
+	u64	remote_key;
+};
+
+static inline
+struct subflow_request_sock *subflow_rsk(const struct request_sock *rsk)
+{
+	return (struct subflow_request_sock *)rsk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
 unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+unsigned int mptcp_synack_options(struct request_sock *req,
+				  u64 *local_key, u64 *remote_key);
 
 void mptcp_finish_connect(struct sock *sk, int mp_capable);
 
@@ -96,6 +115,14 @@ static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 {
 	return 0;
 }
+static inline unsigned int mptcp_synack_options(struct request_sock *sk,
+						u64 *local_key,
+						u64 *remote_key)
+{
+	return 0;
+}
+
+
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +1 -0
index 4cb38904bb5f..9a326729637f 100644
@@ -6441,6 +6441,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	tcp_rsk(req)->is_mptcp = 0;
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
net/ipv4/tcp_output.c +14 -0
index 9919793e293b..780abb11dffd 100644
@@ -767,6 +767,20 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 			remaining -= need;
 		}
 	}
+	if (tcp_rsk(req)->is_mptcp) {
+		u64 local_key;
+		u64 remote_key;
+		if (mptcp_synack_options(req, &local_key, &remote_key)) {
+			if (remaining >= TCPOLEN_MPTCP_MPC_SYNACK) {
+				opts->options |= OPTION_MPTCP;
+				opts->suboptions |= OPTION_MPTCP_MPC_SYNACK;
+				opts->sndr_key = local_key;
+				opts->rcvr_key = remote_key;
+				remaining -= TCPOLEN_MPTCP_MPC_SYNACK;
+			}
+		}
+	}
+
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
 	return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/options.c +14 -0
index 4b1cbc3b3efe..7e48d1d92aac 100644
@@ -161,3 +161,17 @@ unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 	}
 	return subflow->request_mptcp;
 }
+
+unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
+				  u64 *remote_key)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+
+	if (subflow_req->mp_capable) {
+		*local_key = subflow_req->local_key;
+		*remote_key = subflow_req->remote_key;
+		pr_debug("local_key=%llu", *local_key);
+		pr_debug("remote_key=%llu", *remote_key);
+	}
+	return subflow_req->mp_capable;
+}
net/mptcp/protocol.c +95 -7

135 lines not on this page · raw message

net/mptcp/subflow.c +112 -3

151 lines not on this page · raw message

Signature · 1 line
2.19.1
#26 lore raw

Re: [MPTCP] [RFC PATCH v3 08/16] mptcp: Create SUBFLOW socket for incoming connections

Quoted 372 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_request_sock type that extends tcp_request_sock and add an is_mptcp flag to tcp_request_sock distinguish them.

Override the listen() and accept() methods of the MPTCP socket proto_ops so they may act on the subflow socket.

Override the conn_request() and syn_recv_sock() handlers in the inet_connection_sock to handle incoming MPTCP SYNs and the ACK to the response SYN.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN-ACK response for a subflow_request_sock.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   1 +
 include/net/mptcp.h   |  27 ++++++++++
 net/ipv4/tcp_input.c  |   1 +
 net/ipv4/tcp_output.c |  14 +++++
 net/mptcp/options.c   |  14 +++++
 net/mptcp/protocol.c  | 102 ++++++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c   | 115 ++++++++++++++++++++++++++++++++++++++++--
 7 files changed, 264 insertions(+), 10 deletions(-)
include/linux/tcp.h +1 -0
index 7f0dd688376c..b109798482d3 100644
@@ -148,6 +148,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	bool				is_mptcp;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
include/net/mptcp.h +27 -0
index 4b08eb4ccc6f..56883a1ee2fe 100644
@@ -69,11 +69,30 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+struct subflow_request_sock {
+	struct	tcp_request_sock sk;
+	u8	mp_capable : 1,
+		mp_join : 1,
+		checksum : 1,
+		backup : 1,
+		version : 4;
+	u64	local_key;
+	u64	remote_key;
+};
+
+static inline
+struct subflow_request_sock *subflow_rsk(const struct request_sock *rsk)
+{
+	return (struct subflow_request_sock *)rsk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
 unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+unsigned int mptcp_synack_options(struct request_sock *req,
+				  u64 *local_key, u64 *remote_key);
 
 void mptcp_finish_connect(struct sock *sk, int mp_capable);
 
@@ -96,6 +115,14 @@ static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 {
 	return 0;
 }
+static inline unsigned int mptcp_synack_options(struct request_sock *sk,
+						u64 *local_key,
+						u64 *remote_key)
+{
+	return 0;
+}
+
+
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +1 -0
index 4cb38904bb5f..9a326729637f 100644
@@ -6441,6 +6441,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	tcp_rsk(req)->is_mptcp = 0;
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
net/ipv4/tcp_output.c +14 -0
index 9919793e293b..780abb11dffd 100644
@@ -767,6 +767,20 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 			remaining -= need;
 		}
 	}
+	if (tcp_rsk(req)->is_mptcp) {
+		u64 local_key;
+		u64 remote_key;
+		if (mptcp_synack_options(req, &local_key, &remote_key)) {
+			if (remaining >= TCPOLEN_MPTCP_MPC_SYNACK) {
+				opts->options |= OPTION_MPTCP;
+				opts->suboptions |= OPTION_MPTCP_MPC_SYNACK;
+				opts->sndr_key = local_key;
+				opts->rcvr_key = remote_key;
+				remaining -= TCPOLEN_MPTCP_MPC_SYNACK;
+			}
+		}
+	}
+
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
 	return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/options.c +14 -0
index 4b1cbc3b3efe..7e48d1d92aac 100644
@@ -161,3 +161,17 @@ unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 	}
 	return subflow->request_mptcp;
 }
+
+unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
+				  u64 *remote_key)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+
+	if (subflow_req->mp_capable) {
+		*local_key = subflow_req->local_key;
+		*remote_key = subflow_req->remote_key;
+		pr_debug("local_key=%llu", *local_key);
+		pr_debug("remote_key=%llu", *remote_key);
+	}
+	return subflow_req->mp_capable;
+}
net/mptcp/protocol.c +95 -7
index 1a3412a742ea..9f802f69a528 100644
@@ -80,6 +80,45 @@ static void mptcp_close(struct sock *sk, long timeout)
 	}
 }
 
+static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
+				 bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *listener = msk->subflow;
+	struct socket *new_sock;
+	struct socket *mp;
+	struct subflow_sock *subflow;
+
+	pr_debug("msk=%p, listener=%p", msk, listener->sk);
+	*err = kernel_accept(listener, &new_sock, flags);
+	if (*err < 0)
+		return NULL;
+
+	subflow = subflow_sk(new_sock->sk);
+	pr_debug("new_sock=%p", subflow);
+
+	*err = sock_create(PF_INET, SOCK_STREAM, IPPROTO_MPTCP, &mp);
+	if (*err < 0) {
+		kernel_sock_shutdown(new_sock, SHUT_RDWR);
+		sock_release(new_sock);
+		return NULL;
+	}
+
+	msk = mptcp_sk(mp->sk);
+	pr_debug("msk=%p", msk);
+	subflow->conn = mp->sk;
+
+	if (subflow->mp_capable) {
+		msk->remote_key = subflow->remote_key;
+		msk->local_key = subflow->local_key;
+		msk->connection_list = new_sock;
+	} else {
+		msk->subflow = new_sock;
+	}
+
+	return mp->sk;
+}
+
 static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
@@ -129,11 +168,16 @@ static int subflow_create(struct sock *sock)
 int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sock->sk);
-	struct socket *subflow = msk->subflow;
+	int err;
 
-	pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+	pr_debug("msk=%p", msk);
 
-	return inet_bind(subflow, uaddr, addr_len);
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_bind(msk->subflow, uaddr, addr_len);
 }
 
 int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
@@ -153,12 +197,56 @@ int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 	return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
 }
 
+int mptcp_stream_getname(struct socket *sock, struct sockaddr *uaddr, int peer)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	struct socket *subflow;
+	int err = -EPERM;
+
+	if (msk->connection_list)
+		subflow = msk->connection_list;
+	else
+		subflow = msk->subflow;
+
+	err = inet_getname(subflow, uaddr, peer);
+
+	return err;
+}
+
+int mptcp_stream_listen(struct socket *sock, int backlog)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_listen(msk->subflow, backlog);
+}
+
+int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
+			bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		return -EINVAL;
+	}
+	return inet_accept(sock, newsock, flags, kern);
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
 	.init		= mptcp_init_sock,
 	.close		= mptcp_close,
-	.accept		= inet_csk_accept,
+	.accept		= mptcp_accept,
 	.shutdown	= tcp_shutdown,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
@@ -176,11 +264,11 @@ const struct proto_ops mptcp_stream_ops = {
 	.bind		   = mptcp_stream_bind,
 	.connect	   = mptcp_stream_connect,
 	.socketpair	   = sock_no_socketpair,
-	.accept		   = inet_accept,
-	.getname	   = inet_getname,
+	.accept		   = mptcp_stream_accept,
+	.getname	   = mptcp_stream_getname,
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
-	.listen		   = inet_listen,
+	.listen		   = mptcp_stream_listen,
 	.shutdown	   = inet_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
net/mptcp/subflow.c +67 -0
index 5e5fdcb3175f..89fcc3b746eb 100644
@@ -53,6 +53,40 @@ static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
 }
 
+static void subflow_v4_init_req(struct request_sock *req,
+				const struct sock *sk_listener,
+				struct sk_buff *skb)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct subflow_sock *listener = subflow_sk(sk_listener);
+	struct tcp_options_received rx_opt;
+
+	tcp_rsk(req)->is_mptcp = 1;
+	pr_debug("subflow_req=%p, listener=%p", subflow_req, listener);
+
+	tcp_request_sock_ipv4_ops.init_req(req, sk_listener, skb);
+
+	rx_opt.mptcp.flags = 0;
+	rx_opt.mptcp.mp_capable = 0;
+	rx_opt.mptcp.mp_join = 0;
+	rx_opt.mptcp.dss = 0;
+	mptcp_get_options(skb, &rx_opt);
+
+	if (rx_opt.mptcp.mp_capable && listener->request_mptcp) {
+		subflow_req->mp_capable = 1;
+		if (rx_opt.mptcp.version >= listener->version)
+			subflow_req->version = listener->version;
+		else
+			subflow_req->version = rx_opt.mptcp.version;
+		if ((rx_opt.mptcp.flags & MPTCP_CAP_CHECKSUM_REQD) ||
+		    listener->checksum)
+			subflow_req->checksum = 1;
+		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
+	} else {
+		subflow_req->mp_capable = 0;
+	}
+}
+
 static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -68,13 +102,66 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 	}
 }
 
+static struct request_sock_ops subflow_request_sock_ops;
+static struct tcp_request_sock_ops subflow_request_sock_ipv4_ops;
+
+static int subflow_conn_request(struct sock *sk, struct sk_buff *skb)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	/* Never answer to SYNs sent to broadcast or multicast */
+	if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
+		goto drop;
+
+	return tcp_conn_request(&subflow_request_sock_ops,
+				&subflow_request_sock_ipv4_ops,
+				sk, skb);
+drop:
+	tcp_listendrop(sk);
+	return 0;
+}
+
+static struct sock *subflow_syn_recv_sock(const struct sock *sk,
+					  struct sk_buff *skb,
+					  struct request_sock *req,
+					  struct dst_entry *dst,
+					  struct request_sock *req_unhash,
+					  bool *own_req)
+{
+	struct subflow_sock *listener = subflow_sk(sk);
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct sock *child;
+
+	pr_debug("listener=%p, req=%p, conn=%p", sk, req, listener->conn);

net/mptcp/subflow.c static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb) · line 147

Quoted 10 lines
net/mptcp/subflow.c +10 -0
+
+	child = tcp_v4_syn_recv_sock(sk, skb, req, dst, req_unhash, own_req);
+
+	if (child) {
+		struct subflow_sock *subflow = subflow_sk(child);
+
+		pr_debug("child=%p", child);
+		if (subflow_req->mp_capable) {
+			subflow->mp_capable = 1;
+			subflow->fourth_ack = 1;

Where is the fourth ack being triggered?

Christoph

Quoted 75 lines
net/mptcp/subflow.c +35 -3
+			subflow->remote_key = subflow_req->remote_key;
+			subflow->local_key = subflow_req->local_key;
+		} else {
+			subflow->mp_capable = 0;
+		}
+	}
+
+	return child;
+}
+
 const struct inet_connection_sock_af_ops subflow_specific = {
 	.queue_xmit	   = ip_queue_xmit,
 	.send_check	   = tcp_v4_send_check,
 	.rebuild_header	   = inet_sk_rebuild_header,
 	.sk_rx_dst_set	   = subflow_finish_connect,
-	.conn_request	   = tcp_v4_conn_request,
-	.syn_recv_sock	   = tcp_v4_syn_recv_sock,
+	.conn_request	   = subflow_conn_request,
+	.syn_recv_sock	   = subflow_syn_recv_sock,
 	.net_header_len	   = sizeof(struct iphdr),
 	.setsockopt	   = ip_setsockopt,
 	.getsockopt	   = ip_getsockopt,
@@ -112,6 +199,21 @@ static void subflow_close(struct sock *sk, long timeout)
 	tcp_close(sk, timeout);
 }
 
+static struct sock *subflow_accept(struct sock *sk, int flags, int *err,
+				   bool kern)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *child;
+
+	pr_debug("subflow=%p, conn=%p", subflow, subflow->conn);
+
+	child = inet_csk_accept(sk, flags, err, kern);
+
+	pr_debug("child=%p", child);
+
+	return child;
+}
+
 static void subflow_destroy(struct sock *sk)
 {
 	pr_debug("subflow=%p", sk);
@@ -125,7 +227,7 @@ static struct proto subflow_prot = {
 	.close		= subflow_close,
 	.connect	= subflow_connect,
 	.disconnect	= tcp_disconnect,
-	.accept		= inet_csk_accept,
+	.accept		= subflow_accept,
 	.ioctl		= tcp_ioctl,
 	.init		= subflow_init_sock,
 	.destroy	= subflow_destroy,
@@ -169,7 +271,14 @@ int mptcp_subflow_init(void)
 
 	/* TODO: Register path manager callbacks. */
 
+	subflow_request_sock_ops = tcp_request_sock_ops;
+	subflow_request_sock_ops.obj_size = sizeof(struct subflow_request_sock),
+
+	subflow_request_sock_ipv4_ops = tcp_request_sock_ipv4_ops;
+	subflow_request_sock_ipv4_ops.init_req = subflow_v4_init_req;
+
 	subflow_prot.twsk_prot		= tcp_prot.twsk_prot;
+	subflow_prot.rsk_prot		= &subflow_request_sock_ops;
 	subflow_prot.h.hashinfo		= tcp_prot.h.hashinfo;
 	err = proto_register(&subflow_prot, 1);
 	if (err)
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp
#27 lore raw

Re: [MPTCP] [RFC PATCH v3 08/16] mptcp: Create SUBFLOW socket for incoming connections

Quoted 377 lines · Christoph Paasch
Quoted 376 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_request_sock type that extends tcp_request_sock and add an is_mptcp flag to tcp_request_sock distinguish them.

Override the listen() and accept() methods of the MPTCP socket proto_ops so they may act on the subflow socket.

Override the conn_request() and syn_recv_sock() handlers in the inet_connection_sock to handle incoming MPTCP SYNs and the ACK to the response SYN.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN-ACK response for a subflow_request_sock.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   1 +
 include/net/mptcp.h   |  27 ++++++++++
 net/ipv4/tcp_input.c  |   1 +
 net/ipv4/tcp_output.c |  14 +++++
 net/mptcp/options.c   |  14 +++++
 net/mptcp/protocol.c  | 102 ++++++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c   | 115 ++++++++++++++++++++++++++++++++++++++++--
 7 files changed, 264 insertions(+), 10 deletions(-)
include/linux/tcp.h +1 -0
index 7f0dd688376c..b109798482d3 100644
@@ -148,6 +148,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	bool				is_mptcp;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
include/net/mptcp.h +27 -0
index 4b08eb4ccc6f..56883a1ee2fe 100644
@@ -69,11 +69,30 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+struct subflow_request_sock {
+	struct	tcp_request_sock sk;
+	u8	mp_capable : 1,
+		mp_join : 1,
+		checksum : 1,
+		backup : 1,
+		version : 4;
+	u64	local_key;
+	u64	remote_key;
+};
+
+static inline
+struct subflow_request_sock *subflow_rsk(const struct request_sock *rsk)
+{
+	return (struct subflow_request_sock *)rsk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
 unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+unsigned int mptcp_synack_options(struct request_sock *req,
+				  u64 *local_key, u64 *remote_key);
 
 void mptcp_finish_connect(struct sock *sk, int mp_capable);
 
@@ -96,6 +115,14 @@ static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 {
 	return 0;
 }
+static inline unsigned int mptcp_synack_options(struct request_sock *sk,
+						u64 *local_key,
+						u64 *remote_key)
+{
+	return 0;
+}
+
+
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +1 -0
index 4cb38904bb5f..9a326729637f 100644
@@ -6441,6 +6441,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	tcp_rsk(req)->is_mptcp = 0;
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
net/ipv4/tcp_output.c +14 -0
index 9919793e293b..780abb11dffd 100644
@@ -767,6 +767,20 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 			remaining -= need;
 		}
 	}
+	if (tcp_rsk(req)->is_mptcp) {
+		u64 local_key;
+		u64 remote_key;
+		if (mptcp_synack_options(req, &local_key, &remote_key)) {
+			if (remaining >= TCPOLEN_MPTCP_MPC_SYNACK) {
+				opts->options |= OPTION_MPTCP;
+				opts->suboptions |= OPTION_MPTCP_MPC_SYNACK;
+				opts->sndr_key = local_key;
+				opts->rcvr_key = remote_key;
+				remaining -= TCPOLEN_MPTCP_MPC_SYNACK;
+			}
+		}
+	}
+
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
 	return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/options.c +14 -0
index 4b1cbc3b3efe..7e48d1d92aac 100644
@@ -161,3 +161,17 @@ unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 	}
 	return subflow->request_mptcp;
 }
+
+unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
+				  u64 *remote_key)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+
+	if (subflow_req->mp_capable) {
+		*local_key = subflow_req->local_key;
+		*remote_key = subflow_req->remote_key;
+		pr_debug("local_key=%llu", *local_key);
+		pr_debug("remote_key=%llu", *remote_key);
+	}
+	return subflow_req->mp_capable;
+}
net/mptcp/protocol.c +95 -7
index 1a3412a742ea..9f802f69a528 100644
@@ -80,6 +80,45 @@ static void mptcp_close(struct sock *sk, long timeout)
 	}
 }
 
+static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
+				 bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *listener = msk->subflow;
+	struct socket *new_sock;
+	struct socket *mp;
+	struct subflow_sock *subflow;
+
+	pr_debug("msk=%p, listener=%p", msk, listener->sk);
+	*err = kernel_accept(listener, &new_sock, flags);
+	if (*err < 0)
+		return NULL;
+
+	subflow = subflow_sk(new_sock->sk);
+	pr_debug("new_sock=%p", subflow);
+
+	*err = sock_create(PF_INET, SOCK_STREAM, IPPROTO_MPTCP, &mp);
+	if (*err < 0) {
+		kernel_sock_shutdown(new_sock, SHUT_RDWR);
+		sock_release(new_sock);
+		return NULL;
+	}
+
+	msk = mptcp_sk(mp->sk);
+	pr_debug("msk=%p", msk);
+	subflow->conn = mp->sk;
+
+	if (subflow->mp_capable) {
+		msk->remote_key = subflow->remote_key;
+		msk->local_key = subflow->local_key;
+		msk->connection_list = new_sock;
+	} else {
+		msk->subflow = new_sock;
+	}
+
+	return mp->sk;
+}
+
 static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
@@ -129,11 +168,16 @@ static int subflow_create(struct sock *sock)
 int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sock->sk);
-	struct socket *subflow = msk->subflow;
+	int err;
 
-	pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+	pr_debug("msk=%p", msk);
 
-	return inet_bind(subflow, uaddr, addr_len);
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_bind(msk->subflow, uaddr, addr_len);
 }
 
 int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
@@ -153,12 +197,56 @@ int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 	return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
 }
 
+int mptcp_stream_getname(struct socket *sock, struct sockaddr *uaddr, int peer)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	struct socket *subflow;
+	int err = -EPERM;
+
+	if (msk->connection_list)
+		subflow = msk->connection_list;
+	else
+		subflow = msk->subflow;
+
+	err = inet_getname(subflow, uaddr, peer);
+
+	return err;
+}
+
+int mptcp_stream_listen(struct socket *sock, int backlog)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_listen(msk->subflow, backlog);
+}
+
+int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
+			bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		return -EINVAL;
+	}
+	return inet_accept(sock, newsock, flags, kern);
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
 	.init		= mptcp_init_sock,
 	.close		= mptcp_close,
-	.accept		= inet_csk_accept,
+	.accept		= mptcp_accept,
 	.shutdown	= tcp_shutdown,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
@@ -176,11 +264,11 @@ const struct proto_ops mptcp_stream_ops = {
 	.bind		   = mptcp_stream_bind,
 	.connect	   = mptcp_stream_connect,
 	.socketpair	   = sock_no_socketpair,
-	.accept		   = inet_accept,
-	.getname	   = inet_getname,
+	.accept		   = mptcp_stream_accept,
+	.getname	   = mptcp_stream_getname,
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
-	.listen		   = inet_listen,
+	.listen		   = mptcp_stream_listen,
 	.shutdown	   = inet_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
net/mptcp/subflow.c +71 -0
index 5e5fdcb3175f..89fcc3b746eb 100644
@@ -53,6 +53,40 @@ static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
 }
 
+static void subflow_v4_init_req(struct request_sock *req,
+				const struct sock *sk_listener,
+				struct sk_buff *skb)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct subflow_sock *listener = subflow_sk(sk_listener);
+	struct tcp_options_received rx_opt;
+
+	tcp_rsk(req)->is_mptcp = 1;
+	pr_debug("subflow_req=%p, listener=%p", subflow_req, listener);
+
+	tcp_request_sock_ipv4_ops.init_req(req, sk_listener, skb);
+
+	rx_opt.mptcp.flags = 0;
+	rx_opt.mptcp.mp_capable = 0;
+	rx_opt.mptcp.mp_join = 0;
+	rx_opt.mptcp.dss = 0;
+	mptcp_get_options(skb, &rx_opt);
+
+	if (rx_opt.mptcp.mp_capable && listener->request_mptcp) {
+		subflow_req->mp_capable = 1;
+		if (rx_opt.mptcp.version >= listener->version)
+			subflow_req->version = listener->version;
+		else
+			subflow_req->version = rx_opt.mptcp.version;
+		if ((rx_opt.mptcp.flags & MPTCP_CAP_CHECKSUM_REQD) ||
+		    listener->checksum)
+			subflow_req->checksum = 1;
+		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
+	} else {
+		subflow_req->mp_capable = 0;
+	}
+}
+
 static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -68,13 +102,66 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 	}
 }
 
+static struct request_sock_ops subflow_request_sock_ops;
+static struct tcp_request_sock_ops subflow_request_sock_ipv4_ops;
+
+static int subflow_conn_request(struct sock *sk, struct sk_buff *skb)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	/* Never answer to SYNs sent to broadcast or multicast */
+	if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
+		goto drop;
+
+	return tcp_conn_request(&subflow_request_sock_ops,
+				&subflow_request_sock_ipv4_ops,
+				sk, skb);
+drop:
+	tcp_listendrop(sk);
+	return 0;
+}
+
+static struct sock *subflow_syn_recv_sock(const struct sock *sk,
+					  struct sk_buff *skb,
+					  struct request_sock *req,
+					  struct dst_entry *dst,
+					  struct request_sock *req_unhash,
+					  bool *own_req)
+{
+	struct subflow_sock *listener = subflow_sk(sk);
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct sock *child;
+
+	pr_debug("listener=%p, req=%p, conn=%p", sk, req, listener->conn);
+
+	child = tcp_v4_syn_recv_sock(sk, skb, req, dst, req_unhash, own_req);
+
+	if (child) {
Quoted 9 lines

net/mptcp/subflow.c static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb) · line 147

Quoted 6 lines
net/mptcp/subflow.c +6 -0
+		struct subflow_sock *subflow = subflow_sk(child);
+
+		pr_debug("child=%p", child);
+		if (subflow_req->mp_capable) {
+			subflow->mp_capable = 1;
+			subflow->fourth_ack = 1;

Where is the fourth ack being triggered?

For incoming connections this flag is used to prevent sending MP_CAPABLE in any ACKs (see tcp_established_options). The field name is a legacy choice. There is not currently a mechanism to send a duplicate ack with first DSS option after the three-way handshake completes.

Peter.

Quoted 83 lines

Christoph

Quoted 75 lines
net/mptcp/subflow.c +35 -3
+			subflow->remote_key = subflow_req->remote_key;
+			subflow->local_key = subflow_req->local_key;
+		} else {
+			subflow->mp_capable = 0;
+		}
+	}
+
+	return child;
+}
+
 const struct inet_connection_sock_af_ops subflow_specific = {
 	.queue_xmit	   = ip_queue_xmit,
 	.send_check	   = tcp_v4_send_check,
 	.rebuild_header	   = inet_sk_rebuild_header,
 	.sk_rx_dst_set	   = subflow_finish_connect,
-	.conn_request	   = tcp_v4_conn_request,
-	.syn_recv_sock	   = tcp_v4_syn_recv_sock,
+	.conn_request	   = subflow_conn_request,
+	.syn_recv_sock	   = subflow_syn_recv_sock,
 	.net_header_len	   = sizeof(struct iphdr),
 	.setsockopt	   = ip_setsockopt,
 	.getsockopt	   = ip_getsockopt,
@@ -112,6 +199,21 @@ static void subflow_close(struct sock *sk, long timeout)
 	tcp_close(sk, timeout);
 }
 
+static struct sock *subflow_accept(struct sock *sk, int flags, int *err,
+				   bool kern)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *child;
+
+	pr_debug("subflow=%p, conn=%p", subflow, subflow->conn);
+
+	child = inet_csk_accept(sk, flags, err, kern);
+
+	pr_debug("child=%p", child);
+
+	return child;
+}
+
 static void subflow_destroy(struct sock *sk)
 {
 	pr_debug("subflow=%p", sk);
@@ -125,7 +227,7 @@ static struct proto subflow_prot = {
 	.close		= subflow_close,
 	.connect	= subflow_connect,
 	.disconnect	= tcp_disconnect,
-	.accept		= inet_csk_accept,
+	.accept		= subflow_accept,
 	.ioctl		= tcp_ioctl,
 	.init		= subflow_init_sock,
 	.destroy	= subflow_destroy,
@@ -169,7 +271,14 @@ int mptcp_subflow_init(void)
 
 	/* TODO: Register path manager callbacks. */
 
+	subflow_request_sock_ops = tcp_request_sock_ops;
+	subflow_request_sock_ops.obj_size = sizeof(struct subflow_request_sock),
+
+	subflow_request_sock_ipv4_ops = tcp_request_sock_ipv4_ops;
+	subflow_request_sock_ipv4_ops.init_req = subflow_v4_init_req;
+
 	subflow_prot.twsk_prot		= tcp_prot.twsk_prot;
+	subflow_prot.rsk_prot		= &subflow_request_sock_ops;
 	subflow_prot.h.hashinfo		= tcp_prot.h.hashinfo;
 	err = proto_register(&subflow_prot, 1);
 	if (err)
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

#28 lore raw

Re: [MPTCP] [RFC PATCH v3 08/16] mptcp: Create SUBFLOW socket for incoming connections

Quoted 387 lines · Peter
Quoted 377 lines · Christoph Paasch
Quoted 376 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_request_sock type that extends tcp_request_sock and add an is_mptcp flag to tcp_request_sock distinguish them.

Override the listen() and accept() methods of the MPTCP socket proto_ops so they may act on the subflow socket.

Override the conn_request() and syn_recv_sock() handlers in the inet_connection_sock to handle incoming MPTCP SYNs and the ACK to the response SYN.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN-ACK response for a subflow_request_sock.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   1 +
 include/net/mptcp.h   |  27 ++++++++++
 net/ipv4/tcp_input.c  |   1 +
 net/ipv4/tcp_output.c |  14 +++++
 net/mptcp/options.c   |  14 +++++
 net/mptcp/protocol.c  | 102 ++++++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c   | 115 ++++++++++++++++++++++++++++++++++++++++--
 7 files changed, 264 insertions(+), 10 deletions(-)
include/linux/tcp.h +1 -0
index 7f0dd688376c..b109798482d3 100644
@@ -148,6 +148,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	bool				is_mptcp;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
include/net/mptcp.h +27 -0
index 4b08eb4ccc6f..56883a1ee2fe 100644
@@ -69,11 +69,30 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+struct subflow_request_sock {
+	struct	tcp_request_sock sk;
+	u8	mp_capable : 1,
+		mp_join : 1,
+		checksum : 1,
+		backup : 1,
+		version : 4;
+	u64	local_key;
+	u64	remote_key;
+};
+
+static inline
+struct subflow_request_sock *subflow_rsk(const struct request_sock *rsk)
+{
+	return (struct subflow_request_sock *)rsk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
 unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+unsigned int mptcp_synack_options(struct request_sock *req,
+				  u64 *local_key, u64 *remote_key);
 
 void mptcp_finish_connect(struct sock *sk, int mp_capable);
 
@@ -96,6 +115,14 @@ static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 {
 	return 0;
 }
+static inline unsigned int mptcp_synack_options(struct request_sock *sk,
+						u64 *local_key,
+						u64 *remote_key)
+{
+	return 0;
+}
+
+
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +1 -0
index 4cb38904bb5f..9a326729637f 100644
@@ -6441,6 +6441,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	tcp_rsk(req)->is_mptcp = 0;
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
net/ipv4/tcp_output.c +14 -0
index 9919793e293b..780abb11dffd 100644
@@ -767,6 +767,20 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 			remaining -= need;
 		}
 	}
+	if (tcp_rsk(req)->is_mptcp) {
+		u64 local_key;
+		u64 remote_key;
+		if (mptcp_synack_options(req, &local_key, &remote_key)) {
+			if (remaining >= TCPOLEN_MPTCP_MPC_SYNACK) {
+				opts->options |= OPTION_MPTCP;
+				opts->suboptions |= OPTION_MPTCP_MPC_SYNACK;
+				opts->sndr_key = local_key;
+				opts->rcvr_key = remote_key;
+				remaining -= TCPOLEN_MPTCP_MPC_SYNACK;
+			}
+		}
+	}
+
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
 	return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/options.c +14 -0
index 4b1cbc3b3efe..7e48d1d92aac 100644
@@ -161,3 +161,17 @@ unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 	}
 	return subflow->request_mptcp;
 }
+
+unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
+				  u64 *remote_key)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+
+	if (subflow_req->mp_capable) {
+		*local_key = subflow_req->local_key;
+		*remote_key = subflow_req->remote_key;
+		pr_debug("local_key=%llu", *local_key);
+		pr_debug("remote_key=%llu", *remote_key);
+	}
+	return subflow_req->mp_capable;
+}
net/mptcp/protocol.c +95 -7
index 1a3412a742ea..9f802f69a528 100644
@@ -80,6 +80,45 @@ static void mptcp_close(struct sock *sk, long timeout)
 	}
 }
 
+static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
+				 bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *listener = msk->subflow;
+	struct socket *new_sock;
+	struct socket *mp;
+	struct subflow_sock *subflow;
+
+	pr_debug("msk=%p, listener=%p", msk, listener->sk);
+	*err = kernel_accept(listener, &new_sock, flags);
+	if (*err < 0)
+		return NULL;
+
+	subflow = subflow_sk(new_sock->sk);
+	pr_debug("new_sock=%p", subflow);
+
+	*err = sock_create(PF_INET, SOCK_STREAM, IPPROTO_MPTCP, &mp);
+	if (*err < 0) {
+		kernel_sock_shutdown(new_sock, SHUT_RDWR);
+		sock_release(new_sock);
+		return NULL;
+	}
+
+	msk = mptcp_sk(mp->sk);
+	pr_debug("msk=%p", msk);
+	subflow->conn = mp->sk;
+
+	if (subflow->mp_capable) {
+		msk->remote_key = subflow->remote_key;
+		msk->local_key = subflow->local_key;
+		msk->connection_list = new_sock;
+	} else {
+		msk->subflow = new_sock;
+	}
+
+	return mp->sk;
+}
+
 static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
@@ -129,11 +168,16 @@ static int subflow_create(struct sock *sock)
 int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sock->sk);
-	struct socket *subflow = msk->subflow;
+	int err;
 
-	pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+	pr_debug("msk=%p", msk);
 
-	return inet_bind(subflow, uaddr, addr_len);
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_bind(msk->subflow, uaddr, addr_len);
 }
 
 int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
@@ -153,12 +197,56 @@ int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 	return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
 }
 
+int mptcp_stream_getname(struct socket *sock, struct sockaddr *uaddr, int peer)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	struct socket *subflow;
+	int err = -EPERM;
+
+	if (msk->connection_list)
+		subflow = msk->connection_list;
+	else
+		subflow = msk->subflow;
+
+	err = inet_getname(subflow, uaddr, peer);
+
+	return err;
+}
+
+int mptcp_stream_listen(struct socket *sock, int backlog)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_listen(msk->subflow, backlog);
+}
+
+int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
+			bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		return -EINVAL;
+	}
+	return inet_accept(sock, newsock, flags, kern);
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
 	.init		= mptcp_init_sock,
 	.close		= mptcp_close,
-	.accept		= inet_csk_accept,
+	.accept		= mptcp_accept,
 	.shutdown	= tcp_shutdown,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
@@ -176,11 +264,11 @@ const struct proto_ops mptcp_stream_ops = {
 	.bind		   = mptcp_stream_bind,
 	.connect	   = mptcp_stream_connect,
 	.socketpair	   = sock_no_socketpair,
-	.accept		   = inet_accept,
-	.getname	   = inet_getname,
+	.accept		   = mptcp_stream_accept,
+	.getname	   = mptcp_stream_getname,
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
-	.listen		   = inet_listen,
+	.listen		   = mptcp_stream_listen,
 	.shutdown	   = inet_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
net/mptcp/subflow.c +71 -0
index 5e5fdcb3175f..89fcc3b746eb 100644
@@ -53,6 +53,40 @@ static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
 }
 
+static void subflow_v4_init_req(struct request_sock *req,
+				const struct sock *sk_listener,
+				struct sk_buff *skb)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct subflow_sock *listener = subflow_sk(sk_listener);
+	struct tcp_options_received rx_opt;
+
+	tcp_rsk(req)->is_mptcp = 1;
+	pr_debug("subflow_req=%p, listener=%p", subflow_req, listener);
+
+	tcp_request_sock_ipv4_ops.init_req(req, sk_listener, skb);
+
+	rx_opt.mptcp.flags = 0;
+	rx_opt.mptcp.mp_capable = 0;
+	rx_opt.mptcp.mp_join = 0;
+	rx_opt.mptcp.dss = 0;
+	mptcp_get_options(skb, &rx_opt);
+
+	if (rx_opt.mptcp.mp_capable && listener->request_mptcp) {
+		subflow_req->mp_capable = 1;
+		if (rx_opt.mptcp.version >= listener->version)
+			subflow_req->version = listener->version;
+		else
+			subflow_req->version = rx_opt.mptcp.version;
+		if ((rx_opt.mptcp.flags & MPTCP_CAP_CHECKSUM_REQD) ||
+		    listener->checksum)
+			subflow_req->checksum = 1;
+		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
+	} else {
+		subflow_req->mp_capable = 0;
+	}
+}
+
 static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -68,13 +102,66 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 	}
 }
 
+static struct request_sock_ops subflow_request_sock_ops;
+static struct tcp_request_sock_ops subflow_request_sock_ipv4_ops;
+
+static int subflow_conn_request(struct sock *sk, struct sk_buff *skb)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	/* Never answer to SYNs sent to broadcast or multicast */
+	if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
+		goto drop;
+
+	return tcp_conn_request(&subflow_request_sock_ops,
+				&subflow_request_sock_ipv4_ops,
+				sk, skb);
+drop:
+	tcp_listendrop(sk);
+	return 0;
+}
+
+static struct sock *subflow_syn_recv_sock(const struct sock *sk,
+					  struct sk_buff *skb,
+					  struct request_sock *req,
+					  struct dst_entry *dst,
+					  struct request_sock *req_unhash,
+					  bool *own_req)
+{
+	struct subflow_sock *listener = subflow_sk(sk);
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct sock *child;
+
+	pr_debug("listener=%p, req=%p, conn=%p", sk, req, listener->conn);
+
+	child = tcp_v4_syn_recv_sock(sk, skb, req, dst, req_unhash, own_req);
+
+	if (child) {
Quoted 9 lines

net/mptcp/subflow.c static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb) · line 147

Quoted 6 lines
net/mptcp/subflow.c +6 -0
+		struct subflow_sock *subflow = subflow_sk(child);
+
+		pr_debug("child=%p", child);
+		if (subflow_req->mp_capable) {
+			subflow->mp_capable = 1;
+			subflow->fourth_ack = 1;

Where is the fourth ack being triggered?

Quoted 3 lines

For incoming connections this flag is used to prevent sending MP_CAPABLE in any ACKs (see tcp_established_options). The field name is a legacy choice.

I see.

Quoted 2 lines

There is not currently a mechanism to send a duplicate ack with first DSS option after the three-way handshake completes.

Oh - ok that is needed because MPTCP's JOIN-exchange requires a four-way handshake. Which is why I understood the "fourth_ack" as the final ACK of the four-way handshake and thus started looking for where the call to tcp_send_ack() is ;-)

Christoph

Quoted 86 lines

Peter.

Quoted 83 lines

Christoph

Quoted 75 lines
net/mptcp/subflow.c +35 -3
+			subflow->remote_key = subflow_req->remote_key;
+			subflow->local_key = subflow_req->local_key;
+		} else {
+			subflow->mp_capable = 0;
+		}
+	}
+
+	return child;
+}
+
 const struct inet_connection_sock_af_ops subflow_specific = {
 	.queue_xmit	   = ip_queue_xmit,
 	.send_check	   = tcp_v4_send_check,
 	.rebuild_header	   = inet_sk_rebuild_header,
 	.sk_rx_dst_set	   = subflow_finish_connect,
-	.conn_request	   = tcp_v4_conn_request,
-	.syn_recv_sock	   = tcp_v4_syn_recv_sock,
+	.conn_request	   = subflow_conn_request,
+	.syn_recv_sock	   = subflow_syn_recv_sock,
 	.net_header_len	   = sizeof(struct iphdr),
 	.setsockopt	   = ip_setsockopt,
 	.getsockopt	   = ip_getsockopt,
@@ -112,6 +199,21 @@ static void subflow_close(struct sock *sk, long timeout)
 	tcp_close(sk, timeout);
 }
 
+static struct sock *subflow_accept(struct sock *sk, int flags, int *err,
+				   bool kern)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *child;
+
+	pr_debug("subflow=%p, conn=%p", subflow, subflow->conn);
+
+	child = inet_csk_accept(sk, flags, err, kern);
+
+	pr_debug("child=%p", child);
+
+	return child;
+}
+
 static void subflow_destroy(struct sock *sk)
 {
 	pr_debug("subflow=%p", sk);
@@ -125,7 +227,7 @@ static struct proto subflow_prot = {
 	.close		= subflow_close,
 	.connect	= subflow_connect,
 	.disconnect	= tcp_disconnect,
-	.accept		= inet_csk_accept,
+	.accept		= subflow_accept,
 	.ioctl		= tcp_ioctl,
 	.init		= subflow_init_sock,
 	.destroy	= subflow_destroy,
@@ -169,7 +271,14 @@ int mptcp_subflow_init(void)
 
 	/* TODO: Register path manager callbacks. */
 
+	subflow_request_sock_ops = tcp_request_sock_ops;
+	subflow_request_sock_ops.obj_size = sizeof(struct subflow_request_sock),
+
+	subflow_request_sock_ipv4_ops = tcp_request_sock_ipv4_ops;
+	subflow_request_sock_ipv4_ops.init_req = subflow_v4_init_req;
+
 	subflow_prot.twsk_prot		= tcp_prot.twsk_prot;
+	subflow_prot.rsk_prot		= &subflow_request_sock_ops;
 	subflow_prot.h.hashinfo		= tcp_prot.h.hashinfo;
 	err = proto_register(&subflow_prot, 1);
 	if (err)
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

#29 lore raw

Re: [MPTCP] [RFC PATCH v3 08/16] mptcp: Create SUBFLOW socket for incoming connections

Quoted 395 lines · cpaasch(a)apple.com
Quoted 388 lines · Peter
Quoted 377 lines · Christoph Paasch
Quoted 376 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Add subflow_request_sock type that extends tcp_request_sock and add an is_mptcp flag to tcp_request_sock distinguish them.

Override the listen() and accept() methods of the MPTCP socket proto_ops so they may act on the subflow socket.

Override the conn_request() and syn_recv_sock() handlers in the inet_connection_sock to handle incoming MPTCP SYNs and the ACK to the response SYN.

Add handling in tcp_output.c to add MP_CAPABLE to an outgoing SYN-ACK response for a subflow_request_sock.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/linux/tcp.h   |   1 +
 include/net/mptcp.h   |  27 ++++++++++
 net/ipv4/tcp_input.c  |   1 +
 net/ipv4/tcp_output.c |  14 +++++
 net/mptcp/options.c   |  14 +++++
 net/mptcp/protocol.c  | 102 ++++++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c   | 115 ++++++++++++++++++++++++++++++++++++++++--
 7 files changed, 264 insertions(+), 10 deletions(-)
include/linux/tcp.h +1 -0
index 7f0dd688376c..b109798482d3 100644
@@ -148,6 +148,7 @@ struct tcp_request_sock {
 						  * FastOpen it's the seq#
 						  * after data-in-SYN.
 						  */
+	bool				is_mptcp;
 };
 
 static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
include/net/mptcp.h +27 -0
index 4b08eb4ccc6f..56883a1ee2fe 100644
@@ -69,11 +69,30 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+struct subflow_request_sock {
+	struct	tcp_request_sock sk;
+	u8	mp_capable : 1,
+		mp_join : 1,
+		checksum : 1,
+		backup : 1,
+		version : 4;
+	u64	local_key;
+	u64	remote_key;
+};
+
+static inline
+struct subflow_request_sock *subflow_rsk(const struct request_sock *rsk)
+{
+	return (struct subflow_request_sock *)rsk;
+}
+
 #ifdef CONFIG_MPTCP
 
 void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx);
 unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key);
+unsigned int mptcp_synack_options(struct request_sock *req,
+				  u64 *local_key, u64 *remote_key);
 
 void mptcp_finish_connect(struct sock *sk, int mp_capable);
 
@@ -96,6 +115,14 @@ static inline unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 {
 	return 0;
 }
+static inline unsigned int mptcp_synack_options(struct request_sock *sk,
+						u64 *local_key,
+						u64 *remote_key)
+{
+	return 0;
+}
+
+
 
 #endif /* CONFIG_MPTCP */
 #endif /* __NET_MPTCP_H */
net/ipv4/tcp_input.c +1 -0
index 4cb38904bb5f..9a326729637f 100644
@@ -6441,6 +6441,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
 
 	tcp_rsk(req)->af_specific = af_ops;
 	tcp_rsk(req)->ts_off = 0;
+	tcp_rsk(req)->is_mptcp = 0;
 
 	tcp_clear_options(&tmp_opt);
 	tmp_opt.mss_clamp = af_ops->mss_clamp;
net/ipv4/tcp_output.c +14 -0
index 9919793e293b..780abb11dffd 100644
@@ -767,6 +767,20 @@ static unsigned int tcp_synack_options(const struct sock *sk,
 			remaining -= need;
 		}
 	}
+	if (tcp_rsk(req)->is_mptcp) {
+		u64 local_key;
+		u64 remote_key;
+		if (mptcp_synack_options(req, &local_key, &remote_key)) {
+			if (remaining >= TCPOLEN_MPTCP_MPC_SYNACK) {
+				opts->options |= OPTION_MPTCP;
+				opts->suboptions |= OPTION_MPTCP_MPC_SYNACK;
+				opts->sndr_key = local_key;
+				opts->rcvr_key = remote_key;
+				remaining -= TCPOLEN_MPTCP_MPC_SYNACK;
+			}
+		}
+	}
+
 	smc_set_option_cond(tcp_sk(sk), ireq, opts, &remaining);
 
 	return MAX_TCP_OPTION_SPACE - remaining;
net/mptcp/options.c +14 -0
index 4b1cbc3b3efe..7e48d1d92aac 100644
@@ -161,3 +161,17 @@ unsigned int mptcp_syn_options(struct sock *sk, u64 *local_key)
 	}
 	return subflow->request_mptcp;
 }
+
+unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
+				  u64 *remote_key)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+
+	if (subflow_req->mp_capable) {
+		*local_key = subflow_req->local_key;
+		*remote_key = subflow_req->remote_key;
+		pr_debug("local_key=%llu", *local_key);
+		pr_debug("remote_key=%llu", *remote_key);
+	}
+	return subflow_req->mp_capable;
+}
net/mptcp/protocol.c +95 -7
index 1a3412a742ea..9f802f69a528 100644
@@ -80,6 +80,45 @@ static void mptcp_close(struct sock *sk, long timeout)
 	}
 }
 
+static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
+				 bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *listener = msk->subflow;
+	struct socket *new_sock;
+	struct socket *mp;
+	struct subflow_sock *subflow;
+
+	pr_debug("msk=%p, listener=%p", msk, listener->sk);
+	*err = kernel_accept(listener, &new_sock, flags);
+	if (*err < 0)
+		return NULL;
+
+	subflow = subflow_sk(new_sock->sk);
+	pr_debug("new_sock=%p", subflow);
+
+	*err = sock_create(PF_INET, SOCK_STREAM, IPPROTO_MPTCP, &mp);
+	if (*err < 0) {
+		kernel_sock_shutdown(new_sock, SHUT_RDWR);
+		sock_release(new_sock);
+		return NULL;
+	}
+
+	msk = mptcp_sk(mp->sk);
+	pr_debug("msk=%p", msk);
+	subflow->conn = mp->sk;
+
+	if (subflow->mp_capable) {
+		msk->remote_key = subflow->remote_key;
+		msk->local_key = subflow->local_key;
+		msk->connection_list = new_sock;
+	} else {
+		msk->subflow = new_sock;
+	}
+
+	return mp->sk;
+}
+
 static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
@@ -129,11 +168,16 @@ static int subflow_create(struct sock *sock)
 int mptcp_stream_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sock->sk);
-	struct socket *subflow = msk->subflow;
+	int err;
 
-	pr_debug("msk=%p, subflow=%p", msk, subflow->sk);
+	pr_debug("msk=%p", msk);
 
-	return inet_bind(subflow, uaddr, addr_len);
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_bind(msk->subflow, uaddr, addr_len);
 }
 
 int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
@@ -153,12 +197,56 @@ int mptcp_stream_connect(struct socket *sock, struct sockaddr *uaddr,
 	return inet_stream_connect(msk->subflow, uaddr, addr_len, flags);
 }
 
+int mptcp_stream_getname(struct socket *sock, struct sockaddr *uaddr, int peer)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	struct socket *subflow;
+	int err = -EPERM;
+
+	if (msk->connection_list)
+		subflow = msk->connection_list;
+	else
+		subflow = msk->subflow;
+
+	err = inet_getname(subflow, uaddr, peer);
+
+	return err;
+}
+
+int mptcp_stream_listen(struct socket *sock, int backlog)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int err;
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		err = subflow_create(sock->sk);
+		if (err)
+			return err;
+	}
+	return inet_listen(msk->subflow, backlog);
+}
+
+int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
+			bool kern)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+
+	pr_debug("msk=%p", msk);
+
+	if (msk->subflow == NULL) {
+		return -EINVAL;
+	}
+	return inet_accept(sock, newsock, flags, kern);
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
 	.init		= mptcp_init_sock,
 	.close		= mptcp_close,
-	.accept		= inet_csk_accept,
+	.accept		= mptcp_accept,
 	.shutdown	= tcp_shutdown,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
@@ -176,11 +264,11 @@ const struct proto_ops mptcp_stream_ops = {
 	.bind		   = mptcp_stream_bind,
 	.connect	   = mptcp_stream_connect,
 	.socketpair	   = sock_no_socketpair,
-	.accept		   = inet_accept,
-	.getname	   = inet_getname,
+	.accept		   = mptcp_stream_accept,
+	.getname	   = mptcp_stream_getname,
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
-	.listen		   = inet_listen,
+	.listen		   = mptcp_stream_listen,
 	.shutdown	   = inet_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
net/mptcp/subflow.c +71 -0
index 5e5fdcb3175f..89fcc3b746eb 100644
@@ -53,6 +53,40 @@ static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
 }
 
+static void subflow_v4_init_req(struct request_sock *req,
+				const struct sock *sk_listener,
+				struct sk_buff *skb)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct subflow_sock *listener = subflow_sk(sk_listener);
+	struct tcp_options_received rx_opt;
+
+	tcp_rsk(req)->is_mptcp = 1;
+	pr_debug("subflow_req=%p, listener=%p", subflow_req, listener);
+
+	tcp_request_sock_ipv4_ops.init_req(req, sk_listener, skb);
+
+	rx_opt.mptcp.flags = 0;
+	rx_opt.mptcp.mp_capable = 0;
+	rx_opt.mptcp.mp_join = 0;
+	rx_opt.mptcp.dss = 0;
+	mptcp_get_options(skb, &rx_opt);
+
+	if (rx_opt.mptcp.mp_capable && listener->request_mptcp) {
+		subflow_req->mp_capable = 1;
+		if (rx_opt.mptcp.version >= listener->version)
+			subflow_req->version = listener->version;
+		else
+			subflow_req->version = rx_opt.mptcp.version;
+		if ((rx_opt.mptcp.flags & MPTCP_CAP_CHECKSUM_REQD) ||
+		    listener->checksum)
+			subflow_req->checksum = 1;
+		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
+	} else {
+		subflow_req->mp_capable = 0;
+	}
+}
+
 static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -68,13 +102,66 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 	}
 }
 
+static struct request_sock_ops subflow_request_sock_ops;
+static struct tcp_request_sock_ops subflow_request_sock_ipv4_ops;
+
+static int subflow_conn_request(struct sock *sk, struct sk_buff *skb)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	/* Never answer to SYNs sent to broadcast or multicast */
+	if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
+		goto drop;
+
+	return tcp_conn_request(&subflow_request_sock_ops,
+				&subflow_request_sock_ipv4_ops,
+				sk, skb);
+drop:
+	tcp_listendrop(sk);
+	return 0;
+}
+
+static struct sock *subflow_syn_recv_sock(const struct sock *sk,
+					  struct sk_buff *skb,
+					  struct request_sock *req,
+					  struct dst_entry *dst,
+					  struct request_sock *req_unhash,
+					  bool *own_req)
+{
+	struct subflow_sock *listener = subflow_sk(sk);
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+	struct sock *child;
+
+	pr_debug("listener=%p, req=%p, conn=%p", sk, req, listener->conn);
+
+	child = tcp_v4_syn_recv_sock(sk, skb, req, dst, req_unhash, own_req);
+
+	if (child) {
Quoted 9 lines

net/mptcp/subflow.c static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb) · line 147

Quoted 6 lines
net/mptcp/subflow.c +6 -0
+		struct subflow_sock *subflow = subflow_sk(child);
+
+		pr_debug("child=%p", child);
+		if (subflow_req->mp_capable) {
+			subflow->mp_capable = 1;
+			subflow->fourth_ack = 1;

Where is the fourth ack being triggered?

Quoted 3 lines

For incoming connections this flag is used to prevent sending MP_CAPABLE in any ACKs (see tcp_established_options). The field name is a legacy choice.

I see.

Quoted 8 lines
Quoted 2 lines

There is not currently a mechanism to send a duplicate ack with first DSS option after the three-way handshake completes.

Oh - ok that is needed because MPTCP's JOIN-exchange requires a four-way handshake. Which is why I understood the "fourth_ack" as the final ACK of the four-way handshake and thus started looking for where the call to tcp_send_ack() is ;-)

The legacy of the field name is that it was for this purpose (triggering the foruth ack) but the prototpye hasn't got that far yet.

Peter.

Quoted 88 lines

Christoph

Quoted 86 lines

Peter.

Quoted 83 lines

Christoph

Quoted 75 lines
net/mptcp/subflow.c +35 -3
+			subflow->remote_key = subflow_req->remote_key;
+			subflow->local_key = subflow_req->local_key;
+		} else {
+			subflow->mp_capable = 0;
+		}
+	}
+
+	return child;
+}
+
 const struct inet_connection_sock_af_ops subflow_specific = {
 	.queue_xmit	   = ip_queue_xmit,
 	.send_check	   = tcp_v4_send_check,
 	.rebuild_header	   = inet_sk_rebuild_header,
 	.sk_rx_dst_set	   = subflow_finish_connect,
-	.conn_request	   = tcp_v4_conn_request,
-	.syn_recv_sock	   = tcp_v4_syn_recv_sock,
+	.conn_request	   = subflow_conn_request,
+	.syn_recv_sock	   = subflow_syn_recv_sock,
 	.net_header_len	   = sizeof(struct iphdr),
 	.setsockopt	   = ip_setsockopt,
 	.getsockopt	   = ip_getsockopt,
@@ -112,6 +199,21 @@ static void subflow_close(struct sock *sk, long timeout)
 	tcp_close(sk, timeout);
 }
 
+static struct sock *subflow_accept(struct sock *sk, int flags, int *err,
+				   bool kern)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *child;
+
+	pr_debug("subflow=%p, conn=%p", subflow, subflow->conn);
+
+	child = inet_csk_accept(sk, flags, err, kern);
+
+	pr_debug("child=%p", child);
+
+	return child;
+}
+
 static void subflow_destroy(struct sock *sk)
 {
 	pr_debug("subflow=%p", sk);
@@ -125,7 +227,7 @@ static struct proto subflow_prot = {
 	.close		= subflow_close,
 	.connect	= subflow_connect,
 	.disconnect	= tcp_disconnect,
-	.accept		= inet_csk_accept,
+	.accept		= subflow_accept,
 	.ioctl		= tcp_ioctl,
 	.init		= subflow_init_sock,
 	.destroy	= subflow_destroy,
@@ -169,7 +271,14 @@ int mptcp_subflow_init(void)
 
 	/* TODO: Register path manager callbacks. */
 
+	subflow_request_sock_ops = tcp_request_sock_ops;
+	subflow_request_sock_ops.obj_size = sizeof(struct subflow_request_sock),
+
+	subflow_request_sock_ipv4_ops = tcp_request_sock_ipv4_ops;
+	subflow_request_sock_ipv4_ops.init_req = subflow_v4_init_req;
+
 	subflow_prot.twsk_prot		= tcp_prot.twsk_prot;
+	subflow_prot.rsk_prot		= &subflow_request_sock_ops;
 	subflow_prot.h.hashinfo		= tcp_prot.h.hashinfo;
 	err = proto_register(&subflow_prot, 1);
 	if (err)
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

author
#30 lore raw

RFC PATCH v3 9/16 mptcp: Add key generation and token tree

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Generate the local keys, IDSN, and token when creating a new socket. Introduce the token tree to track all tokens in use using a radix tree with the MPTCP token itself as the index.

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 include/net/mptcp.h  |  26 +++++
 net/mptcp/Makefile   |   2 +-
 net/mptcp/crypto.c   | 215 +++++++++++++++++++++++++++++++++++
 net/mptcp/protocol.c |  18 +++
 net/mptcp/subflow.c  |  31 ++++-
 net/mptcp/token.c    | 264 +++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 554 insertions(+), 2 deletions(-)
 create mode 100644 net/mptcp/crypto.c
 create mode 100644 net/mptcp/token.c
include/net/mptcp.h +26 -0
index 56883a1ee2fe..32d22f2f1bbf 100644
@@ -41,6 +41,7 @@ struct mptcp_sock {
 	struct	inet_connection_sock sk;
 	u64	local_key;
 	u64	remote_key;
+	u32	token;
 	struct	socket *connection_list; /* @@ needs to be a list */
 	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
 };
@@ -56,6 +57,7 @@ struct subflow_sock {
 	struct	tcp_sock sk;
 	u64	local_key;
 	u64	remote_key;
+	u32	token;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
@@ -78,6 +80,7 @@ struct subflow_request_sock {
 		version : 4;
 	u64	local_key;
 	u64	remote_key;
+	u32	token;
 };
 
 static inline
@@ -104,6 +107,29 @@ void mptcp_get_options(const struct sk_buff *skb,
 
 extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
 
+void token_init(void);
+void token_new_request(struct request_sock *req, const struct sk_buff *skb);
+void token_destroy_request(u32 token);
+void token_new_connect(struct sock *sk);
+void token_new_accept(struct sock *sk);
+void token_update_accept(struct sock *sk, struct sock *conn);
+void token_destroy(u32 token);
+
+void crypto_init(void);
+u32 crypto_v4_get_nonce(__be32 saddr, __be32 daddr,
+			__be16 sport, __be16 dport);
+u64 crypto_v4_get_key(__be32 saddr, __be32 daddr,
+			__be16 sport, __be16 dport);
+u64 crypto_v6_get_key(const struct in6_addr *saddr,
+		      const struct in6_addr *daddr,
+		      __be16 sport, __be16 dport);
+u32 crypto_v6_get_nonce(const struct in6_addr *saddr,
+			const struct in6_addr *daddr,
+			__be16 sport, __be16 dport);
+void crypto_key_sha1(u64 key, u32 *token, u64 *idsn);
+void crypto_hmac_sha1(u64 key1, u64 key2, u32 *hash_out,
+		     int arg_num, ...);
+
 #else
 
 static inline void mptcp_parse_option(const unsigned char *ptr, int opsize,
net/mptcp/Makefile +1 -1
index 3f0e7163fe80..eaf59dc38b00 100644
@@ -1,3 +1,3 @@
 obj-$(CONFIG_MPTCP) += mptcp.o
 
-mptcp-y := protocol.o subflow.o options.o
+mptcp-y := protocol.o subflow.o options.o token.o crypto.o
net/mptcp/crypto.c new file +215 -0

215 lines not on this page · raw message

net/mptcp/protocol.c +18 -0
index 9f802f69a528..2fa9031a77ca 100644
@@ -111,6 +111,9 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	if (subflow->mp_capable) {
 		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
+		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
+		token_update_accept(new_sock->sk, mp->sk);
 		msk->connection_list = new_sock;
 	} else {
 		msk->subflow = new_sock;
@@ -119,6 +122,15 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	return mp->sk;
 }
 
+static void mptcp_destroy(struct sock *sk)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+
+	pr_debug("msk=%p, subflow=%p", sk, msk->subflow->sk);
+
+	token_destroy(msk->token);
+}
+
 static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
@@ -139,6 +151,8 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	if (mp_capable) {
 		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
+		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
 		msk->connection_list = msk->subflow;
 		msk->subflow = NULL;
 	}
@@ -248,6 +262,7 @@ static struct proto mptcp_prot = {
 	.close		= mptcp_close,
 	.accept		= mptcp_accept,
 	.shutdown	= tcp_shutdown,
+	.destroy	= mptcp_destroy,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
 	.hash		= inet_hash,
@@ -300,6 +315,9 @@ static int __init mptcp_init(void)
 
 	mptcp_prot.h.hashinfo = tcp_prot.h.hashinfo;
 
+	token_init();
+	crypto_init();
+
 	err = mptcp_subflow_init();
 	if (err)
 		goto subflow_failed;
net/mptcp/subflow.c +30 -1
index 89fcc3b746eb..7495e259ce5d 100644
@@ -53,6 +53,29 @@ static int subflow_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 	return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
 }
 
+static int subflow_rebuild_header(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	if (subflow->request_mptcp && !subflow->token) {
+		pr_debug("subflow=%p", sk);
+		token_new_connect(sk);
+	}
+
+	return inet_sk_rebuild_header(sk);
+}
+
+static void subflow_req_destructor(struct request_sock *req)
+{
+	struct subflow_request_sock *subflow_req = subflow_rsk(req);
+
+	pr_debug("subflow_req=%p", subflow_req);
+
+	if (subflow_req->mp_capable)
+		token_destroy_request(subflow_req->token);
+	tcp_request_sock_ops.destructor(req);
+}
+
 static void subflow_v4_init_req(struct request_sock *req,
 				const struct sock *sk_listener,
 				struct sk_buff *skb)
@@ -82,6 +105,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		    listener->checksum)
 			subflow_req->checksum = 1;
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
+		pr_debug("remote_key=%llu", subflow_req->remote_key);
+		token_new_request(req, skb);
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -147,6 +172,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->token = subflow_req->token;
+			pr_debug("token=%u", subflow->token);
+			token_new_accept(child);
 		} else {
 			subflow->mp_capable = 0;
 		}
@@ -158,7 +186,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 const struct inet_connection_sock_af_ops subflow_specific = {
 	.queue_xmit	   = ip_queue_xmit,
 	.send_check	   = tcp_v4_send_check,
-	.rebuild_header	   = inet_sk_rebuild_header,
+	.rebuild_header	   = subflow_rebuild_header,
 	.sk_rx_dst_set	   = subflow_finish_connect,
 	.conn_request	   = subflow_conn_request,
 	.syn_recv_sock	   = subflow_syn_recv_sock,
@@ -273,6 +301,7 @@ int mptcp_subflow_init(void)
 
 	subflow_request_sock_ops = tcp_request_sock_ops;
 	subflow_request_sock_ops.obj_size = sizeof(struct subflow_request_sock),
+	subflow_request_sock_ops.destructor = subflow_req_destructor;
 
 	subflow_request_sock_ipv4_ops = tcp_request_sock_ipv4_ops;
 	subflow_request_sock_ipv4_ops.init_req = subflow_v4_init_req;
net/mptcp/token.c new file +264 -0

264 lines not on this page · raw message

Signature · 1 line
2.19.1
author
#31 lore raw

RFC PATCH v3 10/16 mptcp: Add shutdown() socket operation

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 net/mptcp/protocol.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +20 -1
index 2fa9031a77ca..3d10ab287bd6 100644
@@ -255,6 +255,25 @@ int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
 	return inet_accept(sock, newsock, flags, kern);
 }
 
+int mptcp_stream_shutdown(struct socket *sock, int how)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int ret = 0;
+
+	pr_debug("sk=%p, how=%d", msk, how);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		ret = kernel_sock_shutdown(msk->subflow, how);
+	}
+	if (msk->connection_list) {
+		pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+		ret = kernel_sock_shutdown(msk->connection_list, how);
+	}
+
+	return ret;
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
@@ -284,7 +303,7 @@ const struct proto_ops mptcp_stream_ops = {
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
 	.listen		   = mptcp_stream_listen,
-	.shutdown	   = inet_shutdown,
+	.shutdown	   = mptcp_stream_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
 	.sendmsg	   = inet_sendmsg,
Signature · 1 line
2.19.1
#32 lore raw

Re: [MPTCP] [RFC PATCH v3 10/16] mptcp: Add shutdown() socket operation

Hi Peter,

Quoted 19 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 net/mptcp/protocol.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +4 -0
index 2fa9031a77ca..3d10ab287bd6 100644
@@ -255,6 +255,25 @@ int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
 	return inet_accept(sock, newsock, flags, kern);
 }
 
+int mptcp_stream_shutdown(struct socket *sock, int how)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int ret = 0;

net/mptcp/protocol.c int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags, · line 271

Quoted 10 lines
net/mptcp/protocol.c +10 -0
+
+	pr_debug("sk=%p, how=%d", msk, how);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		ret = kernel_sock_shutdown(msk->subflow, how);
+	}
+	if (msk->connection_list) {
+		pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+		ret = kernel_sock_shutdown(msk->connection_list, how);

Just to be sure, can we have "subflow != NULL" and "connection_list != NULL"?
Because I saw the possible override of "ret" here but it is not possible to have both a "subflow" and a "connection_list" (we could use a "else if" here I guess).

Cheers,
Matthieu

Quoted 18 lines
net/mptcp/protocol.c +6 -1
+	}
+
+	return ret;
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
@@ -284,7 +303,7 @@ const struct proto_ops mptcp_stream_ops = {
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
 	.listen		   = mptcp_stream_listen,
-	.shutdown	   = inet_shutdown,
+	.shutdown	   = mptcp_stream_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
 	.sendmsg	   = inet_sendmsg,
Signature · 5 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
#33 lore raw

Re: [MPTCP] [RFC PATCH v3 10/16] mptcp: Add shutdown() socket operation

Quoted 30 lines · Matthieu Baerts

Hi Peter,

Quoted 27 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 net/mptcp/protocol.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +12 -0
index 2fa9031a77ca..3d10ab287bd6 100644
@@ -255,6 +255,25 @@ int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
 	return inet_accept(sock, newsock, flags, kern);
 }
 
+int mptcp_stream_shutdown(struct socket *sock, int how)
+{
+	struct mptcp_sock *msk = mptcp_sk(sock->sk);
+	int ret = 0;
+
+	pr_debug("sk=%p, how=%d", msk, how);
+
+	if (msk->subflow) {
+		pr_debug("subflow=%p", msk->subflow->sk);
+		ret = kernel_sock_shutdown(msk->subflow, how);
+	}
+	if (msk->connection_list) {
Quoted 8 lines

net/mptcp/protocol.c int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags, · line 271

Quoted 2 lines
net/mptcp/protocol.c +2 -0
+		pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+		ret = kernel_sock_shutdown(msk->connection_list, how);

Just to be sure, can we have "subflow != NULL" and "connection_list != NULL"?
Because I saw the possible override of "ret" here but it is not possible to have both a "subflow" and a "connection_list" (we could use a "else if" here I guess).

Your point is good, it should not be possible for them both to be set.

But in a future commit the connection_list will become a real rcu linked list of subflows and a single lock will be used to protect the list and the subflow field. When the in-progress subflow is done connecting it will be moved to the connection list under lock protection.

Peter.

Quoted 23 lines

Cheers,
Matthieu

Quoted 18 lines
net/mptcp/protocol.c +6 -1
+	}
+
+	return ret;
+}
+
 static struct proto mptcp_prot = {
 	.name		= "MPTCP",
 	.owner		= THIS_MODULE,
@@ -284,7 +303,7 @@ const struct proto_ops mptcp_stream_ops = {
 	.poll		   = tcp_poll,
 	.ioctl		   = inet_ioctl,
 	.listen		   = mptcp_stream_listen,
-	.shutdown	   = inet_shutdown,
+	.shutdown	   = mptcp_stream_shutdown,
 	.setsockopt	   = sock_common_setsockopt,
 	.getsockopt	   = sock_common_getsockopt,
 	.sendmsg	   = inet_sendmsg,
#34 lore raw

Re: [MPTCP] [RFC PATCH v3 10/16] mptcp: Add shutdown() socket operation

Quoted 41 lines · Peter
Quoted 30 lines · Matthieu Baerts

Hi Peter,

Quoted 27 lines · Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 net/mptcp/protocol.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
net/mptcp/protocol.c +12 -0
index 2fa9031a77ca..3d10ab287bd6 100644
@@ -255,6 +255,25 @@ int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags,
    return inet_accept(sock, newsock, flags, kern);
 }
 
+int mptcp_stream_shutdown(struct socket *sock, int how)
+{
+   struct mptcp_sock *msk = mptcp_sk(sock->sk);
+   int ret = 0;
+
+   pr_debug("sk=%p, how=%d", msk, how);
+
+   if (msk->subflow) {
+           pr_debug("subflow=%p", msk->subflow->sk);
+           ret = kernel_sock_shutdown(msk->subflow, how);
+   }
+   if (msk->connection_list) {
Quoted 8 lines

net/mptcp/protocol.c int mptcp_stream_accept(struct socket *sock, struct socket *newsock, int flags, · line 271

Quoted 2 lines
net/mptcp/protocol.c +2 -0
+           pr_debug("conn_list->subflow=%p", msk->connection_list->sk);
+           ret = kernel_sock_shutdown(msk->connection_list, how);

Just to be sure, can we have "subflow != NULL" and "connection_list != NULL"?
Because I saw the possible override of "ret" here but it is not possible to have both a "subflow" and a "connection_list" (we could use a "else if" here I guess).

Quoted 8 lines

Your point is good, it should not be possible for them both to be set.

But in a future commit the connection_list will become a real rcu linked list of subflows and a single lock will be used to protect the list and the subflow field. When the in-progress subflow is done connecting it will be moved to the connection list under lock protection.

Yes, sorry I should have added another comment after have seen that. Thank you for the precision!

Matthieu

Quoted 25 lines

Peter.

Quoted 23 lines

Cheers,
Matthieu

Quoted 18 lines
net/mptcp/protocol.c +6 -1
+   }
+
+   return ret;
+}
+
 static struct proto mptcp_prot = {
    .name           = "MPTCP",
    .owner          = THIS_MODULE,
@@ -284,7 +303,7 @@ const struct proto_ops mptcp_stream_ops = {
    .poll              = tcp_poll,
    .ioctl             = inet_ioctl,
    .listen            = mptcp_stream_listen,
-   .shutdown          = inet_shutdown,
+   .shutdown          = mptcp_stream_shutdown,
    .setsockopt        = sock_common_setsockopt,
    .getsockopt        = sock_common_getsockopt,
    .sendmsg           = inet_sendmsg,
Signature · 7 lines
Matthieu Baerts | R&D Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

Disclaimer: https://www.tessares.net/mail-disclaimer/
author
#35 lore raw

RFC PATCH v3 11/16 mptcp: Add setsockopt()/getsockopt() socket operations

In reply to Mat Martineau

From: Peter Krystad <peter.krystad(a)intel.com>

Signed-off-by Peter Krystad <peter.krystad(a)intel.com>
---
 net/mptcp/protocol.c | 48 ++++++++++++++++++++++++++++++++++++++++++++
 net/mptcp/subflow.c  | 22 ++++++++++++++++++++
 2 files changed, 70 insertions(+)
net/mptcp/protocol.c +48 -0
index 3d10ab287bd6..cbc93dd36588 100644
@@ -131,6 +131,52 @@ static void mptcp_destroy(struct sock *sk)
 	token_destroy(msk->token);
 }
 
+static int mptcp_setsockopt(struct sock *sk, int level, int optname,
+			    char __user *uoptval, unsigned int optlen)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow;
+	char __kernel *optval;
+
+	pr_debug("msk=%p", msk);
+	if (msk->connection_list) {
+		subflow = msk->connection_list;
+		pr_debug("conn_list->subflow=%p", subflow->sk);
+	} else {
+		subflow = msk->subflow;
+		pr_debug("subflow=%p", subflow->sk);
+	}
+
+	/* will be treated as __user in subflow_setsockopt */
+	optval = (char __kernel __force *) uoptval;
+
+	return kernel_setsockopt(subflow, level, optname, optval, optlen);
+}
+
+static int mptcp_getsockopt(struct sock *sk, int level, int optname,
+			    char __user *uoptval, int __user *uoption)
+{
+	struct mptcp_sock *msk = mptcp_sk(sk);
+	struct socket *subflow;
+	char __kernel *optval;
+	int __kernel *option;
+
+	pr_debug("msk=%p", msk);
+	if (msk->connection_list) {
+		subflow = msk->connection_list;
+		pr_debug("conn_list->subflow=%p", subflow->sk);
+	} else {
+		subflow = msk->subflow;
+		pr_debug("subflow=%p", subflow->sk);
+	}
+
+	/* will be treated as __user in subflow_getsockopt */
+	optval = (char __kernel __force *) uoptval;
+	option = (int __kernel __force *) uoption;
+
+	return kernel_getsockopt(subflow, level, optname, optval, option);
+}
+
 static int mptcp_get_port(struct sock *sk, unsigned short snum)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
@@ -282,6 +328,8 @@ static struct proto mptcp_prot = {
 	.accept		= mptcp_accept,
 	.shutdown	= tcp_shutdown,
 	.destroy	= mptcp_destroy,
+	.setsockopt	= mptcp_setsockopt,
+	.getsockopt	= mptcp_getsockopt,
 	.sendmsg	= mptcp_sendmsg,
 	.recvmsg	= mptcp_recvmsg,
 	.hash		= inet_hash,
net/mptcp/subflow.c +22 -0
index 7495e259ce5d..e0b4aa8fe51d 100644
@@ -249,6 +249,26 @@ static void subflow_destroy(struct sock *sk)
 	tcp_v4_destroy_sock(sk);
 }
 
+static int subflow_setsockopt(struct sock *sk, int level, int optname,
+			      char __user *optval, unsigned int optlen)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_setsockopt(sk, level, optname, optval, optlen);
+}
+
+static int subflow_getsockopt(struct sock *sk, int level, int optname,
+			      char __user *optval, int __user *option)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+
+	pr_debug("subflow=%p", subflow);
+
+	return tcp_getsockopt(sk, level, optname, optval, option);
+}
+
 static struct proto subflow_prot = {
 	.name		= "SUBFLOW",
 	.owner		= THIS_MODULE,
@@ -260,6 +280,8 @@ static struct proto subflow_prot = {
 	.init		= subflow_init_sock,
 	.destroy	= subflow_destroy,
 	.shutdown	= tcp_shutdown,
+	.setsockopt	= subflow_setsockopt,
+	.getsockopt	= subflow_getsockopt,
 	.keepalive	= tcp_set_keepalive,
 	.recvmsg	= subflow_recvmsg,
 	.sendmsg	= subflow_sendmsg,
Signature · 1 line
2.19.1
author
#36 lore raw

RFC PATCH v3 12/16 skbuff: Add private data pointer

In reply to Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +3 -0
index 780abb11dffd..9e07afd084c3 100644
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
Signature · 1 line
2.19.1
#37 lore raw

Re: [MPTCP] [RFC PATCH v3 12/16] skbuff: Add private data pointer

Quoted 89 lines · Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +0 -0
index 780abb11dffd..9e07afd084c3 100644

net/ipv4/tcp_output.c static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, · line 1134

Quoted 6 lines
net/ipv4/tcp_output.c +3 -0
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);

Shouldn't this rather be called inside pskb_copy, skb_clone, and friends... ?

Christoph

Quoted 10 lines
net/ipv4/tcp_output.c +0 -0
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp
author
#38 lore raw

Re: [MPTCP] [RFC PATCH v3 12/16] skbuff: Add private data pointer

Quoted 90 lines · Christoph Paasch
Quoted 89 lines · Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +0 -0
index 780abb11dffd..9e07afd084c3 100644
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
Quoted 8 lines

net/ipv4/tcp_output.c static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, · line 1134

Quoted 6 lines
net/ipv4/tcp_output.c +3 -0
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);

Shouldn't this rather be called inside pskb_copy, skb_clone, and friends... ?

I considered that - and thinking about it more, it may be necessary.

For the initial implementation I wanted to regard this "private" data in a very restricted sense and have it belong only to one skb, with propagation to additional clones or copies only where the layer doing so knows what that private data is for.

In this case, the option writing code later in in __tcp_transmit_skb() needs to see the private data, so the private data is shared with the clone. However, the private data wouldn't be needed except in this function and for skbs in the tcp_rtx_queue.

What I'm remembering now is that keeping MPTCP metadata across pskb_copy() calls has been an issue before, so I'll take another look at that.

(and I need to take a look at possible skb collapsing in the rtx queue)

Mat

Quoted 12 lines
Quoted 11 lines
net/ipv4/tcp_output.c +0 -1
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

#39 lore raw

Re: [MPTCP] [RFC PATCH v3 12/16] skbuff: Add private data pointer

Quoted 108 lines · Mat Martineau
Quoted 90 lines · Christoph Paasch
Quoted 89 lines · Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +0 -0
index 780abb11dffd..9e07afd084c3 100644
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
Quoted 8 lines

net/ipv4/tcp_output.c static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, · line 1134

Quoted 6 lines
net/ipv4/tcp_output.c +3 -0
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);

Shouldn't this rather be called inside pskb_copy, skb_clone, and friends... ?

I considered that - and thinking about it more, it may be necessary.

For the initial implementation I wanted to regard this "private" data in a very restricted sense and have it belong only to one skb, with propagation to additional clones or copies only where the layer doing so knows what that private data is for.

Quoted 4 lines

In this case, the option writing code later in in __tcp_transmit_skb() needs to see the private data, so the private data is shared with the clone. However, the private data wouldn't be needed except in this function and for skbs in the tcp_rtx_queue.

Hmmm... Makes actually sense. As TCP is the only one who cares about the private data, he should be the one doing the work. But, in that case we don't even need the callback probably.

Thinking along those lines then: When transmitting the skb, we should clear priv_used, no ?

Quoted 2 lines

What I'm remembering now is that keeping MPTCP metadata across pskb_copy() calls has been an issue before, so I'll take another look at that.

The problem was that we were not sending the DSS-mapping on all segments. However, MPTCP does not mandate that. And Linux & iOS support receiving a DSS-mapping that spreads multiple segments but where the DSS-mapping TCP-option is not present in all those segments.

So, the "old" issues about pskb_copy() with MPTCP are not a real issue.

But, in general, the question is whether we want to make the priv_copy-thing more of an opt-in for each layer where each layer handles the pointers or whether it's handled by the "skb-subsystem".

In the former case we probably don't need the callbacks. In the latter, we do.

Christoph

Quoted 23 lines

(and I need to take a look at possible skb collapsing in the rtx queue)

Mat

Quoted 12 lines
Quoted 11 lines
net/ipv4/tcp_output.c +0 -1
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

author
#40 lore raw

Re: [MPTCP] [RFC PATCH v3 12/16] skbuff: Add private data pointer

Quoted 109 lines · Christoph Paasch
Quoted 108 lines · Mat Martineau
Quoted 90 lines · Christoph Paasch
Quoted 89 lines · Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +0 -0
index 780abb11dffd..9e07afd084c3 100644
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
Quoted 8 lines

net/ipv4/tcp_output.c static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, · line 1134

Quoted 6 lines
net/ipv4/tcp_output.c +3 -0
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);

Shouldn't this rather be called inside pskb_copy, skb_clone, and friends... ?

I considered that - and thinking about it more, it may be necessary.

For the initial implementation I wanted to regard this "private" data in a very restricted sense and have it belong only to one skb, with propagation to additional clones or copies only where the layer doing so knows what that private data is for.

Quoted 8 lines
Quoted 4 lines

In this case, the option writing code later in in __tcp_transmit_skb() needs to see the private data, so the private data is shared with the clone. However, the private data wouldn't be needed except in this function and for skbs in the tcp_rtx_queue.

Hmmm... Makes actually sense. As TCP is the only one who cares about the private data, he should be the one doing the work. But, in that case we don't even need the callback probably.

Looking at it now, there's really no difference between keeping track of what priv_data points to in order to use it (for MPTCP option population) or copy it. Makes sense to get rid of priv_copy.

Quoted 3 lines

Thinking along those lines then: When transmitting the skb, we should clear priv_used, no ?

Good idea, that would leave the priv pointers available for lower layers.

Quoted 10 lines
Quoted 3 lines

What I'm remembering now is that keeping MPTCP metadata across pskb_copy() calls has been an issue before, so I'll take another look at that.

The problem was that we were not sending the DSS-mapping on all segments. However, MPTCP does not mandate that. And Linux & iOS support receiving a DSS-mapping that spreads multiple segments but where the DSS-mapping TCP-option is not present in all those segments.

So, the "old" issues about pskb_copy() with MPTCP are not a real issue.

Ok, good to know.

Quoted 7 lines

But, in general, the question is whether we want to make the priv_copy-thing more of an opt-in for each layer where each layer handles the pointers or whether it's handled by the "skb-subsystem".

In the former case we probably don't need the callbacks. In the latter, we do.

I'm leaning toward the former, to get rid of the copy pointer. The destructor is needed because it's harder to know when or where an skb will get destroyed, but dereferencing the priv_data pointer for copying or reading the memory requires keeping track of its contents much like we're accustomed to doing for skb->cb[].

Mat

Quoted 9 lines
Quoted 9 lines

(and I need to take a look at possible skb collapsing in the rtx queue)

Quoted 6 lines
Quoted 6 lines
net/ipv4/tcp_output.c +0 -1
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
--

2.19.1

--
Mat Martineau
Intel OTC

#41 lore raw

Re: [MPTCP] [RFC PATCH v3 12/16] skbuff: Add private data pointer

Quoted 148 lines · Mat Martineau
Quoted 109 lines · Christoph Paasch
Quoted 108 lines · Mat Martineau
Quoted 90 lines · Christoph Paasch
Quoted 89 lines · Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +0 -0
index 780abb11dffd..9e07afd084c3 100644
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
Quoted 8 lines

net/ipv4/tcp_output.c static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, · line 1134

Quoted 6 lines
net/ipv4/tcp_output.c +3 -0
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);

Shouldn't this rather be called inside pskb_copy, skb_clone, and friends... ?

I considered that - and thinking about it more, it may be necessary.

For the initial implementation I wanted to regard this "private" data in a very restricted sense and have it belong only to one skb, with propagation to additional clones or copies only where the layer doing so knows what that private data is for.

Quoted 8 lines
Quoted 4 lines

In this case, the option writing code later in in __tcp_transmit_skb() needs to see the private data, so the private data is shared with the clone. However, the private data wouldn't be needed except in this function and for skbs in the tcp_rtx_queue.

Hmmm... Makes actually sense. As TCP is the only one who cares about the private data, he should be the one doing the work. But, in that case we don't even need the callback probably.

Looking at it now, there's really no difference between keeping track of what priv_data points to in order to use it (for MPTCP option population) or copy it. Makes sense to get rid of priv_copy.

Quoted 3 lines

Thinking along those lines then: When transmitting the skb, we should clear priv_used, no ?

Good idea, that would leave the priv pointers available for lower layers.

Quoted 10 lines
Quoted 3 lines

What I'm remembering now is that keeping MPTCP metadata across pskb_copy() calls has been an issue before, so I'll take another look at that.

The problem was that we were not sending the DSS-mapping on all segments. However, MPTCP does not mandate that. And Linux & iOS support receiving a DSS-mapping that spreads multiple segments but where the DSS-mapping TCP-option is not present in all those segments.

So, the "old" issues about pskb_copy() with MPTCP are not a real issue.

Ok, good to know.

Quoted 5 lines

But, in general, the question is whether we want to make the priv_copy-thing more of an opt-in for each layer where each layer handles the pointers or whether it's handled by the "skb-subsystem".

Quoted 8 lines
Quoted 2 lines

In the former case we probably don't need the callbacks. In the latter, we do.

I'm leaning toward the former, to get rid of the copy pointer. The destructor is needed because it's harder to know when or where an skb will get destroyed, but dereferencing the priv_data pointer for copying or reading the memory requires keeping track of its contents much like we're accustomed to doing for skb->cb[].

Yes, makes sense to keep the destructor. Already just for safety as an skb could be freed anywhere.

Just wondering: Do we need to "destruct" when we call ip_xmit in tcp_transmit_skb() ? I think so, right? Because we want to set priv_used to 0 above.

Christoph

Quoted 17 lines

Mat

Quoted 9 lines
Quoted 9 lines

(and I need to take a look at possible skb collapsing in the rtx queue)

Quoted 6 lines
Quoted 6 lines
net/ipv4/tcp_output.c +0 -1
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
--

2.19.1

--
Mat Martineau
Intel OTC

author
#42 lore raw

Re: [MPTCP] [RFC PATCH v3 12/16] skbuff: Add private data pointer

Quoted 158 lines · Christoph Paasch
Quoted 148 lines · Mat Martineau
Quoted 109 lines · Christoph Paasch
Quoted 108 lines · Mat Martineau
Quoted 90 lines · Christoph Paasch
Quoted 89 lines · Mat Martineau

Allow struct sk_buff to be optionally extended when the control buffer space is insufficient. New pointers for the extended data and associated destructor are placed at the end of the data structure and left uninitialized so cache behavior of struct sk_buff is unchanged when the new pointers are not accessed.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/skbuff.h | 16 +++++++++++++++-
 net/core/skbuff.c      |  5 +++++
 net/ipv4/tcp_output.c  |  3 +++
 3 files changed, 23 insertions(+), 1 deletion(-)
include/linux/skbuff.h +15 -1
index 17a13e4785fc..5b1f302b1b0f 100644
@@ -631,6 +631,7 @@ typedef unsigned char *sk_buff_data_t;
  *	@queue_mapping: Queue mapping for multiqueue devices
  *	@xmit_more: More SKBs are pending for this queue
  *	@pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
+ *	@priv_used: priv, priv_copy, and priv_destructor contain valid data
  *	@ndisc_nodetype: router type (from link layer)
  *	@ooo_okay: allow the mapping of a socket to a queue to be changed
  *	@l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -660,6 +661,9 @@ typedef unsigned char *sk_buff_data_t;
  *	@data: Data head pointer
  *	@truesize: Buffer size
  *	@users: User count - see {datagram,tcp}.c
+ *	@priv: Optional pointer to skb private data
+ *	@priv_copy: Private data copy function
+ *	@priv_destructor: Private data destruct function
  */
 
 struct sk_buff {
@@ -741,7 +745,8 @@ struct sk_buff {
 				peeked:1,
 				head_frag:1,
 				xmit_more:1,
-				pfmemalloc:1;
+				pfmemalloc:1,
+				priv_used:1;
 
 	/* fields enclosed in headers_start/headers_end are copied
 	 * using a single memcpy() in __copy_skb_header()
@@ -856,6 +861,15 @@ struct sk_buff {
 				*data;
 	unsigned int		truesize;
 	refcount_t		users;
+
+	/* Uninitialized private pointers. At the very end so they do not
+	 * affect cache behavior of the rest of struct sk_buff when left
+	 * unused. All three must be initialized when priv_used is set.
+	 */
+	void			*priv;
+	void			(*priv_copy)(const struct sk_buff *from,
+					     struct sk_buff *to);
+	void			(*priv_destructor)(struct sk_buff *skb);
 };
 
 #ifdef __KERNEL__
net/core/skbuff.c +5 -0
index c996c09d095f..b508e9e5e855 100644
@@ -611,6 +611,10 @@ void skb_release_head_state(struct sk_buff *skb)
 		WARN_ON(in_irq());
 		skb->destructor(skb);
 	}
+	if (skb->priv_used && skb->priv_destructor) {
+		WARN_ON(in_irq());
+		skb->priv_destructor(skb);
+	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
 #endif
@@ -859,6 +863,7 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
 	n->nohdr = 0;
 	n->peeked = 0;
 	C(pfmemalloc);
+	n->priv_used = 0;
 	n->destructor = NULL;
 	C(tail);
 	C(end);
net/ipv4/tcp_output.c +0 -0
index 780abb11dffd..9e07afd084c3 100644
@@ -1129,6 +1129,9 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
Quoted 8 lines

net/ipv4/tcp_output.c static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, · line 1134

Quoted 6 lines
net/ipv4/tcp_output.c +3 -0
 				skb = pskb_copy(oskb, gfp_mask);
 			else
 				skb = skb_clone(oskb, gfp_mask);
+
+			if (oskb->priv_used && oskb->priv_copy)
+				oskb->priv_copy(oskb, skb);

Shouldn't this rather be called inside pskb_copy, skb_clone, and friends... ?

I considered that - and thinking about it more, it may be necessary.

For the initial implementation I wanted to regard this "private" data in a very restricted sense and have it belong only to one skb, with propagation to additional clones or copies only where the layer doing so knows what that private data is for.

Quoted 8 lines
Quoted 4 lines

In this case, the option writing code later in in __tcp_transmit_skb() needs to see the private data, so the private data is shared with the clone. However, the private data wouldn't be needed except in this function and for skbs in the tcp_rtx_queue.

Hmmm... Makes actually sense. As TCP is the only one who cares about the private data, he should be the one doing the work. But, in that case we don't even need the callback probably.

Looking at it now, there's really no difference between keeping track of what priv_data points to in order to use it (for MPTCP option population) or copy it. Makes sense to get rid of priv_copy.

Quoted 3 lines

Thinking along those lines then: When transmitting the skb, we should clear priv_used, no ?

Good idea, that would leave the priv pointers available for lower layers.

Quoted 10 lines
Quoted 3 lines

What I'm remembering now is that keeping MPTCP metadata across pskb_copy() calls has been an issue before, so I'll take another look at that.

The problem was that we were not sending the DSS-mapping on all segments. However, MPTCP does not mandate that. And Linux & iOS support receiving a DSS-mapping that spreads multiple segments but where the DSS-mapping TCP-option is not present in all those segments.

So, the "old" issues about pskb_copy() with MPTCP are not a real issue.

Ok, good to know.

Quoted 5 lines

But, in general, the question is whether we want to make the priv_copy-thing more of an opt-in for each layer where each layer handles the pointers or whether it's handled by the "skb-subsystem".

Quoted 8 lines
Quoted 2 lines

In the former case we probably don't need the callbacks. In the latter, we do.

I'm leaning toward the former, to get rid of the copy pointer. The destructor is needed because it's harder to know when or where an skb will get destroyed, but dereferencing the priv_data pointer for copying or reading the memory requires keeping track of its contents much like we're accustomed to doing for skb->cb[].

Quoted 5 lines

Yes, makes sense to keep the destructor. Already just for safety as an skb could be freed anywhere.

Just wondering: Do we need to "destruct" when we call ip_xmit in tcp_transmit_skb() ? I think so, right? Because we want to set priv_used to 0 above.

Yes, that's a necessary part of clearing it.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

(and I need to take a look at possible skb collapsing in the rtx queue)

Quoted 6 lines
Quoted 6 lines
net/ipv4/tcp_output.c +0 -1
 		} tcp_skb_tsorted_restore(oskb);
 
 		if (unlikely(!skb))
--

2.19.1

--
Mat Martineau
Intel OTC

author
#43 lore raw

RFC PATCH v3 13/16 tcp: Export low-level TCP functions

In reply to Mat Martineau
Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/tcp.h |  4 ++++
 net/ipv4/tcp.c    | 12 ++++++------
 2 files changed, 10 insertions(+), 6 deletions(-)
include/net/tcp.h +4 -0
index 48c59c0d8760..249e3329ee85 100644
@@ -332,6 +332,10 @@ int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset,
 			size_t size, int flags);
 ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset,
 		 size_t size, int flags);
+int tcp_send_mss(struct sock *sk, int *size_goal, int flags);
+void tcp_skb_entail(struct sock *sk, struct sk_buff *skb);
+void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle,
+	      int size_goal);
 void tcp_release_cb(struct sock *sk);
 void tcp_wfree(struct sk_buff *skb);
 void tcp_write_timer_handler(struct sock *sk);
net/ipv4/tcp.c +6 -6
index b8af2fec5ad5..cbf158ae48db 100644
@@ -657,7 +657,7 @@ static inline bool forced_push(const struct tcp_sock *tp)
 	return after(tp->write_seq, tp->pushed_seq + (tp->max_window >> 1));
 }
 
-static void skb_entail(struct sock *sk, struct sk_buff *skb)
+void tcp_skb_entail(struct sock *sk, struct sk_buff *skb)
 {
 	struct tcp_sock *tp = tcp_sk(sk);
 	struct tcp_skb_cb *tcb = TCP_SKB_CB(skb);
@@ -701,8 +701,8 @@ static bool tcp_should_autocork(struct sock *sk, struct sk_buff *skb,
 	       refcount_read(&sk->sk_wmem_alloc) > skb->truesize;
 }
 
-static void tcp_push(struct sock *sk, int flags, int mss_now,
-		     int nonagle, int size_goal)
+void tcp_push(struct sock *sk, int flags, int mss_now,
+	      int nonagle, int size_goal)
 {
 	struct tcp_sock *tp = tcp_sk(sk);
 	struct sk_buff *skb;
@@ -924,7 +924,7 @@ static unsigned int tcp_xmit_size_goal(struct sock *sk, u32 mss_now,
 	return max(size_goal, mss_now);
 }
 
-static int tcp_send_mss(struct sock *sk, int *size_goal, int flags)
+int tcp_send_mss(struct sock *sk, int *size_goal, int flags)
 {
 	int mss_now;
 
@@ -979,7 +979,7 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset,
 			if (!skb)
 				goto wait_for_memory;
 
-			skb_entail(sk, skb);
+			tcp_skb_entail(sk, skb);
 			copy = size_goal;
 		}
 
@@ -1291,7 +1291,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 			process_backlog = true;
 			skb->ip_summed = CHECKSUM_PARTIAL;
 
-			skb_entail(sk, skb);
+			tcp_skb_entail(sk, skb);
 			copy = size_goal;
 
 			/* All packets are restored as if they have
Signature · 1 line
2.19.1
author
#44 lore raw

RFC PATCH v3 14/16 mptcp: Write MPTCP DSS headers to outgoing data packets

In reply to Mat Martineau

Per-packet metadata required to write the MPTCP DSS option is written to the skb shared control buffer. One write to the socket may contain more than one packet of data, in which case the DSS option in the first packet will have a mapping covering all of the data in that write. Packets after the first do not have a DSS option. This is complicated to handle under memory pressure, since the first packet (with the DSS mapping) is pushed to the TCP core before the remaining skbs are allocated.

The current implementation is limited. If a non-blocking write only succeeds in sending part of the data, the mapping that was in the first packet will cover data that was not sent and is not buffered within the kernel. There's no guarantee that the socket user will write the same (or any) data when the socket becomes writable again.

The MPTCP DSS checksum is not yet implemented.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/net/mptcp.h   |  32 ++++++++-
 net/ipv4/tcp_output.c | 148 +++++++++++++++++++++++++++++++++-----
 net/mptcp/protocol.c  | 161 ++++++++++++++++++++++++++++++++++++++----
 net/mptcp/token.c     |  13 ++--
 4 files changed, 315 insertions(+), 39 deletions(-)
include/net/mptcp.h +30 -2
index 32d22f2f1bbf..73f2d25cc42d 100644
@@ -41,6 +41,8 @@ struct mptcp_sock {
 	struct	inet_connection_sock sk;
 	u64	local_key;
 	u64	remote_key;
+	u64	write_seq;
+	u64	ack_seq;
 	u32	token;
 	struct	socket *connection_list; /* @@ needs to be a list */
 	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
@@ -51,13 +53,38 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 	return (struct mptcp_sock *)sk;
 }
 
+/* MPTCP sk_buff private control buffer */
+struct mptcp_skb_cb {
+	refcount_t	refcnt;
+	u64		data_ack;
+	u64		data_seq;
+	u32		subflow_seq;
+	u16		dll;
+	__sum16		checksum;
+	u8		use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			use_ack:1,
+			ack64:1,
+			__unused:2;
+};
+
+static inline struct mptcp_skb_cb *mptcp_skb_priv_cb(struct sk_buff *skb)
+{
+	BUG_ON(!skb->priv_used);
+	return (struct mptcp_skb_cb *)skb->priv;
+}
+
 /* MPTCP subflow sock structure */
 struct subflow_sock {
 	/* tcp_sock must be the first member */
 	struct	tcp_sock sk;
 	u64	local_key;
-	u64	remote_key;
 	u32	token;
+	u64	idsn;
+	u64	remote_key;
+	u32	rel_write_seq;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
@@ -79,8 +106,9 @@ struct subflow_request_sock {
 		backup : 1,
 		version : 4;
 	u64	local_key;
-	u64	remote_key;
 	u32	token;
+	u64	idsn;
+	u64	remote_key;
 };
 
 static inline
net/ipv4/tcp_output.c +129 -19

217 lines not on this page · raw message

net/mptcp/protocol.c +149 -12

190 lines not on this page · raw message

net/mptcp/token.c +7 -6
index 0b4532ab72ef..1a95c71772b4 100644
@@ -90,10 +90,11 @@ static void new_req_token(struct request_sock *req,
 					      ireq->ir_rmt_port);
 #endif
 	}
-	pr_debug("local_key=%llu:%llx", local_key, local_key);
 	subflow_req->local_key = local_key;
-	crypto_key_sha1(subflow_req->local_key, &subflow_req->token, NULL);
-	pr_debug("token=%u", subflow_req->token);
+	crypto_key_sha1(subflow_req->local_key, &subflow_req->token,
+			&subflow_req->idsn);
+	pr_debug("local_key=%llu, token=%u, idsn=%llu", subflow_req->local_key,
+		 subflow_req->token, subflow_req->idsn);
 }
 
 static void new_token(const struct sock *sk)
@@ -114,9 +115,9 @@ static void new_token(const struct sock *sk)
 						       isk->inet_dport);
 #endif
 	}
-	pr_debug("local_key=%llu:%llx", subflow->local_key, subflow->local_key);
-	crypto_key_sha1(subflow->local_key, &subflow->token, NULL);
-	pr_debug("token=%u", subflow->token);
+	crypto_key_sha1(subflow->local_key, &subflow->token, &subflow->idsn);
+	pr_debug("local_key=%llu, token=%u, idsn=%llu", subflow->local_key,
+		 subflow->token, subflow->idsn);
 }
 
 static int insert_req_token(u32 token)
Signature · 1 line
2.19.1
author
#45 lore raw

RFC PATCH v3 15/16 mptcp: Implement MPTCP receive path

In reply to Mat Martineau

Parses incoming DSS options and populates outgoing MPTCP ACK fields. Clone incoming skbs containing MPTCP headers and propagate them to the MMPTCP socket using the error queue. This avoids the need for TCP coalesce/collapse bypass for MPTCP, although cloning incoming skbs does prevent some coalescing.

Keeping DSS map handling separate from data reading makes it easier to add multiple-subflow receiving, since the receive code can determine which subflow contains relevant data or data that needs to be purged without touching the subflow data queues.

MPTCP ACK values are now populated from an atomic value stored in the connection socket rather than carried in the extended control block, which makes use of the most up-to-date sequence number and allows the MPTCP ACK to be populated on TCP ACK packets that have no payload.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/tcp.h           |  12 ++
 include/net/mptcp.h           |  41 +++--
 include/uapi/linux/errqueue.h |   1 +
 net/ipv4/tcp_input.c          |   3 +
 net/ipv4/tcp_output.c         | 123 +++++++-------
 net/mptcp/options.c           |  90 +++++++++-
 net/mptcp/protocol.c          | 304 +++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c           |  29 +++-
 8 files changed, 508 insertions(+), 95 deletions(-)
include/linux/tcp.h +12 -0
index b109798482d3..845da5ee9c44 100644
@@ -112,6 +112,18 @@ struct tcp_options_received {
 		u8      flags;
 		u64     sndr_key;
 		u64     rcvr_key;
+		u64	ack;
+		u64	seq;
+		u32	subflow_seq;
+		u16	dll;
+		u16	checksum;
+		u8	use_ack:1,
+			ack64:1,
+			use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			__unused:2;
 	} mptcp;
 
 };
include/net/mptcp.h +29 -12

93 lines not on this page · raw message

include/uapi/linux/errqueue.h +1 -0
index c0151200f7d1..1a1ec7362a55 100644
@@ -21,6 +21,7 @@ struct sock_extended_err {
 #define SO_EE_ORIGIN_TXSTATUS	4
 #define SO_EE_ORIGIN_ZEROCOPY	5
 #define SO_EE_ORIGIN_TXTIME	6
+#define SO_EE_ORIGIN_MPTCP	7
 #define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS
 
 #define SO_EE_OFFENDER(ee)	((struct sockaddr*)((ee)+1))
net/ipv4/tcp_input.c +3 -0
index 9a326729637f..0904d9baa685 100644
@@ -5642,6 +5642,9 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	tcp_urg(sk, skb, th);
 
 	/* step 7: process the segment text */
+	if (tcp_sk(sk)->is_mptcp)
+		mptcp_queue_headers(sk, skb);
+
 	tcp_data_queue(sk, skb);
 
 	tcp_data_snd_check(sk);
net/ipv4/tcp_output.c +65 -58

191 lines not on this page · raw message

net/mptcp/options.c +88 -2

112 lines not on this page · raw message

net/mptcp/protocol.c +283 -21

366 lines not on this page · raw message

net/mptcp/subflow.c +27 -2
index e0b4aa8fe51d..3c9dca4be26a 100644
@@ -107,6 +107,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
 		pr_debug("remote_key=%llu", subflow_req->remote_key);
 		token_new_request(req, skb);
+		pr_debug("syn seq=%u", TCP_SKB_CB(skb)->seq);
+		subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -120,10 +122,15 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 
 	pr_debug("subflow=%p", subflow);
 
-	if (subflow->conn) {
+	if (!subflow->conn_finished) {
 		pr_debug("remote_key=%llu", subflow->remote_key);
 		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
-		subflow->conn = NULL;
+		subflow->conn_finished = 1;
+
+		if (skb) {
+			pr_debug("synack seq=%u", TCP_SKB_CB(skb)->seq);
+			subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
+		}
 	}
 }
 
@@ -172,7 +179,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->ssn_offset = subflow_req->ssn_offset;
 			subflow->token = subflow_req->token;
+			subflow->idsn = subflow_req->idsn;
 			pr_debug("token=%u", subflow->token);
 			token_new_accept(child);
 		} else {
@@ -202,6 +211,20 @@ const struct inet_connection_sock_af_ops subflow_specific = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
+static void subflow_data_ready(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *parent = subflow->conn;
+
+	pr_debug("sk=%p", sk);
+	subflow->tcp_sk_data_ready(sk);
+
+	if (parent) {
+		pr_debug("parent=%p", parent);
+		parent->sk_data_ready(parent);
+	}
+}
+
 static int subflow_init_sock(struct sock *sk)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -215,6 +238,8 @@ static int subflow_init_sock(struct sock *sk)
 	if (!err) { // @@ AND mptcp is enabled
 		tsk->is_mptcp = 1;
 		icsk->icsk_af_ops = &subflow_specific;
+		subflow->tcp_sk_data_ready = sk->sk_data_ready;
+		sk->sk_data_ready = subflow_data_ready;
 	}
 
 	return err;
Signature · 1 line
2.19.1
#46 lore raw

Re: [MPTCP] [RFC PATCH v3 15/16] mptcp: Implement MPTCP receive path

Quoted 5 lines · Mat Martineau

Parses incoming DSS options and populates outgoing MPTCP ACK fields. Clone incoming skbs containing MPTCP headers and propagate them to the MMPTCP socket using the error queue. This avoids the need for TCP coalesce/collapse bypass for MPTCP, although cloning incoming skbs does prevent some coalescing.

I think it's acceptable to bypass coalesce/collapse for MPTCP instead of cloning every incoming packet over to the error-queue.

The way this is now, makes the error-queue quite loaded, and I don't think it handles packet-reordering correctly.

Christoph

Quoted 950 lines

Keeping DSS map handling separate from data reading makes it easier to add multiple-subflow receiving, since the receive code can determine which subflow contains relevant data or data that needs to be purged without touching the subflow data queues.

MPTCP ACK values are now populated from an atomic value stored in the connection socket rather than carried in the extended control block, which makes use of the most up-to-date sequence number and allows the MPTCP ACK to be populated on TCP ACK packets that have no payload.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/tcp.h           |  12 ++
 include/net/mptcp.h           |  41 +++--
 include/uapi/linux/errqueue.h |   1 +
 net/ipv4/tcp_input.c          |   3 +
 net/ipv4/tcp_output.c         | 123 +++++++-------
 net/mptcp/options.c           |  90 +++++++++-
 net/mptcp/protocol.c          | 304 +++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c           |  29 +++-
 8 files changed, 508 insertions(+), 95 deletions(-)
include/linux/tcp.h +12 -0
index b109798482d3..845da5ee9c44 100644
@@ -112,6 +112,18 @@ struct tcp_options_received {
 		u8      flags;
 		u64     sndr_key;
 		u64     rcvr_key;
+		u64	ack;
+		u64	seq;
+		u32	subflow_seq;
+		u16	dll;
+		u16	checksum;
+		u8	use_ack:1,
+			ack64:1,
+			use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			__unused:2;
 	} mptcp;
 
 };
include/net/mptcp.h +29 -12
index 73f2d25cc42d..6c9516360de4 100644
@@ -38,14 +38,14 @@
 /* MPTCP connection sock */
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
-	struct	inet_connection_sock sk;
-	u64	local_key;
-	u64	remote_key;
-	u64	write_seq;
-	u64	ack_seq;
-	u32	token;
-	struct	socket *connection_list; /* @@ needs to be a list */
-	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
+	struct inet_connection_sock sk;
+	u64		local_key;
+	u64		remote_key;
+	u64		write_seq;
+	atomic64_t	ack_seq;
+	u32		token;
+	struct socket	*connection_list; /* @@ needs to be a list */
+	struct socket	*subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -56,7 +56,6 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 /* MPTCP sk_buff private control buffer */
 struct mptcp_skb_cb {
 	refcount_t	refcnt;
-	u64		data_ack;
 	u64		data_seq;
 	u32		subflow_seq;
 	u16		dll;
@@ -65,9 +64,7 @@ struct mptcp_skb_cb {
 			dsn64:1,
 			use_checksum:1,
 			data_fin:1,
-			use_ack:1,
-			ack64:1,
-			__unused:2;
+			__unused:4;
 };
 
 static inline struct mptcp_skb_cb *mptcp_skb_priv_cb(struct sk_buff *skb)
@@ -81,16 +78,23 @@ struct subflow_sock {
 	/* tcp_sock must be the first member */
 	struct	tcp_sock sk;
 	u64	local_key;
+	u64	map_seq;
+	u32	map_subflow_seq;
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
 	u32	rel_write_seq;
+	u32	ssn_offset;
+	u16	map_dll;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
 	bool	mp_capable;	// remote is MPTCP capable
 	bool	fourth_ack;	// send initial DSS
+	bool	conn_finished;
+	bool	map_valid;
 	struct	sock *conn;	// parent mptcp_sock
+	void	(*tcp_sk_data_ready)(struct sock *sk);
 };
 
 static inline struct subflow_sock *subflow_sk(const struct sock *sk)
@@ -98,6 +102,16 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+static inline struct subflow_sock *subflow_tp(const struct tcp_sock *tp)
+{
+	return (struct subflow_sock *)tp;
+}
+
+static inline struct sock *sock_sk(const struct subflow_sock *sk)
+{
+	return (struct sock *)sk;
+}
+
 struct subflow_request_sock {
 	struct	tcp_request_sock sk;
 	u8	mp_capable : 1,
@@ -109,6 +123,7 @@ struct subflow_request_sock {
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
+	u32	ssn_offset;
 };
 
 static inline
@@ -133,6 +148,8 @@ void mptcp_subflow_exit(void);
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
 
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb);
+
 extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
 
 void token_init(void);
include/uapi/linux/errqueue.h +1 -0
index c0151200f7d1..1a1ec7362a55 100644
@@ -21,6 +21,7 @@ struct sock_extended_err {
 #define SO_EE_ORIGIN_TXSTATUS	4
 #define SO_EE_ORIGIN_ZEROCOPY	5
 #define SO_EE_ORIGIN_TXTIME	6
+#define SO_EE_ORIGIN_MPTCP	7
 #define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS
 
 #define SO_EE_OFFENDER(ee)	((struct sockaddr*)((ee)+1))
net/ipv4/tcp_input.c +3 -0
index 9a326729637f..0904d9baa685 100644
@@ -5642,6 +5642,9 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	tcp_urg(sk, skb, th);
 
 	/* step 7: process the segment text */
+	if (tcp_sk(sk)->is_mptcp)
+		mptcp_queue_headers(sk, skb);
+
 	tcp_data_queue(sk, skb);
 
 	tcp_data_snd_check(sk);
net/ipv4/tcp_output.c +65 -58
index 471f196711fc..1c9517db82c2 100644
@@ -422,8 +422,8 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_MPTCP_MPC_SYN	(1 << 0)
 #define OPTION_MPTCP_MPC_SYNACK	(1 << 1)
 #define OPTION_MPTCP_MPC_ACK	(1 << 2)
-#define OPTION_MPTCP_MPC_DSS_MAP	(1 << 6)
-#define OPTION_MPTCP_MPC_DSS_ACK	(1 << 7)
+#define OPTION_MPTCP_DSS_MAP	(1 << 6)
+#define OPTION_MPTCP_DSS_ACK	(1 << 7)
 
 struct tcp_out_options {
 	u16 options;		/* bit field of OPTION_* */
@@ -472,16 +472,12 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 	}
 
-	if ((OPTION_MPTCP_MPC_DSS_MAP |
-	     OPTION_MPTCP_MPC_DSS_ACK) & opts->suboptions) {
-		struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
-		bool write_ack = (OPTION_MPTCP_MPC_DSS_ACK & opts->suboptions) && skb->priv_used && mcb->use_ack;
-		bool write_map = (OPTION_MPTCP_MPC_DSS_MAP & opts->suboptions) && skb->priv_used && mcb->use_map;
+	if ((OPTION_MPTCP_DSS_MAP | OPTION_MPTCP_DSS_ACK) & opts->suboptions) {
+		bool write_ack = !!(OPTION_MPTCP_DSS_ACK & opts->suboptions);
+		bool write_map = (OPTION_MPTCP_DSS_MAP & opts->suboptions) &&
+			skb->priv_used && mptcp_skb_priv_cb(skb)->use_map;
 		u8 flags = 0;
 		u8 len = 4;
-		u8 *p = (u8 *)ptr;
-
-		BUG_ON(!write_ack && !write_map);
 
 		if (write_ack) {
 			len += 8;
@@ -489,46 +485,60 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 
 		if (write_map) {
+			pr_debug("Updating DSS length and flags for map");
 			len += 14;
 
-			if (mcb->use_checksum)
+			if (mptcp_skb_priv_cb(skb)->use_checksum)
 				len += 2;
 
 			/* Use only 64-bit mapping flags for now, add
 			 * support for optional 32-bit mappings later.
 			 */
 			flags |= 0x0c;
-			if (mcb->data_fin)
+			if (mptcp_skb_priv_cb(skb)->data_fin)
 				flags |= 0x10;
 		}
 
-		*p++ = 0x1e; // TCP option: Multipath TCP
-		*p++ = len;  // length
-		*p++ = 0x20; // subtype=DSS
-		*p++ = flags;
+		*ptr++ = htonl((0x1e << 24) |  // TCP option: Multipath TCP
+			       (len  << 16) |  // length
+			       (0x20 <<  8) |  // subtype=DSS
+			       (flags));
 
 		if (write_ack) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_ack);
-			p += 8;
+			struct mptcp_sock *msk = mptcp_sk(subflow_tp(tp)->conn);
+			u64 ack_seq;
+			__be64 ack;
+
+			if (msk) {
+				ack_seq = atomic64_read(&msk->ack_seq);
+			} else {
+				crypto_key_sha1(subflow_tp(tp)->remote_key,
+						NULL, &ack_seq);
+				ack_seq++;
+			}
+
+			pr_debug("ack=%llu", ack_seq);
+			ack = cpu_to_be64(ack_seq);
+			memcpy((u8 *) ptr, (u8 *) &ack, 8);
+			ptr += 2;
 		}
 
 		if (write_map) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_seq);
-			p += 8;
-
-			*(__be32 *)p = htonl(mcb->subflow_seq);
-			p += 4;
+			struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
+			u16 checksum;
+			__be64 dss;
 
-			*(__be16 *)p = htons(mcb->dll);
-			p += 2;
+			pr_debug("Writing map values");
+			dss = cpu_to_be64(mcb->data_seq);
+			memcpy((u8 *) ptr, &dss, 8);
+			ptr += 2;
+			*ptr++ = htonl(mcb->subflow_seq);
 
-			if (mcb->use_checksum) {
-				*(__be16 *)p = htons(mcb->checksum);
-				p += 2;
-			} else {
-				*p++ = TCPOPT_NOP;
-				*p++ = TCPOPT_NOP;
-			}
+			if (mcb->use_checksum)
+				checksum = (__force u16) mcb->checksum;
+			else
+				checksum = TCPOPT_NOP << 8 | TCPOPT_NOP;
+			*ptr++ = htonl(mcb->dll << 16 | checksum);
 		}
 	}
 }
@@ -899,50 +909,44 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 					size += 20;
 				}
 				subflow->fourth_ack = 1;
-			} else if (skb && skb->priv_used) {
-				struct mptcp_skb_cb *cb;
+			} else if (skb) {
 				unsigned int dss_size = 0;
-				u16 options = 0;
+				unsigned int ack_size = 8;
+				u16 suboptions = 0;
 
-				cb = mptcp_skb_priv_cb(skb);
-
-				if (skb->priv_used && cb->use_map) {
+				if (skb->priv_used &&
+				    mptcp_skb_priv_cb(skb)->use_map) {
 					unsigned int map_size = 18;
 
-					if (cb->use_checksum)
+					if (mptcp_skb_priv_cb(skb)->use_checksum)
 						map_size += 2;
 
 					if (map_size <= remaining) {
 						remaining -= map_size;
 						dss_size = map_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions = OPTION_MPTCP_MPC_DSS_MAP;
+						suboptions = OPTION_MPTCP_DSS_MAP;
 					} else {
 						WARN(1, "MPTCP: Map dropped");
 					}
 				}
 
-				if (cb->use_ack) {
-					unsigned int ack_size = 8;
-
-					/* Add kind/length/subtype/flag
-					 * overhead if mapping not populated
-					 */
-					if (dss_size == 0)
-						ack_size += 4;
-
-					if (ack_size <= remaining) {
-						dss_size += ack_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions |= OPTION_MPTCP_MPC_DSS_ACK;
-					} else {
-						WARN(1, "MPTCP: Ack dropped");
-					}
+				/* Add kind/length/subtype/flag
+				 * overhead if mapping not populated
+				 */
+				if (dss_size == 0)
+					ack_size += 4;
+
+				if (ack_size <= remaining) {
+					dss_size += ack_size;
+					suboptions |= OPTION_MPTCP_DSS_ACK;
+				} else {
+					WARN(1, "MPTCP: Ack dropped");
 				}
 
 				if (dss_size) {
 					size += ALIGN(dss_size, 4);
-					opts->options |= options;
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions = suboptions;
 				}
 			}
 		}
@@ -3804,6 +3808,9 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt)
 	skb_set_tcp_pure_ack(buff);
 
 	/* Send it off, this clears delayed acks for us. */
+	if (tcp_sk(sk)->is_mptcp)
+		pr_debug("mptcp sk=%p", sk);
+
 	__tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0, rcv_nxt);
 }
 EXPORT_SYMBOL_GPL(__tcp_send_ack);
net/mptcp/options.c +88 -2
index 7e48d1d92aac..f5b7be888e84 100644
@@ -14,6 +14,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/errqueue.h>
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
@@ -21,9 +22,8 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx)
 {
 	u8 subtype;
-	u64 *p;
+	int expected_opsize;
 
-	opsize -= 2;
 	subtype = *ptr++;
 
 	/* MPTCPOPT_MP_CAPABLE
@@ -82,6 +82,68 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 	case 0x20:
 		pr_debug("DSS");
 		opt_rx->mptcp.dss = 1;
+
+		opt_rx->mptcp.flags = (*ptr++) & 0x1F;
+		opt_rx->mptcp.data_fin = (opt_rx->mptcp.flags & 0x10) != 0;
+		opt_rx->mptcp.dsn64 = (opt_rx->mptcp.flags & 0x08) != 0;
+		opt_rx->mptcp.use_map = (opt_rx->mptcp.flags & 0x04) != 0;
+		opt_rx->mptcp.ack64 = (opt_rx->mptcp.flags & 0x02) != 0;
+		opt_rx->mptcp.use_ack = (opt_rx->mptcp.flags & 0x01);
+
+		pr_debug("data_fin=%d dsn64=%d use_map=%d ack64=%d use_ack=%d",
+			 opt_rx->mptcp.data_fin, opt_rx->mptcp.dsn64,
+			 opt_rx->mptcp.use_map, opt_rx->mptcp.ack64,
+			 opt_rx->mptcp.use_ack);
+
+		expected_opsize = 0;
+
+		if (opt_rx->mptcp.use_ack) {
+			expected_opsize = 4;
+			if (opt_rx->mptcp.ack64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.ack64) {
+				opt_rx->mptcp.ack = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.ack = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			pr_debug("ack=%llu", opt_rx->mptcp.ack);
+		}
+
+		if (opt_rx->mptcp.use_map) {
+			expected_opsize += 12;
+			if (opt_rx->mptcp.dsn64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.dsn64) {
+				opt_rx->mptcp.seq = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.seq = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			opt_rx->mptcp.subflow_seq = get_unaligned_be32(ptr);
+			ptr += 4;
+
+			opt_rx->mptcp.dll = get_unaligned_be16(ptr);
+			ptr += 2;
+
+			opt_rx->mptcp.checksum = get_unaligned_be16(ptr);
+
+			pr_debug("seq=%llu subflow_seq=%u dll=%u ck=%u",
+				 opt_rx->mptcp.seq, opt_rx->mptcp.subflow_seq,
+				 opt_rx->mptcp.dll, opt_rx->mptcp.checksum);
+		}
 		break;
 
 	/* MPTCPOPT_ADD_ADDR
@@ -175,3 +237,27 @@ unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
 	}
 	return subflow_req->mp_capable;
 }
+
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb)
+{
+	struct sock_exterr_skb *serr;
+	struct sk_buff *skb;
+	int err;
+
+	BUILD_BUG_ON(sizeof(struct sock_exterr_skb) > sizeof(skb->cb));
+
+	skb = skb_clone(original_skb, GFP_ATOMIC);
+	if (!skb) {
+		pr_err("clone failed skb=%p sk=%p", original_skb, sk);
+		return;
+	}
+
+	serr = SKB_EXT_ERR(skb);
+	memset(serr, 0, sizeof(*serr));
+	serr->ee.ee_origin = SO_EE_ORIGIN_MPTCP;
+
+	err = sock_queue_err_skb(sk, skb);
+
+	if (err)
+		kfree_skb(skb);
+}
net/mptcp/protocol.c +283 -21
index dcaaa0dd3784..911428122c9c 100644
@@ -16,6 +16,8 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/netdevice.h>
+#include <linux/sched/signal.h>
+#include <linux/atomic.h>
 #include <net/sock.h>
 #include <net/inet_common.h>
 #include <net/inet_hashtables.h>
@@ -23,6 +25,12 @@
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
+static inline bool before64(__u64 seq1, __u64 seq2)
+{
+	return (__s64)(seq1-seq2) < 0;
+}
+#define after64(seq2, seq1)	before64(seq1, seq2)
+
 static void mptcp_cb_copy(const struct sk_buff *from, struct sk_buff *to)
 {
 	struct mptcp_skb_cb *mcb = from->priv;
@@ -131,15 +139,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	skb = tcp_write_queue_tail(ssk);
 
 	refcount_set(&mcb->refcnt, 1);
-	mcb->data_ack = msk->ack_seq;
 	mcb->data_seq = msk->write_seq;
 	mcb->subflow_seq = subflow_sk(ssk)->rel_write_seq;
 	mcb->dll = ret;
 	mcb->checksum = 0xbeef;
 	mcb->use_map = 1;
 	mcb->dsn64 = 1;
-	mcb->use_ack = 1;
-	mcb->ack64 = 1;
 
 	if (mcb->use_map) {
 		pr_debug("data_seq=%llu subflow_seq=%u "
@@ -170,21 +175,262 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	return ret;
 }
 
+struct mptcp_read_arg {
+	struct msghdr *msg;
+};
+
+static u64 expand_seq(u64 old_seq, u16 old_dll, u64 seq)
+{
+	if ((u32)seq == (u32)old_seq)
+		return old_seq;
+
+	/* Assume map covers data not mapped yet. */
+	return seq | ((old_seq + old_dll + 1) & ~0xFFFFFFFFULL);
+}
+
+static u64 get_mapped_dsn(struct subflow_sock *subflow)
+{
+	u32 map_offset = (tcp_sk(sock_sk(subflow))->copied_seq -
+			  subflow->ssn_offset -
+			  subflow->map_subflow_seq);
+
+	return subflow->map_seq + map_offset;
+}
+
+static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,
+			    unsigned int offset, size_t len)
+{
+	struct mptcp_read_arg *arg = desc->arg.data;
+	size_t copy_len;
+
+	copy_len = min(desc->count, len);
+
+	if (likely(arg->msg)) {
+		int err;
+
+		err = skb_copy_datagram_msg(skb, offset, arg->msg, copy_len);
+		if (err) {
+			pr_debug("error path");
+			desc->error = err;
+			return err;
+		}
+	} else {
+		pr_debug("Flushing skb payload");
+	}
+
+	// MSG_PEEK support? Other flags? MSG_TRUNC?
+
+	desc->count -= copy_len;
+
+	pr_debug("consumed %lu bytes, %lu left", copy_len, desc->count);
+	return copy_len;
+}
+
+static bool mptcp_get_mapping(struct subflow_sock *subflow)
+{
+	struct tcp_options_received opt;
+	struct sk_buff *skb;
+	bool found_map = false;
+
+	do {
+		pr_debug("Dequeue mapping skb");
+		skb = sock_dequeue_err_skb(sock_sk(subflow));
+		if (!skb) {
+			pr_debug("Empty queue");
+			break;
+		}
+
+		opt.mptcp.dss = 0;
+		opt.mptcp.use_map = 0;
+		opt.mptcp.data_fin = 0;
+		mptcp_get_options(skb, &opt);
+
+		if (opt.mptcp.dss && opt.mptcp.use_map) {
+			pr_debug("seq=%llu is64=%d ssn=%u dll=%u ck=%u",
+				 opt.mptcp.seq, opt.mptcp.dsn64,
+				 opt.mptcp.subflow_seq,
+				 opt.mptcp.dll, opt.mptcp.checksum);
+
+			if (opt.mptcp.dll == 0) {
+				pr_err("Infinite mapping not handled");
+			} else if (opt.mptcp.subflow_seq == 0 &&
+				   opt.mptcp.data_fin == 1) {
+				pr_debug("DATA_FIN with no payload");
+				kfree_skb(skb);
+				break;
+			}
+
+			if (!opt.mptcp.dsn64) {
+				subflow->map_seq = expand_seq(subflow->map_seq,
+							      subflow->map_dll,
+							      opt.mptcp.seq);
+				pr_debug("expanded seq=%llu", subflow->map_seq);
+			} else {
+				subflow->map_seq = opt.mptcp.seq;
+			}
+
+			subflow->map_subflow_seq = opt.mptcp.subflow_seq;
+			subflow->map_dll = opt.mptcp.dll;
+			subflow->map_valid = true;
+			pr_debug("new map seq=%llu subflow_seq=%u dll=%u",
+				 subflow->map_seq, subflow->map_subflow_seq,
+				 subflow->map_dll);
+			found_map = true;
+		}
+
+		kfree_skb(skb);
+	} while (!found_map);
+
+	return found_map;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow;
+	struct subflow_sock *subflow;
+	struct mptcp_read_arg arg;
+	read_descriptor_t desc;
+	struct tcp_sock *tp;
+	struct sock *ssk;
+	int copied = 0;
+	long timeo;
 
-	if (msk->connection_list) {
-		subflow = msk->connection_list;
-		pr_debug("conn_list->subflow=%p", subflow->sk);
-	} else {
-		subflow = msk->subflow;
-		pr_debug("subflow=%p", subflow->sk);
+	if (!msk->connection_list) {
+		pr_debug("fallback-read subflow=%p", msk->subflow->sk);
+		return sock_recvmsg(msk->subflow, msg, flags);
 	}
 
-	return sock_recvmsg(subflow, msg, flags);
+	ssk = msk->connection_list->sk;
+	subflow = subflow_sk(ssk);
+	tp = tcp_sk(ssk);
+
+	desc.arg.data = &arg;
+	desc.error = 0;
+
+	timeo = sock_rcvtimeo(sk, nonblock);
+
+	len = min_t(size_t, len, INT_MAX);
+
+	while (copied < len) {
+		size_t discard_len = 0;
+		int bytes_read;
+		u64 ack_seq;
+		u64 old_ack;
+		u32 ssn;
+
+		pr_debug("map_valid=%d", subflow->map_valid);
+		if (!subflow->map_valid && !mptcp_get_mapping(subflow))
+			goto wait_for_data;
+
+		ssn = tcp_sk(ssk)->copied_seq - subflow->ssn_offset;
+		old_ack = atomic64_read(&msk->ack_seq);
+
+		if (unlikely(before(ssn, subflow->map_subflow_seq))) {
+			/* Mapping covers data later in the subflow stream,
+			 * discard unmapped data. */
+			pr_debug("Mapping covers data later in stream");
+			discard_len = subflow->map_subflow_seq - ssn;
+		} else if (unlikely(!before(ssn, (subflow->map_subflow_seq +
+						  subflow->map_dll)))) {
+			/* Mapping ends earlier in the subflow stream.
+			 * Invalidate the mapping and try again.
+			 */
+			subflow->map_valid = false;
+			pr_debug("Invalid mapping ssn=%d map_seq=%d map_dll=%d",
+				 ssn, subflow->map_subflow_seq, subflow->map_dll);
+			continue;
+		} else {
+			ack_seq = get_mapped_dsn(subflow);
+
+			if (before64(ack_seq, old_ack)) {
+				/* Mapping covers data already received,
+				 * discard data in the current mapping
+				 * and invalidate the map
+				 */
+				u64 map_end_dsn = subflow->map_seq +
+					subflow->map_dll;
+				discard_len = min(map_end_dsn - ack_seq,
+						  old_ack - ack_seq);
+				subflow->map_valid = false;
+				pr_debug("Duplicate MPTCP data found");
+			}
+		}
+
+		if (discard_len) {
+			/* Discard data for the current mapping.
+			 */
+			pr_debug("Discard %lu bytes", discard_len);
+
+			arg.msg = NULL;
+			desc.count = discard_len;
+
+			bytes_read = tcp_read_sock(ssk, &desc,
+						   mptcp_read_actor);
+
+			if (bytes_read < 0)
+				break;
+			else if (bytes_read == discard_len)
+				continue;
+			else
+				goto wait_for_data;
+		}
+
+		/* Read mapped data */
+		desc.count = ssn - subflow->map_subflow_seq + subflow->map_dll;
+		arg.msg = msg;
+		bytes_read = tcp_read_sock(ssk, &desc, mptcp_read_actor);
+		if (bytes_read < 0)
+			break;
+
+		/* Refresh current MPTCP sequence number based on subflow seq */
+		ack_seq = get_mapped_dsn(subflow);
+
+		if (before64(old_ack, ack_seq)) {
+			atomic64_set(&msk->ack_seq, ack_seq);
+		}
+
+		if (!before(tcp_sk(ssk)->copied_seq - subflow->ssn_offset,
+			    subflow->map_subflow_seq + subflow->map_dll)) {
+			subflow->map_valid = false;
+			pr_debug("Done with mapping: seq=%u dll=%u",
+				 subflow->map_subflow_seq, subflow->map_dll);
+		}
+
+		copied += bytes_read;
+
+wait_for_data:
+		if (copied)
+			break;
+
+		if (tp->urg_data && tp->urg_seq == tp->copied_seq) {
+			pr_err("Urgent data present, cannot proceed");
+			break;
+		}
+
+		if (ssk->sk_err || ssk->sk_state == TCP_CLOSE ||
+		    (ssk->sk_shutdown & RCV_SHUTDOWN) || !timeo ||
+		    signal_pending(current)) {
+			pr_debug("nonblock or error");
+			break;
+		}
+
+		/* Handle blocking and retry read if needed.
+		 *
+		 * Wait on MPTCP sock, the subflow will notify via data ready.
+		 */
+
+		pr_debug("block");
+		release_sock(ssk);
+		sk_wait_data(sk, &timeo, NULL);
+		lock_sock(ssk);
+
+	}
+
+	release_sock(ssk);
+	release_sock(sk);
+
+	return copied;
 }
 
 static int mptcp_init_sock(struct sock *sk)
@@ -240,18 +486,27 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	subflow->conn = mp->sk;
 
 	if (subflow->mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
 		token_update_accept(new_sock->sk, mp->sk);
+		msk->connection_list = new_sock;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
 		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
 		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
-		msk->connection_list = new_sock;
+		subflow->conn = mp->sk;
 	} else {
 		msk->subflow = new_sock;
 	}
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 
 	return mp->sk;
 }
@@ -329,17 +584,24 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	pr_debug("msk=%p", msk);
 
 	if (mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
-		msk->write_seq = subflow->idsn + 1;
-		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
+		pr_debug("token=%u", msk->token);
 		msk->connection_list = msk->subflow;
 		msk->subflow = NULL;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
+		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
+		subflow->rel_write_seq = 1;
 	}
-	sk->sk_state = TCP_ESTABLISHED;
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 }
 
 static int subflow_create(struct sock *sock)
net/mptcp/subflow.c +27 -2
index e0b4aa8fe51d..3c9dca4be26a 100644
@@ -107,6 +107,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
 		pr_debug("remote_key=%llu", subflow_req->remote_key);
 		token_new_request(req, skb);
+		pr_debug("syn seq=%u", TCP_SKB_CB(skb)->seq);
+		subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -120,10 +122,15 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 
 	pr_debug("subflow=%p", subflow);
 
-	if (subflow->conn) {
+	if (!subflow->conn_finished) {
 		pr_debug("remote_key=%llu", subflow->remote_key);
 		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
-		subflow->conn = NULL;
+		subflow->conn_finished = 1;
+
+		if (skb) {
+			pr_debug("synack seq=%u", TCP_SKB_CB(skb)->seq);
+			subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
+		}
 	}
 }
 
@@ -172,7 +179,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->ssn_offset = subflow_req->ssn_offset;
 			subflow->token = subflow_req->token;
+			subflow->idsn = subflow_req->idsn;
 			pr_debug("token=%u", subflow->token);
 			token_new_accept(child);
 		} else {
@@ -202,6 +211,20 @@ const struct inet_connection_sock_af_ops subflow_specific = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
+static void subflow_data_ready(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *parent = subflow->conn;
+
+	pr_debug("sk=%p", sk);
+	subflow->tcp_sk_data_ready(sk);
+
+	if (parent) {
+		pr_debug("parent=%p", parent);
+		parent->sk_data_ready(parent);
+	}
+}
+
 static int subflow_init_sock(struct sock *sk)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -215,6 +238,8 @@ static int subflow_init_sock(struct sock *sk)
 	if (!err) { // @@ AND mptcp is enabled
 		tsk->is_mptcp = 1;
 		icsk->icsk_af_ops = &subflow_specific;
+		subflow->tcp_sk_data_ready = sk->sk_data_ready;
+		sk->sk_data_ready = subflow_data_ready;
 	}
 
 	return err;
Signature · 6 lines
2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp
author
#47 lore raw

Re: [MPTCP] [RFC PATCH v3 15/16] mptcp: Implement MPTCP receive path

Quoted 7 lines · Christoph Paasch
Quoted 5 lines · Mat Martineau

Parses incoming DSS options and populates outgoing MPTCP ACK fields. Clone incoming skbs containing MPTCP headers and propagate them to the MMPTCP socket using the error queue. This avoids the need for TCP coalesce/collapse bypass for MPTCP, although cloning incoming skbs does prevent some coalescing.

Quoted 5 lines

I think it's acceptable to bypass coalesce/collapse for MPTCP instead of cloning every incoming packet over to the error-queue.

The way this is now, makes the error-queue quite loaded, and I don't think it handles packet-reordering correctly.

While the commit message says skbs with MPTCP headers are cloned, the code clones all incoming skbs right now. Maybe most data packets end up having DSS headers anyway.

I agree that the current method of reading the DSS information is fragile with respect to out-of-order packet arrival.

I'm trying to get away from the previous approach of preventing coalesce/collapse and depending on intact headers in the skb. That technique relies on the integrity of data between sk_buff->head and sk_buff->data, when sk_buff->data is is considered to be the beginning of valid data.

What do you think about using the parallel queue approach once this implementation is further developed (and before you put a lot of thought in to it, read the rest of this reply for an alternative):

* Only packets with DSS headers are cloned

* The DSS mapping queue is emptied on every read, building a data
structure to represent the map for all the queued data on that subflow (deals with ordering issue)

with possible optimization:

* Rather than copy the entire skb, keep a pool of small skbs around and
only copy the TCP header

I tried out using the error queue since it has some similar per-packet use for tx timestamping, but the error queue does take a spinlock for every enqueue/dequeue. Might make more sense to add a lockless linked list to the subflow_sock for this purpose - since my intent is to empty that list every time it's read from, llist_del_all() to pop off all the entries at once is a nice fit. It would also allow use of a much smaller data structure to hold the mapping data.

Thanks,
Mat

Quoted 942 lines
Quoted 942 lines

Keeping DSS map handling separate from data reading makes it easier to add multiple-subflow receiving, since the receive code can determine which subflow contains relevant data or data that needs to be purged without touching the subflow data queues.

MPTCP ACK values are now populated from an atomic value stored in the connection socket rather than carried in the extended control block, which makes use of the most up-to-date sequence number and allows the MPTCP ACK to be populated on TCP ACK packets that have no payload.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/tcp.h           |  12 ++
 include/net/mptcp.h           |  41 +++--
 include/uapi/linux/errqueue.h |   1 +
 net/ipv4/tcp_input.c          |   3 +
 net/ipv4/tcp_output.c         | 123 +++++++-------
 net/mptcp/options.c           |  90 +++++++++-
 net/mptcp/protocol.c          | 304 +++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c           |  29 +++-
 8 files changed, 508 insertions(+), 95 deletions(-)
include/linux/tcp.h +12 -0
index b109798482d3..845da5ee9c44 100644
@@ -112,6 +112,18 @@ struct tcp_options_received {
 		u8      flags;
 		u64     sndr_key;
 		u64     rcvr_key;
+		u64	ack;
+		u64	seq;
+		u32	subflow_seq;
+		u16	dll;
+		u16	checksum;
+		u8	use_ack:1,
+			ack64:1,
+			use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			__unused:2;
 	} mptcp;
 
 };
include/net/mptcp.h +29 -12
index 73f2d25cc42d..6c9516360de4 100644
@@ -38,14 +38,14 @@
 /* MPTCP connection sock */
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
-	struct	inet_connection_sock sk;
-	u64	local_key;
-	u64	remote_key;
-	u64	write_seq;
-	u64	ack_seq;
-	u32	token;
-	struct	socket *connection_list; /* @@ needs to be a list */
-	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
+	struct inet_connection_sock sk;
+	u64		local_key;
+	u64		remote_key;
+	u64		write_seq;
+	atomic64_t	ack_seq;
+	u32		token;
+	struct socket	*connection_list; /* @@ needs to be a list */
+	struct socket	*subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -56,7 +56,6 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 /* MPTCP sk_buff private control buffer */
 struct mptcp_skb_cb {
 	refcount_t	refcnt;
-	u64		data_ack;
 	u64		data_seq;
 	u32		subflow_seq;
 	u16		dll;
@@ -65,9 +64,7 @@ struct mptcp_skb_cb {
 			dsn64:1,
 			use_checksum:1,
 			data_fin:1,
-			use_ack:1,
-			ack64:1,
-			__unused:2;
+			__unused:4;
 };
 
 static inline struct mptcp_skb_cb *mptcp_skb_priv_cb(struct sk_buff *skb)
@@ -81,16 +78,23 @@ struct subflow_sock {
 	/* tcp_sock must be the first member */
 	struct	tcp_sock sk;
 	u64	local_key;
+	u64	map_seq;
+	u32	map_subflow_seq;
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
 	u32	rel_write_seq;
+	u32	ssn_offset;
+	u16	map_dll;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
 	bool	mp_capable;	// remote is MPTCP capable
 	bool	fourth_ack;	// send initial DSS
+	bool	conn_finished;
+	bool	map_valid;
 	struct	sock *conn;	// parent mptcp_sock
+	void	(*tcp_sk_data_ready)(struct sock *sk);
 };
 
 static inline struct subflow_sock *subflow_sk(const struct sock *sk)
@@ -98,6 +102,16 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+static inline struct subflow_sock *subflow_tp(const struct tcp_sock *tp)
+{
+	return (struct subflow_sock *)tp;
+}
+
+static inline struct sock *sock_sk(const struct subflow_sock *sk)
+{
+	return (struct sock *)sk;
+}
+
 struct subflow_request_sock {
 	struct	tcp_request_sock sk;
 	u8	mp_capable : 1,
@@ -109,6 +123,7 @@ struct subflow_request_sock {
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
+	u32	ssn_offset;
 };
 
 static inline
@@ -133,6 +148,8 @@ void mptcp_subflow_exit(void);
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
 
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb);
+
 extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
 
 void token_init(void);
include/uapi/linux/errqueue.h +1 -0
index c0151200f7d1..1a1ec7362a55 100644
@@ -21,6 +21,7 @@ struct sock_extended_err {
 #define SO_EE_ORIGIN_TXSTATUS	4
 #define SO_EE_ORIGIN_ZEROCOPY	5
 #define SO_EE_ORIGIN_TXTIME	6
+#define SO_EE_ORIGIN_MPTCP	7
 #define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS
 
 #define SO_EE_OFFENDER(ee)	((struct sockaddr*)((ee)+1))
net/ipv4/tcp_input.c +3 -0
index 9a326729637f..0904d9baa685 100644
@@ -5642,6 +5642,9 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	tcp_urg(sk, skb, th);
 
 	/* step 7: process the segment text */
+	if (tcp_sk(sk)->is_mptcp)
+		mptcp_queue_headers(sk, skb);
+
 	tcp_data_queue(sk, skb);
 
 	tcp_data_snd_check(sk);
net/ipv4/tcp_output.c +65 -58
index 471f196711fc..1c9517db82c2 100644
@@ -422,8 +422,8 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_MPTCP_MPC_SYN	(1 << 0)
 #define OPTION_MPTCP_MPC_SYNACK	(1 << 1)
 #define OPTION_MPTCP_MPC_ACK	(1 << 2)
-#define OPTION_MPTCP_MPC_DSS_MAP	(1 << 6)
-#define OPTION_MPTCP_MPC_DSS_ACK	(1 << 7)
+#define OPTION_MPTCP_DSS_MAP	(1 << 6)
+#define OPTION_MPTCP_DSS_ACK	(1 << 7)
 
 struct tcp_out_options {
 	u16 options;		/* bit field of OPTION_* */
@@ -472,16 +472,12 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 	}
 
-	if ((OPTION_MPTCP_MPC_DSS_MAP |
-	     OPTION_MPTCP_MPC_DSS_ACK) & opts->suboptions) {
-		struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
-		bool write_ack = (OPTION_MPTCP_MPC_DSS_ACK & opts->suboptions) && skb->priv_used && mcb->use_ack;
-		bool write_map = (OPTION_MPTCP_MPC_DSS_MAP & opts->suboptions) && skb->priv_used && mcb->use_map;
+	if ((OPTION_MPTCP_DSS_MAP | OPTION_MPTCP_DSS_ACK) & opts->suboptions) {
+		bool write_ack = !!(OPTION_MPTCP_DSS_ACK & opts->suboptions);
+		bool write_map = (OPTION_MPTCP_DSS_MAP & opts->suboptions) &&
+			skb->priv_used && mptcp_skb_priv_cb(skb)->use_map;
 		u8 flags = 0;
 		u8 len = 4;
-		u8 *p = (u8 *)ptr;
-
-		BUG_ON(!write_ack && !write_map);
 
 		if (write_ack) {
 			len += 8;
@@ -489,46 +485,60 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 
 		if (write_map) {
+			pr_debug("Updating DSS length and flags for map");
 			len += 14;
 
-			if (mcb->use_checksum)
+			if (mptcp_skb_priv_cb(skb)->use_checksum)
 				len += 2;
 
 			/* Use only 64-bit mapping flags for now, add
 			 * support for optional 32-bit mappings later.
 			 */
 			flags |= 0x0c;
-			if (mcb->data_fin)
+			if (mptcp_skb_priv_cb(skb)->data_fin)
 				flags |= 0x10;
 		}
 
-		*p++ = 0x1e; // TCP option: Multipath TCP
-		*p++ = len;  // length
-		*p++ = 0x20; // subtype=DSS
-		*p++ = flags;
+		*ptr++ = htonl((0x1e << 24) |  // TCP option: Multipath TCP
+			       (len  << 16) |  // length
+			       (0x20 <<  8) |  // subtype=DSS
+			       (flags));
 
 		if (write_ack) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_ack);
-			p += 8;
+			struct mptcp_sock *msk = mptcp_sk(subflow_tp(tp)->conn);
+			u64 ack_seq;
+			__be64 ack;
+
+			if (msk) {
+				ack_seq = atomic64_read(&msk->ack_seq);
+			} else {
+				crypto_key_sha1(subflow_tp(tp)->remote_key,
+						NULL, &ack_seq);
+				ack_seq++;
+			}
+
+			pr_debug("ack=%llu", ack_seq);
+			ack = cpu_to_be64(ack_seq);
+			memcpy((u8 *) ptr, (u8 *) &ack, 8);
+			ptr += 2;
 		}
 
 		if (write_map) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_seq);
-			p += 8;
-
-			*(__be32 *)p = htonl(mcb->subflow_seq);
-			p += 4;
+			struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
+			u16 checksum;
+			__be64 dss;
 
-			*(__be16 *)p = htons(mcb->dll);
-			p += 2;
+			pr_debug("Writing map values");
+			dss = cpu_to_be64(mcb->data_seq);
+			memcpy((u8 *) ptr, &dss, 8);
+			ptr += 2;
+			*ptr++ = htonl(mcb->subflow_seq);
 
-			if (mcb->use_checksum) {
-				*(__be16 *)p = htons(mcb->checksum);
-				p += 2;
-			} else {
-				*p++ = TCPOPT_NOP;
-				*p++ = TCPOPT_NOP;
-			}
+			if (mcb->use_checksum)
+				checksum = (__force u16) mcb->checksum;
+			else
+				checksum = TCPOPT_NOP << 8 | TCPOPT_NOP;
+			*ptr++ = htonl(mcb->dll << 16 | checksum);
 		}
 	}
 }
@@ -899,50 +909,44 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 					size += 20;
 				}
 				subflow->fourth_ack = 1;
-			} else if (skb && skb->priv_used) {
-				struct mptcp_skb_cb *cb;
+			} else if (skb) {
 				unsigned int dss_size = 0;
-				u16 options = 0;
+				unsigned int ack_size = 8;
+				u16 suboptions = 0;
 
-				cb = mptcp_skb_priv_cb(skb);
-
-				if (skb->priv_used && cb->use_map) {
+				if (skb->priv_used &&
+				    mptcp_skb_priv_cb(skb)->use_map) {
 					unsigned int map_size = 18;
 
-					if (cb->use_checksum)
+					if (mptcp_skb_priv_cb(skb)->use_checksum)
 						map_size += 2;
 
 					if (map_size <= remaining) {
 						remaining -= map_size;
 						dss_size = map_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions = OPTION_MPTCP_MPC_DSS_MAP;
+						suboptions = OPTION_MPTCP_DSS_MAP;
 					} else {
 						WARN(1, "MPTCP: Map dropped");
 					}
 				}
 
-				if (cb->use_ack) {
-					unsigned int ack_size = 8;
-
-					/* Add kind/length/subtype/flag
-					 * overhead if mapping not populated
-					 */
-					if (dss_size == 0)
-						ack_size += 4;
-
-					if (ack_size <= remaining) {
-						dss_size += ack_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions |= OPTION_MPTCP_MPC_DSS_ACK;
-					} else {
-						WARN(1, "MPTCP: Ack dropped");
-					}
+				/* Add kind/length/subtype/flag
+				 * overhead if mapping not populated
+				 */
+				if (dss_size == 0)
+					ack_size += 4;
+
+				if (ack_size <= remaining) {
+					dss_size += ack_size;
+					suboptions |= OPTION_MPTCP_DSS_ACK;
+				} else {
+					WARN(1, "MPTCP: Ack dropped");
 				}
 
 				if (dss_size) {
 					size += ALIGN(dss_size, 4);
-					opts->options |= options;
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions = suboptions;
 				}
 			}
 		}
@@ -3804,6 +3808,9 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt)
 	skb_set_tcp_pure_ack(buff);
 
 	/* Send it off, this clears delayed acks for us. */
+	if (tcp_sk(sk)->is_mptcp)
+		pr_debug("mptcp sk=%p", sk);
+
 	__tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0, rcv_nxt);
 }
 EXPORT_SYMBOL_GPL(__tcp_send_ack);
net/mptcp/options.c +88 -2
index 7e48d1d92aac..f5b7be888e84 100644
@@ -14,6 +14,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/errqueue.h>
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
@@ -21,9 +22,8 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx)
 {
 	u8 subtype;
-	u64 *p;
+	int expected_opsize;
 
-	opsize -= 2;
 	subtype = *ptr++;
 
 	/* MPTCPOPT_MP_CAPABLE
@@ -82,6 +82,68 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 	case 0x20:
 		pr_debug("DSS");
 		opt_rx->mptcp.dss = 1;
+
+		opt_rx->mptcp.flags = (*ptr++) & 0x1F;
+		opt_rx->mptcp.data_fin = (opt_rx->mptcp.flags & 0x10) != 0;
+		opt_rx->mptcp.dsn64 = (opt_rx->mptcp.flags & 0x08) != 0;
+		opt_rx->mptcp.use_map = (opt_rx->mptcp.flags & 0x04) != 0;
+		opt_rx->mptcp.ack64 = (opt_rx->mptcp.flags & 0x02) != 0;
+		opt_rx->mptcp.use_ack = (opt_rx->mptcp.flags & 0x01);
+
+		pr_debug("data_fin=%d dsn64=%d use_map=%d ack64=%d use_ack=%d",
+			 opt_rx->mptcp.data_fin, opt_rx->mptcp.dsn64,
+			 opt_rx->mptcp.use_map, opt_rx->mptcp.ack64,
+			 opt_rx->mptcp.use_ack);
+
+		expected_opsize = 0;
+
+		if (opt_rx->mptcp.use_ack) {
+			expected_opsize = 4;
+			if (opt_rx->mptcp.ack64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.ack64) {
+				opt_rx->mptcp.ack = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.ack = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			pr_debug("ack=%llu", opt_rx->mptcp.ack);
+		}
+
+		if (opt_rx->mptcp.use_map) {
+			expected_opsize += 12;
+			if (opt_rx->mptcp.dsn64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.dsn64) {
+				opt_rx->mptcp.seq = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.seq = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			opt_rx->mptcp.subflow_seq = get_unaligned_be32(ptr);
+			ptr += 4;
+
+			opt_rx->mptcp.dll = get_unaligned_be16(ptr);
+			ptr += 2;
+
+			opt_rx->mptcp.checksum = get_unaligned_be16(ptr);
+
+			pr_debug("seq=%llu subflow_seq=%u dll=%u ck=%u",
+				 opt_rx->mptcp.seq, opt_rx->mptcp.subflow_seq,
+				 opt_rx->mptcp.dll, opt_rx->mptcp.checksum);
+		}
 		break;
 
 	/* MPTCPOPT_ADD_ADDR
@@ -175,3 +237,27 @@ unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
 	}
 	return subflow_req->mp_capable;
 }
+
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb)
+{
+	struct sock_exterr_skb *serr;
+	struct sk_buff *skb;
+	int err;
+
+	BUILD_BUG_ON(sizeof(struct sock_exterr_skb) > sizeof(skb->cb));
+
+	skb = skb_clone(original_skb, GFP_ATOMIC);
+	if (!skb) {
+		pr_err("clone failed skb=%p sk=%p", original_skb, sk);
+		return;
+	}
+
+	serr = SKB_EXT_ERR(skb);
+	memset(serr, 0, sizeof(*serr));
+	serr->ee.ee_origin = SO_EE_ORIGIN_MPTCP;
+
+	err = sock_queue_err_skb(sk, skb);
+
+	if (err)
+		kfree_skb(skb);
+}
net/mptcp/protocol.c +283 -21
index dcaaa0dd3784..911428122c9c 100644
@@ -16,6 +16,8 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/netdevice.h>
+#include <linux/sched/signal.h>
+#include <linux/atomic.h>
 #include <net/sock.h>
 #include <net/inet_common.h>
 #include <net/inet_hashtables.h>
@@ -23,6 +25,12 @@
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
+static inline bool before64(__u64 seq1, __u64 seq2)
+{
+	return (__s64)(seq1-seq2) < 0;
+}
+#define after64(seq2, seq1)	before64(seq1, seq2)
+
 static void mptcp_cb_copy(const struct sk_buff *from, struct sk_buff *to)
 {
 	struct mptcp_skb_cb *mcb = from->priv;
@@ -131,15 +139,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	skb = tcp_write_queue_tail(ssk);
 
 	refcount_set(&mcb->refcnt, 1);
-	mcb->data_ack = msk->ack_seq;
 	mcb->data_seq = msk->write_seq;
 	mcb->subflow_seq = subflow_sk(ssk)->rel_write_seq;
 	mcb->dll = ret;
 	mcb->checksum = 0xbeef;
 	mcb->use_map = 1;
 	mcb->dsn64 = 1;
-	mcb->use_ack = 1;
-	mcb->ack64 = 1;
 
 	if (mcb->use_map) {
 		pr_debug("data_seq=%llu subflow_seq=%u "
@@ -170,21 +175,262 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	return ret;
 }
 
+struct mptcp_read_arg {
+	struct msghdr *msg;
+};
+
+static u64 expand_seq(u64 old_seq, u16 old_dll, u64 seq)
+{
+	if ((u32)seq == (u32)old_seq)
+		return old_seq;
+
+	/* Assume map covers data not mapped yet. */
+	return seq | ((old_seq + old_dll + 1) & ~0xFFFFFFFFULL);
+}
+
+static u64 get_mapped_dsn(struct subflow_sock *subflow)
+{
+	u32 map_offset = (tcp_sk(sock_sk(subflow))->copied_seq -
+			  subflow->ssn_offset -
+			  subflow->map_subflow_seq);
+
+	return subflow->map_seq + map_offset;
+}
+
+static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,
+			    unsigned int offset, size_t len)
+{
+	struct mptcp_read_arg *arg = desc->arg.data;
+	size_t copy_len;
+
+	copy_len = min(desc->count, len);
+
+	if (likely(arg->msg)) {
+		int err;
+
+		err = skb_copy_datagram_msg(skb, offset, arg->msg, copy_len);
+		if (err) {
+			pr_debug("error path");
+			desc->error = err;
+			return err;
+		}
+	} else {
+		pr_debug("Flushing skb payload");
+	}
+
+	// MSG_PEEK support? Other flags? MSG_TRUNC?
+
+	desc->count -= copy_len;
+
+	pr_debug("consumed %lu bytes, %lu left", copy_len, desc->count);
+	return copy_len;
+}
+
+static bool mptcp_get_mapping(struct subflow_sock *subflow)
+{
+	struct tcp_options_received opt;
+	struct sk_buff *skb;
+	bool found_map = false;
+
+	do {
+		pr_debug("Dequeue mapping skb");
+		skb = sock_dequeue_err_skb(sock_sk(subflow));
+		if (!skb) {
+			pr_debug("Empty queue");
+			break;
+		}
+
+		opt.mptcp.dss = 0;
+		opt.mptcp.use_map = 0;
+		opt.mptcp.data_fin = 0;
+		mptcp_get_options(skb, &opt);
+
+		if (opt.mptcp.dss && opt.mptcp.use_map) {
+			pr_debug("seq=%llu is64=%d ssn=%u dll=%u ck=%u",
+				 opt.mptcp.seq, opt.mptcp.dsn64,
+				 opt.mptcp.subflow_seq,
+				 opt.mptcp.dll, opt.mptcp.checksum);
+
+			if (opt.mptcp.dll == 0) {
+				pr_err("Infinite mapping not handled");
+			} else if (opt.mptcp.subflow_seq == 0 &&
+				   opt.mptcp.data_fin == 1) {
+				pr_debug("DATA_FIN with no payload");
+				kfree_skb(skb);
+				break;
+			}
+
+			if (!opt.mptcp.dsn64) {
+				subflow->map_seq = expand_seq(subflow->map_seq,
+							      subflow->map_dll,
+							      opt.mptcp.seq);
+				pr_debug("expanded seq=%llu", subflow->map_seq);
+			} else {
+				subflow->map_seq = opt.mptcp.seq;
+			}
+
+			subflow->map_subflow_seq = opt.mptcp.subflow_seq;
+			subflow->map_dll = opt.mptcp.dll;
+			subflow->map_valid = true;
+			pr_debug("new map seq=%llu subflow_seq=%u dll=%u",
+				 subflow->map_seq, subflow->map_subflow_seq,
+				 subflow->map_dll);
+			found_map = true;
+		}
+
+		kfree_skb(skb);
+	} while (!found_map);
+
+	return found_map;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow;
+	struct subflow_sock *subflow;
+	struct mptcp_read_arg arg;
+	read_descriptor_t desc;
+	struct tcp_sock *tp;
+	struct sock *ssk;
+	int copied = 0;
+	long timeo;
 
-	if (msk->connection_list) {
-		subflow = msk->connection_list;
-		pr_debug("conn_list->subflow=%p", subflow->sk);
-	} else {
-		subflow = msk->subflow;
-		pr_debug("subflow=%p", subflow->sk);
+	if (!msk->connection_list) {
+		pr_debug("fallback-read subflow=%p", msk->subflow->sk);
+		return sock_recvmsg(msk->subflow, msg, flags);
 	}
 
-	return sock_recvmsg(subflow, msg, flags);
+	ssk = msk->connection_list->sk;
+	subflow = subflow_sk(ssk);
+	tp = tcp_sk(ssk);
+
+	desc.arg.data = &arg;
+	desc.error = 0;
+
+	timeo = sock_rcvtimeo(sk, nonblock);
+
+	len = min_t(size_t, len, INT_MAX);
+
+	while (copied < len) {
+		size_t discard_len = 0;
+		int bytes_read;
+		u64 ack_seq;
+		u64 old_ack;
+		u32 ssn;
+
+		pr_debug("map_valid=%d", subflow->map_valid);
+		if (!subflow->map_valid && !mptcp_get_mapping(subflow))
+			goto wait_for_data;
+
+		ssn = tcp_sk(ssk)->copied_seq - subflow->ssn_offset;
+		old_ack = atomic64_read(&msk->ack_seq);
+
+		if (unlikely(before(ssn, subflow->map_subflow_seq))) {
+			/* Mapping covers data later in the subflow stream,
+			 * discard unmapped data. */
+			pr_debug("Mapping covers data later in stream");
+			discard_len = subflow->map_subflow_seq - ssn;
+		} else if (unlikely(!before(ssn, (subflow->map_subflow_seq +
+						  subflow->map_dll)))) {
+			/* Mapping ends earlier in the subflow stream.
+			 * Invalidate the mapping and try again.
+			 */
+			subflow->map_valid = false;
+			pr_debug("Invalid mapping ssn=%d map_seq=%d map_dll=%d",
+				 ssn, subflow->map_subflow_seq, subflow->map_dll);
+			continue;
+		} else {
+			ack_seq = get_mapped_dsn(subflow);
+
+			if (before64(ack_seq, old_ack)) {
+				/* Mapping covers data already received,
+				 * discard data in the current mapping
+				 * and invalidate the map
+				 */
+				u64 map_end_dsn = subflow->map_seq +
+					subflow->map_dll;
+				discard_len = min(map_end_dsn - ack_seq,
+						  old_ack - ack_seq);
+				subflow->map_valid = false;
+				pr_debug("Duplicate MPTCP data found");
+			}
+		}
+
+		if (discard_len) {
+			/* Discard data for the current mapping.
+			 */
+			pr_debug("Discard %lu bytes", discard_len);
+
+			arg.msg = NULL;
+			desc.count = discard_len;
+
+			bytes_read = tcp_read_sock(ssk, &desc,
+						   mptcp_read_actor);
+
+			if (bytes_read < 0)
+				break;
+			else if (bytes_read == discard_len)
+				continue;
+			else
+				goto wait_for_data;
+		}
+
+		/* Read mapped data */
+		desc.count = ssn - subflow->map_subflow_seq + subflow->map_dll;
+		arg.msg = msg;
+		bytes_read = tcp_read_sock(ssk, &desc, mptcp_read_actor);
+		if (bytes_read < 0)
+			break;
+
+		/* Refresh current MPTCP sequence number based on subflow seq */
+		ack_seq = get_mapped_dsn(subflow);
+
+		if (before64(old_ack, ack_seq)) {
+			atomic64_set(&msk->ack_seq, ack_seq);
+		}
+
+		if (!before(tcp_sk(ssk)->copied_seq - subflow->ssn_offset,
+			    subflow->map_subflow_seq + subflow->map_dll)) {
+			subflow->map_valid = false;
+			pr_debug("Done with mapping: seq=%u dll=%u",
+				 subflow->map_subflow_seq, subflow->map_dll);
+		}
+
+		copied += bytes_read;
+
+wait_for_data:
+		if (copied)
+			break;
+
+		if (tp->urg_data && tp->urg_seq == tp->copied_seq) {
+			pr_err("Urgent data present, cannot proceed");
+			break;
+		}
+
+		if (ssk->sk_err || ssk->sk_state == TCP_CLOSE ||
+		    (ssk->sk_shutdown & RCV_SHUTDOWN) || !timeo ||
+		    signal_pending(current)) {
+			pr_debug("nonblock or error");
+			break;
+		}
+
+		/* Handle blocking and retry read if needed.
+		 *
+		 * Wait on MPTCP sock, the subflow will notify via data ready.
+		 */
+
+		pr_debug("block");
+		release_sock(ssk);
+		sk_wait_data(sk, &timeo, NULL);
+		lock_sock(ssk);
+
+	}
+
+	release_sock(ssk);
+	release_sock(sk);
+
+	return copied;
 }
 
 static int mptcp_init_sock(struct sock *sk)
@@ -240,18 +486,27 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	subflow->conn = mp->sk;
 
 	if (subflow->mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
 		token_update_accept(new_sock->sk, mp->sk);
+		msk->connection_list = new_sock;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
 		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
 		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
-		msk->connection_list = new_sock;
+		subflow->conn = mp->sk;
 	} else {
 		msk->subflow = new_sock;
 	}
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 
 	return mp->sk;
 }
@@ -329,17 +584,24 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	pr_debug("msk=%p", msk);
 
 	if (mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
-		msk->write_seq = subflow->idsn + 1;
-		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
+		pr_debug("token=%u", msk->token);
 		msk->connection_list = msk->subflow;
 		msk->subflow = NULL;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
+		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
+		subflow->rel_write_seq = 1;
 	}
-	sk->sk_state = TCP_ESTABLISHED;
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 }
 
 static int subflow_create(struct sock *sock)
net/mptcp/subflow.c +27 -2
index e0b4aa8fe51d..3c9dca4be26a 100644
@@ -107,6 +107,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
 		pr_debug("remote_key=%llu", subflow_req->remote_key);
 		token_new_request(req, skb);
+		pr_debug("syn seq=%u", TCP_SKB_CB(skb)->seq);
+		subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -120,10 +122,15 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 
 	pr_debug("subflow=%p", subflow);
 
-	if (subflow->conn) {
+	if (!subflow->conn_finished) {
 		pr_debug("remote_key=%llu", subflow->remote_key);
 		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
-		subflow->conn = NULL;
+		subflow->conn_finished = 1;
+
+		if (skb) {
+			pr_debug("synack seq=%u", TCP_SKB_CB(skb)->seq);
+			subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
+		}
 	}
 }
 
@@ -172,7 +179,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->ssn_offset = subflow_req->ssn_offset;
 			subflow->token = subflow_req->token;
+			subflow->idsn = subflow_req->idsn;
 			pr_debug("token=%u", subflow->token);
 			token_new_accept(child);
 		} else {
@@ -202,6 +211,20 @@ const struct inet_connection_sock_af_ops subflow_specific = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
+static void subflow_data_ready(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *parent = subflow->conn;
+
+	pr_debug("sk=%p", sk);
+	subflow->tcp_sk_data_ready(sk);
+
+	if (parent) {
+		pr_debug("parent=%p", parent);
+		parent->sk_data_ready(parent);
+	}
+}
+
 static int subflow_init_sock(struct sock *sk)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -215,6 +238,8 @@ static int subflow_init_sock(struct sock *sk)
 	if (!err) { // @@ AND mptcp is enabled
 		tsk->is_mptcp = 1;
 		icsk->icsk_af_ops = &subflow_specific;
+		subflow->tcp_sk_data_ready = sk->sk_data_ready;
+		sk->sk_data_ready = subflow_data_ready;
 	}
 
Quoted 9 lines
Quoted 8 lines
net/mptcp/subflow.c +0 -1
 	return err;
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

#48 lore raw

Re: [MPTCP] [RFC PATCH v3 15/16] mptcp: Implement MPTCP receive path

Quoted 47 lines · Mat Martineau
Quoted 7 lines · Christoph Paasch
Quoted 5 lines · Mat Martineau

Parses incoming DSS options and populates outgoing MPTCP ACK fields. Clone incoming skbs containing MPTCP headers and propagate them to the MMPTCP socket using the error queue. This avoids the need for TCP coalesce/collapse bypass for MPTCP, although cloning incoming skbs does prevent some coalescing.

Quoted 5 lines

I think it's acceptable to bypass coalesce/collapse for MPTCP instead of cloning every incoming packet over to the error-queue.

The way this is now, makes the error-queue quite loaded, and I don't think it handles packet-reordering correctly.

While the commit message says skbs with MPTCP headers are cloned, the code clones all incoming skbs right now. Maybe most data packets end up having DSS headers anyway.

I agree that the current method of reading the DSS information is fragile with respect to out-of-order packet arrival.

I'm trying to get away from the previous approach of preventing coalesce/collapse and depending on intact headers in the skb. That technique relies on the integrity of data between sk_buff->head and sk_buff->data, when sk_buff->data is is considered to be the beginning of valid data.

What do you think about using the parallel queue approach once this implementation is further developed (and before you put a lot of thought in to it, read the rest of this reply for an alternative):

* Only packets with DSS headers are cloned

* The DSS mapping queue is emptied on every read, building a data structure
to represent the map for all the queued data on that subflow (deals with ordering issue)

with possible optimization:

* Rather than copy the entire skb, keep a pool of small skbs around and
only copy the TCP header

Quoted 7 lines

I tried out using the error queue since it has some similar per-packet use for tx timestamping, but the error queue does take a spinlock for every enqueue/dequeue. Might make more sense to add a lockless linked list to the subflow_sock for this purpose - since my intent is to empty that list every time it's read from, llist_del_all() to pop off all the entries at once is a nice fit. It would also allow use of a much smaller data structure to hold the mapping data.

Did you consider using the priv-pointer in the skb to take care of this instead?

The problem here is that the DSS-information is tightly linked to a byte in the payload. When we start sending this DSS-information asynchronously over to the meta-socket (which is what the error-queue or another queue is doing), we start making things very complex (e.g., need to keep ordering right,...).

E.g., in skb_try_coalesce, we can simply do a check on the priv-used field. And then decide based on the DSS-mapping (or do that in tcp_try_coalesce).

When coalescing is happening, the issue is that we anyways need to "de-coalesce" after if non-continuous mappings have been coalesced. This is another reason why it's good to do a "smart" coalescing that takes DSS-mappings into account. priv_used is the perfect infrastructure for that.

Doing it that way will be much less error-prone.

The parallel queue is great for asynchronous signals like ADD_ADDR,...

Christoph

Quoted 961 lines

Thanks,
Mat

Quoted 942 lines
Quoted 942 lines

Keeping DSS map handling separate from data reading makes it easier to add multiple-subflow receiving, since the receive code can determine which subflow contains relevant data or data that needs to be purged without touching the subflow data queues.

MPTCP ACK values are now populated from an atomic value stored in the connection socket rather than carried in the extended control block, which makes use of the most up-to-date sequence number and allows the MPTCP ACK to be populated on TCP ACK packets that have no payload.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/tcp.h           |  12 ++
 include/net/mptcp.h           |  41 +++--
 include/uapi/linux/errqueue.h |   1 +
 net/ipv4/tcp_input.c          |   3 +
 net/ipv4/tcp_output.c         | 123 +++++++-------
 net/mptcp/options.c           |  90 +++++++++-
 net/mptcp/protocol.c          | 304 +++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c           |  29 +++-
 8 files changed, 508 insertions(+), 95 deletions(-)
include/linux/tcp.h +12 -0
index b109798482d3..845da5ee9c44 100644
@@ -112,6 +112,18 @@ struct tcp_options_received {
 		u8      flags;
 		u64     sndr_key;
 		u64     rcvr_key;
+		u64	ack;
+		u64	seq;
+		u32	subflow_seq;
+		u16	dll;
+		u16	checksum;
+		u8	use_ack:1,
+			ack64:1,
+			use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			__unused:2;
 	} mptcp;
 
 };
include/net/mptcp.h +29 -12
index 73f2d25cc42d..6c9516360de4 100644
@@ -38,14 +38,14 @@
 /* MPTCP connection sock */
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
-	struct	inet_connection_sock sk;
-	u64	local_key;
-	u64	remote_key;
-	u64	write_seq;
-	u64	ack_seq;
-	u32	token;
-	struct	socket *connection_list; /* @@ needs to be a list */
-	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
+	struct inet_connection_sock sk;
+	u64		local_key;
+	u64		remote_key;
+	u64		write_seq;
+	atomic64_t	ack_seq;
+	u32		token;
+	struct socket	*connection_list; /* @@ needs to be a list */
+	struct socket	*subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -56,7 +56,6 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 /* MPTCP sk_buff private control buffer */
 struct mptcp_skb_cb {
 	refcount_t	refcnt;
-	u64		data_ack;
 	u64		data_seq;
 	u32		subflow_seq;
 	u16		dll;
@@ -65,9 +64,7 @@ struct mptcp_skb_cb {
 			dsn64:1,
 			use_checksum:1,
 			data_fin:1,
-			use_ack:1,
-			ack64:1,
-			__unused:2;
+			__unused:4;
 };
 
 static inline struct mptcp_skb_cb *mptcp_skb_priv_cb(struct sk_buff *skb)
@@ -81,16 +78,23 @@ struct subflow_sock {
 	/* tcp_sock must be the first member */
 	struct	tcp_sock sk;
 	u64	local_key;
+	u64	map_seq;
+	u32	map_subflow_seq;
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
 	u32	rel_write_seq;
+	u32	ssn_offset;
+	u16	map_dll;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
 	bool	mp_capable;	// remote is MPTCP capable
 	bool	fourth_ack;	// send initial DSS
+	bool	conn_finished;
+	bool	map_valid;
 	struct	sock *conn;	// parent mptcp_sock
+	void	(*tcp_sk_data_ready)(struct sock *sk);
 };
 
 static inline struct subflow_sock *subflow_sk(const struct sock *sk)
@@ -98,6 +102,16 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+static inline struct subflow_sock *subflow_tp(const struct tcp_sock *tp)
+{
+	return (struct subflow_sock *)tp;
+}
+
+static inline struct sock *sock_sk(const struct subflow_sock *sk)
+{
+	return (struct sock *)sk;
+}
+
 struct subflow_request_sock {
 	struct	tcp_request_sock sk;
 	u8	mp_capable : 1,
@@ -109,6 +123,7 @@ struct subflow_request_sock {
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
+	u32	ssn_offset;
 };
 
 static inline
@@ -133,6 +148,8 @@ void mptcp_subflow_exit(void);
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
 
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb);
+
 extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
 
 void token_init(void);
include/uapi/linux/errqueue.h +1 -0
index c0151200f7d1..1a1ec7362a55 100644
@@ -21,6 +21,7 @@ struct sock_extended_err {
 #define SO_EE_ORIGIN_TXSTATUS	4
 #define SO_EE_ORIGIN_ZEROCOPY	5
 #define SO_EE_ORIGIN_TXTIME	6
+#define SO_EE_ORIGIN_MPTCP	7
 #define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS
 
 #define SO_EE_OFFENDER(ee)	((struct sockaddr*)((ee)+1))
net/ipv4/tcp_input.c +3 -0
index 9a326729637f..0904d9baa685 100644
@@ -5642,6 +5642,9 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	tcp_urg(sk, skb, th);
 
 	/* step 7: process the segment text */
+	if (tcp_sk(sk)->is_mptcp)
+		mptcp_queue_headers(sk, skb);
+
 	tcp_data_queue(sk, skb);
 
 	tcp_data_snd_check(sk);
net/ipv4/tcp_output.c +65 -58
index 471f196711fc..1c9517db82c2 100644
@@ -422,8 +422,8 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_MPTCP_MPC_SYN	(1 << 0)
 #define OPTION_MPTCP_MPC_SYNACK	(1 << 1)
 #define OPTION_MPTCP_MPC_ACK	(1 << 2)
-#define OPTION_MPTCP_MPC_DSS_MAP	(1 << 6)
-#define OPTION_MPTCP_MPC_DSS_ACK	(1 << 7)
+#define OPTION_MPTCP_DSS_MAP	(1 << 6)
+#define OPTION_MPTCP_DSS_ACK	(1 << 7)
 
 struct tcp_out_options {
 	u16 options;		/* bit field of OPTION_* */
@@ -472,16 +472,12 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 	}
 
-	if ((OPTION_MPTCP_MPC_DSS_MAP |
-	     OPTION_MPTCP_MPC_DSS_ACK) & opts->suboptions) {
-		struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
-		bool write_ack = (OPTION_MPTCP_MPC_DSS_ACK & opts->suboptions) && skb->priv_used && mcb->use_ack;
-		bool write_map = (OPTION_MPTCP_MPC_DSS_MAP & opts->suboptions) && skb->priv_used && mcb->use_map;
+	if ((OPTION_MPTCP_DSS_MAP | OPTION_MPTCP_DSS_ACK) & opts->suboptions) {
+		bool write_ack = !!(OPTION_MPTCP_DSS_ACK & opts->suboptions);
+		bool write_map = (OPTION_MPTCP_DSS_MAP & opts->suboptions) &&
+			skb->priv_used && mptcp_skb_priv_cb(skb)->use_map;
 		u8 flags = 0;
 		u8 len = 4;
-		u8 *p = (u8 *)ptr;
-
-		BUG_ON(!write_ack && !write_map);
 
 		if (write_ack) {
 			len += 8;
@@ -489,46 +485,60 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 
 		if (write_map) {
+			pr_debug("Updating DSS length and flags for map");
 			len += 14;
 
-			if (mcb->use_checksum)
+			if (mptcp_skb_priv_cb(skb)->use_checksum)
 				len += 2;
 
 			/* Use only 64-bit mapping flags for now, add
 			 * support for optional 32-bit mappings later.
 			 */
 			flags |= 0x0c;
-			if (mcb->data_fin)
+			if (mptcp_skb_priv_cb(skb)->data_fin)
 				flags |= 0x10;
 		}
 
-		*p++ = 0x1e; // TCP option: Multipath TCP
-		*p++ = len;  // length
-		*p++ = 0x20; // subtype=DSS
-		*p++ = flags;
+		*ptr++ = htonl((0x1e << 24) |  // TCP option: Multipath TCP
+			       (len  << 16) |  // length
+			       (0x20 <<  8) |  // subtype=DSS
+			       (flags));
 
 		if (write_ack) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_ack);
-			p += 8;
+			struct mptcp_sock *msk = mptcp_sk(subflow_tp(tp)->conn);
+			u64 ack_seq;
+			__be64 ack;
+
+			if (msk) {
+				ack_seq = atomic64_read(&msk->ack_seq);
+			} else {
+				crypto_key_sha1(subflow_tp(tp)->remote_key,
+						NULL, &ack_seq);
+				ack_seq++;
+			}
+
+			pr_debug("ack=%llu", ack_seq);
+			ack = cpu_to_be64(ack_seq);
+			memcpy((u8 *) ptr, (u8 *) &ack, 8);
+			ptr += 2;
 		}
 
 		if (write_map) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_seq);
-			p += 8;
-
-			*(__be32 *)p = htonl(mcb->subflow_seq);
-			p += 4;
+			struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
+			u16 checksum;
+			__be64 dss;
 
-			*(__be16 *)p = htons(mcb->dll);
-			p += 2;
+			pr_debug("Writing map values");
+			dss = cpu_to_be64(mcb->data_seq);
+			memcpy((u8 *) ptr, &dss, 8);
+			ptr += 2;
+			*ptr++ = htonl(mcb->subflow_seq);
 
-			if (mcb->use_checksum) {
-				*(__be16 *)p = htons(mcb->checksum);
-				p += 2;
-			} else {
-				*p++ = TCPOPT_NOP;
-				*p++ = TCPOPT_NOP;
-			}
+			if (mcb->use_checksum)
+				checksum = (__force u16) mcb->checksum;
+			else
+				checksum = TCPOPT_NOP << 8 | TCPOPT_NOP;
+			*ptr++ = htonl(mcb->dll << 16 | checksum);
 		}
 	}
 }
@@ -899,50 +909,44 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 					size += 20;
 				}
 				subflow->fourth_ack = 1;
-			} else if (skb && skb->priv_used) {
-				struct mptcp_skb_cb *cb;
+			} else if (skb) {
 				unsigned int dss_size = 0;
-				u16 options = 0;
+				unsigned int ack_size = 8;
+				u16 suboptions = 0;
 
-				cb = mptcp_skb_priv_cb(skb);
-
-				if (skb->priv_used && cb->use_map) {
+				if (skb->priv_used &&
+				    mptcp_skb_priv_cb(skb)->use_map) {
 					unsigned int map_size = 18;
 
-					if (cb->use_checksum)
+					if (mptcp_skb_priv_cb(skb)->use_checksum)
 						map_size += 2;
 
 					if (map_size <= remaining) {
 						remaining -= map_size;
 						dss_size = map_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions = OPTION_MPTCP_MPC_DSS_MAP;
+						suboptions = OPTION_MPTCP_DSS_MAP;
 					} else {
 						WARN(1, "MPTCP: Map dropped");
 					}
 				}
 
-				if (cb->use_ack) {
-					unsigned int ack_size = 8;
-
-					/* Add kind/length/subtype/flag
-					 * overhead if mapping not populated
-					 */
-					if (dss_size == 0)
-						ack_size += 4;
-
-					if (ack_size <= remaining) {
-						dss_size += ack_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions |= OPTION_MPTCP_MPC_DSS_ACK;
-					} else {
-						WARN(1, "MPTCP: Ack dropped");
-					}
+				/* Add kind/length/subtype/flag
+				 * overhead if mapping not populated
+				 */
+				if (dss_size == 0)
+					ack_size += 4;
+
+				if (ack_size <= remaining) {
+					dss_size += ack_size;
+					suboptions |= OPTION_MPTCP_DSS_ACK;
+				} else {
+					WARN(1, "MPTCP: Ack dropped");
 				}
 
 				if (dss_size) {
 					size += ALIGN(dss_size, 4);
-					opts->options |= options;
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions = suboptions;
 				}
 			}
 		}
@@ -3804,6 +3808,9 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt)
 	skb_set_tcp_pure_ack(buff);
 
 	/* Send it off, this clears delayed acks for us. */
+	if (tcp_sk(sk)->is_mptcp)
+		pr_debug("mptcp sk=%p", sk);
+
 	__tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0, rcv_nxt);
 }
 EXPORT_SYMBOL_GPL(__tcp_send_ack);
net/mptcp/options.c +88 -2
index 7e48d1d92aac..f5b7be888e84 100644
@@ -14,6 +14,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/errqueue.h>
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
@@ -21,9 +22,8 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx)
 {
 	u8 subtype;
-	u64 *p;
+	int expected_opsize;
 
-	opsize -= 2;
 	subtype = *ptr++;
 
 	/* MPTCPOPT_MP_CAPABLE
@@ -82,6 +82,68 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 	case 0x20:
 		pr_debug("DSS");
 		opt_rx->mptcp.dss = 1;
+
+		opt_rx->mptcp.flags = (*ptr++) & 0x1F;
+		opt_rx->mptcp.data_fin = (opt_rx->mptcp.flags & 0x10) != 0;
+		opt_rx->mptcp.dsn64 = (opt_rx->mptcp.flags & 0x08) != 0;
+		opt_rx->mptcp.use_map = (opt_rx->mptcp.flags & 0x04) != 0;
+		opt_rx->mptcp.ack64 = (opt_rx->mptcp.flags & 0x02) != 0;
+		opt_rx->mptcp.use_ack = (opt_rx->mptcp.flags & 0x01);
+
+		pr_debug("data_fin=%d dsn64=%d use_map=%d ack64=%d use_ack=%d",
+			 opt_rx->mptcp.data_fin, opt_rx->mptcp.dsn64,
+			 opt_rx->mptcp.use_map, opt_rx->mptcp.ack64,
+			 opt_rx->mptcp.use_ack);
+
+		expected_opsize = 0;
+
+		if (opt_rx->mptcp.use_ack) {
+			expected_opsize = 4;
+			if (opt_rx->mptcp.ack64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.ack64) {
+				opt_rx->mptcp.ack = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.ack = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			pr_debug("ack=%llu", opt_rx->mptcp.ack);
+		}
+
+		if (opt_rx->mptcp.use_map) {
+			expected_opsize += 12;
+			if (opt_rx->mptcp.dsn64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.dsn64) {
+				opt_rx->mptcp.seq = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.seq = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			opt_rx->mptcp.subflow_seq = get_unaligned_be32(ptr);
+			ptr += 4;
+
+			opt_rx->mptcp.dll = get_unaligned_be16(ptr);
+			ptr += 2;
+
+			opt_rx->mptcp.checksum = get_unaligned_be16(ptr);
+
+			pr_debug("seq=%llu subflow_seq=%u dll=%u ck=%u",
+				 opt_rx->mptcp.seq, opt_rx->mptcp.subflow_seq,
+				 opt_rx->mptcp.dll, opt_rx->mptcp.checksum);
+		}
 		break;
 
 	/* MPTCPOPT_ADD_ADDR
@@ -175,3 +237,27 @@ unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
 	}
 	return subflow_req->mp_capable;
 }
+
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb)
+{
+	struct sock_exterr_skb *serr;
+	struct sk_buff *skb;
+	int err;
+
+	BUILD_BUG_ON(sizeof(struct sock_exterr_skb) > sizeof(skb->cb));
+
+	skb = skb_clone(original_skb, GFP_ATOMIC);
+	if (!skb) {
+		pr_err("clone failed skb=%p sk=%p", original_skb, sk);
+		return;
+	}
+
+	serr = SKB_EXT_ERR(skb);
+	memset(serr, 0, sizeof(*serr));
+	serr->ee.ee_origin = SO_EE_ORIGIN_MPTCP;
+
+	err = sock_queue_err_skb(sk, skb);
+
+	if (err)
+		kfree_skb(skb);
+}
net/mptcp/protocol.c +283 -21
index dcaaa0dd3784..911428122c9c 100644
@@ -16,6 +16,8 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/netdevice.h>
+#include <linux/sched/signal.h>
+#include <linux/atomic.h>
 #include <net/sock.h>
 #include <net/inet_common.h>
 #include <net/inet_hashtables.h>
@@ -23,6 +25,12 @@
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
+static inline bool before64(__u64 seq1, __u64 seq2)
+{
+	return (__s64)(seq1-seq2) < 0;
+}
+#define after64(seq2, seq1)	before64(seq1, seq2)
+
 static void mptcp_cb_copy(const struct sk_buff *from, struct sk_buff *to)
 {
 	struct mptcp_skb_cb *mcb = from->priv;
@@ -131,15 +139,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	skb = tcp_write_queue_tail(ssk);
 
 	refcount_set(&mcb->refcnt, 1);
-	mcb->data_ack = msk->ack_seq;
 	mcb->data_seq = msk->write_seq;
 	mcb->subflow_seq = subflow_sk(ssk)->rel_write_seq;
 	mcb->dll = ret;
 	mcb->checksum = 0xbeef;
 	mcb->use_map = 1;
 	mcb->dsn64 = 1;
-	mcb->use_ack = 1;
-	mcb->ack64 = 1;
 
 	if (mcb->use_map) {
 		pr_debug("data_seq=%llu subflow_seq=%u "
@@ -170,21 +175,262 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	return ret;
 }
 
+struct mptcp_read_arg {
+	struct msghdr *msg;
+};
+
+static u64 expand_seq(u64 old_seq, u16 old_dll, u64 seq)
+{
+	if ((u32)seq == (u32)old_seq)
+		return old_seq;
+
+	/* Assume map covers data not mapped yet. */
+	return seq | ((old_seq + old_dll + 1) & ~0xFFFFFFFFULL);
+}
+
+static u64 get_mapped_dsn(struct subflow_sock *subflow)
+{
+	u32 map_offset = (tcp_sk(sock_sk(subflow))->copied_seq -
+			  subflow->ssn_offset -
+			  subflow->map_subflow_seq);
+
+	return subflow->map_seq + map_offset;
+}
+
+static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,
+			    unsigned int offset, size_t len)
+{
+	struct mptcp_read_arg *arg = desc->arg.data;
+	size_t copy_len;
+
+	copy_len = min(desc->count, len);
+
+	if (likely(arg->msg)) {
+		int err;
+
+		err = skb_copy_datagram_msg(skb, offset, arg->msg, copy_len);
+		if (err) {
+			pr_debug("error path");
+			desc->error = err;
+			return err;
+		}
+	} else {
+		pr_debug("Flushing skb payload");
+	}
+
+	// MSG_PEEK support? Other flags? MSG_TRUNC?
+
+	desc->count -= copy_len;
+
+	pr_debug("consumed %lu bytes, %lu left", copy_len, desc->count);
+	return copy_len;
+}
+
+static bool mptcp_get_mapping(struct subflow_sock *subflow)
+{
+	struct tcp_options_received opt;
+	struct sk_buff *skb;
+	bool found_map = false;
+
+	do {
+		pr_debug("Dequeue mapping skb");
+		skb = sock_dequeue_err_skb(sock_sk(subflow));
+		if (!skb) {
+			pr_debug("Empty queue");
+			break;
+		}
+
+		opt.mptcp.dss = 0;
+		opt.mptcp.use_map = 0;
+		opt.mptcp.data_fin = 0;
+		mptcp_get_options(skb, &opt);
+
+		if (opt.mptcp.dss && opt.mptcp.use_map) {
+			pr_debug("seq=%llu is64=%d ssn=%u dll=%u ck=%u",
+				 opt.mptcp.seq, opt.mptcp.dsn64,
+				 opt.mptcp.subflow_seq,
+				 opt.mptcp.dll, opt.mptcp.checksum);
+
+			if (opt.mptcp.dll == 0) {
+				pr_err("Infinite mapping not handled");
+			} else if (opt.mptcp.subflow_seq == 0 &&
+				   opt.mptcp.data_fin == 1) {
+				pr_debug("DATA_FIN with no payload");
+				kfree_skb(skb);
+				break;
+			}
+
+			if (!opt.mptcp.dsn64) {
+				subflow->map_seq = expand_seq(subflow->map_seq,
+							      subflow->map_dll,
+							      opt.mptcp.seq);
+				pr_debug("expanded seq=%llu", subflow->map_seq);
+			} else {
+				subflow->map_seq = opt.mptcp.seq;
+			}
+
+			subflow->map_subflow_seq = opt.mptcp.subflow_seq;
+			subflow->map_dll = opt.mptcp.dll;
+			subflow->map_valid = true;
+			pr_debug("new map seq=%llu subflow_seq=%u dll=%u",
+				 subflow->map_seq, subflow->map_subflow_seq,
+				 subflow->map_dll);
+			found_map = true;
+		}
+
+		kfree_skb(skb);
+	} while (!found_map);
+
+	return found_map;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow;
+	struct subflow_sock *subflow;
+	struct mptcp_read_arg arg;
+	read_descriptor_t desc;
+	struct tcp_sock *tp;
+	struct sock *ssk;
+	int copied = 0;
+	long timeo;
 
-	if (msk->connection_list) {
-		subflow = msk->connection_list;
-		pr_debug("conn_list->subflow=%p", subflow->sk);
-	} else {
-		subflow = msk->subflow;
-		pr_debug("subflow=%p", subflow->sk);
+	if (!msk->connection_list) {
+		pr_debug("fallback-read subflow=%p", msk->subflow->sk);
+		return sock_recvmsg(msk->subflow, msg, flags);
 	}
 
-	return sock_recvmsg(subflow, msg, flags);
+	ssk = msk->connection_list->sk;
+	subflow = subflow_sk(ssk);
+	tp = tcp_sk(ssk);
+
+	desc.arg.data = &arg;
+	desc.error = 0;
+
+	timeo = sock_rcvtimeo(sk, nonblock);
+
+	len = min_t(size_t, len, INT_MAX);
+
+	while (copied < len) {
+		size_t discard_len = 0;
+		int bytes_read;
+		u64 ack_seq;
+		u64 old_ack;
+		u32 ssn;
+
+		pr_debug("map_valid=%d", subflow->map_valid);
+		if (!subflow->map_valid && !mptcp_get_mapping(subflow))
+			goto wait_for_data;
+
+		ssn = tcp_sk(ssk)->copied_seq - subflow->ssn_offset;
+		old_ack = atomic64_read(&msk->ack_seq);
+
+		if (unlikely(before(ssn, subflow->map_subflow_seq))) {
+			/* Mapping covers data later in the subflow stream,
+			 * discard unmapped data. */
+			pr_debug("Mapping covers data later in stream");
+			discard_len = subflow->map_subflow_seq - ssn;
+		} else if (unlikely(!before(ssn, (subflow->map_subflow_seq +
+						  subflow->map_dll)))) {
+			/* Mapping ends earlier in the subflow stream.
+			 * Invalidate the mapping and try again.
+			 */
+			subflow->map_valid = false;
+			pr_debug("Invalid mapping ssn=%d map_seq=%d map_dll=%d",
+				 ssn, subflow->map_subflow_seq, subflow->map_dll);
+			continue;
+		} else {
+			ack_seq = get_mapped_dsn(subflow);
+
+			if (before64(ack_seq, old_ack)) {
+				/* Mapping covers data already received,
+				 * discard data in the current mapping
+				 * and invalidate the map
+				 */
+				u64 map_end_dsn = subflow->map_seq +
+					subflow->map_dll;
+				discard_len = min(map_end_dsn - ack_seq,
+						  old_ack - ack_seq);
+				subflow->map_valid = false;
+				pr_debug("Duplicate MPTCP data found");
+			}
+		}
+
+		if (discard_len) {
+			/* Discard data for the current mapping.
+			 */
+			pr_debug("Discard %lu bytes", discard_len);
+
+			arg.msg = NULL;
+			desc.count = discard_len;
+
+			bytes_read = tcp_read_sock(ssk, &desc,
+						   mptcp_read_actor);
+
+			if (bytes_read < 0)
+				break;
+			else if (bytes_read == discard_len)
+				continue;
+			else
+				goto wait_for_data;
+		}
+
+		/* Read mapped data */
+		desc.count = ssn - subflow->map_subflow_seq + subflow->map_dll;
+		arg.msg = msg;
+		bytes_read = tcp_read_sock(ssk, &desc, mptcp_read_actor);
+		if (bytes_read < 0)
+			break;
+
+		/* Refresh current MPTCP sequence number based on subflow seq */
+		ack_seq = get_mapped_dsn(subflow);
+
+		if (before64(old_ack, ack_seq)) {
+			atomic64_set(&msk->ack_seq, ack_seq);
+		}
+
+		if (!before(tcp_sk(ssk)->copied_seq - subflow->ssn_offset,
+			    subflow->map_subflow_seq + subflow->map_dll)) {
+			subflow->map_valid = false;
+			pr_debug("Done with mapping: seq=%u dll=%u",
+				 subflow->map_subflow_seq, subflow->map_dll);
+		}
+
+		copied += bytes_read;
+
+wait_for_data:
+		if (copied)
+			break;
+
+		if (tp->urg_data && tp->urg_seq == tp->copied_seq) {
+			pr_err("Urgent data present, cannot proceed");
+			break;
+		}
+
+		if (ssk->sk_err || ssk->sk_state == TCP_CLOSE ||
+		    (ssk->sk_shutdown & RCV_SHUTDOWN) || !timeo ||
+		    signal_pending(current)) {
+			pr_debug("nonblock or error");
+			break;
+		}
+
+		/* Handle blocking and retry read if needed.
+		 *
+		 * Wait on MPTCP sock, the subflow will notify via data ready.
+		 */
+
+		pr_debug("block");
+		release_sock(ssk);
+		sk_wait_data(sk, &timeo, NULL);
+		lock_sock(ssk);
+
+	}
+
+	release_sock(ssk);
+	release_sock(sk);
+
+	return copied;
 }
 
 static int mptcp_init_sock(struct sock *sk)
@@ -240,18 +486,27 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	subflow->conn = mp->sk;
 
 	if (subflow->mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
 		token_update_accept(new_sock->sk, mp->sk);
+		msk->connection_list = new_sock;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
 		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
 		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
-		msk->connection_list = new_sock;
+		subflow->conn = mp->sk;
 	} else {
 		msk->subflow = new_sock;
 	}
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 
 	return mp->sk;
 }
@@ -329,17 +584,24 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	pr_debug("msk=%p", msk);
 
 	if (mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
-		msk->write_seq = subflow->idsn + 1;
-		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
+		pr_debug("token=%u", msk->token);
 		msk->connection_list = msk->subflow;
 		msk->subflow = NULL;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
+		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
+		subflow->rel_write_seq = 1;
 	}
-	sk->sk_state = TCP_ESTABLISHED;
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 }
 
 static int subflow_create(struct sock *sock)
net/mptcp/subflow.c +27 -2
index e0b4aa8fe51d..3c9dca4be26a 100644
@@ -107,6 +107,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
 		pr_debug("remote_key=%llu", subflow_req->remote_key);
 		token_new_request(req, skb);
+		pr_debug("syn seq=%u", TCP_SKB_CB(skb)->seq);
+		subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -120,10 +122,15 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 
 	pr_debug("subflow=%p", subflow);
 
-	if (subflow->conn) {
+	if (!subflow->conn_finished) {
 		pr_debug("remote_key=%llu", subflow->remote_key);
 		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
-		subflow->conn = NULL;
+		subflow->conn_finished = 1;
+
+		if (skb) {
+			pr_debug("synack seq=%u", TCP_SKB_CB(skb)->seq);
+			subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
+		}
 	}
 }
 
@@ -172,7 +179,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->ssn_offset = subflow_req->ssn_offset;
 			subflow->token = subflow_req->token;
+			subflow->idsn = subflow_req->idsn;
 			pr_debug("token=%u", subflow->token);
 			token_new_accept(child);
 		} else {
@@ -202,6 +211,20 @@ const struct inet_connection_sock_af_ops subflow_specific = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
+static void subflow_data_ready(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *parent = subflow->conn;
+
+	pr_debug("sk=%p", sk);
+	subflow->tcp_sk_data_ready(sk);
+
+	if (parent) {
+		pr_debug("parent=%p", parent);
+		parent->sk_data_ready(parent);
+	}
+}
+
 static int subflow_init_sock(struct sock *sk)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -215,6 +238,8 @@ static int subflow_init_sock(struct sock *sk)
 	if (!err) { // @@ AND mptcp is enabled
 		tsk->is_mptcp = 1;
 		icsk->icsk_af_ops = &subflow_specific;
+		subflow->tcp_sk_data_ready = sk->sk_data_ready;
+		sk->sk_data_ready = subflow_data_ready;
 	}
 
Quoted 9 lines
Quoted 8 lines
net/mptcp/subflow.c +0 -1
 	return err;
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

--
Mat Martineau
Intel OTC

author
#49 lore raw

Re: [MPTCP] [RFC PATCH v3 15/16] mptcp: Implement MPTCP receive path

Quoted 74 lines · Christoph Paasch
Quoted 47 lines · Mat Martineau
Quoted 7 lines · Christoph Paasch
Quoted 5 lines · Mat Martineau

Parses incoming DSS options and populates outgoing MPTCP ACK fields. Clone incoming skbs containing MPTCP headers and propagate them to the MMPTCP socket using the error queue. This avoids the need for TCP coalesce/collapse bypass for MPTCP, although cloning incoming skbs does prevent some coalescing.

Quoted 5 lines

I think it's acceptable to bypass coalesce/collapse for MPTCP instead of cloning every incoming packet over to the error-queue.

The way this is now, makes the error-queue quite loaded, and I don't think it handles packet-reordering correctly.

While the commit message says skbs with MPTCP headers are cloned, the code clones all incoming skbs right now. Maybe most data packets end up having DSS headers anyway.

I agree that the current method of reading the DSS information is fragile with respect to out-of-order packet arrival.

I'm trying to get away from the previous approach of preventing coalesce/collapse and depending on intact headers in the skb. That technique relies on the integrity of data between sk_buff->head and sk_buff->data, when sk_buff->data is is considered to be the beginning of valid data.

What do you think about using the parallel queue approach once this implementation is further developed (and before you put a lot of thought in to it, read the rest of this reply for an alternative):

* Only packets with DSS headers are cloned

* The DSS mapping queue is emptied on every read, building a data structure
to represent the map for all the queued data on that subflow (deals with ordering issue)

with possible optimization:

* Rather than copy the entire skb, keep a pool of small skbs around and
only copy the TCP header

Quoted 7 lines

I tried out using the error queue since it has some similar per-packet use for tx timestamping, but the error queue does take a spinlock for every enqueue/dequeue. Might make more sense to add a lockless linked list to the subflow_sock for this purpose - since my intent is to empty that list every time it's read from, llist_del_all() to pop off all the entries at once is a nice fit. It would also allow use of a much smaller data structure to hold the mapping data.

Did you consider using the priv-pointer in the skb to take care of this instead?

The problem here is that the DSS-information is tightly linked to a byte in the payload. When we start sending this DSS-information asynchronously over to the meta-socket (which is what the error-queue or another queue is doing), we start making things very complex (e.g., need to keep ordering right,...).

E.g., in skb_try_coalesce, we can simply do a check on the priv-used field. And then decide based on the DSS-mapping (or do that in tcp_try_coalesce).

When coalescing is happening, the issue is that we anyways need to "de-coalesce" after if non-continuous mappings have been coalesced. This is another reason why it's good to do a "smart" coalescing that takes DSS-mappings into account. priv_used is the perfect infrastructure for that.

Quoted 4 lines

Doing it that way will be much less error-prone.

The parallel queue is great for asynchronous signals like ADD_ADDR,...

It depends on whether you want to read from a subflow and then figure out where the payload belongs in the MPTCP stream, or if you want to have the mapping info for all the subflows available before reading in order to decide which subflow to look at next.

I think there's something to be said for leaving the data in the subflow rx queue until it's requested by userspace. That would let TCP handle a lot of the buffering in a more normal (for TCP) way. While the MPTCP layer would have to handle some out-of-order reassembly and buffering, I think the common case would be to pass reads through to the appropriate subflow. MPTCP-layer reassembly and buffering would be the uncommon case (I assume...) and wouldn't need to be super optimized.

Speaking of assumptions, I should be more specific about what I expect from normal traffic patterns. For the mobile connectivity use case, I'd expect one subflow to be receiving all the incoming packets in-order until there was a change in the available interfaces, when the subflow in use might change. For the hybrid access use case, I would expect increasing sequence numbers (with wrap-around) on all subflows. In both cases, a jump backwards in the sequence numbers must be handled but is unlikely, and no buffering is required at the MPTCP layer. For hybrid access, knowing the sequence numbers prior to reading from the subflow seems like a win.

Does this fit with what you (or researchers) have seen from MPTCP in practice?

Thanks,

Mat

Quoted 958 lines
Quoted 958 lines

Thanks,
Mat

Quoted 951 lines
Quoted 950 lines

Keeping DSS map handling separate from data reading makes it easier to add multiple-subflow receiving, since the receive code can determine which subflow contains relevant data or data that needs to be purged without touching the subflow data queues.

MPTCP ACK values are now populated from an atomic value stored in the connection socket rather than carried in the extended control block, which makes use of the most up-to-date sequence number and allows the MPTCP ACK to be populated on TCP ACK packets that have no payload.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/tcp.h           |  12 ++
 include/net/mptcp.h           |  41 +++--
 include/uapi/linux/errqueue.h |   1 +
 net/ipv4/tcp_input.c          |   3 +
 net/ipv4/tcp_output.c         | 123 +++++++-------
 net/mptcp/options.c           |  90 +++++++++-
 net/mptcp/protocol.c          | 304 +++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c           |  29 +++-
 8 files changed, 508 insertions(+), 95 deletions(-)
include/linux/tcp.h +12 -0
index b109798482d3..845da5ee9c44 100644
@@ -112,6 +112,18 @@ struct tcp_options_received {
 		u8      flags;
 		u64     sndr_key;
 		u64     rcvr_key;
+		u64	ack;
+		u64	seq;
+		u32	subflow_seq;
+		u16	dll;
+		u16	checksum;
+		u8	use_ack:1,
+			ack64:1,
+			use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			__unused:2;
 	} mptcp;
 
 };
include/net/mptcp.h +29 -12
index 73f2d25cc42d..6c9516360de4 100644
@@ -38,14 +38,14 @@
 /* MPTCP connection sock */
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
-	struct	inet_connection_sock sk;
-	u64	local_key;
-	u64	remote_key;
-	u64	write_seq;
-	u64	ack_seq;
-	u32	token;
-	struct	socket *connection_list; /* @@ needs to be a list */
-	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
+	struct inet_connection_sock sk;
+	u64		local_key;
+	u64		remote_key;
+	u64		write_seq;
+	atomic64_t	ack_seq;
+	u32		token;
+	struct socket	*connection_list; /* @@ needs to be a list */
+	struct socket	*subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -56,7 +56,6 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 /* MPTCP sk_buff private control buffer */
 struct mptcp_skb_cb {
 	refcount_t	refcnt;
-	u64		data_ack;
 	u64		data_seq;
 	u32		subflow_seq;
 	u16		dll;
@@ -65,9 +64,7 @@ struct mptcp_skb_cb {
 			dsn64:1,
 			use_checksum:1,
 			data_fin:1,
-			use_ack:1,
-			ack64:1,
-			__unused:2;
+			__unused:4;
 };
 
 static inline struct mptcp_skb_cb *mptcp_skb_priv_cb(struct sk_buff *skb)
@@ -81,16 +78,23 @@ struct subflow_sock {
 	/* tcp_sock must be the first member */
 	struct	tcp_sock sk;
 	u64	local_key;
+	u64	map_seq;
+	u32	map_subflow_seq;
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
 	u32	rel_write_seq;
+	u32	ssn_offset;
+	u16	map_dll;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
 	bool	mp_capable;	// remote is MPTCP capable
 	bool	fourth_ack;	// send initial DSS
+	bool	conn_finished;
+	bool	map_valid;
 	struct	sock *conn;	// parent mptcp_sock
+	void	(*tcp_sk_data_ready)(struct sock *sk);
 };
 
 static inline struct subflow_sock *subflow_sk(const struct sock *sk)
@@ -98,6 +102,16 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+static inline struct subflow_sock *subflow_tp(const struct tcp_sock *tp)
+{
+	return (struct subflow_sock *)tp;
+}
+
+static inline struct sock *sock_sk(const struct subflow_sock *sk)
+{
+	return (struct sock *)sk;
+}
+
 struct subflow_request_sock {
 	struct	tcp_request_sock sk;
 	u8	mp_capable : 1,
@@ -109,6 +123,7 @@ struct subflow_request_sock {
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
+	u32	ssn_offset;
 };
 
 static inline
@@ -133,6 +148,8 @@ void mptcp_subflow_exit(void);
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
 
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb);
+
 extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
 
 void token_init(void);
include/uapi/linux/errqueue.h +1 -0
index c0151200f7d1..1a1ec7362a55 100644
@@ -21,6 +21,7 @@ struct sock_extended_err {
 #define SO_EE_ORIGIN_TXSTATUS	4
 #define SO_EE_ORIGIN_ZEROCOPY	5
 #define SO_EE_ORIGIN_TXTIME	6
+#define SO_EE_ORIGIN_MPTCP	7
 #define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS
 
 #define SO_EE_OFFENDER(ee)	((struct sockaddr*)((ee)+1))
net/ipv4/tcp_input.c +3 -0
index 9a326729637f..0904d9baa685 100644
@@ -5642,6 +5642,9 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	tcp_urg(sk, skb, th);
 
 	/* step 7: process the segment text */
+	if (tcp_sk(sk)->is_mptcp)
+		mptcp_queue_headers(sk, skb);
+
 	tcp_data_queue(sk, skb);
 
 	tcp_data_snd_check(sk);
net/ipv4/tcp_output.c +65 -58
index 471f196711fc..1c9517db82c2 100644
@@ -422,8 +422,8 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_MPTCP_MPC_SYN	(1 << 0)
 #define OPTION_MPTCP_MPC_SYNACK	(1 << 1)
 #define OPTION_MPTCP_MPC_ACK	(1 << 2)
-#define OPTION_MPTCP_MPC_DSS_MAP	(1 << 6)
-#define OPTION_MPTCP_MPC_DSS_ACK	(1 << 7)
+#define OPTION_MPTCP_DSS_MAP	(1 << 6)
+#define OPTION_MPTCP_DSS_ACK	(1 << 7)
 
 struct tcp_out_options {
 	u16 options;		/* bit field of OPTION_* */
@@ -472,16 +472,12 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 	}
 
-	if ((OPTION_MPTCP_MPC_DSS_MAP |
-	     OPTION_MPTCP_MPC_DSS_ACK) & opts->suboptions) {
-		struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
-		bool write_ack = (OPTION_MPTCP_MPC_DSS_ACK & opts->suboptions) && skb->priv_used && mcb->use_ack;
-		bool write_map = (OPTION_MPTCP_MPC_DSS_MAP & opts->suboptions) && skb->priv_used && mcb->use_map;
+	if ((OPTION_MPTCP_DSS_MAP | OPTION_MPTCP_DSS_ACK) & opts->suboptions) {
+		bool write_ack = !!(OPTION_MPTCP_DSS_ACK & opts->suboptions);
+		bool write_map = (OPTION_MPTCP_DSS_MAP & opts->suboptions) &&
+			skb->priv_used && mptcp_skb_priv_cb(skb)->use_map;
 		u8 flags = 0;
 		u8 len = 4;
-		u8 *p = (u8 *)ptr;
-
-		BUG_ON(!write_ack && !write_map);
 
 		if (write_ack) {
 			len += 8;
@@ -489,46 +485,60 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 
 		if (write_map) {
+			pr_debug("Updating DSS length and flags for map");
 			len += 14;
 
-			if (mcb->use_checksum)
+			if (mptcp_skb_priv_cb(skb)->use_checksum)
 				len += 2;
 
 			/* Use only 64-bit mapping flags for now, add
 			 * support for optional 32-bit mappings later.
 			 */
 			flags |= 0x0c;
-			if (mcb->data_fin)
+			if (mptcp_skb_priv_cb(skb)->data_fin)
 				flags |= 0x10;
 		}
 
-		*p++ = 0x1e; // TCP option: Multipath TCP
-		*p++ = len;  // length
-		*p++ = 0x20; // subtype=DSS
-		*p++ = flags;
+		*ptr++ = htonl((0x1e << 24) |  // TCP option: Multipath TCP
+			       (len  << 16) |  // length
+			       (0x20 <<  8) |  // subtype=DSS
+			       (flags));
 
 		if (write_ack) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_ack);
-			p += 8;
+			struct mptcp_sock *msk = mptcp_sk(subflow_tp(tp)->conn);
+			u64 ack_seq;
+			__be64 ack;
+
+			if (msk) {
+				ack_seq = atomic64_read(&msk->ack_seq);
+			} else {
+				crypto_key_sha1(subflow_tp(tp)->remote_key,
+						NULL, &ack_seq);
+				ack_seq++;
+			}
+
+			pr_debug("ack=%llu", ack_seq);
+			ack = cpu_to_be64(ack_seq);
+			memcpy((u8 *) ptr, (u8 *) &ack, 8);
+			ptr += 2;
 		}
 
 		if (write_map) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_seq);
-			p += 8;
-
-			*(__be32 *)p = htonl(mcb->subflow_seq);
-			p += 4;
+			struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
+			u16 checksum;
+			__be64 dss;
 
-			*(__be16 *)p = htons(mcb->dll);
-			p += 2;
+			pr_debug("Writing map values");
+			dss = cpu_to_be64(mcb->data_seq);
+			memcpy((u8 *) ptr, &dss, 8);
+			ptr += 2;
+			*ptr++ = htonl(mcb->subflow_seq);
 
-			if (mcb->use_checksum) {
-				*(__be16 *)p = htons(mcb->checksum);
-				p += 2;
-			} else {
-				*p++ = TCPOPT_NOP;
-				*p++ = TCPOPT_NOP;
-			}
+			if (mcb->use_checksum)
+				checksum = (__force u16) mcb->checksum;
+			else
+				checksum = TCPOPT_NOP << 8 | TCPOPT_NOP;
+			*ptr++ = htonl(mcb->dll << 16 | checksum);
 		}
 	}
 }
@@ -899,50 +909,44 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 					size += 20;
 				}
 				subflow->fourth_ack = 1;
-			} else if (skb && skb->priv_used) {
-				struct mptcp_skb_cb *cb;
+			} else if (skb) {
 				unsigned int dss_size = 0;
-				u16 options = 0;
+				unsigned int ack_size = 8;
+				u16 suboptions = 0;
 
-				cb = mptcp_skb_priv_cb(skb);
-
-				if (skb->priv_used && cb->use_map) {
+				if (skb->priv_used &&
+				    mptcp_skb_priv_cb(skb)->use_map) {
 					unsigned int map_size = 18;
 
-					if (cb->use_checksum)
+					if (mptcp_skb_priv_cb(skb)->use_checksum)
 						map_size += 2;
 
 					if (map_size <= remaining) {
 						remaining -= map_size;
 						dss_size = map_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions = OPTION_MPTCP_MPC_DSS_MAP;
+						suboptions = OPTION_MPTCP_DSS_MAP;
 					} else {
 						WARN(1, "MPTCP: Map dropped");
 					}
 				}
 
-				if (cb->use_ack) {
-					unsigned int ack_size = 8;
-
-					/* Add kind/length/subtype/flag
-					 * overhead if mapping not populated
-					 */
-					if (dss_size == 0)
-						ack_size += 4;
-
-					if (ack_size <= remaining) {
-						dss_size += ack_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions |= OPTION_MPTCP_MPC_DSS_ACK;
-					} else {
-						WARN(1, "MPTCP: Ack dropped");
-					}
+				/* Add kind/length/subtype/flag
+				 * overhead if mapping not populated
+				 */
+				if (dss_size == 0)
+					ack_size += 4;
+
+				if (ack_size <= remaining) {
+					dss_size += ack_size;
+					suboptions |= OPTION_MPTCP_DSS_ACK;
+				} else {
+					WARN(1, "MPTCP: Ack dropped");
 				}
 
 				if (dss_size) {
 					size += ALIGN(dss_size, 4);
-					opts->options |= options;
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions = suboptions;
 				}
 			}
 		}
@@ -3804,6 +3808,9 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt)
 	skb_set_tcp_pure_ack(buff);
 
 	/* Send it off, this clears delayed acks for us. */
+	if (tcp_sk(sk)->is_mptcp)
+		pr_debug("mptcp sk=%p", sk);
+
 	__tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0, rcv_nxt);
 }
 EXPORT_SYMBOL_GPL(__tcp_send_ack);
net/mptcp/options.c +88 -2
index 7e48d1d92aac..f5b7be888e84 100644
@@ -14,6 +14,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/errqueue.h>
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
@@ -21,9 +22,8 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx)
 {
 	u8 subtype;
-	u64 *p;
+	int expected_opsize;
 
-	opsize -= 2;
 	subtype = *ptr++;
 
 	/* MPTCPOPT_MP_CAPABLE
@@ -82,6 +82,68 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 	case 0x20:
 		pr_debug("DSS");
 		opt_rx->mptcp.dss = 1;
+
+		opt_rx->mptcp.flags = (*ptr++) & 0x1F;
+		opt_rx->mptcp.data_fin = (opt_rx->mptcp.flags & 0x10) != 0;
+		opt_rx->mptcp.dsn64 = (opt_rx->mptcp.flags & 0x08) != 0;
+		opt_rx->mptcp.use_map = (opt_rx->mptcp.flags & 0x04) != 0;
+		opt_rx->mptcp.ack64 = (opt_rx->mptcp.flags & 0x02) != 0;
+		opt_rx->mptcp.use_ack = (opt_rx->mptcp.flags & 0x01);
+
+		pr_debug("data_fin=%d dsn64=%d use_map=%d ack64=%d use_ack=%d",
+			 opt_rx->mptcp.data_fin, opt_rx->mptcp.dsn64,
+			 opt_rx->mptcp.use_map, opt_rx->mptcp.ack64,
+			 opt_rx->mptcp.use_ack);
+
+		expected_opsize = 0;
+
+		if (opt_rx->mptcp.use_ack) {
+			expected_opsize = 4;
+			if (opt_rx->mptcp.ack64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.ack64) {
+				opt_rx->mptcp.ack = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.ack = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			pr_debug("ack=%llu", opt_rx->mptcp.ack);
+		}
+
+		if (opt_rx->mptcp.use_map) {
+			expected_opsize += 12;
+			if (opt_rx->mptcp.dsn64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.dsn64) {
+				opt_rx->mptcp.seq = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.seq = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			opt_rx->mptcp.subflow_seq = get_unaligned_be32(ptr);
+			ptr += 4;
+
+			opt_rx->mptcp.dll = get_unaligned_be16(ptr);
+			ptr += 2;
+
+			opt_rx->mptcp.checksum = get_unaligned_be16(ptr);
+
+			pr_debug("seq=%llu subflow_seq=%u dll=%u ck=%u",
+				 opt_rx->mptcp.seq, opt_rx->mptcp.subflow_seq,
+				 opt_rx->mptcp.dll, opt_rx->mptcp.checksum);
+		}
 		break;
 
 	/* MPTCPOPT_ADD_ADDR
@@ -175,3 +237,27 @@ unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
 	}
 	return subflow_req->mp_capable;
 }
+
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb)
+{
+	struct sock_exterr_skb *serr;
+	struct sk_buff *skb;
+	int err;
+
+	BUILD_BUG_ON(sizeof(struct sock_exterr_skb) > sizeof(skb->cb));
+
+	skb = skb_clone(original_skb, GFP_ATOMIC);
+	if (!skb) {
+		pr_err("clone failed skb=%p sk=%p", original_skb, sk);
+		return;
+	}
+
+	serr = SKB_EXT_ERR(skb);
+	memset(serr, 0, sizeof(*serr));
+	serr->ee.ee_origin = SO_EE_ORIGIN_MPTCP;
+
+	err = sock_queue_err_skb(sk, skb);
+
+	if (err)
+		kfree_skb(skb);
+}
net/mptcp/protocol.c +283 -21
index dcaaa0dd3784..911428122c9c 100644
@@ -16,6 +16,8 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/netdevice.h>
+#include <linux/sched/signal.h>
+#include <linux/atomic.h>
 #include <net/sock.h>
 #include <net/inet_common.h>
 #include <net/inet_hashtables.h>
@@ -23,6 +25,12 @@
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
+static inline bool before64(__u64 seq1, __u64 seq2)
+{
+	return (__s64)(seq1-seq2) < 0;
+}
+#define after64(seq2, seq1)	before64(seq1, seq2)
+
 static void mptcp_cb_copy(const struct sk_buff *from, struct sk_buff *to)
 {
 	struct mptcp_skb_cb *mcb = from->priv;
@@ -131,15 +139,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	skb = tcp_write_queue_tail(ssk);
 
 	refcount_set(&mcb->refcnt, 1);
-	mcb->data_ack = msk->ack_seq;
 	mcb->data_seq = msk->write_seq;
 	mcb->subflow_seq = subflow_sk(ssk)->rel_write_seq;
 	mcb->dll = ret;
 	mcb->checksum = 0xbeef;
 	mcb->use_map = 1;
 	mcb->dsn64 = 1;
-	mcb->use_ack = 1;
-	mcb->ack64 = 1;
 
 	if (mcb->use_map) {
 		pr_debug("data_seq=%llu subflow_seq=%u "
@@ -170,21 +175,262 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	return ret;
 }
 
+struct mptcp_read_arg {
+	struct msghdr *msg;
+};
+
+static u64 expand_seq(u64 old_seq, u16 old_dll, u64 seq)
+{
+	if ((u32)seq == (u32)old_seq)
+		return old_seq;
+
+	/* Assume map covers data not mapped yet. */
+	return seq | ((old_seq + old_dll + 1) & ~0xFFFFFFFFULL);
+}
+
+static u64 get_mapped_dsn(struct subflow_sock *subflow)
+{
+	u32 map_offset = (tcp_sk(sock_sk(subflow))->copied_seq -
+			  subflow->ssn_offset -
+			  subflow->map_subflow_seq);
+
+	return subflow->map_seq + map_offset;
+}
+
+static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,
+			    unsigned int offset, size_t len)
+{
+	struct mptcp_read_arg *arg = desc->arg.data;
+	size_t copy_len;
+
+	copy_len = min(desc->count, len);
+
+	if (likely(arg->msg)) {
+		int err;
+
+		err = skb_copy_datagram_msg(skb, offset, arg->msg, copy_len);
+		if (err) {
+			pr_debug("error path");
+			desc->error = err;
+			return err;
+		}
+	} else {
+		pr_debug("Flushing skb payload");
+	}
+
+	// MSG_PEEK support? Other flags? MSG_TRUNC?
+
+	desc->count -= copy_len;
+
+	pr_debug("consumed %lu bytes, %lu left", copy_len, desc->count);
+	return copy_len;
+}
+
+static bool mptcp_get_mapping(struct subflow_sock *subflow)
+{
+	struct tcp_options_received opt;
+	struct sk_buff *skb;
+	bool found_map = false;
+
+	do {
+		pr_debug("Dequeue mapping skb");
+		skb = sock_dequeue_err_skb(sock_sk(subflow));
+		if (!skb) {
+			pr_debug("Empty queue");
+			break;
+		}
+
+		opt.mptcp.dss = 0;
+		opt.mptcp.use_map = 0;
+		opt.mptcp.data_fin = 0;
+		mptcp_get_options(skb, &opt);
+
+		if (opt.mptcp.dss && opt.mptcp.use_map) {
+			pr_debug("seq=%llu is64=%d ssn=%u dll=%u ck=%u",
+				 opt.mptcp.seq, opt.mptcp.dsn64,
+				 opt.mptcp.subflow_seq,
+				 opt.mptcp.dll, opt.mptcp.checksum);
+
+			if (opt.mptcp.dll == 0) {
+				pr_err("Infinite mapping not handled");
+			} else if (opt.mptcp.subflow_seq == 0 &&
+				   opt.mptcp.data_fin == 1) {
+				pr_debug("DATA_FIN with no payload");
+				kfree_skb(skb);
+				break;
+			}
+
+			if (!opt.mptcp.dsn64) {
+				subflow->map_seq = expand_seq(subflow->map_seq,
+							      subflow->map_dll,
+							      opt.mptcp.seq);
+				pr_debug("expanded seq=%llu", subflow->map_seq);
+			} else {
+				subflow->map_seq = opt.mptcp.seq;
+			}
+
+			subflow->map_subflow_seq = opt.mptcp.subflow_seq;
+			subflow->map_dll = opt.mptcp.dll;
+			subflow->map_valid = true;
+			pr_debug("new map seq=%llu subflow_seq=%u dll=%u",
+				 subflow->map_seq, subflow->map_subflow_seq,
+				 subflow->map_dll);
+			found_map = true;
+		}
+
+		kfree_skb(skb);
+	} while (!found_map);
+
+	return found_map;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow;
+	struct subflow_sock *subflow;
+	struct mptcp_read_arg arg;
+	read_descriptor_t desc;
+	struct tcp_sock *tp;
+	struct sock *ssk;
+	int copied = 0;
+	long timeo;
 
-	if (msk->connection_list) {
-		subflow = msk->connection_list;
-		pr_debug("conn_list->subflow=%p", subflow->sk);
-	} else {
-		subflow = msk->subflow;
-		pr_debug("subflow=%p", subflow->sk);
+	if (!msk->connection_list) {
+		pr_debug("fallback-read subflow=%p", msk->subflow->sk);
+		return sock_recvmsg(msk->subflow, msg, flags);
 	}
 
-	return sock_recvmsg(subflow, msg, flags);
+	ssk = msk->connection_list->sk;
+	subflow = subflow_sk(ssk);
+	tp = tcp_sk(ssk);
+
+	desc.arg.data = &arg;
+	desc.error = 0;
+
+	timeo = sock_rcvtimeo(sk, nonblock);
+
+	len = min_t(size_t, len, INT_MAX);
+
+	while (copied < len) {
+		size_t discard_len = 0;
+		int bytes_read;
+		u64 ack_seq;
+		u64 old_ack;
+		u32 ssn;
+
+		pr_debug("map_valid=%d", subflow->map_valid);
+		if (!subflow->map_valid && !mptcp_get_mapping(subflow))
+			goto wait_for_data;
+
+		ssn = tcp_sk(ssk)->copied_seq - subflow->ssn_offset;
+		old_ack = atomic64_read(&msk->ack_seq);
+
+		if (unlikely(before(ssn, subflow->map_subflow_seq))) {
+			/* Mapping covers data later in the subflow stream,
+			 * discard unmapped data. */
+			pr_debug("Mapping covers data later in stream");
+			discard_len = subflow->map_subflow_seq - ssn;
+		} else if (unlikely(!before(ssn, (subflow->map_subflow_seq +
+						  subflow->map_dll)))) {
+			/* Mapping ends earlier in the subflow stream.
+			 * Invalidate the mapping and try again.
+			 */
+			subflow->map_valid = false;
+			pr_debug("Invalid mapping ssn=%d map_seq=%d map_dll=%d",
+				 ssn, subflow->map_subflow_seq, subflow->map_dll);
+			continue;
+		} else {
+			ack_seq = get_mapped_dsn(subflow);
+
+			if (before64(ack_seq, old_ack)) {
+				/* Mapping covers data already received,
+				 * discard data in the current mapping
+				 * and invalidate the map
+				 */
+				u64 map_end_dsn = subflow->map_seq +
+					subflow->map_dll;
+				discard_len = min(map_end_dsn - ack_seq,
+						  old_ack - ack_seq);
+				subflow->map_valid = false;
+				pr_debug("Duplicate MPTCP data found");
+			}
+		}
+
+		if (discard_len) {
+			/* Discard data for the current mapping.
+			 */
+			pr_debug("Discard %lu bytes", discard_len);
+
+			arg.msg = NULL;
+			desc.count = discard_len;
+
+			bytes_read = tcp_read_sock(ssk, &desc,
+						   mptcp_read_actor);
+
+			if (bytes_read < 0)
+				break;
+			else if (bytes_read == discard_len)
+				continue;
+			else
+				goto wait_for_data;
+		}
+
+		/* Read mapped data */
+		desc.count = ssn - subflow->map_subflow_seq + subflow->map_dll;
+		arg.msg = msg;
+		bytes_read = tcp_read_sock(ssk, &desc, mptcp_read_actor);
+		if (bytes_read < 0)
+			break;
+
+		/* Refresh current MPTCP sequence number based on subflow seq */
+		ack_seq = get_mapped_dsn(subflow);
+
+		if (before64(old_ack, ack_seq)) {
+			atomic64_set(&msk->ack_seq, ack_seq);
+		}
+
+		if (!before(tcp_sk(ssk)->copied_seq - subflow->ssn_offset,
+			    subflow->map_subflow_seq + subflow->map_dll)) {
+			subflow->map_valid = false;
+			pr_debug("Done with mapping: seq=%u dll=%u",
+				 subflow->map_subflow_seq, subflow->map_dll);
+		}
+
+		copied += bytes_read;
+
+wait_for_data:
+		if (copied)
+			break;
+
+		if (tp->urg_data && tp->urg_seq == tp->copied_seq) {
+			pr_err("Urgent data present, cannot proceed");
+			break;
+		}
+
+		if (ssk->sk_err || ssk->sk_state == TCP_CLOSE ||
+		    (ssk->sk_shutdown & RCV_SHUTDOWN) || !timeo ||
+		    signal_pending(current)) {
+			pr_debug("nonblock or error");
+			break;
+		}
+
+		/* Handle blocking and retry read if needed.
+		 *
+		 * Wait on MPTCP sock, the subflow will notify via data ready.
+		 */
+
+		pr_debug("block");
+		release_sock(ssk);
+		sk_wait_data(sk, &timeo, NULL);
+		lock_sock(ssk);
+
+	}
+
+	release_sock(ssk);
+	release_sock(sk);
+
+	return copied;
 }
 
 static int mptcp_init_sock(struct sock *sk)
@@ -240,18 +486,27 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	subflow->conn = mp->sk;
 
 	if (subflow->mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
 		token_update_accept(new_sock->sk, mp->sk);
+		msk->connection_list = new_sock;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
 		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
 		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
-		msk->connection_list = new_sock;
+		subflow->conn = mp->sk;
 	} else {
 		msk->subflow = new_sock;
 	}
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 
 	return mp->sk;
 }
@@ -329,17 +584,24 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	pr_debug("msk=%p", msk);
 
 	if (mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
-		msk->write_seq = subflow->idsn + 1;
-		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
+		pr_debug("token=%u", msk->token);
 		msk->connection_list = msk->subflow;
 		msk->subflow = NULL;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
+		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
+		subflow->rel_write_seq = 1;
 	}
-	sk->sk_state = TCP_ESTABLISHED;
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 }
 
 static int subflow_create(struct sock *sock)
net/mptcp/subflow.c +27 -3
index e0b4aa8fe51d..3c9dca4be26a 100644
@@ -107,6 +107,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
 		pr_debug("remote_key=%llu", subflow_req->remote_key);
 		token_new_request(req, skb);
+		pr_debug("syn seq=%u", TCP_SKB_CB(skb)->seq);
+		subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -120,10 +122,15 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 
 	pr_debug("subflow=%p", subflow);
 
-	if (subflow->conn) {
+	if (!subflow->conn_finished) {
 		pr_debug("remote_key=%llu", subflow->remote_key);
 		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
-		subflow->conn = NULL;
+		subflow->conn_finished = 1;
+
+		if (skb) {
+			pr_debug("synack seq=%u", TCP_SKB_CB(skb)->seq);
+			subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
+		}
 	}
 }
 
@@ -172,7 +179,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->ssn_offset = subflow_req->ssn_offset;
 			subflow->token = subflow_req->token;
+			subflow->idsn = subflow_req->idsn;
 			pr_debug("token=%u", subflow->token);
 			token_new_accept(child);
 		} else {
@@ -202,6 +211,20 @@ const struct inet_connection_sock_af_ops subflow_specific = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
+static void subflow_data_ready(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *parent = subflow->conn;
+
+	pr_debug("sk=%p", sk);
+	subflow->tcp_sk_data_ready(sk);
+
+	if (parent) {
+		pr_debug("parent=%p", parent);
+		parent->sk_data_ready(parent);
+	}
+}
+
 static int subflow_init_sock(struct sock *sk)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -215,6 +238,8 @@ static int subflow_init_sock(struct sock *sk)
 	if (!err) { // @@ AND mptcp is enabled
 		tsk->is_mptcp = 1;
 		icsk->icsk_af_ops = &subflow_specific;
+		subflow->tcp_sk_data_ready = sk->sk_data_ready;
+		sk->sk_data_ready = subflow_data_ready;
 	}
 
 	return err;
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Quoted 4 lines
Quoted 3 lines

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC

#50 lore raw

Re: [MPTCP] [RFC PATCH v3 15/16] mptcp: Implement MPTCP receive path

Quoted 79 lines · Mat Martineau
Quoted 77 lines · Christoph Paasch
Quoted 47 lines · Mat Martineau
Quoted 7 lines · Christoph Paasch
Quoted 5 lines · Mat Martineau

Parses incoming DSS options and populates outgoing MPTCP ACK fields. Clone incoming skbs containing MPTCP headers and propagate them to the MMPTCP socket using the error queue. This avoids the need for TCP coalesce/collapse bypass for MPTCP, although cloning incoming skbs does prevent some coalescing.

Quoted 5 lines

I think it's acceptable to bypass coalesce/collapse for MPTCP instead of cloning every incoming packet over to the error-queue.

The way this is now, makes the error-queue quite loaded, and I don't think it handles packet-reordering correctly.

While the commit message says skbs with MPTCP headers are cloned, the code clones all incoming skbs right now. Maybe most data packets end up having DSS headers anyway.

I agree that the current method of reading the DSS information is fragile with respect to out-of-order packet arrival.

I'm trying to get away from the previous approach of preventing coalesce/collapse and depending on intact headers in the skb. That technique relies on the integrity of data between sk_buff->head and sk_buff->data, when sk_buff->data is is considered to be the beginning of valid data.

What do you think about using the parallel queue approach once this implementation is further developed (and before you put a lot of thought in to it, read the rest of this reply for an alternative):

* Only packets with DSS headers are cloned

* The DSS mapping queue is emptied on every read, building a data structure
to represent the map for all the queued data on that subflow (deals with ordering issue)

with possible optimization:

* Rather than copy the entire skb, keep a pool of small skbs around and
only copy the TCP header

Quoted 7 lines

I tried out using the error queue since it has some similar per-packet use for tx timestamping, but the error queue does take a spinlock for every enqueue/dequeue. Might make more sense to add a lockless linked list to the subflow_sock for this purpose - since my intent is to empty that list every time it's read from, llist_del_all() to pop off all the entries at once is a nice fit. It would also allow use of a much smaller data structure to hold the mapping data.

Did you consider using the priv-pointer in the skb to take care of this instead?

The problem here is that the DSS-information is tightly linked to a byte in the payload. When we start sending this DSS-information asynchronously over to the meta-socket (which is what the error-queue or another queue is doing), we start making things very complex (e.g., need to keep ordering right,...).

E.g., in skb_try_coalesce, we can simply do a check on the priv-used field. And then decide based on the DSS-mapping (or do that in tcp_try_coalesce).

When coalescing is happening, the issue is that we anyways need to "de-coalesce" after if non-continuous mappings have been coalesced. This is another reason why it's good to do a "smart" coalescing that takes DSS-mappings into account. priv_used is the perfect infrastructure for that.

Doing it that way will be much less error-prone.

Quoted 6 lines
Quoted 1 line

The parallel queue is great for asynchronous signals like ADD_ADDR,...

It depends on whether you want to read from a subflow and then figure out where the payload belongs in the MPTCP stream, or if you want to have the mapping info for all the subflows available before reading in order to decide which subflow to look at next.

I see the difference. But, out-of-order handling would still need to be managed at the MPTCP-layer.

Quoted 3 lines

I think there's something to be said for leaving the data in the subflow rx queue until it's requested by userspace. That would let TCP handle a lot of the buffering in a more normal (for TCP) way.

What do you mean by that? I'm not sure what you mean by "buffering".

I am actually thinking that we can run into a deadlock when we leave things in the subflows. Because, how would you handle the receive-window announcment and the checks for incoming packets whether they are within the bounds of the window (at the subflow as well as the MPTCP-layer) ?

I believe, a subflow can fill up entirely and literally prevent the connection from making progress because the upper layer is not reading out of the subflow (waiting for a missing mapping).
As the subflow is full, the sender is not able to send the missing DSS-mapping which would trigger the purge of the subflow at the receiver-side.

Reading out of the subflow early allows to free up things at the subflow-window level and actually drop the data that has been filling up the subflow uselessly.

Quoted 16 lines

While the MPTCP layer would
have to handle some out-of-order reassembly and buffering, I think the common case would be to pass reads through to the appropriate subflow. MPTCP-layer reassembly and buffering would be the uncommon case (I assume...) and wouldn't need to be super optimized.

Speaking of assumptions, I should be more specific about what I expect from normal traffic patterns. For the mobile connectivity use case, I'd expect one subflow to be receiving all the incoming packets in-order until there was a change in the available interfaces, when the subflow in use might change. For the hybrid access use case, I would expect increasing sequence numbers (with wrap-around) on all subflows. In both cases, a jump backwards in the sequence numbers must be handled but is unlikely, and no buffering is required at the MPTCP layer. For hybrid access, knowing the sequence numbers prior to reading from the subflow seems like a win.

Quoted 2 lines

Does this fit with what you (or researchers) have seen from MPTCP in practice?

I think the assumptions are valid. Nevertheless, the un-common case still needs to be handled correctly and needs to be implemented in a functional implementation.

Christoph

Quoted 972 lines

Thanks,

Mat

Quoted 958 lines
Quoted 958 lines

Thanks,
Mat

Quoted 951 lines
Quoted 950 lines

Keeping DSS map handling separate from data reading makes it easier to add multiple-subflow receiving, since the receive code can determine which subflow contains relevant data or data that needs to be purged without touching the subflow data queues.

MPTCP ACK values are now populated from an atomic value stored in the connection socket rather than carried in the extended control block, which makes use of the most up-to-date sequence number and allows the MPTCP ACK to be populated on TCP ACK packets that have no payload.

Signed-off-by Mat Martineau <mathew.j.martineau(a)linux.intel.com>
---
 include/linux/tcp.h           |  12 ++
 include/net/mptcp.h           |  41 +++--
 include/uapi/linux/errqueue.h |   1 +
 net/ipv4/tcp_input.c          |   3 +
 net/ipv4/tcp_output.c         | 123 +++++++-------
 net/mptcp/options.c           |  90 +++++++++-
 net/mptcp/protocol.c          | 304 +++++++++++++++++++++++++++++++---
 net/mptcp/subflow.c           |  29 +++-
 8 files changed, 508 insertions(+), 95 deletions(-)
include/linux/tcp.h +12 -0
index b109798482d3..845da5ee9c44 100644
@@ -112,6 +112,18 @@ struct tcp_options_received {
 		u8      flags;
 		u64     sndr_key;
 		u64     rcvr_key;
+		u64	ack;
+		u64	seq;
+		u32	subflow_seq;
+		u16	dll;
+		u16	checksum;
+		u8	use_ack:1,
+			ack64:1,
+			use_map:1,
+			dsn64:1,
+			use_checksum:1,
+			data_fin:1,
+			__unused:2;
 	} mptcp;
 
 };
include/net/mptcp.h +29 -12
index 73f2d25cc42d..6c9516360de4 100644
@@ -38,14 +38,14 @@
 /* MPTCP connection sock */
 struct mptcp_sock {
 	/* inet_connection_sock must be the first member */
-	struct	inet_connection_sock sk;
-	u64	local_key;
-	u64	remote_key;
-	u64	write_seq;
-	u64	ack_seq;
-	u32	token;
-	struct	socket *connection_list; /* @@ needs to be a list */
-	struct	socket *subflow; /* outgoing connect, listener or !mp_capable */
+	struct inet_connection_sock sk;
+	u64		local_key;
+	u64		remote_key;
+	u64		write_seq;
+	atomic64_t	ack_seq;
+	u32		token;
+	struct socket	*connection_list; /* @@ needs to be a list */
+	struct socket	*subflow; /* outgoing connect, listener or !mp_capable */
 };
 
 static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
@@ -56,7 +56,6 @@ static inline struct mptcp_sock *mptcp_sk(const struct sock *sk)
 /* MPTCP sk_buff private control buffer */
 struct mptcp_skb_cb {
 	refcount_t	refcnt;
-	u64		data_ack;
 	u64		data_seq;
 	u32		subflow_seq;
 	u16		dll;
@@ -65,9 +64,7 @@ struct mptcp_skb_cb {
 			dsn64:1,
 			use_checksum:1,
 			data_fin:1,
-			use_ack:1,
-			ack64:1,
-			__unused:2;
+			__unused:4;
 };
 
 static inline struct mptcp_skb_cb *mptcp_skb_priv_cb(struct sk_buff *skb)
@@ -81,16 +78,23 @@ struct subflow_sock {
 	/* tcp_sock must be the first member */
 	struct	tcp_sock sk;
 	u64	local_key;
+	u64	map_seq;
+	u32	map_subflow_seq;
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
 	u32	rel_write_seq;
+	u32	ssn_offset;
+	u16	map_dll;
 	bool	request_mptcp;	// send MP_CAPABLE
 	bool	checksum;
 	bool	version;
 	bool	mp_capable;	// remote is MPTCP capable
 	bool	fourth_ack;	// send initial DSS
+	bool	conn_finished;
+	bool	map_valid;
 	struct	sock *conn;	// parent mptcp_sock
+	void	(*tcp_sk_data_ready)(struct sock *sk);
 };
 
 static inline struct subflow_sock *subflow_sk(const struct sock *sk)
@@ -98,6 +102,16 @@ static inline struct subflow_sock *subflow_sk(const struct sock *sk)
 	return (struct subflow_sock *)sk;
 }
 
+static inline struct subflow_sock *subflow_tp(const struct tcp_sock *tp)
+{
+	return (struct subflow_sock *)tp;
+}
+
+static inline struct sock *sock_sk(const struct subflow_sock *sk)
+{
+	return (struct sock *)sk;
+}
+
 struct subflow_request_sock {
 	struct	tcp_request_sock sk;
 	u8	mp_capable : 1,
@@ -109,6 +123,7 @@ struct subflow_request_sock {
 	u32	token;
 	u64	idsn;
 	u64	remote_key;
+	u32	ssn_offset;
 };
 
 static inline
@@ -133,6 +148,8 @@ void mptcp_subflow_exit(void);
 void mptcp_get_options(const struct sk_buff *skb,
 		       struct tcp_options_received *options);
 
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb);
+
 extern const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops;
 
 void token_init(void);
include/uapi/linux/errqueue.h +1 -0
index c0151200f7d1..1a1ec7362a55 100644
@@ -21,6 +21,7 @@ struct sock_extended_err {
 #define SO_EE_ORIGIN_TXSTATUS	4
 #define SO_EE_ORIGIN_ZEROCOPY	5
 #define SO_EE_ORIGIN_TXTIME	6
+#define SO_EE_ORIGIN_MPTCP	7
 #define SO_EE_ORIGIN_TIMESTAMPING SO_EE_ORIGIN_TXSTATUS
 
 #define SO_EE_OFFENDER(ee)	((struct sockaddr*)((ee)+1))
net/ipv4/tcp_input.c +3 -0
index 9a326729637f..0904d9baa685 100644
@@ -5642,6 +5642,9 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	tcp_urg(sk, skb, th);
 
 	/* step 7: process the segment text */
+	if (tcp_sk(sk)->is_mptcp)
+		mptcp_queue_headers(sk, skb);
+
 	tcp_data_queue(sk, skb);
 
 	tcp_data_snd_check(sk);
net/ipv4/tcp_output.c +65 -58
index 471f196711fc..1c9517db82c2 100644
@@ -422,8 +422,8 @@ static inline bool tcp_urg_mode(const struct tcp_sock *tp)
 #define OPTION_MPTCP_MPC_SYN	(1 << 0)
 #define OPTION_MPTCP_MPC_SYNACK	(1 << 1)
 #define OPTION_MPTCP_MPC_ACK	(1 << 2)
-#define OPTION_MPTCP_MPC_DSS_MAP	(1 << 6)
-#define OPTION_MPTCP_MPC_DSS_ACK	(1 << 7)
+#define OPTION_MPTCP_DSS_MAP	(1 << 6)
+#define OPTION_MPTCP_DSS_ACK	(1 << 7)
 
 struct tcp_out_options {
 	u16 options;		/* bit field of OPTION_* */
@@ -472,16 +472,12 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 	}
 
-	if ((OPTION_MPTCP_MPC_DSS_MAP |
-	     OPTION_MPTCP_MPC_DSS_ACK) & opts->suboptions) {
-		struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
-		bool write_ack = (OPTION_MPTCP_MPC_DSS_ACK & opts->suboptions) && skb->priv_used && mcb->use_ack;
-		bool write_map = (OPTION_MPTCP_MPC_DSS_MAP & opts->suboptions) && skb->priv_used && mcb->use_map;
+	if ((OPTION_MPTCP_DSS_MAP | OPTION_MPTCP_DSS_ACK) & opts->suboptions) {
+		bool write_ack = !!(OPTION_MPTCP_DSS_ACK & opts->suboptions);
+		bool write_map = (OPTION_MPTCP_DSS_MAP & opts->suboptions) &&
+			skb->priv_used && mptcp_skb_priv_cb(skb)->use_map;
 		u8 flags = 0;
 		u8 len = 4;
-		u8 *p = (u8 *)ptr;
-
-		BUG_ON(!write_ack && !write_map);
 
 		if (write_ack) {
 			len += 8;
@@ -489,46 +485,60 @@ static void mptcp_options_write(__be32 *ptr, struct sk_buff *skb,
 		}
 
 		if (write_map) {
+			pr_debug("Updating DSS length and flags for map");
 			len += 14;
 
-			if (mcb->use_checksum)
+			if (mptcp_skb_priv_cb(skb)->use_checksum)
 				len += 2;
 
 			/* Use only 64-bit mapping flags for now, add
 			 * support for optional 32-bit mappings later.
 			 */
 			flags |= 0x0c;
-			if (mcb->data_fin)
+			if (mptcp_skb_priv_cb(skb)->data_fin)
 				flags |= 0x10;
 		}
 
-		*p++ = 0x1e; // TCP option: Multipath TCP
-		*p++ = len;  // length
-		*p++ = 0x20; // subtype=DSS
-		*p++ = flags;
+		*ptr++ = htonl((0x1e << 24) |  // TCP option: Multipath TCP
+			       (len  << 16) |  // length
+			       (0x20 <<  8) |  // subtype=DSS
+			       (flags));
 
 		if (write_ack) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_ack);
-			p += 8;
+			struct mptcp_sock *msk = mptcp_sk(subflow_tp(tp)->conn);
+			u64 ack_seq;
+			__be64 ack;
+
+			if (msk) {
+				ack_seq = atomic64_read(&msk->ack_seq);
+			} else {
+				crypto_key_sha1(subflow_tp(tp)->remote_key,
+						NULL, &ack_seq);
+				ack_seq++;
+			}
+
+			pr_debug("ack=%llu", ack_seq);
+			ack = cpu_to_be64(ack_seq);
+			memcpy((u8 *) ptr, (u8 *) &ack, 8);
+			ptr += 2;
 		}
 
 		if (write_map) {
-			*(__be64 *)p = cpu_to_be64(mcb->data_seq);
-			p += 8;
-
-			*(__be32 *)p = htonl(mcb->subflow_seq);
-			p += 4;
+			struct mptcp_skb_cb *mcb = mptcp_skb_priv_cb(skb);
+			u16 checksum;
+			__be64 dss;
 
-			*(__be16 *)p = htons(mcb->dll);
-			p += 2;
+			pr_debug("Writing map values");
+			dss = cpu_to_be64(mcb->data_seq);
+			memcpy((u8 *) ptr, &dss, 8);
+			ptr += 2;
+			*ptr++ = htonl(mcb->subflow_seq);
 
-			if (mcb->use_checksum) {
-				*(__be16 *)p = htons(mcb->checksum);
-				p += 2;
-			} else {
-				*p++ = TCPOPT_NOP;
-				*p++ = TCPOPT_NOP;
-			}
+			if (mcb->use_checksum)
+				checksum = (__force u16) mcb->checksum;
+			else
+				checksum = TCPOPT_NOP << 8 | TCPOPT_NOP;
+			*ptr++ = htonl(mcb->dll << 16 | checksum);
 		}
 	}
 }
@@ -899,50 +909,44 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
 					size += 20;
 				}
 				subflow->fourth_ack = 1;
-			} else if (skb && skb->priv_used) {
-				struct mptcp_skb_cb *cb;
+			} else if (skb) {
 				unsigned int dss_size = 0;
-				u16 options = 0;
+				unsigned int ack_size = 8;
+				u16 suboptions = 0;
 
-				cb = mptcp_skb_priv_cb(skb);
-
-				if (skb->priv_used && cb->use_map) {
+				if (skb->priv_used &&
+				    mptcp_skb_priv_cb(skb)->use_map) {
 					unsigned int map_size = 18;
 
-					if (cb->use_checksum)
+					if (mptcp_skb_priv_cb(skb)->use_checksum)
 						map_size += 2;
 
 					if (map_size <= remaining) {
 						remaining -= map_size;
 						dss_size = map_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions = OPTION_MPTCP_MPC_DSS_MAP;
+						suboptions = OPTION_MPTCP_DSS_MAP;
 					} else {
 						WARN(1, "MPTCP: Map dropped");
 					}
 				}
 
-				if (cb->use_ack) {
-					unsigned int ack_size = 8;
-
-					/* Add kind/length/subtype/flag
-					 * overhead if mapping not populated
-					 */
-					if (dss_size == 0)
-						ack_size += 4;
-
-					if (ack_size <= remaining) {
-						dss_size += ack_size;
-						opts->options |= OPTION_MPTCP;
-						opts->suboptions |= OPTION_MPTCP_MPC_DSS_ACK;
-					} else {
-						WARN(1, "MPTCP: Ack dropped");
-					}
+				/* Add kind/length/subtype/flag
+				 * overhead if mapping not populated
+				 */
+				if (dss_size == 0)
+					ack_size += 4;
+
+				if (ack_size <= remaining) {
+					dss_size += ack_size;
+					suboptions |= OPTION_MPTCP_DSS_ACK;
+				} else {
+					WARN(1, "MPTCP: Ack dropped");
 				}
 
 				if (dss_size) {
 					size += ALIGN(dss_size, 4);
-					opts->options |= options;
+					opts->options |= OPTION_MPTCP;
+					opts->suboptions = suboptions;
 				}
 			}
 		}
@@ -3804,6 +3808,9 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt)
 	skb_set_tcp_pure_ack(buff);
 
 	/* Send it off, this clears delayed acks for us. */
+	if (tcp_sk(sk)->is_mptcp)
+		pr_debug("mptcp sk=%p", sk);
+
 	__tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0, rcv_nxt);
 }
 EXPORT_SYMBOL_GPL(__tcp_send_ack);
net/mptcp/options.c +88 -2
index 7e48d1d92aac..f5b7be888e84 100644
@@ -14,6 +14,7 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/errqueue.h>
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
@@ -21,9 +22,8 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 			struct tcp_options_received *opt_rx)
 {
 	u8 subtype;
-	u64 *p;
+	int expected_opsize;
 
-	opsize -= 2;
 	subtype = *ptr++;
 
 	/* MPTCPOPT_MP_CAPABLE
@@ -82,6 +82,68 @@ void mptcp_parse_option(const unsigned char *ptr, int opsize,
 	case 0x20:
 		pr_debug("DSS");
 		opt_rx->mptcp.dss = 1;
+
+		opt_rx->mptcp.flags = (*ptr++) & 0x1F;
+		opt_rx->mptcp.data_fin = (opt_rx->mptcp.flags & 0x10) != 0;
+		opt_rx->mptcp.dsn64 = (opt_rx->mptcp.flags & 0x08) != 0;
+		opt_rx->mptcp.use_map = (opt_rx->mptcp.flags & 0x04) != 0;
+		opt_rx->mptcp.ack64 = (opt_rx->mptcp.flags & 0x02) != 0;
+		opt_rx->mptcp.use_ack = (opt_rx->mptcp.flags & 0x01);
+
+		pr_debug("data_fin=%d dsn64=%d use_map=%d ack64=%d use_ack=%d",
+			 opt_rx->mptcp.data_fin, opt_rx->mptcp.dsn64,
+			 opt_rx->mptcp.use_map, opt_rx->mptcp.ack64,
+			 opt_rx->mptcp.use_ack);
+
+		expected_opsize = 0;
+
+		if (opt_rx->mptcp.use_ack) {
+			expected_opsize = 4;
+			if (opt_rx->mptcp.ack64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.ack64) {
+				opt_rx->mptcp.ack = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.ack = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			pr_debug("ack=%llu", opt_rx->mptcp.ack);
+		}
+
+		if (opt_rx->mptcp.use_map) {
+			expected_opsize += 12;
+			if (opt_rx->mptcp.dsn64)
+				expected_opsize += 4;
+
+			if (opsize < expected_opsize)
+				break;
+
+			if (opt_rx->mptcp.dsn64) {
+				opt_rx->mptcp.seq = get_unaligned_be64(ptr);
+				ptr += 8;
+			} else {
+				opt_rx->mptcp.seq = get_unaligned_be32(ptr);
+				ptr += 4;
+			}
+
+			opt_rx->mptcp.subflow_seq = get_unaligned_be32(ptr);
+			ptr += 4;
+
+			opt_rx->mptcp.dll = get_unaligned_be16(ptr);
+			ptr += 2;
+
+			opt_rx->mptcp.checksum = get_unaligned_be16(ptr);
+
+			pr_debug("seq=%llu subflow_seq=%u dll=%u ck=%u",
+				 opt_rx->mptcp.seq, opt_rx->mptcp.subflow_seq,
+				 opt_rx->mptcp.dll, opt_rx->mptcp.checksum);
+		}
 		break;
 
 	/* MPTCPOPT_ADD_ADDR
@@ -175,3 +237,27 @@ unsigned int mptcp_synack_options(struct request_sock *req, u64 *local_key,
 	}
 	return subflow_req->mp_capable;
 }
+
+void mptcp_queue_headers(struct sock *sk, struct sk_buff *original_skb)
+{
+	struct sock_exterr_skb *serr;
+	struct sk_buff *skb;
+	int err;
+
+	BUILD_BUG_ON(sizeof(struct sock_exterr_skb) > sizeof(skb->cb));
+
+	skb = skb_clone(original_skb, GFP_ATOMIC);
+	if (!skb) {
+		pr_err("clone failed skb=%p sk=%p", original_skb, sk);
+		return;
+	}
+
+	serr = SKB_EXT_ERR(skb);
+	memset(serr, 0, sizeof(*serr));
+	serr->ee.ee_origin = SO_EE_ORIGIN_MPTCP;
+
+	err = sock_queue_err_skb(sk, skb);
+
+	if (err)
+		kfree_skb(skb);
+}
net/mptcp/protocol.c +283 -21
index dcaaa0dd3784..911428122c9c 100644
@@ -16,6 +16,8 @@
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/netdevice.h>
+#include <linux/sched/signal.h>
+#include <linux/atomic.h>
 #include <net/sock.h>
 #include <net/inet_common.h>
 #include <net/inet_hashtables.h>
@@ -23,6 +25,12 @@
 #include <net/tcp.h>
 #include <net/mptcp.h>
 
+static inline bool before64(__u64 seq1, __u64 seq2)
+{
+	return (__s64)(seq1-seq2) < 0;
+}
+#define after64(seq2, seq1)	before64(seq1, seq2)
+
 static void mptcp_cb_copy(const struct sk_buff *from, struct sk_buff *to)
 {
 	struct mptcp_skb_cb *mcb = from->priv;
@@ -131,15 +139,12 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	skb = tcp_write_queue_tail(ssk);
 
 	refcount_set(&mcb->refcnt, 1);
-	mcb->data_ack = msk->ack_seq;
 	mcb->data_seq = msk->write_seq;
 	mcb->subflow_seq = subflow_sk(ssk)->rel_write_seq;
 	mcb->dll = ret;
 	mcb->checksum = 0xbeef;
 	mcb->use_map = 1;
 	mcb->dsn64 = 1;
-	mcb->use_ack = 1;
-	mcb->ack64 = 1;
 
 	if (mcb->use_map) {
 		pr_debug("data_seq=%llu subflow_seq=%u "
@@ -170,21 +175,262 @@ static int mptcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
 	return ret;
 }
 
+struct mptcp_read_arg {
+	struct msghdr *msg;
+};
+
+static u64 expand_seq(u64 old_seq, u16 old_dll, u64 seq)
+{
+	if ((u32)seq == (u32)old_seq)
+		return old_seq;
+
+	/* Assume map covers data not mapped yet. */
+	return seq | ((old_seq + old_dll + 1) & ~0xFFFFFFFFULL);
+}
+
+static u64 get_mapped_dsn(struct subflow_sock *subflow)
+{
+	u32 map_offset = (tcp_sk(sock_sk(subflow))->copied_seq -
+			  subflow->ssn_offset -
+			  subflow->map_subflow_seq);
+
+	return subflow->map_seq + map_offset;
+}
+
+static int mptcp_read_actor(read_descriptor_t *desc, struct sk_buff *skb,
+			    unsigned int offset, size_t len)
+{
+	struct mptcp_read_arg *arg = desc->arg.data;
+	size_t copy_len;
+
+	copy_len = min(desc->count, len);
+
+	if (likely(arg->msg)) {
+		int err;
+
+		err = skb_copy_datagram_msg(skb, offset, arg->msg, copy_len);
+		if (err) {
+			pr_debug("error path");
+			desc->error = err;
+			return err;
+		}
+	} else {
+		pr_debug("Flushing skb payload");
+	}
+
+	// MSG_PEEK support? Other flags? MSG_TRUNC?
+
+	desc->count -= copy_len;
+
+	pr_debug("consumed %lu bytes, %lu left", copy_len, desc->count);
+	return copy_len;
+}
+
+static bool mptcp_get_mapping(struct subflow_sock *subflow)
+{
+	struct tcp_options_received opt;
+	struct sk_buff *skb;
+	bool found_map = false;
+
+	do {
+		pr_debug("Dequeue mapping skb");
+		skb = sock_dequeue_err_skb(sock_sk(subflow));
+		if (!skb) {
+			pr_debug("Empty queue");
+			break;
+		}
+
+		opt.mptcp.dss = 0;
+		opt.mptcp.use_map = 0;
+		opt.mptcp.data_fin = 0;
+		mptcp_get_options(skb, &opt);
+
+		if (opt.mptcp.dss && opt.mptcp.use_map) {
+			pr_debug("seq=%llu is64=%d ssn=%u dll=%u ck=%u",
+				 opt.mptcp.seq, opt.mptcp.dsn64,
+				 opt.mptcp.subflow_seq,
+				 opt.mptcp.dll, opt.mptcp.checksum);
+
+			if (opt.mptcp.dll == 0) {
+				pr_err("Infinite mapping not handled");
+			} else if (opt.mptcp.subflow_seq == 0 &&
+				   opt.mptcp.data_fin == 1) {
+				pr_debug("DATA_FIN with no payload");
+				kfree_skb(skb);
+				break;
+			}
+
+			if (!opt.mptcp.dsn64) {
+				subflow->map_seq = expand_seq(subflow->map_seq,
+							      subflow->map_dll,
+							      opt.mptcp.seq);
+				pr_debug("expanded seq=%llu", subflow->map_seq);
+			} else {
+				subflow->map_seq = opt.mptcp.seq;
+			}
+
+			subflow->map_subflow_seq = opt.mptcp.subflow_seq;
+			subflow->map_dll = opt.mptcp.dll;
+			subflow->map_valid = true;
+			pr_debug("new map seq=%llu subflow_seq=%u dll=%u",
+				 subflow->map_seq, subflow->map_subflow_seq,
+				 subflow->map_dll);
+			found_map = true;
+		}
+
+		kfree_skb(skb);
+	} while (!found_map);
+
+	return found_map;
+}
+
 static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
 			 int nonblock, int flags, int *addr_len)
 {
 	struct mptcp_sock *msk = mptcp_sk(sk);
-	struct socket *subflow;
+	struct subflow_sock *subflow;
+	struct mptcp_read_arg arg;
+	read_descriptor_t desc;
+	struct tcp_sock *tp;
+	struct sock *ssk;
+	int copied = 0;
+	long timeo;
 
-	if (msk->connection_list) {
-		subflow = msk->connection_list;
-		pr_debug("conn_list->subflow=%p", subflow->sk);
-	} else {
-		subflow = msk->subflow;
-		pr_debug("subflow=%p", subflow->sk);
+	if (!msk->connection_list) {
+		pr_debug("fallback-read subflow=%p", msk->subflow->sk);
+		return sock_recvmsg(msk->subflow, msg, flags);
 	}
 
-	return sock_recvmsg(subflow, msg, flags);
+	ssk = msk->connection_list->sk;
+	subflow = subflow_sk(ssk);
+	tp = tcp_sk(ssk);
+
+	desc.arg.data = &arg;
+	desc.error = 0;
+
+	timeo = sock_rcvtimeo(sk, nonblock);
+
+	len = min_t(size_t, len, INT_MAX);
+
+	while (copied < len) {
+		size_t discard_len = 0;
+		int bytes_read;
+		u64 ack_seq;
+		u64 old_ack;
+		u32 ssn;
+
+		pr_debug("map_valid=%d", subflow->map_valid);
+		if (!subflow->map_valid && !mptcp_get_mapping(subflow))
+			goto wait_for_data;
+
+		ssn = tcp_sk(ssk)->copied_seq - subflow->ssn_offset;
+		old_ack = atomic64_read(&msk->ack_seq);
+
+		if (unlikely(before(ssn, subflow->map_subflow_seq))) {
+			/* Mapping covers data later in the subflow stream,
+			 * discard unmapped data. */
+			pr_debug("Mapping covers data later in stream");
+			discard_len = subflow->map_subflow_seq - ssn;
+		} else if (unlikely(!before(ssn, (subflow->map_subflow_seq +
+						  subflow->map_dll)))) {
+			/* Mapping ends earlier in the subflow stream.
+			 * Invalidate the mapping and try again.
+			 */
+			subflow->map_valid = false;
+			pr_debug("Invalid mapping ssn=%d map_seq=%d map_dll=%d",
+				 ssn, subflow->map_subflow_seq, subflow->map_dll);
+			continue;
+		} else {
+			ack_seq = get_mapped_dsn(subflow);
+
+			if (before64(ack_seq, old_ack)) {
+				/* Mapping covers data already received,
+				 * discard data in the current mapping
+				 * and invalidate the map
+				 */
+				u64 map_end_dsn = subflow->map_seq +
+					subflow->map_dll;
+				discard_len = min(map_end_dsn - ack_seq,
+						  old_ack - ack_seq);
+				subflow->map_valid = false;
+				pr_debug("Duplicate MPTCP data found");
+			}
+		}
+
+		if (discard_len) {
+			/* Discard data for the current mapping.
+			 */
+			pr_debug("Discard %lu bytes", discard_len);
+
+			arg.msg = NULL;
+			desc.count = discard_len;
+
+			bytes_read = tcp_read_sock(ssk, &desc,
+						   mptcp_read_actor);
+
+			if (bytes_read < 0)
+				break;
+			else if (bytes_read == discard_len)
+				continue;
+			else
+				goto wait_for_data;
+		}
+
+		/* Read mapped data */
+		desc.count = ssn - subflow->map_subflow_seq + subflow->map_dll;
+		arg.msg = msg;
+		bytes_read = tcp_read_sock(ssk, &desc, mptcp_read_actor);
+		if (bytes_read < 0)
+			break;
+
+		/* Refresh current MPTCP sequence number based on subflow seq */
+		ack_seq = get_mapped_dsn(subflow);
+
+		if (before64(old_ack, ack_seq)) {
+			atomic64_set(&msk->ack_seq, ack_seq);
+		}
+
+		if (!before(tcp_sk(ssk)->copied_seq - subflow->ssn_offset,
+			    subflow->map_subflow_seq + subflow->map_dll)) {
+			subflow->map_valid = false;
+			pr_debug("Done with mapping: seq=%u dll=%u",
+				 subflow->map_subflow_seq, subflow->map_dll);
+		}
+
+		copied += bytes_read;
+
+wait_for_data:
+		if (copied)
+			break;
+
+		if (tp->urg_data && tp->urg_seq == tp->copied_seq) {
+			pr_err("Urgent data present, cannot proceed");
+			break;
+		}
+
+		if (ssk->sk_err || ssk->sk_state == TCP_CLOSE ||
+		    (ssk->sk_shutdown & RCV_SHUTDOWN) || !timeo ||
+		    signal_pending(current)) {
+			pr_debug("nonblock or error");
+			break;
+		}
+
+		/* Handle blocking and retry read if needed.
+		 *
+		 * Wait on MPTCP sock, the subflow will notify via data ready.
+		 */
+
+		pr_debug("block");
+		release_sock(ssk);
+		sk_wait_data(sk, &timeo, NULL);
+		lock_sock(ssk);
+
+	}
+
+	release_sock(ssk);
+	release_sock(sk);
+
+	return copied;
 }
 
 static int mptcp_init_sock(struct sock *sk)
@@ -240,18 +486,27 @@ static struct sock *mptcp_accept(struct sock *sk, int flags, int *err,
 	subflow->conn = mp->sk;
 
 	if (subflow->mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
+		pr_debug("token=%u", msk->token);
 		token_update_accept(new_sock->sk, mp->sk);
+		msk->connection_list = new_sock;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
 		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
 		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
-		msk->connection_list = new_sock;
+		subflow->conn = mp->sk;
 	} else {
 		msk->subflow = new_sock;
 	}
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 
 	return mp->sk;
 }
@@ -329,17 +584,24 @@ void mptcp_finish_connect(struct sock *sk, int mp_capable)
 	pr_debug("msk=%p", msk);
 
 	if (mp_capable) {
+		u64 ack_seq;
+
+		msk->remote_key = subflow->remote_key;
 		msk->local_key = subflow->local_key;
 		msk->token = subflow->token;
-		msk->write_seq = subflow->idsn + 1;
-		subflow->rel_write_seq = 1;
-		msk->remote_key = subflow->remote_key;
-		crypto_key_sha1(msk->remote_key, NULL, &msk->ack_seq);
-		msk->ack_seq++;
+		pr_debug("token=%u", msk->token);
 		msk->connection_list = msk->subflow;
 		msk->subflow = NULL;
+
+		crypto_key_sha1(msk->remote_key, NULL, &ack_seq);
+		msk->write_seq = subflow->idsn + 1;
+		ack_seq++;
+		atomic64_set(&msk->ack_seq, ack_seq);
+		subflow->map_seq = ack_seq;
+		subflow->map_subflow_seq = 1;
+		subflow->rel_write_seq = 1;
 	}
-	sk->sk_state = TCP_ESTABLISHED;
+	inet_sk_state_store(sk, TCP_ESTABLISHED);
 }
 
 static int subflow_create(struct sock *sock)
net/mptcp/subflow.c +27 -3
index e0b4aa8fe51d..3c9dca4be26a 100644
@@ -107,6 +107,8 @@ static void subflow_v4_init_req(struct request_sock *req,
 		subflow_req->remote_key = rx_opt.mptcp.sndr_key;
 		pr_debug("remote_key=%llu", subflow_req->remote_key);
 		token_new_request(req, skb);
+		pr_debug("syn seq=%u", TCP_SKB_CB(skb)->seq);
+		subflow_req->ssn_offset = TCP_SKB_CB(skb)->seq;
 	} else {
 		subflow_req->mp_capable = 0;
 	}
@@ -120,10 +122,15 @@ static void subflow_finish_connect(struct sock *sk, const struct sk_buff *skb)
 
 	pr_debug("subflow=%p", subflow);
 
-	if (subflow->conn) {
+	if (!subflow->conn_finished) {
 		pr_debug("remote_key=%llu", subflow->remote_key);
 		mptcp_finish_connect(subflow->conn, subflow->mp_capable);
-		subflow->conn = NULL;
+		subflow->conn_finished = 1;
+
+		if (skb) {
+			pr_debug("synack seq=%u", TCP_SKB_CB(skb)->seq);
+			subflow->ssn_offset = TCP_SKB_CB(skb)->seq;
+		}
 	}
 }
 
@@ -172,7 +179,9 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk,
 			subflow->fourth_ack = 1;
 			subflow->remote_key = subflow_req->remote_key;
 			subflow->local_key = subflow_req->local_key;
+			subflow->ssn_offset = subflow_req->ssn_offset;
 			subflow->token = subflow_req->token;
+			subflow->idsn = subflow_req->idsn;
 			pr_debug("token=%u", subflow->token);
 			token_new_accept(child);
 		} else {
@@ -202,6 +211,20 @@ const struct inet_connection_sock_af_ops subflow_specific = {
 	.mtu_reduced	   = tcp_v4_mtu_reduced,
 };
 
+static void subflow_data_ready(struct sock *sk)
+{
+	struct subflow_sock *subflow = subflow_sk(sk);
+	struct sock *parent = subflow->conn;
+
+	pr_debug("sk=%p", sk);
+	subflow->tcp_sk_data_ready(sk);
+
+	if (parent) {
+		pr_debug("parent=%p", parent);
+		parent->sk_data_ready(parent);
+	}
+}
+
 static int subflow_init_sock(struct sock *sk)
 {
 	struct subflow_sock *subflow = subflow_sk(sk);
@@ -215,6 +238,8 @@ static int subflow_init_sock(struct sock *sk)
 	if (!err) { // @@ AND mptcp is enabled
 		tsk->is_mptcp = 1;
 		icsk->icsk_af_ops = &subflow_specific;
+		subflow->tcp_sk_data_ready = sk->sk_data_ready;
+		sk->sk_data_ready = subflow_data_ready;
 	}
 
 	return err;
--

2.19.1

_______________________________________________
mptcp mailing list
mptcp(a)lists.01.org
https://lists.01.org/mailman/listinfo/mptcp

Quoted 4 lines
Quoted 3 lines

--
Mat Martineau
Intel OTC

--
Mat Martineau
Intel OTC