MPTCP-API
Christoph Paasch
Hello everyone,
I am bringing up something that we already once discussed quite a while back. Apache Traffic Server has an option to enable MPTCP (currently, through the socket-option MPTCP_ENABLED that is exposed on the multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer of ATS.
Now, ATS will be doing a 9.0.0-release soon that will be the first release with this MPTCP-option in the ATS-config.
We want to make sure that it would also work with the upcoming upstream MPTCP implementation. So, we can easily make this use IPPROTO_MPTCP. However, there currently isn't a 100% guarantee that the API is stable (as upstream might request changes).
So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
What are your thoughts?
Thanks,
Christoph
Mat Martineau
Hi Christoph and Leif,
Quoted 14 lines · Christoph Paasch
I am bringing up something that we already once discussed quite a while back. Apache Traffic Server has an option to enable MPTCP (currently, through the socket-option
MPTCP_ENABLEDthat is exposed on the multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer of ATS.Now, ATS will be doing a 9.0.0-release soon that will be the first release with this MPTCP-option in the ATS-config.
We want to make sure that it would also work with the upcoming upstream MPTCP implementation. So, we can easily make this use
IPPROTO_MPTCP. However, there currently isn't a 100% guarantee that the API is stable (as upstream might request changes).
Quoted 7 lines
So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
What are your thoughts?
There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
One is that we should pick a likely-to-get-merged value for IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP | 0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.
My impression is that the maintainers would not want a "MPTCP API version" sysctl where you'd read a number from a file and have v0, v1, v2 etc API since it would mainly exist to accomodate non-upstream kernels. The upstream API (once MPTCP is merged) is not supposed to change in ways that break userspace, after all.
Given that the MPTCP features will get added to the upstream kernel over time, we may have an easier time justifying a MPTCP feature flag sysctl, kind of like the CPU flags in /proc/cpuinfo. That could indicate support for things like TFO, netlink path management, break-before-make, customizable schedulers, etc. The combination of features may be enough for userspace programs to figure out if they can make use of that MPTCP implementation.
--
Mat Martineau
Intel
Christoph Paasch
Hello,
Quoted 27 lines · Mat Martineau
Hi Christoph and Leif,
Quoted 20 lines · Christoph Paasch
I am bringing up something that we already once discussed quite a while back. Apache Traffic Server has an option to enable MPTCP (currently, through the socket-option
MPTCP_ENABLEDthat is exposed on the multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer of ATS.Now, ATS will be doing a 9.0.0-release soon that will be the first release with this MPTCP-option in the ATS-config.
We want to make sure that it would also work with the upcoming upstream MPTCP implementation. So, we can easily make this use
IPPROTO_MPTCP. However, there currently isn't a 100% guarantee that the API is stable (as upstream might request changes).So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
Quoted 5 lines
Quoted 1 line
What are your thoughts?
There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
One is that we should pick a likely-to-get-merged value for
IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP|0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.
Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?
Quoted 3 lines
My impression is that the maintainers would not want a "MPTCP API version" sysctl where you'd read a number from a file and have v0, v1, v2 etc API since it would mainly exist to accomodate non-upstream kernels. The upstream API (once MPTCP is merged) is not supposed to change in ways that break userspace, after all.
Given that the MPTCP features will get added to the upstream kernel over time, we may have an easier time justifying a MPTCP feature flag sysctl, kind of like the CPU flags in
/proc/cpuinfo. That could indicate support for things like TFO, netlink path management, break-before-make, customizable schedulers, etc. The combination of features may be enough for userspace programs to figure out if they can make use of that MPTCP implementation.
That sounds like a fine approach to me.
In general, I am wondering, how do userspace applications handle runtime feature-checking as they may be running on different kernels? Is it always a try-and-error approach?
Christoph
Mat Martineau
Quoted 29 lines · Christoph Paasch
Hello,
Quoted 27 lines · Mat Martineau
Hi Christoph and Leif,
Quoted 20 lines · Christoph Paasch
I am bringing up something that we already once discussed quite a while back. Apache Traffic Server has an option to enable MPTCP (currently, through the socket-option
MPTCP_ENABLEDthat is exposed on the multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer of ATS.Now, ATS will be doing a 9.0.0-release soon that will be the first release with this MPTCP-option in the ATS-config.
We want to make sure that it would also work with the upcoming upstream MPTCP implementation. So, we can easily make this use
IPPROTO_MPTCP. However, there currently isn't a 100% guarantee that the API is stable (as upstream might request changes).So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
What are your thoughts?
There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
One is that we should pick a likely-to-get-merged value for
IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP|0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?
I'll test it out and send the patches, so they're ready to merge if we have a consensus.
Quoted 7 lines
Quoted 3 lines
My impression is that the maintainers would not want a "MPTCP API version" sysctl where you'd read a number from a file and have v0, v1, v2 etc API since it would mainly exist to accomodate non-upstream kernels. The upstream API (once MPTCP is merged) is not supposed to change in ways that break userspace, after all.
Given that the MPTCP features will get added to the upstream kernel over time, we may have an easier time justifying a MPTCP feature flag sysctl, kind of like the CPU flags in
/proc/cpuinfo. That could indicate support for things like TFO, netlink path management, break-before-make, customizable schedulers, etc. The combination of features may be enough for userspace programs to figure out if they can make use of that MPTCP implementation.That sounds like a fine approach to me.
In general, I am wondering, how do userspace applications handle runtime feature-checking as they may be running on different kernels? Is it always a try-and-error approach?
A project I am involved in makes use of recent additions to the keyctl API and various crypto algorithms on AF_ALG sockets. The checks are basically trial and error, if the feature is not there we expect EOPNOTSUPP from keyctl or bind() to fail:
https://git.kernel.org/pub/scm/libs/ell/ell.git/tree/ell/key.c#n788
https://git.kernel.org/pub/scm/libs/ell/ell.git/tree/ell/cipher.c#n621
--
Mat Martineau
Intel
Matthieu Baerts
Hi Mat,
Quoted 52 lines · Mat Martineau
Quoted 49 lines · Christoph Paasch
Hello,
Quoted 46 lines
On Jun 11, 2019, at 3:24 PM, Mat Martineau
<mathew.j.martineau(a)linux.intel.com> wrote:Hi Christoph and Leif,
Quoted 27 lines · Christoph Paasch
I am bringing up something that we already once discussed quite a while back. Apache Traffic Server has an option to enable MPTCP (currently, through the socket-option
MPTCP_ENABLEDthat is exposed on the multipath-tcp.org kernel). Leif (in CC) is the main contributor & maintainer
of ATS.Now, ATS will be doing a 9.0.0-release soon that will be the first release
with this MPTCP-option in the ATS-config.We want to make sure that it would also work with the upcoming upstream MPTCP
implementation. So, we can easily make this useIPPROTO_MPTCP. However, there currently isn't a 100% guarantee that the API is stable (as upstream
might request changes).So, we are wondering how we can do binary compatibility without having to
wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl
that indicates the "API-version" ? That way, upon startup ATS can simply
check the API-version and if it doesn't match, just disable MPTCP.Quoted 1 line
What are your thoughts?
There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
One is that we should pick a likely-to-get-merged value for
IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP|0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.
Quoted 6 lines
Quoted 3 lines
Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?
I'll test it out and send the patches, so they're ready to merge if we have a consensus.
Should we not request an official protocol number from the IANA? It seems that all the other ones in the upstream Linux kernel are official. Why not having one for MPTCP then. Especially if this will need to be added later in the different libC.
I can check what to do and ask around how our TCP option official number for MPTCP (30) was requested and assigned.
Best regards,
Matt
Signature · 5 lines
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net Tessares SA | Hybrid Access Solutions www.tessares.net 1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
Mat Martineau
Quoted 42 lines · Matthieu Baerts
Hi Mat,
Quoted 32 lines · Mat Martineau
Quoted 29 lines · Christoph Paasch
Hello,
Quoted 26 lines
On Jun 11, 2019, at 3:24 PM, Mat Martineau
<mathew.j.martineau(a)linux.intel.com> wrote:Hi Christoph and Leif,
Quoted 8 lines · Christoph Paasch
So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
What are your thoughts?
There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
One is that we should pick a likely-to-get-merged value for
IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP|0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.Quoted 6 lines
Quoted 3 lines
Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?
I'll test it out and send the patches, so they're ready to merge if we have a consensus.
Quoted 7 lines
Should we not request an official protocol number from the IANA? It seems that all the other ones in the upstream Linux kernel are official. Why not having one for MPTCP then. Especially if this will need to be added later in the different libC.
I can check what to do and ask around how our TCP option official number for MPTCP (30) was requested and assigned.
I think the IANA would expect a new protocol number to be used over-the-wire, which would dramatically affect MPTCP's ability to pretend to be TCP as far as middleboxes and firewalls are concerned!
The Linux definitions already deviate somewhat from the IANA list (for example, IPPROTO_IP), which leads me to believe there is some flexibility in future definitions.
--
Mat Martineau
Intel
Matthieu Baerts
Hi Mat,
Quoted 53 lines · Mat Martineau
Quoted 42 lines · Matthieu Baerts
Hi Mat,
Quoted 32 lines · Mat Martineau
Quoted 29 lines · Christoph Paasch
Hello,
Quoted 26 lines
On Jun 11, 2019, at 3:24 PM, Mat Martineau
<mathew.j.martineau(a)linux.intel.com> wrote:Hi Christoph and Leif,
Quoted 8 lines · Christoph Paasch
So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
What are your thoughts?
There are a couple of approaches we could use to help ATS and other projects as upstream MPTCP evolves.
One is that we should pick a likely-to-get-merged value for
IPPROTO_MPTCP. The current prototype uses 99, but the IANA defines that as "any private encryption scheme). We had earlier prototypes with 262 (IPPROTO_TCP|0x100) since the socket API uses 'int proto' instead of a single byte. ATS could consider a runtime-configurable proto value for MPTCP to help with this.Quoted 6 lines
Quoted 3 lines
Yes, let's pick a "likely-to-get-merged" value ASAP. Do you think 262 could work out?
I'll test it out and send the patches, so they're ready to merge if we have a consensus.
Quoted 7 lines
Should we not request an official protocol number from the IANA? It seems that all the other ones in the upstream Linux kernel are official. Why not having one for MPTCP then. Especially if this will need to be added later in the different libC.
I can check what to do and ask around how our TCP option official number for MPTCP (30) was requested and assigned.
Quoted 7 lines
I think the IANA would expect a new protocol number to be used over-the-wire, which would dramatically affect MPTCP's ability to pretend to be TCP as far as middleboxes and firewalls are concerned!
The Linux definitions already deviate somewhat from the IANA list (for example,
IPPROTO_IP), which leads me to believe there is some flexibility in future definitions.
I agree with you, I didn't want to set a new protocol number visible on the wire. I guess we cannot do that regarding the RFC6824 anyway.
My point is more to get an official number for this IPPROTO_MPTCP only for libc and keep the same between the different OS.
For example, we have IPPROTO_RAW (255) which is marked as "Reserved" by IANA.
We can discuss about that at our weekly meeting if you want to.
Cheers,
Matt
Quoted 3 lines
Signature · 2 lines
Mat Martineau Intel
Signature · 5 lines
Matthieu Baerts | R&D Engineer matthieu.baerts(a)tessares.net Tessares SA | Hybrid Access Solutions www.tessares.net 1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
Paolo Abeni
Quoted 2 lines · Christoph Paasch
Now, ATS will be doing a 9.0.0-release soon that will be the first release with this MPTCP-option in the ATS-config.
This is probably a question more for Leif, how far/near is such release?
Quoted 6 lines
So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
What are your thoughts?
I think we may look at other pieces of the kernel/networking stack that alreday implement "versioning"
- AF_PACKET. A getsockopt() is used to retrieve the API version (specifying e.g. the metadata layout). There is an additional possible
difference: packet sockets allow user space to set the API version via setsockopt().
- nftables exposes an API version number via netlink (@Florian can you give more details?)
- schedstat exposes the stats layout version number via a field in the same procfs entry
- Others?!?
First 2 approaches look viable to me.
Cheers,
Paolo
Paolo Abeni
Hi again,
Quoted 7 lines · Christoph Paasch
So, we are wondering how we can do binary compatibility without having to wait for MPTCP to land upstream. Could the MPTCP-code expose maybe a sysctl that indicates the "API-version" ? That way, upon startup ATS can simply check the API-version and if it doesn't match, just disable MPTCP.
What are your thoughts?
Reporting some additional thoughts from Florian. He can't reply directly ATM, but told me his opinion:
As far as upstream is concerned api should not change, so he is not sure why any of the proposals would fly.
AF_PACKET is really a special case, we should not take inspiration from it, and there is no no abi version in nftables as far as he knows.
Paolo (voicing Florian ;)