Skip to contentSkip to conversation

Next steps discussion

MPTCP Christoph Paasch 3 participants 15 messages Feb 26, 2018 – Mar 5, 2018 Networking MPTCP

lore
Share & tools
lore

Christoph Paasch

author
#0 lore raw

Hello,

as for next steps after the submission of the TCP-option framework to netdev and DaveM's feedback on it.

Even if the submission got rejected, I think we still have a very useful set of patches here. The need for a framework might pop up again in the future, and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches? I can also create a github repo if you prefer that.

As for DaveM's feedback, the main takeaway - as Mat already noted on his other mail - is that fast-path performance he the highest priority. Branching and indirect function calls are hardly accepted there.

So, in that spirit I think we need to work towards reducing MPTCP's intrusiveness to the TCP stack.

* Stop taking meta-lock when receiving subflow data (all the changes where we check for mptcp(tp) and then do bh_lock_sock(meta_sk)).
The reason we do this in today's implementation is because it allows to access the meta data-structure at any point. If we stop taking the meta-lock a few things need to change:
1. Do lockless accesses for selected fields (e.g., for the DATA_ACK).
2. Group the more intrusive accesses to few select points in the TCP-stack where we then take the meta-lock (e.g., when receiving data). (this would be equivalent as if the TCP-option framework would be there
- thus we need to move code to these or similar points in the stack)
3. Sometimes schedule work-queues when we need to avoid deadlocks due to lock-ordering issues (e.g., when we can't take the meta-lock because it's already held by another thread).

I think, the way to approach this here, is by working iteratively and start moving code in such a way that accesses to the meta-socket are grouped together.

Also, we have a few callbacks that we added (cfr., struct tcp_sock_ops). We added them to avoid duplicating the code. Let's review those and see if we can get rid of them. (as an example: .send_fin could be removed as it is only called from tcp_shutdown, called from the .shutdown callback in tcp_prot - thus if we expose a separate MPTCP socket-type with its own struct proto, we can get rid of the .send_fin callback)

* Investigate how/if we can make MPTCP adopt KCM or ULP.

* There is still the open question of the API, path-management,... Tessares has some experience with that, so maybe they can provide some ideas here.

* The size of the skb. Well, we have been discussing this for quite a while :) One option is always to have a lookup table as they do for the TLS-records. That will hurt performance, but at least it's a step forward. And we have a bunch of other ideas that might be worth exploring as well. If I'm not mistaken, Rao had an approach that could work as well, right?

Any other comments, suggestions,...? :-)

On my side, as a first concrete step, I will work towards lockless subflow establishment. In tcp_v4_rcv, we are currently taking the meta-level lock when the socket-lookup matches on a request-socket. Now that TCP supports lockless listeners, MPTCP should do that as well.

Cheers,
Christoph

#1 lore raw

Hi Christoph,

Quoted 10 lines · Christoph Paasch

Hello,

as for next steps after the submission of the TCP-option framework to netdev and DaveM's feedback on it.

Even if the submission got rejected, I think we still have a very useful set of patches here. The need for a framework might pop up again in the future, and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Done:
https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/log/?h=md5

Quoted 29 lines

I can also create a github repo if you prefer that.

As for DaveM's feedback, the main takeaway - as Mat already noted on his other mail - is that fast-path performance he the highest priority. Branching and indirect function calls are hardly accepted there.

So, in that spirit I think we need to work towards reducing MPTCP's intrusiveness to the TCP stack.

* Stop taking meta-lock when receiving subflow data (all the changes where we check for mptcp(tp) and then do bh_lock_sock(meta_sk)).
The reason we do this in today's implementation is because it allows to access the meta data-structure at any point. If we stop taking the meta-lock a few things need to change:
1. Do lockless accesses for selected fields (e.g., for the DATA_ACK).
2. Group the more intrusive accesses to few select points in the TCP-stack where we then take the meta-lock (e.g., when receiving data). (this would be equivalent as if the TCP-option framework would be there
- thus we need to move code to these or similar points in the stack)
3. Sometimes schedule work-queues when we need to avoid deadlocks due to lock-ordering issues (e.g., when we can't take the meta-lock because it's already held by another thread).

I think, the way to approach this here, is by working iteratively and start moving code in such a way that accesses to the meta-socket are grouped together.

Quoted 7 lines

Also, we have a few callbacks that we added (cfr., struct tcp_sock_ops). We added them to avoid duplicating the code. Let's review those and see if we can get rid of them. (as an example: .send_fin could be removed as it is only called from tcp_shutdown, called from the .shutdown callback in tcp_prot - thus if we expose a separate MPTCP socket-type with its own struct proto, we can get rid of the .send_fin callback)

I think a separate MPTCP socket type will be important for upstream acceptance. My team has been working on some code with this separate socket type that we can share. I'm thinking that it will be useful to share once a connection can stay up without falling back to TCP.

Quoted 1 line

* Investigate how/if we can make MPTCP adopt KCM or ULP.

My main concern about ULP is that only one upper layer protocol can be set up (at least as the code is now), so you wouldn't be able to do something like use in-kernel TLS over MPTCP. Other than that, it seems like a natural fit for MPTCP.

So far I've been looking at KCM as a source of good ideas rather than something we could use directly. KCM uses SOCK_SEQPACKET or SOCK_DGRAM, but maybe it could be extended to include SOCK_STREAM. Where MPTCP places DSS mappings in the TCP options, KCM handles message boundaries within the data stream - that made me ponder using XDP to place the DSS mappings in the data payload (with the necessary TCP sequence number adjustments). I'm not sure it's workable because it can be expensive to change the length of an incoming skb and adjusting the acks gets complicated, but it's at least an interesting thought experiment :)

Quoted 2 lines

* There is still the open question of the API, path-management,... Tessares has some experience with that, so maybe they can provide some ideas here.

We (at OTC) are working on a generic netlink proposal for path management as well.

Quoted 6 lines

* The size of the skb. Well, we have been discussing this for quite a while :) One option is always to have a lookup table as they do for the TLS-records. That will hurt performance, but at least it's a step forward. And we have a bunch of other ideas that might be worth exploring as well. If I'm not mistaken, Rao had an approach that could work as well, right?

This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

Quoted 1 line

Any other comments, suggestions,...? :-)

I had these thoughts on evolving the multipath-tcp.org kernel fork last summer (excerpt from
https://lists.01.org/pipermail/mptcp/2017-July/000064.html), which I think are still relevant:

"""

One approach is to attempt to merge the multipath-tcp.org fork. This is an implementation in which the multipath-tcp.org community has invested a lot of time and effort, and it is in production for major applications (see
https://tools.ietf.org/html/rfc8041). This is a tremendous amount of code to review at once (even separating out modules), and currently doesn't fit with what the maintainers have asked for (non-intrusive, sk_buff size, MPTCP by default). I don't think the maintainers would consider merging such an extensive piece of git history, especially where there are a fair number of commits without an "mptcp:" label on the subject line or without a DCO signoff
(https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin).
Today, the fork is at kernel v4.4 and current upstream development is at v4.13-rc1, so the fork would have to catch up and stay current. (2018
note: Christoph has merged up to more recent kernels now)

The other extreme is to rewrite from scratch. This would allow incremental development with maintainer review from the start, but doesn't take advantage of existing code.

The most realistic approach is somewhere in between, where we write new code that fits maintainer expectations and utilize components from the fork where licensing allows and the code fits. We'll have to find the right balance: over-reliance on new code could take extra time, but constantly reworking the fork and keeping it up-to-date with net-next is also a lot of overhead.

"""

Gregory and Matthieu, do you have any thoughts on where the right balance is on evolving the fork vs. adding new code?

Quoted 4 lines

On my side, as a first concrete step, I will work towards lockless subflow establishment. In tcp_v4_rcv, we are currently taking the meta-level lock when the socket-lookup matches on a request-socket. Now that TCP supports lockless listeners, MPTCP should do that as well.

I'll work on getting my team's MPTCP socket type code posted to git.kernel.org, and getting our generic netlink proposal posted to this list.

Thanks,

--
Mat Martineau
Intel OTC

#2 lore raw

Hi Mat, Christoph,

Quoted 20 lines · Mat Martineau

Hi Christoph,

On Mon, 26 Feb 2018, Christoph Paasch wrote:

Hello,

Quoted 5 lines

as for next steps after the submission of the TCP-option framework to netdev
and DaveM's feedback on it.

Quoted 8 lines

Even if the submission got rejected, I think we still have a very useful set
of patches here. The need for a framework might pop up again in the future,
and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Quoted 2 lines

Done: https://git.kernel.org/pub/scm/linux/kernel/git/martineau/li
nux.git/log/?h=md5

By chance, did you get some feedbacks from people interested by this framework?
Or even from Eric Dumazet who wanted to clean TCP MD5 part if I am not wrong. It is maybe not what he wanted.

* There is still the open question of the API, path-management,... Tessares

Quoted 6 lines
Quoted 2 lines

has some experience with that, so maybe they can provide some ideas here.

We (at OTC) are working on a generic netlink proposal for path management as well.

At Tessares, we were going to propose a new PM supporting Netlink for a management from userspace. There is a short paper describing it:
http://www.tessares.net/tessares-founders-win-an-ietf-award/

Even if we think that having a PM customisable via BPF could be even better, this PM has still some values. It is mature, should be ready to be shared but we first need to rebase it on the latest kernel. We can discuss about that in a separate email if you want :)

Note that the PDF of the short paper is also available there:
http://www.tessares.net/wp-content/uploads/2016/06/SMAPP-Towards-Smart-Multipath-TCP-enabled-APPlications.pdf

Quoted 9 lines

* The size of the skb. Well, we have been discussing this for quite a

Quoted 7 lines

while :)
One option is always to have a lookup table as they do for the TLS-records. That will hurt performance, but at least it's a step
forward.
And we have a bunch of other ideas that might be worth exploring as well. If I'm not mistaken, Rao had an approach that could work as well, right?

Quoted 7 lines

This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

That looks interesting!

On some systems with hardware acceleration to bypass the main CPU for established connections, I guess they also have to modify the skb to share info between the main CPU and another component.
Do you have any ideas how they are doing that? It is maybe not a real problem for them to increase the size of the skb if most of the traffic goes in the "fast path".

Quoted 35 lines

Any other comments, suggestions,...? :-)

Quoted 1 line

I had these thoughts on evolving the multipath-tcp.org kernel fork last summer (excerpt from https://lists.01.org/pipermail
/mptcp/2017-July/000064.html), which I think are still relevant:

"""

One approach is to attempt to merge the multipath-tcp.org fork. This is an implementation in which the multipath-tcp.org community has invested a lot of time and effort, and it is in production for major applications (see
https://tools.ietf.org/html/rfc8041). This is a tremendous amount of code to review at once (even separating out modules), and currently doesn't fit with what the maintainers have asked for (non-intrusive, sk_buff size, MPTCP by default). I don't think the maintainers would consider merging such an extensive piece of git history, especially where there are a fair number of commits without an "mptcp:" label on the subject line or without a DCO signoff (https://www.kernel.org/doc/html/latest/process/submitting-p
atches.html#sign-your-work-the-developer-s-certificate-of-origin). Today, the fork is at kernel v4.4 and current upstream development is at v4.13-rc1, so the fork would have to catch up and stay current. (2018 note:
Christoph has merged up to more recent kernels now)

The other extreme is to rewrite from scratch. This would allow incremental development with maintainer review from the start, but doesn't take advantage of existing code.

The most realistic approach is somewhere in between, where we write new code that fits maintainer expectations and utilize components from the fork where licensing allows and the code fits. We'll have to find the right balance: over-reliance on new code could take extra time, but constantly reworking the fork and keeping it up-to-date with net-next is also a lot of overhead.

Quoted 5 lines

"""

Gregory and Matthieu, do you have any thoughts on where the right balance is on evolving the fork vs. adding new code?

We think we don't need to rewrite it from scratch. MPTCP code is consequent and complex but having a fork will not simplify it a lot if we want to continue supporting middle boxes, fallback to TCP, etc. Most of the MPTCP code is then reusable and some didn't really change for years now.

We clearly understand that the current implementation cannot go upstream. It is not just a question of git history but mostly the fact the code is too intrusive as you both said. We agree that this doesn't need a full rewrite but some adaptations from both the upstream side and the current MPTCP implementation.

Sorry for not really giving new thoughts but we agree with all points written by Christoph :-)

What we also think is that this upstreaming job is not easy. The best is certainly to well organise that and prepare as a team here questions/RFC to netdev.
I am personally not used to work on big changes on the same subject between different organisations but we guess it would be helpful to organise regular (virtual) meetings to share plans of what can be done by who.

Unlike Gregory, I guess I don't have the same high knowledge you all have here regarding the overall Linux network stack but then I would be happy to also help on improving the organisation if this can allow me to be quickly efficient on the help I can provide here :)

On my side, as a first concrete step, I will work towards lockless subflow

Quoted 9 lines
Quoted 4 lines

establishment. In tcp_v4_rcv, we are currently taking the meta-level lock when the socket-lookup matches on a request-socket. Now that TCP supports lockless listeners, MPTCP should do that as well.

I'll work on getting my team's MPTCP socket type code posted to git.kernel.org, and getting our generic netlink proposal posted to this list.

We can also work on our side to accelerate the sharing of our netlink PM.

Best regards,
Matt, ok Matthieu :-)

Signature · 23 lines
[image: Tessares SA] <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
<https://www.google.com/maps?q=1+Avenue+Jean+Monnet,+1348+Ottignies-Louvain-la-Neuve,+Belgium>

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.
author
#3 lore raw
Quoted 25 lines · Matthieu Baerts

Hi Mat, Christoph,

Quoted 20 lines · Mat Martineau

Hi Christoph,

On Mon, 26 Feb 2018, Christoph Paasch wrote:

Hello,

Quoted 13 lines

as for next steps after the submission of the TCP-option framework to netdev
and DaveM's feedback on it.

Even if the submission got rejected, I think we still have a very useful set
of patches here. The need for a framework might pop up again in the future,
and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Quoted 8 lines
Quoted 2 lines

Done: https://git.kernel.org/pub/scm/linux/kernel/git/martineau/li
nux.git/log/?h=md5

By chance, did you get some feedbacks from people interested by this framework?
Or even from Eric Dumazet who wanted to clean TCP MD5 part if I am not wrong. It is maybe not what he wanted.

Yeah, feedback was disappointingly scarce :/

It probably was not what he had completely in mind, as he didn't talk about a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Quoted 39 lines

* There is still the open question of the API, path-management,... Tessares

Quoted 6 lines
Quoted 2 lines

has some experience with that, so maybe they can provide some ideas here.

We (at OTC) are working on a generic netlink proposal for path management as well.

At Tessares, we were going to propose a new PM supporting Netlink for a management from userspace. There is a short paper describing it:
http://www.tessares.net/tessares-founders-win-an-ietf-award/

Even if we think that having a PM customisable via BPF could be even better, this PM has still some values. It is mature, should be ready to be shared but we first need to rebase it on the latest kernel. We can discuss about that in a separate email if you want :)

Note that the PDF of the short paper is also available there:
http://www.tessares.net/wp-content/uploads/2016/06/SMAPP-Towards-Smart-Multipath-TCP-enabled-APPlications.pdf

Quoted 9 lines

* The size of the skb. Well, we have been discussing this for quite a

Quoted 7 lines

while :)
One option is always to have a lookup table as they do for the TLS-records. That will hurt performance, but at least it's a step
forward.
And we have a bunch of other ideas that might be worth exploring as well. If I'm not mistaken, Rao had an approach that could work as well, right?

Quoted 7 lines

This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

Quoted 8 lines

That looks interesting!

On some systems with hardware acceleration to bypass the main CPU for established connections, I guess they also have to modify the skb to share info between the main CPU and another component.
Do you have any ideas how they are doing that? It is maybe not a real problem for them to increase the size of the skb if most of the traffic goes in the "fast path".

TLS is a good example. They actually do have a lookup table where the driver is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the
TLS hardware offloading: https://lists.01.org/pipermail/mptcp/2017-November/000165.html

Quoted 55 lines
Quoted 35 lines

Any other comments, suggestions,...? :-)

Quoted 1 line

I had these thoughts on evolving the multipath-tcp.org kernel fork last summer (excerpt from https://lists.01.org/pipermail
/mptcp/2017-July/000064.html), which I think are still relevant:

"""

One approach is to attempt to merge the multipath-tcp.org fork. This is an implementation in which the multipath-tcp.org community has invested a lot of time and effort, and it is in production for major applications (see
https://tools.ietf.org/html/rfc8041). This is a tremendous amount of code to review at once (even separating out modules), and currently doesn't fit with what the maintainers have asked for (non-intrusive, sk_buff size, MPTCP by default). I don't think the maintainers would consider merging such an extensive piece of git history, especially where there are a fair number of commits without an "mptcp:" label on the subject line or without a DCO signoff (https://www.kernel.org/doc/html/latest/process/submitting-p
atches.html#sign-your-work-the-developer-s-certificate-of-origin). Today, the fork is at kernel v4.4 and current upstream development is at v4.13-rc1, so the fork would have to catch up and stay current. (2018 note:
Christoph has merged up to more recent kernels now)

The other extreme is to rewrite from scratch. This would allow incremental development with maintainer review from the start, but doesn't take advantage of existing code.

The most realistic approach is somewhere in between, where we write new code that fits maintainer expectations and utilize components from the fork where licensing allows and the code fits. We'll have to find the right balance: over-reliance on new code could take extra time, but constantly reworking the fork and keeping it up-to-date with net-next is also a lot of overhead.

Quoted 5 lines

"""

Gregory and Matthieu, do you have any thoughts on where the right balance is on evolving the fork vs. adding new code?

We think we don't need to rewrite it from scratch. MPTCP code is consequent and complex but having a fork will not simplify it a lot if we want to continue supporting middle boxes, fallback to TCP, etc. Most of the MPTCP code is then reusable and some didn't really change for years now.

We clearly understand that the current implementation cannot go upstream. It is not just a question of git history but mostly the fact the code is too intrusive as you both said. We agree that this doesn't need a full rewrite but some adaptations from both the upstream side and the current MPTCP implementation.

Sorry for not really giving new thoughts but we agree with all points written by Christoph :-)

Quoted 6 lines

What we also think is that this upstreaming job is not easy. The best is certainly to well organise that and prepare as a team here questions/RFC to netdev.
I am personally not used to work on big changes on the same subject between different organisations but we guess it would be helpful to organise regular (virtual) meetings to share plans of what can be done by who.

Yes, that would be good! Just a short weekly sync could be useful.

Do you have a Webex or other means to set this up?

Looking forward to the netlink PM! :-)

Christoph

Quoted 46 lines

Unlike Gregory, I guess I don't have the same high knowledge you all have here regarding the overall Linux network stack but then I would be happy to also help on improving the organisation if this can allow me to be quickly efficient on the help I can provide here :)

On my side, as a first concrete step, I will work towards lockless subflow

Quoted 9 lines
Quoted 4 lines

establishment. In tcp_v4_rcv, we are currently taking the meta-level lock when the socket-lookup matches on a request-socket. Now that TCP supports lockless listeners, MPTCP should do that as well.

I'll work on getting my team's MPTCP socket type code posted to git.kernel.org, and getting our generic netlink proposal posted to this list.

We can also work on our side to accelerate the sharing of our netlink PM.

Best regards,
Matt, ok Matthieu :-)

Signature · 23 lines
[image: Tessares SA] <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
<https://www.google.com/maps?q=1+Avenue+Jean+Monnet,+1348+Ottignies-Louvain-la-Neuve,+Belgium>

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.
author
#11 lore raw
In reply to Mat Martineau
Quoted 10 lines · Mat Martineau

Hi Christoph,

Quoted 5 lines · Christoph Paasch

Hello,

as for next steps after the submission of the TCP-option framework to netdev and DaveM's feedback on it.

Quoted 7 lines
Quoted 5 lines

Even if the submission got rejected, I think we still have a very useful set of patches here. The need for a framework might pop up again in the future, and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Done: https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/log/?h=md5

Thanks!

Quoted 37 lines
Quoted 29 lines

I can also create a github repo if you prefer that.

As for DaveM's feedback, the main takeaway - as Mat already noted on his other mail - is that fast-path performance he the highest priority. Branching and indirect function calls are hardly accepted there.

So, in that spirit I think we need to work towards reducing MPTCP's intrusiveness to the TCP stack.

* Stop taking meta-lock when receiving subflow data (all the changes where we check for mptcp(tp) and then do bh_lock_sock(meta_sk)).
The reason we do this in today's implementation is because it allows to access the meta data-structure at any point. If we stop taking the meta-lock a few things need to change:
1. Do lockless accesses for selected fields (e.g., for the DATA_ACK).
2. Group the more intrusive accesses to few select points in the TCP-stack where we then take the meta-lock (e.g., when receiving data). (this would be equivalent as if the TCP-option framework would be there
- thus we need to move code to these or similar points in the stack)
3. Sometimes schedule work-queues when we need to avoid deadlocks due to lock-ordering issues (e.g., when we can't take the meta-lock because it's already held by another thread).

I think, the way to approach this here, is by working iteratively and start moving code in such a way that accesses to the meta-socket are grouped together.

Quoted 7 lines

Also, we have a few callbacks that we added (cfr., struct tcp_sock_ops). We added them to avoid duplicating the code. Let's review those and see if we can get rid of them. (as an example: .send_fin could be removed as it is only called from tcp_shutdown, called from the .shutdown callback in tcp_prot - thus if we expose a separate MPTCP socket-type with its own struct proto, we can get rid of the .send_fin callback)

Quoted 3 lines

I think a separate MPTCP socket type will be important for upstream acceptance. My team has been working on some code with this separate socket type that we can share.

Great! I would love to move MPTCP to a separate socket-type.

Quoted 2 lines

I'm thinking that it will be useful to share once a connection can stay up without falling back to TCP.

Hmm... I'm not sure I understand. What do you mean with "connection can stay up without falling back to TCP".

Quoted 7 lines
Quoted 2 lines

* Investigate how/if we can make MPTCP adopt KCM or ULP.

My main concern about ULP is that only one upper layer protocol can be set up (at least as the code is now), so you wouldn't be able to do something like use in-kernel TLS over MPTCP. Other than that, it seems like a natural fit for MPTCP.

Do you think it would be feasible to make ULP use multiple ULPs ?

Quoted 71 lines

So far I've been looking at KCM as a source of good ideas rather than something we could use directly. KCM uses SOCK_SEQPACKET or SOCK_DGRAM, but maybe it could be extended to include SOCK_STREAM. Where MPTCP places DSS mappings in the TCP options, KCM handles message boundaries within the data stream - that made me ponder using XDP to place the DSS mappings in the data payload (with the necessary TCP sequence number adjustments). I'm not sure it's workable because it can be expensive to change the length of an incoming skb and adjusting the acks gets complicated, but it's at least an interesting thought experiment :)

Quoted 2 lines

* There is still the open question of the API, path-management,... Tessares has some experience with that, so maybe they can provide some ideas here.

We (at OTC) are working on a generic netlink proposal for path management as well.

Quoted 6 lines

* The size of the skb. Well, we have been discussing this for quite a while :) One option is always to have a lookup table as they do for the TLS-records. That will hurt performance, but at least it's a step forward. And we have a bunch of other ideas that might be worth exploring as well. If I'm not mistaken, Rao had an approach that could work as well, right?

This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

Quoted 1 line

Any other comments, suggestions,...? :-)

I had these thoughts on evolving the multipath-tcp.org kernel fork last summer (excerpt from
https://lists.01.org/pipermail/mptcp/2017-July/000064.html), which I think are still relevant:

"""

One approach is to attempt to merge the multipath-tcp.org fork. This is an
implementation in which the multipath-tcp.org community has invested a lot
of time and effort, and it is in production for major applications (see
https://tools.ietf.org/html/rfc8041). This is a tremendous amount of code to
review at once (even separating out modules), and currently doesn't fit with
what the maintainers have asked for (non-intrusive, sk_buff size, MPTCP by
default). I don't think the maintainers would consider merging such an
extensive piece of git history, especially where there are a fair number of
commits without an "mptcp:" label on the subject line or without a DCO
signoff (https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin).
Today, the fork is at kernel v4.4 and current upstream development is at
v4.13-rc1, so the fork would have to catch up and stay current. (2018 note:
Christoph has merged up to more recent kernels now)

The other extreme is to rewrite from scratch. This would allow incremental development with maintainer review from the start, but doesn't take advantage of existing code.

The most realistic approach is somewhere in between, where we write new code that fits maintainer expectations and utilize components from the fork where licensing allows and the code fits. We'll have to find the right balance: over-reliance on new code could take extra time, but constantly reworking the fork and keeping it up-to-date with net-next is also a lot of overhead.

"""

Gregory and Matthieu, do you have any thoughts on where the right balance is on evolving the fork vs. adding new code?

Quoted 8 lines
Quoted 4 lines

On my side, as a first concrete step, I will work towards lockless subflow establishment. In tcp_v4_rcv, we are currently taking the meta-level lock when the socket-lookup matches on a request-socket. Now that TCP supports lockless listeners, MPTCP should do that as well.

I'll work on getting my team's MPTCP socket type code posted to git.kernel.org, and getting our generic netlink proposal posted to this list.

Cool! I think, the MPTCP socket-type code should also find its way into mptcp-dev. It would allow to move that one forward as well.

Christoph

#12 lore raw
Quoted 62 lines · Christoph Paasch
Quoted 10 lines · Mat Martineau

Hi Christoph,

Quoted 5 lines · Christoph Paasch

Hello,

as for next steps after the submission of the TCP-option framework to netdev and DaveM's feedback on it.

Quoted 7 lines
Quoted 5 lines

Even if the submission got rejected, I think we still have a very useful set of patches here. The need for a framework might pop up again in the future, and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Done: https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/log/?h=md5

Thanks!

Quoted 37 lines
Quoted 29 lines

I can also create a github repo if you prefer that.

As for DaveM's feedback, the main takeaway - as Mat already noted on his other mail - is that fast-path performance he the highest priority. Branching and indirect function calls are hardly accepted there.

So, in that spirit I think we need to work towards reducing MPTCP's intrusiveness to the TCP stack.

* Stop taking meta-lock when receiving subflow data (all the changes where we check for mptcp(tp) and then do bh_lock_sock(meta_sk)).
The reason we do this in today's implementation is because it allows to access the meta data-structure at any point. If we stop taking the meta-lock a few things need to change:
1. Do lockless accesses for selected fields (e.g., for the DATA_ACK).
2. Group the more intrusive accesses to few select points in the TCP-stack where we then take the meta-lock (e.g., when receiving data). (this would be equivalent as if the TCP-option framework would be there
- thus we need to move code to these or similar points in the stack)
3. Sometimes schedule work-queues when we need to avoid deadlocks due to lock-ordering issues (e.g., when we can't take the meta-lock because it's already held by another thread).

I think, the way to approach this here, is by working iteratively and start moving code in such a way that accesses to the meta-socket are grouped together.

Quoted 7 lines

Also, we have a few callbacks that we added (cfr., struct tcp_sock_ops). We added them to avoid duplicating the code. Let's review those and see if we can get rid of them. (as an example: .send_fin could be removed as it is only called from tcp_shutdown, called from the .shutdown callback in tcp_prot - thus if we expose a separate MPTCP socket-type with its own struct proto, we can get rid of the .send_fin callback)

Quoted 3 lines

I think a separate MPTCP socket type will be important for upstream acceptance. My team has been working on some code with this separate socket type that we can share.

Quoted 7 lines

Great! I would love to move MPTCP to a separate socket-type.

Quoted 2 lines

I'm thinking that it will be useful to share once a connection can stay up without falling back to TCP.

Hmm... I'm not sure I understand. What do you mean with "connection can stay up without falling back to TCP".

Sorry, to clarify:

The code I have today does not process incoming DSS options or send a data ack, so the connection falls back to regular TCP after a couple of packets. It's based on net-next.

I would like to get my code to a point where it sends a correct data ack so the connection does not fall back, and at that point post the code on git.kernel.org and as a patch set on this list.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

* Investigate how/if we can make MPTCP adopt KCM or ULP.

My main concern about ULP is that only one upper layer protocol can be set up (at least as the code is now), so you wouldn't be able to do something like use in-kernel TLS over MPTCP. Other than that, it seems like a natural fit for MPTCP.

Do you think it would be feasible to make ULP use multiple ULPs ?

I think it's possible to have a layered approach to ULP, maybe by providing a list of protocols to setsockopt. It would be nice to have some infrastructure to help implement it correctly, rather than depending on each ULP to correctly call in to the layer below it whether it's the "normal" TCP code or some other ULP. The combinatorics would be a headache for sure: running TLS over an MPTCP connection makes sense, but MPTCP layered on top of TLS doesn't make sense. At least if all the protocols were specified in one setsockopt we could limit which combinations are allowed.

Quoted 76 lines
Quoted 76 lines

So far I've been looking at KCM as a source of good ideas rather than something we could use directly. KCM uses SOCK_SEQPACKET or SOCK_DGRAM, but maybe it could be extended to include SOCK_STREAM. Where MPTCP places DSS mappings in the TCP options, KCM handles message boundaries within the data stream - that made me ponder using XDP to place the DSS mappings in the data payload (with the necessary TCP sequence number adjustments). I'm not sure it's workable because it can be expensive to change the length of an incoming skb and adjusting the acks gets complicated, but it's at least an interesting thought experiment :)

Quoted 2 lines

* There is still the open question of the API, path-management,... Tessares has some experience with that, so maybe they can provide some ideas here.

We (at OTC) are working on a generic netlink proposal for path management as well.

Quoted 6 lines

* The size of the skb. Well, we have been discussing this for quite a while :) One option is always to have a lookup table as they do for the TLS-records. That will hurt performance, but at least it's a step forward. And we have a bunch of other ideas that might be worth exploring as well. If I'm not mistaken, Rao had an approach that could work as well, right?

This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

Quoted 1 line

Any other comments, suggestions,...? :-)

I had these thoughts on evolving the multipath-tcp.org kernel fork last summer (excerpt from
https://lists.01.org/pipermail/mptcp/2017-July/000064.html), which I think are still relevant:

"""

One approach is to attempt to merge the multipath-tcp.org fork. This is an
implementation in which the multipath-tcp.org community has invested a lot
of time and effort, and it is in production for major applications (see
https://tools.ietf.org/html/rfc8041). This is a tremendous amount of code to
review at once (even separating out modules), and currently doesn't fit with
what the maintainers have asked for (non-intrusive, sk_buff size, MPTCP by
default). I don't think the maintainers would consider merging such an
extensive piece of git history, especially where there are a fair number of
commits without an "mptcp:" label on the subject line or without a DCO
signoff (https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin).
Today, the fork is at kernel v4.4 and current upstream development is at
v4.13-rc1, so the fork would have to catch up and stay current. (2018 note:
Christoph has merged up to more recent kernels now)

The other extreme is to rewrite from scratch. This would allow incremental development with maintainer review from the start, but doesn't take advantage of existing code.

The most realistic approach is somewhere in between, where we write new code that fits maintainer expectations and utilize components from the fork where licensing allows and the code fits. We'll have to find the right balance: over-reliance on new code could take extra time, but constantly reworking the fork and keeping it up-to-date with net-next is also a lot of overhead.

"""

Gregory and Matthieu, do you have any thoughts on where the right balance is on evolving the fork vs. adding new code?

Quoted 4 lines

On my side, as a first concrete step, I will work towards lockless subflow establishment. In tcp_v4_rcv, we are currently taking the meta-level lock when the socket-lookup matches on a request-socket. Now that TCP supports lockless listeners, MPTCP should do that as well.

Quoted 6 lines
Quoted 3 lines

I'll work on getting my team's MPTCP socket type code posted to git.kernel.org, and getting our generic netlink proposal posted to this list.

Cool! I think, the MPTCP socket-type code should also find its way into mptcp-dev. It would allow to move that one forward as well.

Thanks. It's a fairly deep change from the metasocket architecture, so we will see how it works out.

--
Mat Martineau
Intel OTC

author
#13 lore raw
Quoted 69 lines · Mat Martineau
Quoted 67 lines · Christoph Paasch
Quoted 10 lines · Mat Martineau

Hi Christoph,

Quoted 5 lines · Christoph Paasch

Hello,

as for next steps after the submission of the TCP-option framework to netdev and DaveM's feedback on it.

Quoted 7 lines
Quoted 5 lines

Even if the submission got rejected, I think we still have a very useful set of patches here. The need for a framework might pop up again in the future, and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Done: https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/log/?h=md5

Thanks!

Quoted 37 lines
Quoted 29 lines

I can also create a github repo if you prefer that.

As for DaveM's feedback, the main takeaway - as Mat already noted on his other mail - is that fast-path performance he the highest priority. Branching and indirect function calls are hardly accepted there.

So, in that spirit I think we need to work towards reducing MPTCP's intrusiveness to the TCP stack.

* Stop taking meta-lock when receiving subflow data (all the changes where we check for mptcp(tp) and then do bh_lock_sock(meta_sk)).
The reason we do this in today's implementation is because it allows to access the meta data-structure at any point. If we stop taking the meta-lock a few things need to change:
1. Do lockless accesses for selected fields (e.g., for the DATA_ACK).
2. Group the more intrusive accesses to few select points in the TCP-stack where we then take the meta-lock (e.g., when receiving data). (this would be equivalent as if the TCP-option framework would be there
- thus we need to move code to these or similar points in the stack)
3. Sometimes schedule work-queues when we need to avoid deadlocks due to lock-ordering issues (e.g., when we can't take the meta-lock because it's already held by another thread).

I think, the way to approach this here, is by working iteratively and start moving code in such a way that accesses to the meta-socket are grouped together.

Quoted 7 lines

Also, we have a few callbacks that we added (cfr., struct tcp_sock_ops). We added them to avoid duplicating the code. Let's review those and see if we can get rid of them. (as an example: .send_fin could be removed as it is only called from tcp_shutdown, called from the .shutdown callback in tcp_prot - thus if we expose a separate MPTCP socket-type with its own struct proto, we can get rid of the .send_fin callback)

Quoted 3 lines

I think a separate MPTCP socket type will be important for upstream acceptance. My team has been working on some code with this separate socket type that we can share.

Great! I would love to move MPTCP to a separate socket-type.

Quoted 2 lines

I'm thinking that it will be useful to share once a connection can stay up without falling back to TCP.

Quoted 8 lines
Quoted 2 lines

Hmm... I'm not sure I understand. What do you mean with "connection can stay up without falling back to TCP".

Sorry, to clarify:

The code I have today does not process incoming DSS options or send a data ack, so the connection falls back to regular TCP after a couple of packets. It's based on net-next.

Oh, so it is not integrated on top of the MPTCP-code and rather a fresh start from scratch?

Quoted 3 lines

I would like to get my code to a point where it sends a correct data ack so the connection does not fall back, and at that point post the code on git.kernel.org and as a patch set on this list.

Maybe, even if the data-reception is not yet working, I still would be intersted in the MPTCP socket-type patches. Because that could be integrated into multipath-tcp.org.

Because, getting the data-reception to work if you started from scratch is going to take you a lot of time.

Quoted 9 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

* Investigate how/if we can make MPTCP adopt KCM or ULP.

My main concern about ULP is that only one upper layer protocol can be set up (at least as the code is now), so you wouldn't be able to do something like use in-kernel TLS over MPTCP. Other than that, it seems like a natural fit for MPTCP.

Do you think it would be feasible to make ULP use multiple ULPs ?

Quoted 8 lines

I think it's possible to have a layered approach to ULP, maybe by providing a list of protocols to setsockopt. It would be nice to have some infrastructure to help implement it correctly, rather than depending on each ULP to correctly call in to the layer below it whether it's the "normal" TCP code or some other ULP. The combinatorics would be a headache for sure:
running TLS over an MPTCP connection makes sense, but MPTCP layered on top of TLS doesn't make sense. At least if all the protocols were specified in one setsockopt we could limit which combinations are allowed.

Yes, MPTCP in this case would be special as it is so tightly coupled to TCP.

MPTCP is always the protocol that would be at the bottom of the stack I think.

Christoph

#4 lore raw
In reply to Christoph Paasch

Hi Christoph,

Quoted 6 lines · Christoph Paasch

Yeah, feedback was disappointingly scarce :/

It probably was not what he had completely in mind, as he didn't talk about a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Or maybe simply unlucky with who was available at this time to have a look at them :)

Quoted 7 lines

TLS is a good example. They actually do have a lookup table where the driver
is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the TLS hardware offloading: https://lists.01.org/
pipermail/mptcp/2017-November/000165.html

Thank you for the link!

Yes, that would be good! Just a short weekly sync could be useful.

Quoted 3 lines

Do you have a Webex or other means to set this up?

I think I can setup a webex (even if it is less fun to use that on a Linux desktop :) ). We already used talky.io, more Linux friendly if we need to share screen and others, maybe not needed.

Another issue may be the time. I think you are all in the west coast of the US :-)
But I am sure I can find time on the evening for a short meeting!

Quoted 2 lines

Looking forward to the netlink PM! :-)

I hope I will have time very soon to clean this.
Mat, do you prefer to talk about that directly by commenting the patches on mptcp-dev ML?

Matthieu

Signature · 23 lines
[image: Tessares SA] <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
<https://www.google.com/maps?q=1+Avenue+Jean+Monnet,+1348+Ottignies-Louvain-la-Neuve,+Belgium>

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.
#14 lore raw
In reply to Christoph Paasch
Quoted 73 lines · Christoph Paasch
Quoted 72 lines · Mat Martineau
Quoted 69 lines · Christoph Paasch
Quoted 10 lines · Mat Martineau

Hi Christoph,

Quoted 5 lines · Christoph Paasch

Hello,

as for next steps after the submission of the TCP-option framework to netdev and DaveM's feedback on it.

Quoted 7 lines
Quoted 5 lines

Even if the submission got rejected, I think we still have a very useful set of patches here. The need for a framework might pop up again in the future, and so these patches could come in handy.
Mat, maybe you can put our latest submission on your kernel.org-git repo just so that we don't lose track of these patches?

Done: https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/log/?h=md5

Thanks!

Quoted 37 lines
Quoted 29 lines

I can also create a github repo if you prefer that.

As for DaveM's feedback, the main takeaway - as Mat already noted on his other mail - is that fast-path performance he the highest priority. Branching and indirect function calls are hardly accepted there.

So, in that spirit I think we need to work towards reducing MPTCP's intrusiveness to the TCP stack.

* Stop taking meta-lock when receiving subflow data (all the changes where we check for mptcp(tp) and then do bh_lock_sock(meta_sk)).
The reason we do this in today's implementation is because it allows to access the meta data-structure at any point. If we stop taking the meta-lock a few things need to change:
1. Do lockless accesses for selected fields (e.g., for the DATA_ACK).
2. Group the more intrusive accesses to few select points in the TCP-stack where we then take the meta-lock (e.g., when receiving data). (this would be equivalent as if the TCP-option framework would be there
- thus we need to move code to these or similar points in the stack)
3. Sometimes schedule work-queues when we need to avoid deadlocks due to lock-ordering issues (e.g., when we can't take the meta-lock because it's already held by another thread).

I think, the way to approach this here, is by working iteratively and start moving code in such a way that accesses to the meta-socket are grouped together.

Quoted 7 lines

Also, we have a few callbacks that we added (cfr., struct tcp_sock_ops). We added them to avoid duplicating the code. Let's review those and see if we can get rid of them. (as an example: .send_fin could be removed as it is only called from tcp_shutdown, called from the .shutdown callback in tcp_prot - thus if we expose a separate MPTCP socket-type with its own struct proto, we can get rid of the .send_fin callback)

Quoted 3 lines

I think a separate MPTCP socket type will be important for upstream acceptance. My team has been working on some code with this separate socket type that we can share.

Great! I would love to move MPTCP to a separate socket-type.

Quoted 2 lines

I'm thinking that it will be useful to share once a connection can stay up without falling back to TCP.

Hmm... I'm not sure I understand. What do you mean with "connection can stay up without falling back to TCP".

Quoted 8 lines
Quoted 5 lines

Sorry, to clarify:

The code I have today does not process incoming DSS options or send a data ack, so the connection falls back to regular TCP after a couple of packets. It's based on net-next.

Oh, so it is not integrated on top of the MPTCP-code and rather a fresh start from scratch?

Not entirely from scratch, but mostly.

Quoted 10 lines
Quoted 3 lines

I would like to get my code to a point where it sends a correct data ack so the connection does not fall back, and at that point post the code on git.kernel.org and as a patch set on this list.

Maybe, even if the data-reception is not yet working, I still would be intersted in the MPTCP socket-type patches. Because that could be integrated into multipath-tcp.org.

Because, getting the data-reception to work if you started from scratch is going to take you a lot of time.

We've been working on it for a while, and I'm pretty close to having data reception working (just a single subflow, no join, various other early implementation constraints).

Quoted 19 lines
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

* Investigate how/if we can make MPTCP adopt KCM or ULP.

My main concern about ULP is that only one upper layer protocol can be set up (at least as the code is now), so you wouldn't be able to do something like use in-kernel TLS over MPTCP. Other than that, it seems like a natural fit for MPTCP.

Do you think it would be feasible to make ULP use multiple ULPs ?

Quoted 8 lines

I think it's possible to have a layered approach to ULP, maybe by providing a list of protocols to setsockopt. It would be nice to have some infrastructure to help implement it correctly, rather than depending on each ULP to correctly call in to the layer below it whether it's the "normal" TCP code or some other ULP. The combinatorics would be a headache for sure:
running TLS over an MPTCP connection makes sense, but MPTCP layered on top of TLS doesn't make sense. At least if all the protocols were specified in one setsockopt we could limit which combinations are allowed.

Quoted 4 lines

Yes, MPTCP in this case would be special as it is so tightly coupled to TCP.

MPTCP is always the protocol that would be at the bottom of the stack I think.

Ok, thanks for the confirmation.

--
Mat Martineau
Intel OTC

author
#5 lore raw
In reply to Matthieu Baerts
Quoted 27 lines · Matthieu Baerts

Hi Christoph,

Quoted 6 lines · Christoph Paasch

Yeah, feedback was disappointingly scarce :/

It probably was not what he had completely in mind, as he didn't talk about a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Or maybe simply unlucky with who was available at this time to have a look at them :)

Quoted 7 lines

TLS is a good example. They actually do have a lookup table where the driver
is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the TLS hardware offloading: https://lists.01.org/
pipermail/mptcp/2017-November/000165.html

Thank you for the link!

Quoted 8 lines

Yes, that would be good! Just a short weekly sync could be useful.

Quoted 3 lines

Do you have a Webex or other means to set this up?

I think I can setup a webex (even if it is less fun to use that on a Linux desktop :) ). We already used talky.io, more Linux friendly if we need to share screen and others, maybe not needed.

talky.io is fine for me as well.

Quoted 3 lines

Another issue may be the time. I think you are all in the west coast of the US :-)
But I am sure I can find time on the evening for a short meeting!

Ideally for me, it would be at 9am (for you 6pm). Otherwise, I can make 8am as well (5pm for you).

I don't know what the opinion of the others is on the timing.

Christoph

Quoted 34 lines
Quoted 4 lines

Looking forward to the netlink PM! :-)

I hope I will have time very soon to clean this.
Mat, do you prefer to talk about that directly by commenting the patches on mptcp-dev ML?

Matthieu

Signature · 23 lines
[image: Tessares SA] <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
<https://www.google.com/maps?q=1+Avenue+Jean+Monnet,+1348+Ottignies-Louvain-la-Neuve,+Belgium>

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.
#6 lore raw
Quoted 30 lines · Christoph Paasch
Quoted 29 lines · Matthieu Baerts

Hi Christoph,

Quoted 6 lines · Christoph Paasch

Yeah, feedback was disappointingly scarce :/

It probably was not what he had completely in mind, as he didn't talk about a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Or maybe simply unlucky with who was available at this time to have a look at them :)

Quoted 7 lines

TLS is a good example. They actually do have a lookup table where the driver
is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the TLS hardware offloading: https://lists.01.org/
pipermail/mptcp/2017-November/000165.html

Thank you for the link!

Yes, that would be good! Just a short weekly sync could be useful.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Do you have a Webex or other means to set this up?

I think I can setup a webex (even if it is less fun to use that on a Linux desktop :) ). We already used talky.io, more Linux friendly if we need to share screen and others, maybe not needed.

talky.io is fine for me as well.

I haven't used it before, but it looks like a fine option. Thank you for setting this up.

Quoted 9 lines
Quoted 4 lines

Another issue may be the time. I think you are all in the west coast of the US :-)
But I am sure I can find time on the evening for a short meeting!

Ideally for me, it would be at 9am (for you 6pm). Otherwise, I can make 8am as well (5pm for you).

I don't know what the opinion of the others is on the timing.

I'm also in the Pacific time zone, as are my coworkers Peter and Ossama. Personally, 9am works better.

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines

Looking forward to the netlink PM! :-)

I hope I will have time very soon to clean this.
Mat, do you prefer to talk about that directly by commenting the patches on mptcp-dev ML?

That's fine with me. I did read the paper (unfortunately I hadn't come across it before!), and it looks like our approaches have a lot in common
- it will be informative to compare the details.

From earlier in this sub-thread:

Quoted 7 lines

This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

Quoted 8 lines

That looks interesting!

On some systems with hardware acceleration to bypass the main CPU for established connections, I guess they also have to modify the skb to share info between the main CPU and another component. Do you have any ideas how they are doing that? It is maybe not a real problem for them to increase the size of the skb if most of the traffic goes in the "fast path".

Are you referring to userspace stacks (like DPDK), or optimizations like GRO? I'm not sure of the specific answer to your question, but from what I see the drivers try to allocate enough space in the skbs to accomodate the storage needs through the skb's lifetime.

Thanks,

--
Mat Martineau
Intel OTC

#7 lore raw

Hi Mat, Christoph,

Quoted 47 lines · Mat Martineau

On Thu, 1 Mar 2018, Christoph Paasch wrote:

Quoted 44 lines · Matthieu Baerts
Quoted 43 lines

Hi Christoph,

On Wed, Feb 28, 2018 at 11:41 PM, Christoph Paasch <cpaasch(a)apple.com> wrote:

Yeah, feedback was disappointingly scarce :/

Quoted 7 lines

It probably was not what he had completely in mind, as he didn't talk about
a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Or maybe simply unlucky with who was available at this time to have a look
at them :)

TLS is a good example. They actually do have a lookup table where the

Quoted 7 lines

driver
is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the TLS hardware offloading: https://lists.01.org/
pipermail/mptcp/2017-November/000165.html

Thank you for the link!

Yes, that would be good! Just a short weekly sync could be useful.

Quoted 4 lines

Do you have a Webex or other means to set this up?

I think I can setup a webex (even if it is less fun to use that on a Linux
desktop :) ). We already used talky.io, more Linux friendly if we need to
share screen and others, maybe not needed.

Quoted 6 lines
Quoted 2 lines

talky.io is fine for me as well.

I haven't used it before, but it looks like a fine option. Thank you for setting this up.

The good thing is that we can have a public URL, just in case someone else would help for this upstreaming task: https://talky.io/mptcp_upstreaming

I can also define a shared key if needed.

Another issue may be the time. I think you are all in the west coast of the

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

US :-)
But I am sure I can find time on the evening for a short meeting!

Ideally for me, it would be at 9am (for you 6pm). Otherwise, I can make 8am
as well (5pm for you).

Quoted 6 lines
Quoted 2 lines

I don't know what the opinion of the others is on the timing.

I'm also in the Pacific time zone, as are my coworkers Peter and Ossama. Personally, 9am works better.

9am (6pm here) is good for me!

When would you like to have our first meeting? At 6pm, I should not have any other meetings.
Exceptionally I will not be available this Tuesday 6th of March. Just to propose a date, what about Wednesday 7th of March?

Looking forward to the netlink PM! :-)

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

I hope I will have time very soon to clean this.
Mat, do you prefer to talk about that directly by commenting the patches on
mptcp-dev ML?

Quoted 4 lines

That's fine with me. I did read the paper (unfortunately I hadn't come across it before!), and it looks like our approaches have a lot in common - it will be informative to compare the details.

Good idea! I will work on the rebase/clean-up and send it to mptcp-dev ML. I will add you in cc.

From earlier in this sub-thread:

Quoted 17 lines

This is what I'm working on now. For outgoing packets, I have a way

Quoted 6 lines

to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

Quoted 8 lines

That looks interesting!

On some systems with hardware acceleration to bypass the main CPU for established connections, I guess they also have to modify the skb to share info between the main CPU and another component. Do you have any ideas how they are doing that? It is maybe not a real problem for them to increase the size of the skb if most of the traffic goes in the "fast path".

Quoted 4 lines

Are you referring to userspace stacks (like DPDK), or optimizations like GRO? I'm not sure of the specific answer to your question, but from what I see the drivers try to allocate enough space in the skbs to accomodate the storage needs through the skb's lifetime.

Sorry, I was not clear. I don't know if it is common to do that, maybe not because it is not upstream.

In short, the NIC (or another external HW module) of a router/switch is doing some learning about the flows it needs to forward. Once this NIC/module knows the flow, the Linux network stack no longer see the rest of a traffic because it has been "accelerated".
Here is a longer description when this is done by the NIC:
https://www.netronome.com/media/documents/WP_Hardware_
Acceleration.pdf#WP_Hardware_Acceleration.indd%3A.3203%3A27

But I guess there are some info that the NIC/module cannot learn by itself when looking at the beginning of the flow like what QoS to apply, if a specific flow can be "accelerated" or not, etc. These info are certainly shared in the skb.

Now that I am thinking more about this scenario, it will maybe not help for our MPTCP case. I was supposing these info shared via the skb could be randomly present. But even if the skb is bigger all the time, I guess this is not a problem for them because at the end, the Linux network stack will only see a very small part of the traffic, not a big deal to have more cache misses for the slow path only (and some other exceptions). In other words, their skb is maybe too big as well but that's not a problem.

Best regards,
Matthieu

Signature · 23 lines
[image: Tessares SA] <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
<https://www.google.com/maps?q=1+Avenue+Jean+Monnet,+1348+Ottignies-Louvain-la-Neuve,+Belgium>

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.
#8 lore raw
Quoted 47 lines · Matthieu Baerts

Hi Mat, Christoph,

On Thu, Mar 1, 2018 at 9:05 PM, Mat Martineau <mathew.j.martineau(a)linux.intel.com> wrote:
On Thu, 1 Mar 2018, Christoph Paasch wrote:

On 01/03/18 - 18:05:40, Matthieu Baerts wrote:
Hi Christoph,

On Wed, Feb 28, 2018 at 11:41 PM, Christoph Paasch <cpaasch(a)apple.com> wrote:

Yeah, feedback was disappointingly scarce :/

It probably was not what he had completely in mind, as he didn't talk about a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Or maybe simply unlucky with who was available at this time to have a look at them :)

TLS is a good example. They actually do have a lookup table where the driver
is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the TLS hardware offloading: https://lists.01.org/
pipermail/mptcp/2017-November/000165.html

Thank you for the link!

Yes, that would be good! Just a short weekly sync could be useful.

Do you have a Webex or other means to set this up?

I think I can setup a webex (even if it is less fun to use that on a Linux desktop :) ). We already used talky.io, more Linux friendly if we need to share screen and others, maybe not needed.

talky.io is fine for me as well.

Quoted 7 lines

I haven't used it before, but it looks like a fine option. Thank you for setting this up.

The good thing is that we can have a public URL, just in case someone else would help for this upstreaming task:
https://talky.io/mptcp_upstreaming

I can also define a shared key if needed.

Yes, since this is an open list it's important to have a public URL.

Quoted 12 lines

Another issue may be the time. I think you are all in the west coast of the US :-)
But I am sure I can find time on the evening for a short meeting!

Ideally for me, it would be at 9am (for you 6pm). Otherwise, I can make 8am as well (5pm for you).

I don't know what the opinion of the others is on the timing.

Quoted 7 lines

I'm also in the Pacific time zone, as are my coworkers Peter and Ossama. Personally, 9am works better.

9am (6pm here) is good for me!

When would you like to have our first meeting? At 6pm, I should not have any other meetings. Exceptionally I will not be available this Tuesday 6th of March. Just to propose a date, what about Wednesday 7th of March?

Wednesday the 7th of March is the one morning next week I have a conflict at 9am, but I am available at 9:30. Other mornings at 9 are ok.

Quoted 44 lines

Looking forward to the netlink PM! :-)

I hope I will have time very soon to clean this.
Mat, do you prefer to talk about that directly by commenting the patches on mptcp-dev ML?

That's fine with me. I did read the paper (unfortunately I hadn't come across it before!), and it looks like our approaches have a lot in common - it will be informative to compare the details.

Good idea! I will work on the rebase/clean-up and send it to mptcp-dev ML. I will add you in cc.

From earlier in this sub-thread:

     This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

That looks interesting!

On some systems with hardware acceleration to bypass the main CPU for established connections, I guess they also have to modify the skb to share info between the main CPU and another component. Do you have any ideas how they are doing that? It is maybe not a real problem for them to increase the size of the skb if most of the traffic goes in the "fast path".

Are you referring to userspace stacks (like DPDK), or optimizations like GRO? I'm not sure of the specific answer to your question, but from what I see the drivers try to allocate enough space in the skbs to accomodate the storage needs through the skb's lifetime.

Sorry, I was not clear. I don't know if it is common to do that, maybe not because it is not upstream.

In short, the NIC (or another external HW module) of a router/switch is doing some learning about the flows it needs to forward. Once this NIC/module knows the flow, the Linux network stack no longer see the rest of a traffic because it has been "accelerated". Here is a longer description when this is done by the NIC:
https://www.netronome.com/media/documents/WP_Hardware_Acceleration.pdf#WP_Hardware_Acceleration.indd%3A.3203%3A27

Quoted 8 lines

But I guess there are some info that the NIC/module cannot learn by itself when looking at the beginning of the flow like what QoS to apply, if a specific flow can be "accelerated" or not, etc. These info are certainly shared in the skb.

Now that I am thinking more about this scenario, it will maybe not help for our MPTCP case. I was supposing these info shared via the skb could be randomly present. But even if the skb is bigger all the time, I guess this is not a problem for them because at the end, the Linux network stack will only see a very small part of the traffic, not a big deal to have more cache misses for the slow path only (and some other exceptions).
In other words, their skb is maybe too big as well but that's not a problem.

The only case where I'm allocating larger skbs is on the outgoing path where MPTCP code can control allocation, and it happens in a way that's transparent to most users of the skb. While it would be great to have the space on received skbs I wanted to leave control of incoming skb allocation with the drivers.

I'm using an updated version of this:

https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/commit/?h=sharedcb&id=8fffadb6b1ee0dc23e8eed43e40e15f5a6277307

In short, the shinfo part of the skb has additional bytes allocated but code that expects a "normal" skb sees only the normal part of the skb. It's fine for the payload part of the skb to be any size, since the shinfo area is tracked separately.

Thanks,

--
Mat Martineau
Intel OTC

author
#9 lore raw
Quoted 75 lines · Mat Martineau
Quoted 47 lines · Matthieu Baerts

Hi Mat, Christoph,

On Thu, Mar 1, 2018 at 9:05 PM, Mat Martineau <mathew.j.martineau(a)linux.intel.com> wrote:
On Thu, 1 Mar 2018, Christoph Paasch wrote:

On 01/03/18 - 18:05:40, Matthieu Baerts wrote:
Hi Christoph,

On Wed, Feb 28, 2018 at 11:41 PM, Christoph Paasch <cpaasch(a)apple.com> wrote:

Yeah, feedback was disappointingly scarce :/

It probably was not what he had completely in mind, as he didn't talk about a TCP-option framework but simply about moving TCP-MD5 code out of the TCP-stack. So, with our proposal we went a bit further.

Or maybe simply unlucky with who was available at this time to have a look at them :)

TLS is a good example. They actually do have a lookup table where the driver
is calling back into the TCP-stack to get the TLS-record information.

We had discussed this with people from Mellanox that proposed the TLS hardware offloading: https://lists.01.org/
pipermail/mptcp/2017-November/000165.html

Thank you for the link!

Yes, that would be good! Just a short weekly sync could be useful.

Do you have a Webex or other means to set this up?

I think I can setup a webex (even if it is less fun to use that on a Linux desktop :) ). We already used talky.io, more Linux friendly if we need to share screen and others, maybe not needed.

talky.io is fine for me as well.

Quoted 7 lines

I haven't used it before, but it looks like a fine option. Thank you for setting this up.

The good thing is that we can have a public URL, just in case someone else would help for this upstreaming task:
https://talky.io/mptcp_upstreaming

I can also define a shared key if needed.

Yes, since this is an open list it's important to have a public URL.

Quoted 15 lines

Another issue may be the time. I think you are all in the west coast of the US :-)
But I am sure I can find time on the evening for a short meeting!

Ideally for me, it would be at 9am (for you 6pm). Otherwise, I can make 8am as well (5pm for you).

I don't know what the opinion of the others is on the timing.

I'm also in the Pacific time zone, as are my coworkers Peter and Ossama. Personally, 9am works better.

Quoted 7 lines
Quoted 4 lines

9am (6pm here) is good for me!

When would you like to have our first meeting? At 6pm, I should not have any other meetings. Exceptionally I will not be available this Tuesday 6th of March. Just to propose a date, what about Wednesday 7th of March?

Wednesday the 7th of March is the one morning next week I have a conflict at 9am, but I am available at 9:30. Other mornings at 9 are ok.

On my side, any day at 9am next week is fine.

Cheers,
Christoph

Quoted 74 lines
Quoted 45 lines

Looking forward to the netlink PM! :-)

I hope I will have time very soon to clean this.
Mat, do you prefer to talk about that directly by commenting the patches on mptcp-dev ML?

That's fine with me. I did read the paper (unfortunately I hadn't come across it before!), and it looks like our approaches have a lot in common - it will be informative to compare the details.

Good idea! I will work on the rebase/clean-up and send it to mptcp-dev ML. I will add you in cc.

From earlier in this sub-thread:

     This is what I'm working on now. For outgoing packets, I have a way to optionally allocate sk_buffs with extra control block space. For incoming packets, my initial experiment is with preventing packet coalesce/collapse so TCP options are still in the skb headroom. I don't consider that a long-term solution, though. Some kind of lookup table will probably be needed.

That looks interesting!

On some systems with hardware acceleration to bypass the main CPU for established connections, I guess they also have to modify the skb to share info between the main CPU and another component. Do you have any ideas how they are doing that? It is maybe not a real problem for them to increase the size of the skb if most of the traffic goes in the "fast path".

Are you referring to userspace stacks (like DPDK), or optimizations like GRO? I'm not sure of the specific answer to your question, but from what I see the drivers try to allocate enough space in the skbs to accomodate the storage needs through the skb's lifetime.

Sorry, I was not clear. I don't know if it is common to do that, maybe not because it is not upstream.

In short, the NIC (or another external HW module) of a router/switch is doing some learning about the flows it needs to forward. Once this NIC/module knows the flow, the Linux network stack no longer see the rest of a traffic because it has been "accelerated". Here is a longer description when this is done by the NIC:
https://www.netronome.com/media/documents/WP_Hardware_Acceleration.pdf#WP_Hardware_Acceleration.indd%3A.3203%3A27

Quoted 8 lines

But I guess there are some info that the NIC/module cannot learn by itself when looking at the beginning of the flow like what QoS to apply, if a specific flow can be "accelerated" or not, etc. These info are certainly shared in the skb.

Now that I am thinking more about this scenario, it will maybe not help for our MPTCP case. I was supposing these info shared via the skb could be randomly present. But even if the skb is bigger all the time, I guess this is not a problem for them because at the end, the Linux network stack will only see a very small part of the traffic, not a big deal to have more cache misses for the slow path only (and some other exceptions).
In other words, their skb is maybe too big as well but that's not a problem.

The only case where I'm allocating larger skbs is on the outgoing path where MPTCP code can control allocation, and it happens in a way that's transparent to most users of the skb. While it would be great to have the space on received skbs I wanted to leave control of incoming skb allocation with the drivers.

I'm using an updated version of this:

https://git.kernel.org/pub/scm/linux/kernel/git/martineau/linux.git/commit/?h=sharedcb&id=8fffadb6b1ee0dc23e8eed43e40e15f5a6277307

In short, the shinfo part of the skb has additional bytes allocated but code that expects a "normal" skb sees only the normal part of the skb. It's fine for the payload part of the skb to be any size, since the shinfo area is tracked separately.

Thanks,

--
Mat Martineau
Intel OTC

#10 lore raw
In reply to Mat Martineau

Hi Mat, Christoph,

Quoted 20 lines · Mat Martineau

Another issue may be the time. I think you are all in

Quoted 19 lines

the west coast of the
US :-)
But I am sure I can find time on the evening for a
short meeting!

Ideally for me, it would be at 9am (for you 6pm). Otherwise,
I can make 8am
as well (5pm for you).

I don't know what the opinion of the others is on the timing.

I'm also in the Pacific time zone, as are my coworkers Peter and
Ossama. Personally, 9am works better.

9am (6pm here) is good for me!

Quoted 8 lines
Quoted 5 lines

When would you like to have our first meeting? At 6pm, I should not have any other meetings.
Exceptionally I will not be available this Tuesday 6th of March. Just to propose a date, what about Wednesday 7th of March?

Wednesday the 7th of March is the one morning next week I have a conflict at 9am, but I am available at 9:30. Other mornings at 9 are ok.

Just to be safe and not force you to rush finishing your meeting on time, we can do that this Thursday at 9am? (8th of March, International Women's Day)

I am going to send a calendar invitation to both of you. Can I add someone else? Rao maybe?

From earlier in this sub-thread:

Quoted 66 lines
Quoted 47 lines

This is what I'm working on now. For outgoing packets, I
have a way
to optionally allocate sk_buffs with extra control block
space. For
incoming packets, my initial experiment is with
preventing packet
coalesce/collapse so TCP options are still in the skb
headroom. I
don't consider that a long-term solution, though. Some
kind of
lookup table will probably be needed.

That looks interesting!

On some systems with hardware acceleration to bypass the main
CPU for established connections, I guess they also
have to modify the skb to share info between the main CPU and
another component. Do you have any ideas how they
are doing that? It is maybe not a real problem for them to
increase the size of the skb if most of the traffic
goes in the "fast path".

Are you referring to userspace stacks (like DPDK), or optimizations
like GRO? I'm not sure of the specific answer to your
question, but from what I see the drivers try to allocate enough
space in the skbs to accomodate the storage needs through
the skb's lifetime.

Sorry, I was not clear. I don't know if it is common to do that, maybe not because it is not upstream.

In short, the NIC (or another external HW module) of a router/switch is doing some learning about the flows it needs to forward. Once this NIC/module knows the flow, the Linux network stack no longer see the rest of a traffic because it has been "accelerated".
Here is a longer description when this is done by the NIC:
https://www.netronome.com/media/documents/WP_Hardware_Accele
ration.pdf#WP_Hardware_Acceleration.indd%3A.3203%3A27

But I guess there are some info that the NIC/module cannot learn by itself when looking at the beginning of the flow like what QoS to apply, if a specific flow can be "accelerated" or not, etc. These info are certainly shared in the skb.

Quoted 10 lines

Now that I am thinking more about this scenario, it will maybe not help for our MPTCP case. I was supposing these info shared via the skb could be randomly present. But even if the skb is bigger all the time, I guess this is not a problem for them because at the end, the Linux network stack will only see a very small part of the traffic, not a big deal to have more cache misses for the slow path only (and some other exceptions).
In other words, their skb is maybe too big as well but that's not a problem.

The only case where I'm allocating larger skbs is on the outgoing path where MPTCP code can control allocation, and it happens in a way that's transparent to most users of the skb. While it would be great to have the space on received skbs I wanted to leave control of incoming skb allocation with the drivers.

I'm using an updated version of this:

Quoted 7 lines

https://git.kernel.org/pub/scm/linux/kernel/git/martineau/
linux.git/commit/?h=sharedcb&id=8fffadb6b1ee0dc23e8eed43e40e15f5a6277307

In short, the shinfo part of the skb has additional bytes allocated but code that expects a "normal" skb sees only the normal part of the skb. It's fine for the payload part of the skb to be any size, since the shinfo area is tracked separately.

Thank you for the explanations! That looks promising!
Do you think this approach could also be interesting for TCP TLS project?

Best regards,
Matthieu

Signature · 23 lines
[image: Tessares SA] <http://www.tessares.net> Matthieu Baerts | R&D
Engineer
matthieu.baerts(a)tessares.net
Tessares SA | Hybrid Access Solutions
www.tessares.net
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium
<https://www.google.com/maps?q=1+Avenue+Jean+Monnet,+1348+Ottignies-Louvain-la-Neuve,+Belgium>

-- 

------------------------------
DISCLAIMER.
This email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the system manager. 
This message contains confidential information and is intended only for the 
individual named. If you are not the named addressee you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately by e-mail if you have received this e-mail by mistake and 
delete this e-mail from your system. If you are not the intended recipient 
you are notified that disclosing, copying, distributing or taking any 
action in reliance on the contents of this information is strictly 
prohibited.