Network Event Notifier Mechanism
Subsystems (3)
cover Network Event Notifier Mechanism
Round 3 Changes:
- changed netlink msg for neighbour change to (RTM_NEIGHUPD)
- added netlink msg for PMTU change events (RTM_ROUTEUPD)
- added netlink messages for redirect (RTM_DELROUTE + RTM_NEWROUTE)
- tested neighbour change events via netlink for ipv4 and ipv6.
- tested redirect change events via netlink for ipv4.
Round 2 Changes:
- cleaned up event structures per review feedback.
- began integration with netlink (see neighbour changes in patch 2).
- added IPv6 support.
TODO:
- review feedback changes, if any
- more testing
- retest with RDMA NIC
------
This patch implements a mechanism that allows interested clients to register for notification of certain network events. The intended use is to allow RDMA devices (linux/drivers/infiniband) to be notified of neighbour updates, ICMP redirects, path MTU changes, and route changes.
The reason these devices need update events is because they typically cache this information in hardware and need to be notified when this information has been updated. For information on RDMA protocols, see:
http://www.ietf.org/html.charters/rddp-charter.html.
The key events of interest are:
- neighbour mac address change
- routing redirect (the next hop neighbour changes for a dst_entry)
- path mtu change (the path mtu for a dst_entry changes).
- route add/deletes
NOTE: These new netevents are also passed up to user space via netlink.
We would like to get this or similar functionality included in 2.6.19 and request comments.
This patchset consists of 2 patches:
1) New files implementing the Network Event Notifier
2) Core network changes to generate network event notifications
PATCH 1/2 Network Event Notifier Mechanism.
This patch uses notifier blocks to implement a network event notifier mechanism.
Clients register their callback function by calling register_netevent_notifier() like this:
static struct notifier_block nb = {
.notifier_call = my_callback_func
};
...
register_netevent_notifier(&nb);
---
include/net/netevent.h | 49 +++++++++++++++++++++++++++++++++++ net/core/netevent.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+), 0 deletions(-)
include/net/netevent.h new file +49 -0
@@ -0,0 +1,49 @@ +#ifndef _NET_EVENT_H +#define _NET_EVENT_H + +/* +* Generic netevent notifiers +* +* Authors: +* Tom Tucker <[email protected]> +* +* Changes: +*/ + +#ifdef __KERNEL__ + +#include <net/dst.h> + +/* +* Generic route info structure. +* +* Family Data ptr type +* -------------------------------- +* AF_INET - struct fib_info * +* AF_INET6 - struct rt6_info * +* AF_DECnet - struct dn_route * +*/ +struct netevent_route_info { +u16 family; +void *data; +}; + +struct netevent_redirect { +struct dst_entry *old; +struct dst_entry *new; +}; + +enum netevent_notif_type { +NETEVENT_NEIGH_UPDATE =1 ,/* arg is struct neighbour ptr */ +NETEVENT_ROUTE_ADD ,/* arg is struct netevent_route_info ptr */ +NETEVENT_ROUTE_DEL ,/* arg is struct netevent_route_info ptr */ +NETEVENT_PMTU_UPDATE ,/* arg is struct dst_entry ptr */ +NETEVENT_REDIRECT ,/* arg is struct netevent_redirect ptr */ +}; + +extern int register_netevent_notifier (struct notifier_block *nb); +extern int unregister_netevent_notifier (struct notifier_block *nb); +extern int call_netevent_notifiers (unsigned long val,void *v); + +#endif +#endif
net/core/netevent.c new file +68 -0
68 lines not on this page · raw message
PATCH 2/2 Core network changes to support network event notification.
In reply to Steve WiseThis patch adds netevent and netlink calls for neighbour change, route add/del, pmtu change, and routing redirect events.
Netlink Details:
Neighbour change events are broadcast as a new ndmsg type RTM_NEIGHUPD.
Path mtu change events are broadcast as a new rtmsg type RTM_ROUTEUPD.
Routing redirect events are broadcast as a pair of rtmsgs, RTM_DELROUTE and RTM_NEWROUTE.
---
include/linux/rtnetlink.h | 4 ++ net/core/Makefile | 2 + net/core/neighbour.c | 37 ++++++++++++++++--- net/ipv4/fib_semantics.c | 9 +++++ net/ipv4/route.c | 86 ++++++++++++++++++++++++++++++++++++++++++-- net/ipv6/route.c | 87 +++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 213 insertions(+), 12 deletions(-)
include/linux/rtnetlink.h +4 -0
@@ -35,6 +35,8 @@ #define RTM_NEWROUTE RTM_NEWROUTE#define RTM_DELROUTE RTM_DELROUTE RTM_GETROUTE,#define RTM_GETROUTE RTM_GETROUTE + RTM_ROUTEUPD, +#define RTM_ROUTEUPD RTM_ROUTEUPD RTM_NEWNEIGH = 28,#define RTM_NEWNEIGH RTM_NEWNEIGH
@@ -42,6 +44,8 @@ #define RTM_NEWNEIGH RTM_NEWNEIGH#define RTM_DELNEIGH RTM_DELNEIGH RTM_GETNEIGH,#define RTM_GETNEIGH RTM_GETNEIGH + RTM_NEIGHUPD, +#define RTM_NEIGHUPD RTM_NEIGHUPD RTM_NEWRULE = 32,#define RTM_NEWRULE RTM_NEWRULE
net/core/Makefile +1 -1
@@ -7,7 +7,7 @@ obj-y := sock.o request_sock.o skbuff.o obj-$(CONFIG_SYSCTL) += sysctl_net_core.o -obj-y += dev.o ethtool.o dev_mcast.o dst.o \ +obj-y += dev.o ethtool.o dev_mcast.o dst.o netevent.o \ neighbour.o rtnetlink.o utils.o link_watch.o filter.o obj-$(CONFIG_XFRM) += flow.o
net/core/neighbour.c +30 -7
@@ -30,9 +30,11 @@ #include <linux/times.h>#include <net/neighbour.h> #include <net/dst.h> #include <net/sock.h> +#include <net/netevent.h> #include <linux/rtnetlink.h> #include <linux/random.h> #include <linux/string.h> +#include <linux/notifier.h> #define NEIGH_DEBUG 1
@@ -59,6 +61,7 @@ static void neigh_app_notify(struct neig#endif static int pneigh_ifdown (struct neigh_table *tbl,struct net_device *dev);void neigh_changeaddr (struct neigh_table *tbl,struct net_device *dev); +static void rtm_neigh_change (struct neighbour *n);static struct neigh_table *neigh_tables;#ifdef CONFIG_PROC_FS
@@ -755,6 +758,7 @@ #endif neigh->nud_state =NUD_STALE ; neigh->updated = jiffies;neigh_suspect (neigh); + notify =1 ; } }else if (state &NUD_DELAY ) { if (time_before_eq(now,
@@ -763,6 +767,7 @@ #endif neigh->nud_state =NUD_REACHABLE ; neigh->updated = jiffies;neigh_connect (neigh); + notify =1 ; next = neigh->confirmed + neigh->parms->reachable_time; }else {NEIGH_PRINTK2 ("neigh %p is probed.\n" , neigh);
@@ -820,6 +825,8 @@ #endif out:write_unlock (&neigh->lock); } +if (notify) +rtm_neigh_change (neigh);#ifdef CONFIG_ARPD if (notify && neigh->parms->app_probes)
@@ -927,9 +934,7 @@ int neigh_update(struct neighbour *neigh {u8 old;int err; -#ifdef CONFIG_ARPD int notify =0 ; -#endif struct net_device *dev;int update_isrouter =0 ;
@@ -949,9 +954,7 @@ #endifneigh_suspect (neigh); neigh->nud_state = new; err =0 ; -#ifdef CONFIG_ARPD notify = old &NUD_VALID ; -#endif goto out; }
@@ -1023,9 +1026,7 @@ #endifif (!(new &NUD_CONNECTED )) neigh->confirmed = jiffies - (neigh->parms->base_reachable_time <<1 ); -#ifdef CONFIG_ARPD notify =1 ; -#endif }if (new == old) goto out;
@@ -1056,7 +1057,11 @@ out: (neigh->flags | NTF_ROUTER) : (neigh->flags & ~NTF_ROUTER ); } +write_unlock_bh (&neigh->lock); + +if (notify) +rtm_neigh_change (neigh);#ifdef CONFIG_ARPD if (notify && neigh->parms->app_probes)neigh_app_notify (neigh);
@@ -2370,9 +2375,27 @@ static void neigh_app_notify(struct neigNETLINK_CB (skb).dst_group =RTNLGRP_NEIGH ;netlink_broadcast (rtnl, skb,0 ,RTNLGRP_NEIGH ,GFP_ATOMIC ); } -#endif /* CONFIG_ARPD */ +static void rtm_neigh_change (struct neighbour *n) +{ +struct nlmsghdr *nlh; +int size =NLMSG_SPACE (sizeof (struct ndmsg ) +256 ); +struct sk_buff *skb =alloc_skb (size,GFP_ATOMIC ); + +call_netevent_notifiers (NETEVENT_NEIGH_UPDATE , n); +if (!skb) +return ; + +if (neigh_fill_info (skb, n,0 ,0 ,RTM_NEIGHUPD ,0 ) <0 ) { +kfree_skb (skb); +return ; + } + nlh = (struct nlmsghdr *)skb->data; +NETLINK_CB (skb).dst_group =RTNLGRP_NEIGH ; +netlink_broadcast (rtnl, skb,0 ,RTNLGRP_NEIGH ,GFP_ATOMIC ); +} + #ifdef CONFIG_SYSCTL static struct neigh_sysctl_table {
net/ipv4/fib_semantics.c +9 -0
@@ -45,6 +45,7 @@ #include <net/tcp.h>#include <net/sock.h> #include <net/ip_fib.h> #include <net/ip_mp_alg.h> +#include <net/netevent.h> #include "fib_lookup.h"
@@ -280,6 +281,14 @@ void rtmsg_fib(int event, u32 key, strucstruct sk_buff *skb;u32 pid = req ? req->pid : n->nlmsg_pid;int size =NLMSG_SPACE (sizeof (struct rtmsg )+256 ); +struct netevent_route_info nri; +int netevent; + + nri.family =AF_INET ; + nri.data = &fa->fa_info; + netevent = event ==RTM_NEWROUTE ?NETEVENT_ROUTE_ADD + :NETEVENT_ROUTE_DEL ; +call_netevent_notifiers (netevent, &nri); skb =alloc_skb (size,GFP_KERNEL );if (!skb)
net/ipv4/route.c +82 -4
146 lines not on this page · raw message
net/ipv6/route.c +87 -0
147 lines not on this page · raw message
Quoted 5 lines · Steve Wise
The reason these devices need update events is because they typically cache this information in hardware and need to be notified when this information has been updated. For information on RDMA protocols, see:
http://www.ietf.org/html.charters/rddp-charter.html.
Please give more specific reasons for needing these events because it is certainly far from obvious from reading those documents.
Without reasons these invasive changes may turn out to be completely inappropriate.
Thanks,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 3 lines · Herbert Xu
Please give more specific reasons for needing these events because it is certainly far from obvious from reading those documents.
Never mind, I've found your earlier messages on the list which explains your reasons more clearly. It would be nice if you could include those explanations in your patch description.
BTW, does this mean that we're now comfortable with full TOE?
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 9 lines · Herbert Xu
Quoted 2 lines · Herbert Xu
Please give more specific reasons for needing these events because it is certainly far from obvious from reading those documents.
Never mind, I've found your earlier messages on the list which explains your reasons more clearly. It would be nice if you could include those explanations in your patch description.
BTW, does this mean that we're now comfortable with full TOE?
I don't see how that position has changed?
http://linux-net.osdl.org/index.php/TOE
Jeff
Quoted 4 lines · Jeff Garzik
I don't see how that position has changed?
Well I must say that RDMA over TCP smells very much like TOE. They've got an ARP table, a routing table, and presumably a TCP stack.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtTOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)
Quoted 7 lines · Herbert Xu
Quoted 3 lines · Jeff Garzik
I don't see how that position has changed?
Well I must say that RDMA over TCP smells very much like TOE. They've got an ARP table, a routing table, and presumably a TCP stack.
A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)
If a TOE vendors wants to do TOE in a way that is transparent to the kernel, more power to them. Such non-Linux TCP stack solutions still suffer many of the problems listed at the web page above, but at least they impose no burden on kernel maintenance.
i.e. we really _do not_ want to get into the habit of co-managing arp tables, routing tables, filtering rules, and dozens of other such resources with multiple remote, independent TCP stack. We have enough complexity as it is today, coordinating between the random variations of SMP, uniprocessor, and NUMA machines out there. Not to mention competing with under-the-hood firmware actions (ASF) on NICs.
As an aside, RDMA over TCP just seems silly. TCP was _not_ meant to do the things that RDMA users want. The infiniband/RDMA programming model is an ultra-low-latency polling model where one or two apps are allowed to completely consume the machine, either busy-waiting or processing messages.
Unfortunately I don't have more details, so you just get a generalized rant :)
Jeff
Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)
Quoted 5 lines · Jeff Garzik
A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)
Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?
Quoted 2 lines
Unfortunately I don't have more details, so you just get a generalized rant :)
OK, the patch under discussion here adds hooks to all the stuff in the previous paragraph for the purpose of RDMA over TCP (well I must say that the exact RDMA application/hardware has never been clearly given but this is what I can gather from the previous posts).
Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtRe: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)
Quoted 6 lines · Herbert Xu
Quoted 1 line · Jeff Garzik
A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)
Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?
Well, you've always been able to implement userspace (or otherwise completely-virtualized) network stack. tuntap and the packet socket enable that, if nothing else. But, like you characterize below, those are existing, well-defined, easily contained interfaces.
Quoted 5 lines
Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.
Strongly agreed.
Jeff
David Miller
maintainerRe: TOE, etc.
In reply to Herbert XuFrom: Herbert Xu <[email protected]>
Date: Wed, 28 Jun 2006 14:29:59 +1000
Quoted 10 lines · Jeff Garzik
Quoted 5 lines
A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)
Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?
Socket state, and that is one thing I don't see them doing yet.
Quoted 5 lines
Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.
Yeah, it's starting to smell really bad.
But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.
Right?
Re: TOE, etc.
Quoted 2 lines · David Miller
Socket state, and that is one thing I don't see them doing yet.
I wonder what happens when the Linux TCP stack attempts to open a connection to a remote host when that connection is already open in the RDMA NIC? For that matter what happens if a Linux application decides to listen on a TCP port already listened on by the RDMA NIC?
The only saving grace is that they're only doing RDMA rather than arbitrary TCP. However, exactly the same infrastructure can be used to do arbitrary TCP should they wish to.
Quoted 5 lines
But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.
Right?
Yes, however I think the same argument could be applied to TOE.
With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtDavid Miller
maintainerRe: TOE, etc.
From: Herbert Xu <[email protected]>
Date: Wed, 28 Jun 2006 15:35:54 +1000
Quoted 4 lines
With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.
That's true. I don't think we should really add any more help for these kinds of things then.
Re: TOE, etc.
In reply to Herbert XuQuoted 18 lines · Herbert Xu
Quoted 2 lines · David Miller
Socket state, and that is one thing I don't see them doing yet.
I wonder what happens when the Linux TCP stack attempts to open a connection to a remote host when that connection is already open in the RDMA NIC? For that matter what happens if a Linux application decides to listen on a TCP port already listened on by the RDMA NIC?
The only saving grace is that they're only doing RDMA rather than arbitrary TCP. However, exactly the same infrastructure can be used to do arbitrary TCP should they wish to.
Quoted 4 lines
But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.
Quoted 9 lines
Quoted 1 line
Right?
Yes, however I think the same argument could be applied to TOE.
With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.
Doesn't iSCSI have the same issue? No netfilter, IPsec, tcpdump, etc...
Re: TOE, etc.
In reply to Herbert XuQuoted 10 lines · Herbert Xu
Quoted 2 lines · David Miller
Socket state, and that is one thing I don't see them doing yet.
I wonder what happens when the Linux TCP stack attempts to open a connection to a remote host when that connection is already open in the RDMA NIC? For that matter what happens if a Linux application decides to listen on a TCP port already listened on by the RDMA NIC?
This issue would have to be handled by using seperate IP addresses for RDMA connections vs native stack TCP.
Consider NFS-RDMA server. Through administration, it would be configured to listen on the specific rdma ip addresses, and the native stack tcp ip addresses and thus support both TCP and RDMA NFS connections.
There are definitely issues with this that could be resolved via tighter integration, but that seems to not be a goal of the linux community at this time...
Quoted 8 lines
The only saving grace is that they're only doing RDMA rather than arbitrary TCP. However, exactly the same infrastructure can be used to do arbitrary TCP should they wish to.
Quoted 4 lines
But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.
Quoted 9 lines
Quoted 1 line
Right?
Yes, however I think the same argument could be applied to TOE.
With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.
Doesn't iSCSI have this same issue?
Steve.
David Miller
maintainerRe: TOE, etc.
From: Steve Wise <[email protected]>
Date: Wed, 28 Jun 2006 09:54:57 -0500
Quoted 1 line
Doesn't iSCSI have this same issue?
Software iSCSI implementations don't have the issue because they go through the stack using normal sockets and normal device send and receive.
Re: TOE, etc.
Quoted 9 lines · David Miller
From: Steve Wise <[email protected]>
Date: Wed, 28 Jun 2006 09:54:57 -0500Quoted 1 line
Doesn't iSCSI have this same issue?
Software iSCSI implementations don't have the issue because they go through the stack using normal sockets and normal device send and receive.
-
Right. I was assuming, in this thread we were talking about iSCSI devices where the TCP stack is in HW/FW on the adapter...
Steve.
Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)
In reply to Herbert XuQuoted 11 lines · Herbert Xu
Quoted 5 lines · Jeff Garzik
A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)
Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?
Quoted 7 lines
Quoted 2 lines
Unfortunately I don't have more details, so you just get a generalized rant :)
OK, the patch under discussion here adds hooks to all the stuff in the previous paragraph for the purpose of RDMA over TCP (well I must say that the exact RDMA application/hardware has never been clearly given but this is what I can gather from the previous posts).
There are Ammasso and Chelsio RDMA/Ethernet drivers in the openib.org svn iwarp branch today. The goal is to submit them for review and inclusion into linux. The Ammasso driver has been through 3 review cycles on lkml and netdev. There are other vendors with drivers, but they're currently not disclosing any information to me about their status.
Applications:
kernel: NFS-RDMA, iSER, RDP.
user: MPI, uDAPL (both middle ware).
The Ammasso driver is a different model. It actually has a full TCP/ARP/ICMP stack and doesn't require these hooks. But the RDMA/TCP model defined and implemented, I think, by most vendors is a model where the HW is doing a limited TCP offload, relying on the native stack for L2 and L3 integration (as described in the netevent patch).
Quoted 6 lines
Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.
Don't these netevent hooks have utility for other purposes? IE: Should we really shoot changes to linux _just because_ they might possibly enable TOE?
Steve.
Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)
In reply to Jeff GarzikQuoted 6 lines · Jeff Garzik
Quoted 5 lines · Herbert Xu
Quoted 3 lines · Jeff Garzik
I don't see how that position has changed?
Quoted 8 lines
Quoted 2 lines
Well I must say that RDMA over TCP smells very much like TOE. They've got an ARP table, a routing table, and presumably a TCP stack.
A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)
I wonder if the existing iSCSI solutions handle next host mac addr changes, pmtu changes, etc? They _can_, if they implement the entire ARP and ICMP suites in HW/FW. Just curious if these vendors also see merit in the netevent changes I'm proposing...
Quoted 12 lines
If a TOE vendors wants to do TOE in a way that is transparent to the kernel, more power to them. Such non-Linux TCP stack solutions still suffer many of the problems listed at the web page above, but at least they impose no burden on kernel maintenance.
i.e. we really
_donot_want to get into the habit of co-managing arp tables, routing tables, filtering rules, and dozens of other such resources with multiple remote, independent TCP stack. We have enough complexity as it is today, coordinating between the random variations of SMP, uniprocessor, and NUMA machines out there. Not to mention competing with under-the-hood firmware actions (ASF) on NICs.
Quoted 6 lines
As an aside, RDMA over TCP just seems silly. TCP was _not_ meant to do the things that RDMA users want. The infiniband/RDMA programming model is an ultra-low-latency polling model where one or two apps are allowed to completely consume the machine, either busy-waiting or processing messages.
With RDMA over TCP, you can get the same ultra-low-latency, interrupt coalescing or avoidance, and copy avoidance as with Infiniband (with newly arriving 10Gb RDMA NICs). The benefit over IB, IMO, is the fact that your infrastructure is all IP.
If you're interested in a kernel-mode app that benfits from RDMA, then check out NFS-RDMA, which runs today transparently over Infiniband and RDMA/TCP devices using the Infiniband RDMA-CM and verbs with minor changes to support RDMA/TCP devices.
http://sourceforge.net/projects/nfs-rdma/
Steve.