Skip to contentSkip to conversation

Network Event Notifier Mechanism

RFC PATCH In review Steve Wise 4 participants 19 messages Jun 27, 2006 – Jun 28, 2006

Subsystems (3)
lore
Share & tools
lore

Steve Wise

author
#0 lore raw

cover Network Event Notifier Mechanism

Round 3 Changes:

- changed netlink msg for neighbour change to (RTM_NEIGHUPD)
- added netlink msg for PMTU change events (RTM_ROUTEUPD)
- added netlink messages for redirect (RTM_DELROUTE + RTM_NEWROUTE)
- tested neighbour change events via netlink for ipv4 and ipv6.
- tested redirect change events via netlink for ipv4.

Round 2 Changes:

- cleaned up event structures per review feedback.
- began integration with netlink (see neighbour changes in patch 2).
- added IPv6 support.

TODO:

- review feedback changes, if any
- more testing
- retest with RDMA NIC

------

This patch implements a mechanism that allows interested clients to register for notification of certain network events. The intended use is to allow RDMA devices (linux/drivers/infiniband) to be notified of neighbour updates, ICMP redirects, path MTU changes, and route changes.

The reason these devices need update events is because they typically cache this information in hardware and need to be notified when this information has been updated. For information on RDMA protocols, see:
http://www.ietf.org/html.charters/rddp-charter.html.

The key events of interest are:

- neighbour mac address change
- routing redirect (the next hop neighbour changes for a dst_entry)
- path mtu change (the path mtu for a dst_entry changes).
- route add/deletes

NOTE: These new netevents are also passed up to user space via netlink.

We would like to get this or similar functionality included in 2.6.19 and request comments.

This patchset consists of 2 patches:

1) New files implementing the Network Event Notifier
2) Core network changes to generate network event notifications

author
#1 lore raw

PATCH 1/2 Network Event Notifier Mechanism.

This patch uses notifier blocks to implement a network event notifier mechanism.

Clients register their callback function by calling register_netevent_notifier() like this:

static struct notifier_block nb = {
.notifier_call = my_callback_func
};

...

register_netevent_notifier(&nb);
---

 include/net/netevent.h |   49 +++++++++++++++++++++++++++++++++++
 net/core/netevent.c    |   68 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 117 insertions(+), 0 deletions(-)
include/net/netevent.h new file +49 -0
new file mode 100644
index 0000000..22214c8
@@ -0,0 +1,49 @@
+#ifndef _NET_EVENT_H
+#define _NET_EVENT_H
+
+/*
+ *	Generic netevent notifiers
+ *
+ *	Authors:
+ *      Tom Tucker              <[email protected]>
+ *
+ * 	Changes:
+ */
+
+#ifdef __KERNEL__
+
+#include <net/dst.h>
+
+/* 
+ * Generic route info structure.
+ *
+ * Family	  Data ptr type
+ * --------------------------------
+ * AF_INET 	- struct fib_info *
+ * AF_INET6	- struct rt6_info *
+ * AF_DECnet	- struct dn_route *
+ */
+struct netevent_route_info {
+	u16 family;
+	void *data;	
+};
+
+struct netevent_redirect {
+	struct dst_entry *old;
+	struct dst_entry *new;
+};
+
+enum netevent_notif_type {
+	NETEVENT_NEIGH_UPDATE = 1, /* arg is struct neighbour ptr */
+	NETEVENT_ROUTE_ADD,   	   /* arg is struct netevent_route_info ptr */
+	NETEVENT_ROUTE_DEL,   	   /* arg is struct netevent_route_info ptr */
+	NETEVENT_PMTU_UPDATE,	   /* arg is struct dst_entry ptr */
+	NETEVENT_REDIRECT,	   /* arg is struct netevent_redirect ptr */
+};
+
+extern int register_netevent_notifier(struct notifier_block *nb);
+extern int unregister_netevent_notifier(struct notifier_block *nb);
+extern int call_netevent_notifiers(unsigned long val, void *v);
+
+#endif
+#endif
net/core/netevent.c new file +68 -0

68 lines not on this page · raw message

author
#2 lore raw

PATCH 2/2 Core network changes to support network event notification.

In reply to Steve Wise

This patch adds netevent and netlink calls for neighbour change, route add/del, pmtu change, and routing redirect events.

Netlink Details:

Neighbour change events are broadcast as a new ndmsg type RTM_NEIGHUPD.

Path mtu change events are broadcast as a new rtmsg type RTM_ROUTEUPD.

Routing redirect events are broadcast as a pair of rtmsgs, RTM_DELROUTE and RTM_NEWROUTE.
---

 include/linux/rtnetlink.h |    4 ++
 net/core/Makefile         |    2 +
 net/core/neighbour.c      |   37 ++++++++++++++++---
 net/ipv4/fib_semantics.c  |    9 +++++
 net/ipv4/route.c          |   86 ++++++++++++++++++++++++++++++++++++++++++--
 net/ipv6/route.c          |   87 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 213 insertions(+), 12 deletions(-)
include/linux/rtnetlink.h +4 -0
index facd9ee..340ca4f 100644
@@ -35,6 +35,8 @@ #define RTM_NEWROUTE	RTM_NEWROUTE
 #define RTM_DELROUTE	RTM_DELROUTE
 	RTM_GETROUTE,
 #define RTM_GETROUTE	RTM_GETROUTE
+	RTM_ROUTEUPD,
+#define RTM_ROUTEUPD	RTM_ROUTEUPD
 
 	RTM_NEWNEIGH	= 28,
 #define RTM_NEWNEIGH	RTM_NEWNEIGH
@@ -42,6 +44,8 @@ #define RTM_NEWNEIGH	RTM_NEWNEIGH
 #define RTM_DELNEIGH	RTM_DELNEIGH
 	RTM_GETNEIGH,
 #define RTM_GETNEIGH	RTM_GETNEIGH
+	RTM_NEIGHUPD,
+#define RTM_NEIGHUPD	RTM_NEIGHUPD
 
 	RTM_NEWRULE	= 32,
 #define RTM_NEWRULE	RTM_NEWRULE
net/core/Makefile +1 -1
index e9bd246..2645ba4 100644
@@ -7,7 +7,7 @@ obj-y := sock.o request_sock.o skbuff.o 
 
 obj-$(CONFIG_SYSCTL) += sysctl_net_core.o
 
-obj-y		     += dev.o ethtool.o dev_mcast.o dst.o \
+obj-y		     += dev.o ethtool.o dev_mcast.o dst.o netevent.o \
 			neighbour.o rtnetlink.o utils.o link_watch.o filter.o
 
 obj-$(CONFIG_XFRM) += flow.o
net/core/neighbour.c +30 -7
index 50a8c73..bf70981 100644
@@ -30,9 +30,11 @@ #include <linux/times.h>
 #include <net/neighbour.h>
 #include <net/dst.h>
 #include <net/sock.h>
+#include <net/netevent.h>
 #include <linux/rtnetlink.h>
 #include <linux/random.h>
 #include <linux/string.h>
+#include <linux/notifier.h>
 
 #define NEIGH_DEBUG 1
 
@@ -59,6 +61,7 @@ static void neigh_app_notify(struct neig
 #endif
 static int pneigh_ifdown(struct neigh_table *tbl, struct net_device *dev);
 void neigh_changeaddr(struct neigh_table *tbl, struct net_device *dev);
+static void rtm_neigh_change(struct neighbour *n);
 
 static struct neigh_table *neigh_tables;
 #ifdef CONFIG_PROC_FS
@@ -755,6 +758,7 @@ #endif
 			neigh->nud_state = NUD_STALE;
 			neigh->updated = jiffies;
 			neigh_suspect(neigh);
+			notify = 1;
 		}
 	} else if (state & NUD_DELAY) {
 		if (time_before_eq(now, 
@@ -763,6 +767,7 @@ #endif
 			neigh->nud_state = NUD_REACHABLE;
 			neigh->updated = jiffies;
 			neigh_connect(neigh);
+			notify = 1;
 			next = neigh->confirmed + neigh->parms->reachable_time;
 		} else {
 			NEIGH_PRINTK2("neigh %p is probed.\n", neigh);
@@ -820,6 +825,8 @@ #endif
 out:
 		write_unlock(&neigh->lock);
 	}
+	if (notify)
+		rtm_neigh_change(neigh);
 
 #ifdef CONFIG_ARPD
 	if (notify && neigh->parms->app_probes)
@@ -927,9 +934,7 @@ int neigh_update(struct neighbour *neigh
 {
 	u8 old;
 	int err;
-#ifdef CONFIG_ARPD
 	int notify = 0;
-#endif
 	struct net_device *dev;
 	int update_isrouter = 0;
 
@@ -949,9 +954,7 @@ #endif
 			neigh_suspect(neigh);
 		neigh->nud_state = new;
 		err = 0;
-#ifdef CONFIG_ARPD
 		notify = old & NUD_VALID;
-#endif
 		goto out;
 	}
 
@@ -1023,9 +1026,7 @@ #endif
 		if (!(new & NUD_CONNECTED))
 			neigh->confirmed = jiffies -
 				      (neigh->parms->base_reachable_time << 1);
-#ifdef CONFIG_ARPD
 		notify = 1;
-#endif
 	}
 	if (new == old)
 		goto out;
@@ -1056,7 +1057,11 @@ out:
 			(neigh->flags | NTF_ROUTER) :
 			(neigh->flags & ~NTF_ROUTER);
 	}
+
 	write_unlock_bh(&neigh->lock);
+
+	if (notify)
+		rtm_neigh_change(neigh);
 #ifdef CONFIG_ARPD
 	if (notify && neigh->parms->app_probes)
 		neigh_app_notify(neigh);
@@ -2370,9 +2375,27 @@ static void neigh_app_notify(struct neig
 	NETLINK_CB(skb).dst_group  = RTNLGRP_NEIGH;
 	netlink_broadcast(rtnl, skb, 0, RTNLGRP_NEIGH, GFP_ATOMIC);
 }
-
 #endif /* CONFIG_ARPD */
 
+static void rtm_neigh_change(struct neighbour *n)
+{
+	struct nlmsghdr *nlh;
+	int size = NLMSG_SPACE(sizeof(struct ndmsg) + 256);
+	struct sk_buff *skb = alloc_skb(size, GFP_ATOMIC);
+
+	call_netevent_notifiers(NETEVENT_NEIGH_UPDATE, n);
+	if (!skb)
+		return;
+
+	if (neigh_fill_info(skb, n, 0, 0, RTM_NEIGHUPD, 0) < 0) {
+		kfree_skb(skb);
+		return;
+	}
+	nlh			   = (struct nlmsghdr *)skb->data;
+	NETLINK_CB(skb).dst_group  = RTNLGRP_NEIGH;
+	netlink_broadcast(rtnl, skb, 0, RTNLGRP_NEIGH, GFP_ATOMIC);
+}
+
 #ifdef CONFIG_SYSCTL
 
 static struct neigh_sysctl_table {
net/ipv4/fib_semantics.c +9 -0
index 0f4145b..197c365 100644
@@ -45,6 +45,7 @@ #include <net/tcp.h>
 #include <net/sock.h>
 #include <net/ip_fib.h>
 #include <net/ip_mp_alg.h>
+#include <net/netevent.h>
 
 #include "fib_lookup.h"
 
@@ -280,6 +281,14 @@ void rtmsg_fib(int event, u32 key, struc
 	struct sk_buff *skb;
 	u32 pid = req ? req->pid : n->nlmsg_pid;
 	int size = NLMSG_SPACE(sizeof(struct rtmsg)+256);
+	struct netevent_route_info nri;
+	int netevent;
+
+	nri.family = AF_INET;
+	nri.data = &fa->fa_info;
+	netevent = event == RTM_NEWROUTE ? NETEVENT_ROUTE_ADD 
+					 : NETEVENT_ROUTE_DEL;
+	call_netevent_notifiers(netevent, &nri);
 
 	skb = alloc_skb(size, GFP_KERNEL);
 	if (!skb)
net/ipv4/route.c +82 -4

146 lines not on this page · raw message

net/ipv6/route.c +87 -0

147 lines not on this page · raw message

#3 lore raw
In reply to Steve Wise
Quoted 5 lines · Steve Wise

The reason these devices need update events is because they typically cache this information in hardware and need to be notified when this information has been updated. For information on RDMA protocols, see:
http://www.ietf.org/html.charters/rddp-charter.html.

Please give more specific reasons for needing these events because it is certainly far from obvious from reading those documents.

Without reasons these invasive changes may turn out to be completely inappropriate.

Thanks,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#4 lore raw
Quoted 3 lines · Herbert Xu

Please give more specific reasons for needing these events because it is certainly far from obvious from reading those documents.

Never mind, I've found your earlier messages on the list which explains your reasons more clearly. It would be nice if you could include those explanations in your patch description.

BTW, does this mean that we're now comfortable with full TOE?

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#5 lore raw
Quoted 9 lines · Herbert Xu
Quoted 2 lines · Herbert Xu

Please give more specific reasons for needing these events because it is certainly far from obvious from reading those documents.

Never mind, I've found your earlier messages on the list which explains your reasons more clearly. It would be nice if you could include those explanations in your patch description.

BTW, does this mean that we're now comfortable with full TOE?

I don't see how that position has changed?

http://linux-net.osdl.org/index.php/TOE

Jeff

#6 lore raw
Quoted 4 lines · Jeff Garzik

I don't see how that position has changed?

http://linux-net.osdl.org/index.php/TOE

Well I must say that RDMA over TCP smells very much like TOE. They've got an ARP table, a routing table, and presumably a TCP stack.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#7 lore raw

TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

Quoted 7 lines · Herbert Xu
Quoted 3 lines · Jeff Garzik

I don't see how that position has changed?

http://linux-net.osdl.org/index.php/TOE

Well I must say that RDMA over TCP smells very much like TOE. They've got an ARP table, a routing table, and presumably a TCP stack.

A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)

If a TOE vendors wants to do TOE in a way that is transparent to the kernel, more power to them. Such non-Linux TCP stack solutions still suffer many of the problems listed at the web page above, but at least they impose no burden on kernel maintenance.

i.e. we really _do not_ want to get into the habit of co-managing arp tables, routing tables, filtering rules, and dozens of other such resources with multiple remote, independent TCP stack. We have enough complexity as it is today, coordinating between the random variations of SMP, uniprocessor, and NUMA machines out there. Not to mention competing with under-the-hood firmware actions (ASF) on NICs.

As an aside, RDMA over TCP just seems silly. TCP was _not_ meant to do the things that RDMA users want. The infiniband/RDMA programming model is an ultra-low-latency polling model where one or two apps are allowed to completely consume the machine, either busy-waiting or processing messages.

Unfortunately I don't have more details, so you just get a generalized rant :)

Jeff

#8 lore raw

Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

Quoted 5 lines · Jeff Garzik

A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)

Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?

Quoted 2 lines

Unfortunately I don't have more details, so you just get a generalized rant :)

OK, the patch under discussion here adds hooks to all the stuff in the previous paragraph for the purpose of RDMA over TCP (well I must say that the exact RDMA application/hardware has never been clearly given but this is what I can gather from the previous posts).

Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#9 lore raw

Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

Quoted 6 lines · Herbert Xu
Quoted 1 line · Jeff Garzik

A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)

Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?

Well, you've always been able to implement userspace (or otherwise completely-virtualized) network stack. tuntap and the packet socket enable that, if nothing else. But, like you characterize below, those are existing, well-defined, easily contained interfaces.

Quoted 5 lines

Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.

Strongly agreed.

Jeff

maintainer
#10 lore raw

Re: TOE, etc.

In reply to Herbert Xu

From: Herbert Xu <[email protected]>
Date: Wed, 28 Jun 2006 14:29:59 +1000

Quoted 10 lines · Jeff Garzik
Quoted 5 lines

A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)

Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?

Socket state, and that is one thing I don't see them doing yet.

Quoted 5 lines

Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.

Yeah, it's starting to smell really bad.

But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.

Right?

#11 lore raw

Re: TOE, etc.

Quoted 2 lines · David Miller

Socket state, and that is one thing I don't see them doing yet.

I wonder what happens when the Linux TCP stack attempts to open a connection to a remote host when that connection is already open in the RDMA NIC? For that matter what happens if a Linux application decides to listen on a TCP port already listened on by the RDMA NIC?

The only saving grace is that they're only doing RDMA rather than arbitrary TCP. However, exactly the same infrastructure can be used to do arbitrary TCP should they wish to.

Quoted 5 lines

But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.

Right?

Yes, however I think the same argument could be applied to TOE.

With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#12 lore raw

Re: TOE, etc.

From: Herbert Xu <[email protected]>
Date: Wed, 28 Jun 2006 15:35:54 +1000

Quoted 4 lines

With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.

That's true. I don't think we should really add any more help for these kinds of things then.

author
#13 lore raw

Re: TOE, etc.

In reply to Herbert Xu
Quoted 18 lines · Herbert Xu
Quoted 2 lines · David Miller

Socket state, and that is one thing I don't see them doing yet.

I wonder what happens when the Linux TCP stack attempts to open a connection to a remote host when that connection is already open in the RDMA NIC? For that matter what happens if a Linux application decides to listen on a TCP port already listened on by the RDMA NIC?

The only saving grace is that they're only doing RDMA rather than arbitrary TCP. However, exactly the same infrastructure can be used to do arbitrary TCP should they wish to.

Quoted 4 lines

But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.

Quoted 9 lines
Quoted 1 line

Right?

Yes, however I think the same argument could be applied to TOE.

With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.

Doesn't iSCSI have the same issue? No netfilter, IPsec, tcpdump, etc...

author
#14 lore raw

Re: TOE, etc.

In reply to Herbert Xu
Quoted 10 lines · Herbert Xu
Quoted 2 lines · David Miller

Socket state, and that is one thing I don't see them doing yet.

I wonder what happens when the Linux TCP stack attempts to open a connection to a remote host when that connection is already open in the RDMA NIC? For that matter what happens if a Linux application decides to listen on a TCP port already listened on by the RDMA NIC?

This issue would have to be handled by using seperate IP addresses for RDMA connections vs native stack TCP.

Consider NFS-RDMA server. Through administration, it would be configured to listen on the specific rdma ip addresses, and the native stack tcp ip addresses and thus support both TCP and RDMA NFS connections.

There are definitely issues with this that could be resolved via tighter integration, but that seems to not be a goal of the linux community at this time...

Quoted 8 lines

The only saving grace is that they're only doing RDMA rather than arbitrary TCP. However, exactly the same infrastructure can be used to do arbitrary TCP should they wish to.

Quoted 4 lines

But we have to realize they've already been given %95 of the interfaces they need to speak IP using our routes and our neighbour entries.

Quoted 9 lines
Quoted 1 line

Right?

Yes, however I think the same argument could be applied to TOE.

With their RDMA NIC, we'll have TCP/SCTP connections that bypass netfilter, tc, IPsec, AF_PACKET/tcpdump and the rest of our stack while at the same time it is using the same IP address as us and deciding what packets we will or won't see.

Doesn't iSCSI have this same issue?

Steve.

maintainer
#15 lore raw

Re: TOE, etc.

From: Steve Wise <[email protected]>
Date: Wed, 28 Jun 2006 09:54:57 -0500

Quoted 1 line

Doesn't iSCSI have this same issue?

Software iSCSI implementations don't have the issue because they go through the stack using normal sockets and normal device send and receive.

author
#16 lore raw

Re: TOE, etc.

Quoted 9 lines · David Miller

From: Steve Wise <[email protected]>
Date: Wed, 28 Jun 2006 09:54:57 -0500

Quoted 1 line

Doesn't iSCSI have this same issue?

Software iSCSI implementations don't have the issue because they go through the stack using normal sockets and normal device send and receive.
-

Right. I was assuming, in this thread we were talking about iSCSI devices where the TCP stack is in HW/FW on the adapter...

Steve.

author
#17 lore raw

Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

In reply to Herbert Xu
Quoted 11 lines · Herbert Xu
Quoted 5 lines · Jeff Garzik

A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)

Agreed. However, when they start adding hooks to the ARP table, the routing table, and PMTU management, it begs the question what more is there to add for TOE (well, user-space driven TOE at least)?

Quoted 7 lines
Quoted 2 lines

Unfortunately I don't have more details, so you just get a generalized rant :)

OK, the patch under discussion here adds hooks to all the stuff in the previous paragraph for the purpose of RDMA over TCP (well I must say that the exact RDMA application/hardware has never been clearly given but this is what I can gather from the previous posts).

There are Ammasso and Chelsio RDMA/Ethernet drivers in the openib.org svn iwarp branch today. The goal is to submit them for review and inclusion into linux. The Ammasso driver has been through 3 review cycles on lkml and netdev. There are other vendors with drivers, but they're currently not disclosing any information to me about their status.

Applications:

kernel: NFS-RDMA, iSER, RDP.
user: MPI, uDAPL (both middle ware).

The Ammasso driver is a different model. It actually has a full TCP/ARP/ICMP stack and doesn't require these hooks. But the RDMA/TCP model defined and implemented, I think, by most vendors is a model where the HW is doing a limited TCP offload, relying on the native stack for L2 and L3 integration (as described in the netevent patch).

Quoted 6 lines

Put it another way, I think the dividing line between TOE and iSCSI or virtualisation is exactly the interface between them and the Linux kernel. If the interface is an existing one such as SCSI or standard IP then it's OK. However, when it starts poking in the guts of the Linux stack I'd say that it has crossed the line.

Don't these netevent hooks have utility for other purposes? IE: Should we really shoot changes to linux _just because_ they might possibly enable TOE?

Steve.

author
#18 lore raw

Re: TOE, etc. (was Re: [PATCH Round 3 0/2][RFC] Network Event Notifier Mechanism)

In reply to Jeff Garzik
Quoted 6 lines · Jeff Garzik
Quoted 5 lines · Herbert Xu
Quoted 3 lines · Jeff Garzik

I don't see how that position has changed?

http://linux-net.osdl.org/index.php/TOE

Quoted 8 lines
Quoted 2 lines

Well I must say that RDMA over TCP smells very much like TOE. They've got an ARP table, a routing table, and presumably a TCP stack.

A PCI device that presents itself as a SCSI controller, but under the hood is really iSCSI-over-TCP smells like TOE. Running a virtualized Linux guest on top of a proprietary stack [which provides networking services to guests] also smells like TOE. :)

I wonder if the existing iSCSI solutions handle next host mac addr changes, pmtu changes, etc? They _can_, if they implement the entire ARP and ICMP suites in HW/FW. Just curious if these vendors also see merit in the netevent changes I'm proposing...

Quoted 12 lines

If a TOE vendors wants to do TOE in a way that is transparent to the kernel, more power to them. Such non-Linux TCP stack solutions still suffer many of the problems listed at the web page above, but at least they impose no burden on kernel maintenance.

i.e. we really _do not_ want to get into the habit of co-managing arp tables, routing tables, filtering rules, and dozens of other such resources with multiple remote, independent TCP stack. We have enough complexity as it is today, coordinating between the random variations of SMP, uniprocessor, and NUMA machines out there. Not to mention competing with under-the-hood firmware actions (ASF) on NICs.

Quoted 6 lines

As an aside, RDMA over TCP just seems silly. TCP was _not_ meant to do the things that RDMA users want. The infiniband/RDMA programming model is an ultra-low-latency polling model where one or two apps are allowed to completely consume the machine, either busy-waiting or processing messages.

With RDMA over TCP, you can get the same ultra-low-latency, interrupt coalescing or avoidance, and copy avoidance as with Infiniband (with newly arriving 10Gb RDMA NICs). The benefit over IB, IMO, is the fact that your infrastructure is all IP.

If you're interested in a kernel-mode app that benfits from RDMA, then check out NFS-RDMA, which runs today transparently over Infiniband and RDMA/TCP devices using the Infiniband RDMA-CM and verbs with minor changes to support RDMA/TCP devices.

http://sourceforge.net/projects/nfs-rdma/

Steve.