NET: Add TCP connection abort IOCTL

Adds an IOCTL for aborting established TCP connections, and is designed to be an HA performance improvement for cleaning up, failure notification, and application termination.

---

 include/linux/ipv6.h          |    8 ++++
 include/linux/socket.h        |    5 ++
 include/linux/sockios.h       |    1 
 include/net/inet_hashtables.h |    6 +++
 net/ipv4/tcp.c                |   15 +++++++
 net/ipv4/tcp_ipv4.c           |   86 +++++++++++++++++++++++++++++++++++++++++
 6 files changed, 121 insertions(+), 0 deletions(-)

---

index f824113..42f6765 100644
@@ -467,6 +467,14 @@ #define INET6_MATCH(__sk, __hash, __sadd
 	 ipv6_addr_equal(&inet6_sk(__sk)->rcv_saddr, (__daddr))	&& \
 	 (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
 
+#define TCP_IPV6_WILDCARD_MATCH(__sk, __saddr, __sport,__daddr, __dport) \
+	((ipv6_addr_any(&__saddr) 					|| \
+	  ipv6_addr_equal(&__saddr, &inet6_sk(__sk)->saddr))		&& \
+	 ((!__sport) || (ntohs(__sport) == inet_sk(__sk)->num))		&& \
+	 (ipv6_addr_any(&__daddr)					|| \
+	   ipv6_addr_equal(&__daddr, &inet6_sk(__sk)->daddr))		&& \
+	 ((!__dport) || (__dport ==  inet_sk(__sk)->dport)))
+
 #endif /* __KERNEL__ */
 
 #endif /* _IPV6_H */

index fcd35a2..0bf7b0a 100644
@@ -48,6 +48,11 @@ struct linger {
 
 #define sockaddr_storage __kernel_sockaddr_storage
 
+struct tcp_abort_sockaddr_storage {
+	struct sockaddr_storage local;	/* local address for lookup */
+	struct sockaddr_storage remote;	/* Remote address for lookup */
+};
+
 /*
  *	As we do 4.4BSD message passing we use a 4.4BSD message passing
  *	system, not 4.3. Thus msg_accrights(len) are now missing. They

index abef759..b850577 100644
@@ -140,4 +140,5 @@ #define SIOCDEVPRIVATE	0x89F0	/* to 89FF
  */
  
 #define SIOCPROTOPRIVATE 0x89E0 /* to 89EF */
+#define SIOCABORTCONN SIOCPROTOPRIVATE + 1
 #endif	/* _LINUX_SOCKIOS_H */

index d27ee8c..735739a 100644
@@ -339,6 +339,12 @@ #define INET_TW_MATCH(__sk, __hash,__coo
 	 (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
 #endif /* 64-bit arch */
 
+#define TCP_IPV4_WILDCARD_MATCH(__sk, __saddr, __sport,__daddr, __dport)    \
+	(((__saddr == INADDR_ANY) || (__saddr ==  inet_sk(__sk)->saddr)) && \
+	 ((!__sport) || (ntohs(__sport) == inet_sk(__sk)->num)) 	 && \
+	 ((__daddr == INADDR_ANY) || (__daddr == inet_sk(__sk)->daddr))  && \
+	 ((!__dport) || (__dport ==  inet_sk(__sk)->dport)))
+
 /*
  * Sockets in TCP_CLOSE state are _always_ taken out of the hash, so we need
  * not check it for lookups anymore, thanks Alexey. -DaveM

index 3834b10..b3e7a6f 100644
@@ -442,6 +442,21 @@ int tcp_ioctl(struct sock *sk, int cmd, 
 		else
 			answ = tp->write_seq - tp->snd_una;
 		break;
+	case SIOCABORTCONN:
+	{
+		struct tcp_abort_sockaddr_storage tcp_abort;
+		void __user *data = (void __user *)arg;
+
+		if (!capable(CAP_NET_ADMIN)){
+			return -EACCES;
+		}
+
+		/* Check permissions */
+		if (copy_from_user(&tcp_abort, data, sizeof(tcp_abort))) {
+			return -EFAULT;
+		}
+		return tcp_handle_abort_req(&tcp_abort.local, &tcp_abort.remote);
+	}
 	default:
 		return -ENOIOCTLCMD;
 	};

1.4.1

From: [email protected] (David Griego)
Date: Tue, 27 Mar 2007 14:47:54 -0700

SO_LINGER with a zero linger time plus close() isn't working properly?

There is no reason for this ioctl at all. Either existing facilities provide what you need or what you want is a protocol violation we can't do.

Actually, there are legitimate uses for this sort of API. The patch allows an administrator to kill specific connections that are in use by other applications, where the close is not available, since the socket is owned by another process. Say one of your large applications has hundreds or even thousands of open connections and you have determined that a particular connection is causing trouble. This API allows the admin to kill that particular connection, and doesn't appear to violate any RFC offhand, since an abort is sent to the peer.

One may argue that the applications should be modified, but that is not always possible in the case of various ISVs. As Linux gains market share in the large server market, more and more applications are being ported from other platforms that have this sort of management/administrative interfaces.

Mark Huth

I also believe this is a useful thing to have. I'm not 100% sure this ioctl is the way to go, but it seems reasonable. This directly corresponds to writing deleteTcb to the tcpConnectionState variable in the TCP MIB (RFC 4022). I don't think it constitutes a protocol violation.

As a concrete example of a way I've used this type of feature is to defend against a netkill [1] style attack, where the defense involves making decisions about which connections to kill when memory gets scarce. It makes sense to do this with a system daemon, since an admin might have an arbitrarily complicated policy as to which applications and peers have priority for the memory. This is too complicated to distribute and enforce across all applications. You could do this in the kernel, but why if you don't have to?

-John

[1] http://shlang.com/netkill/

Responding to myself in good form :P I'll add that there are other ways to do this currently but all I know of are hackish, f.e. using a raw socket to send RST packets to yourself.

-John

While not pretty, it is easy enough to ptrace a process using gdb and call shutdown(2) on the fd.

Cheers,

Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

From: John Heffner <[email protected]>
Date: Tue, 27 Mar 2007 20:27:44 -0400

On the contrary this sounds like an excellent task for a netfilter based solution.

From: Mark Huth <[email protected]>
Date: Tue, 27 Mar 2007 16:09:30 -0700

Anything that wants to act as an external agent to manipulate or terminate connections should use netfilter.

This is what I thought too at the begining.

But after some thinking I recalled having to reboot machines just because netfilter was not in (because of noticeable performance hit), and I could find
the tree to compile netfilter as modules..

When I saw revoke() work in progess, I did react like you : This is coming from hell...

From: Eric Dumazet <[email protected]>
Date: Wed, 28 Mar 2007 08:02:21 +0200

Another option, similar in vain to what Herbert suggested (the gdb hack) is to provide a way for root to open file descriptors of other processes and handle it like that.

Then you could just set the linger socket option properly and close it up.

Unfortunately we prevent openning of /proc/${PID}/fd/${FD} objects when they are sockets for well documented reasons.

Using an ioctl with a socket demux key is just beyond disgusting, it's as simple as that. We have abstractions already for these objects so let's use them.

I agree that 99 times out of ten such a mechanism serves only as a massive KLUDGE to paper-over application bugs. I'll also sadly point-out that such a mechanism exists in HP-UX 11.X and I suspect Solaris !-( I've spent probably the last decade or so attempting to discourage its use in the HP-UX space, but like some daemon from hell it just refuses to die.

rick jones

An ioctl like that is sorely missed for high-availability clustering on Linux and things like the Carrier Grade Linux. Other Unices do have something similar - there is TCP_IOC_ABORT_CONN ioctl on Solaris and on FreeBSD/OpenBSD there is TCPCTL_DROP ioctl (accompanied by tcpdrop utility).

Need for such an API is to a degree indicated in the Carrier Grade Linux requirements by The Linux Foundation (former OSDL). CGL Clustering requirements document addresses the need to deal with open TCP sessions in events of node failure. For example, Carrier Grade Linux Clustering Requirements Definition (at
http://groups.osdl.org/apps/group_public/download.php/1981/cgl-cluster.pdf)
has two such requirements on its roadmap list:

CAF.2.3 Deliberate TCP Session Takeover CAF.2.4 TCP Session Takeover on Node Failure

(There they talk about session takeover, which is in my opinion quite difficult to implement, and before such a thing is achieved there is a clear need for ioctl similar to the one proposed - to tell all clients with TCP sessions towards the IP address of a failed node that their sessions are broken, and that they should handle that instantly, not wait for timeouts to expire. And a client might be a general 3rd party software where one does not set socket options oneself.)

I'll give a possible use case - let's say we have a highly-available cluster in active/passive configuration. The active node runs some services and listens on cluster's IP address. Clients use the cluster's services by establishing TCP sessions to that IP address. When the active node fails, the other node should as fast as possible take over the same IP address and offer the same services, with clients not noticing anything or noticing as little as possible. Ideally, TCP sessions should be taken over (as indicated in the Carrier Grade Linux requirements), enabling clients to continue transparently. But, TCP session takeover is a rather tough requirement, and before something like that exists (if ever) it would be quite useful with such ioctl to enable clients to discover broken sessions and recover in matter of seconds instead of minutes. Obviously, if one envisages failover of any kind of service, then you can't control what socket options are used by someone.

Cheers,
Predrag Hodoba

From: "Predrag Hodoba" <[email protected]>
Date: Thu, 29 Mar 2007 16:56:22 +0200

Something being in the CGL specification is to me exactly a great reason NOT to add it. That specification is so full of garbage it's unbelievable.

Thanks, you've given me one more reason not to even remotely consider adding this feature.

Agreed, CGL is a vendor driven group that has always wanted to replicate proprietary misfeatures onto Linux. There is a real requirement to provide high availability but there should be no requirement to implement the solution in the same crap way as legacy Unix.

OK, let's put aside CGL and legacy Unices.

Still, I don't see how the case I mentioned can easily be handled. (The case being - effective clean up of all affected client TCP connections, following failover of the server IP address from active to passive node in a highly available cluster).

Why clean them up? The client connections will timeout and they can reconnect. Actively killing them early does nothing helpful. Just like the CGL requirement for forced unmount, the forced operation introduces a whole bunch of race conditions and shared file descriptor problems.

Well, it depends on how fast you have to react on failure. For data-center grade high-availability it is, as you said, enough to wait for clients to timeout. For telco (or similar, more demanding) grade of high-availability, timeout just takes too long. You typically have to discover failure using some kind of heartbeat mechanism and clean up immediately ...

If the switchover from active to standby is "commanded" then there is the opportunity to "tell" the applications on the server to close their connections - either explicitly with some sort of defined interface, or implicitly by killing the processes. Then the IP can be brought-up on the standby and processes started/enabled/whatever and the clients can establish their new connections. The ioctl here (at least if it is like the tcp_discon options in HP-UX/Solaris) wouldn't be any better than just killing the process in so far as what happens on the network - in fact, it could be worse since the RST will not be retransmitted if lost, but FINs would. So, the ioctl could still leave clients twisting in the ether waiting for their application-level heartbeats to kick-in anyway.
Heck, depending on their heartbeat lengths, even the FIN stuff if lost
could leave them depending on their heartbeats.

If the switchover from active to standby is "uncommanded" it probably means the primary went belly-up which means you don't have the opportunity to make an ioctl call anyway, and you are back to the heartbeats.

rick jones

What I meant is - it could be used on ***client***. Because clients are left stranded with invalid connections when a primary fails (your "uncommanded" switchover scenario). If you wait for them to timeout, that will indeed happen, but it takes time and you are not back online as fast as you would like. If cluster's services running on a client already know about the failover (by means of "heartbeat" and observing change in cluster membership), then they can propagate that knowledge to all processes uneccessarily blocked in their socket calls towards the failed IP address. If these connections are forcibly disconnected, the respective sockets' calls would return with error code and their processes can reconnect in few seconds after the failure and continue to do what they are meant to do.

predrag