net/hsr: Add support for the High-availability Seamless Redundancy protocol (HSRv0)

High-availability Seamless Redundancy ("HSR") provides instant failover redundancy for Ethernet networks. It requires a special network topology where all nodes are connected in a ring (each node having two physical network interfaces). It is suited for applications that demand high availability and very short reaction time.

HSR acts on the Ethernet layer, using a registered Ethernet protocol type to send special HSR frames in both directions over the ring. The driver creates virtual network interfaces that can be used just like any ordinary Linux network interface, for IP/TCP/UDP traffic etc. All nodes in the network ring must be HSR capable.

This code is a "best effort" to comply with the HSR standard as described in IEC 62439-3:2010 (HSRv0).

---

This is a patch against net-next (2013-09-18).

The code has been statically analysed using sparse (no problems detected).

kmemleak has been used to detect memory leaks (no known leaks exist).

The code passes checkpatch.pl with the options --strict --max-line-length=83 except for CHECKs, which are all intentional; e.g. CamelCase is used in the IEC standard to describe field names, and these names have been used in the code.

The code has been tested extensively on rings of up to 32 avr32-based nodes. Less intense testing has included an x86_64 (SMP) system, and running HSR over a USB-to-Ethernet device.

Changes in v4:

* Fixed racy test-and-modify lock for device operstate in __hsr_set_operstate().

Changes in v3:

* hsr_dev_open() no longer calls dev_open() on the HSR slave interfaces. This is to let user space decide how to treat any errors on slave open.
* is_operstate_up() is now called is_slave_up() and checks both admin state and operational state. Slaves going DOWN could otherwise be detected as still UP when HSR decided its own state.
* Files under net/ now have header comments that follow the network code standard.

Changes in v2:

* Aligned newlines with open parenthesis (Joe Perches)
* Removed unneccessary WARN_ON_ONCE from is_hsr_master() (Joe Perches)
* Removed deprecated comments about "actually signed" for HSR_A_IFn_AGE (Joe Perches)
* Broke out userspace hsr_netlink.h into include/uapi/linux/ (Joe Perches)
* Use netif_oper_up() to check for operational state UP (Stephen Hemminger)

Possibly unresolved:

* dev_base_lock vs rtnl_mutex when setting operstate? (Stephen Hemminger) David Miller asked about this on 2013-09-03; no response.
* Move more (local) header files into include/net/? (checkpatch still warns about CamelCase) (Joe Perches / with help from David Miller?)

Earlier RFC:s:
RFC v1: http://www.spinics.net/lists/netdev/msg192817.html
RFC v2: http://www.spinics.net/lists/netdev/msg203397.html
RFC v3: http://www.spinics.net/lists/netdev/msg207816.html
RFC v4: http://www.spinics.net/lists/netdev/msg213309.html

Thanks to Stephen Hemminger, Joe Perches, and others for their comments on these RFC:s.

 include/uapi/linux/hsr_netlink.h |  50 ++++
 include/uapi/linux/if_ether.h    |   1 +
 include/uapi/linux/if_link.h     |  13 +
 net/Kconfig                      |   1 +
 net/Makefile                     |   1 +
 net/hsr/Kconfig                  |  27 ++
 net/hsr/Makefile                 |   7 +
 net/hsr/hsr_device.c             | 594 +++++++++++++++++++++++++++++++++++++++
 net/hsr/hsr_device.h             |  29 ++
 net/hsr/hsr_framereg.c           | 501 +++++++++++++++++++++++++++++++++
 net/hsr/hsr_framereg.h           |  53 ++++
 net/hsr/hsr_main.c               | 458 ++++++++++++++++++++++++++++++
 net/hsr/hsr_main.h               | 166 +++++++++++
 net/hsr/hsr_netlink.c            | 457 ++++++++++++++++++++++++++++++
 net/hsr/hsr_netlink.h            |  30 ++
 15 files changed, 2388 insertions(+)
 create mode 100644 include/uapi/linux/hsr_netlink.h
 create mode 100644 net/hsr/Kconfig
 create mode 100644 net/hsr/Makefile
 create mode 100644 net/hsr/hsr_device.c
 create mode 100644 net/hsr/hsr_device.h
 create mode 100644 net/hsr/hsr_framereg.c
 create mode 100644 net/hsr/hsr_framereg.h
 create mode 100644 net/hsr/hsr_main.c
 create mode 100644 net/hsr/hsr_main.h
 create mode 100644 net/hsr/hsr_netlink.c
 create mode 100644 net/hsr/hsr_netlink.h

new file mode 100644
index 0000000..2475cb8
@@ -0,0 +1,50 @@
+/*
+ * Copyright 2011-2013 Autronica Fire and Security AS
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * Author(s):
+ *	2011-2013 Arvid Brodin, [email protected]
+ */
+
+#ifndef __UAPI_HSR_NETLINK_H
+#define __UAPI_HSR_NETLINK_H
+
+/* Generic Netlink HSR family definition
+ */
+
+/* attributes */
+enum {
+	HSR_A_UNSPEC,
+	HSR_A_NODE_ADDR,
+	HSR_A_IFINDEX,
+	HSR_A_IF1_AGE,
+	HSR_A_IF2_AGE,
+	HSR_A_NODE_ADDR_B,
+	HSR_A_IF1_SEQ,
+	HSR_A_IF2_SEQ,
+	HSR_A_IF1_IFINDEX,
+	HSR_A_IF2_IFINDEX,
+	HSR_A_ADDR_B_IFINDEX,
+	__HSR_A_MAX,
+};
+#define HSR_A_MAX (__HSR_A_MAX - 1)
+
+
+/* commands */
+enum {
+	HSR_C_UNSPEC,
+	HSR_C_RING_ERROR,
+	HSR_C_NODE_DOWN,
+	HSR_C_GET_NODE_STATUS,
+	HSR_C_SET_NODE_STATUS,
+	HSR_C_GET_NODE_LIST,
+	HSR_C_SET_NODE_LIST,
+	__HSR_C_MAX,
+};
+#define HSR_C_MAX (__HSR_C_MAX - 1)
+
+#endif /* __UAPI_HSR_NETLINK_H */

index ade07f1..2ce0f6a 100644
@@ -85,6 +85,7 @@
 #define ETH_P_8021AH	0x88E7          /* 802.1ah Backbone Service Tag */
 #define ETH_P_MVRP	0x88F5          /* 802.1Q MVRP                  */
 #define ETH_P_1588	0x88F7		/* IEEE 1588 Timesync */
+#define ETH_P_PRP	0x88FB		/* IEC 62439-3 PRP/HSRv0	*/
 #define ETH_P_FCOE	0x8906		/* Fibre Channel over Ethernet  */
 #define ETH_P_TDLS	0x890D          /* TDLS */
 #define ETH_P_FIP	0x8914		/* FCoE Initialization Protocol */

index 80394e8..1fb8ce5 100644
@@ -470,4 +470,17 @@ enum {
 
 #define IFLA_IPOIB_MAX (__IFLA_IPOIB_MAX - 1)
 
+
+/* HSR section */
+
+enum {
+	IFLA_HSR_UNSPEC,
+	IFLA_HSR_SLAVE1,
+	IFLA_HSR_SLAVE2,
+	IFLA_HSR_MULTICAST_SPEC,
+	__IFLA_HSR_MAX,
+};
+
+#define IFLA_HSR_MAX (__IFLA_HSR_MAX - 1)
+
 #endif /* _UAPI_LINUX_IF_LINK_H */

index ee02136..718935c 100644
@@ -220,6 +220,7 @@ source "net/openvswitch/Kconfig"
 source "net/vmw_vsock/Kconfig"
 source "net/netlink/Kconfig"
 source "net/mpls/Kconfig"
+source "net/hsr/Kconfig"
 
 config RPS
 	boolean

index 9492e8c..8fa2f91 100644
@@ -71,3 +71,4 @@ obj-$(CONFIG_NFC)		+= nfc/
 obj-$(CONFIG_OPENVSWITCH)	+= openvswitch/
 obj-$(CONFIG_VSOCKETS)	+= vmw_vsock/
 obj-$(CONFIG_NET_MPLS_GSO)	+= mpls/
+obj-$(CONFIG_HSR)		+= hsr/

new file mode 100644
index 0000000..0d3d709
@@ -0,0 +1,27 @@
+#
+# IEC 62439-3 High-availability Seamless Redundancy
+#
+
+config HSR
+	tristate "High-availability Seamless Redundancy (HSR)"
+	---help---
+	  If you say Y here, then your Linux box will be able to act as a
+	  DANH ("Doubly attached node implementing HSR"). For this to work,
+	  your Linux box needs (at least) two physical Ethernet interfaces,
+	  and it must be connected as a node in a ring network together with
+	  other HSR capable nodes.
+
+	  All Ethernet frames sent over the hsr device will be sent in both
+	  directions on the ring (over both slave ports), giving a redundant,
+	  instant fail-over network. Each HSR node in the ring acts like a
+	  bridge for HSR frames, but filters frames that have been forwarded
+	  earlier.
+
+	  This code is a "best effort" to comply with the HSR standard as
+	  described in IEC 62439-3:2010 (HSRv0), but no compliancy tests have
+	  been made.
+
+	  You need to perform any and all necessary tests yourself before
+	  relying on this code in a safety critical system!
+
+	  If unsure, say N.

new file mode 100644
index 0000000..b68359f
@@ -0,0 +1,7 @@
+#
+# Makefile for HSR
+#
+
+obj-$(CONFIG_HSR)	+= hsr.o
+
+hsr-y			:= hsr_main.o hsr_framereg.o hsr_device.o hsr_netlink.o

new file mode 100644
index 0000000..2c7148e
@@ -0,0 +1,29 @@
+/* Copyright 2011-2013 Autronica Fire and Security AS
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * Author(s):
+ *	2011-2013 Arvid Brodin, [email protected]
+ */
+
+#ifndef __HSR_DEVICE_H
+#define __HSR_DEVICE_H
+
+#include <linux/netdevice.h>
+#include "hsr_main.h"
+
+void hsr_dev_setup(struct net_device *dev);
+int hsr_dev_finalize(struct net_device *hsr_dev, struct net_device *slave[2],
+		     unsigned char multicast_spec);
+void hsr_set_operstate(struct net_device *hsr_dev, struct net_device *slave1,
+		       struct net_device *slave2);
+void hsr_set_carrier(struct net_device *hsr_dev, struct net_device *slave1,
+		     struct net_device *slave2);
+void hsr_check_announce(struct net_device *hsr_dev, int old_operstate);
+bool is_hsr_master(struct net_device *dev);
+int hsr_get_max_mtu(struct hsr_priv *hsr_priv);
+
+#endif /* __HSR_DEVICE_H */

new file mode 100644
index 0000000..d4579dc
@@ -0,0 +1,30 @@
+/* Copyright 2011-2013 Autronica Fire and Security AS
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * Author(s):
+ *	2011-2013 Arvid Brodin, [email protected]
+ */
+
+#ifndef __HSR_NETLINK_H
+#define __HSR_NETLINK_H
+
+#include <linux/if_ether.h>
+#include <linux/module.h>
+#include <uapi/linux/hsr_netlink.h>
+
+struct hsr_priv;
+
+int __init hsr_netlink_init(void);
+void __exit hsr_netlink_exit(void);
+
+void hsr_nl_ringerror(struct hsr_priv *hsr_priv, unsigned char addr[ETH_ALEN],
+		      int dev_idx);
+void hsr_nl_nodedown(struct hsr_priv *hsr_priv, unsigned char addr[ETH_ALEN]);
+void hsr_nl_framedrop(int dropcount, int dev_idx);
+void hsr_nl_linkdown(int dev_idx);
+
+#endif /* __HSR_NETLINK_H */

1.8.1.5

-- 
Arvid Brodin | Consultant (Linux)
XDIN AB | Knarrarnäsgatan 7 | SE-164 40 Kista | Sweden | xdin.com

From: Arvid Brodin <[email protected]>
Date: Thu, 19 Sep 2013 03:11:58 +0200

Isn't it reasonable to assume that skb points to an sk_buff with a valid Ethernet header, and with skb->data pointing to the end of that header, at the start of ETH_P_... protocol handlers?

Are you saying packet data may contain offsets from itself to the mac header? Or are you talking about fragments? This being the Ethernet protocol handler, e.g. skb->network_header and skb->transport_header should not have been set yet.

Anyway, I should do

	skb_deliver->len -= HSR_TAGLEN;

here! (I'm removing 6 bytes after the mac header from the packet, not just realigning it.)

Given all this, and looking at e.g. __pskb_pull_tail(), it should be correct to memmove() only the linear part of the packet after the removed bytes - any fragments will just be appended directly after the end of the (shortened) linear part on use?

It's weird though, the "len" bug doesn't show up in my practical tests - scp of 16 MiB file (sha1sum compared afterwards) and ping with default and 65 KB packets work just fine before and after this change - even between one fixed system and one with the bug remaining. Could it be that none of the network device drivers we use (e1000e and macb) make use of nonlinear skbs?

Arvid Brodin | Consultant (Linux)
XDIN AB | Knarrarnäsgatan 7 | SE-164 40 Kista | Sweden | xdin.com

Forget about this, e1000e is running on an architecture with HAVE_EFFICIENT_UNALIGNED_ACCESS, and on macb it does affect things - with the "fix" nothing works. :-|

Arvid Brodin | Consultant (Linux)
XDIN AB | Knarrarnäsgatan 7 | SE-164 40 Kista | Sweden | xdin.com

... and that is because I have already shortened the packet through the call to strip_hsr_tag() a few lines above, which calls skb_pull(..., HSR_TAGLEN) in turn. So ->len is already correct.

Moving both ->data and ->tail, and all data between them, HSR_TAGLEN bytes lower, should not be a problem if

a) the space ahead of ->data is vacant (not part of the packet),
b) there is enough headroom,
c) it does not break 32-bit alignment, and
d) there are no references to headers etc in the data being moved.

Right?

Arvid Brodin | Consultant (Linux)
XDIN AB | Knarrarnäsgatan 7 | SE-164 40 Kista | Sweden | xdin.com

(Shorter and more to the point than my previous replies:)

I _do_ know: this can't possibly underflow since strip_hsr_tag() a few lines above pulled the same amount of data. I will rename strip_hsr_tag() to hsr_pull_tag() to make this clearer.

How does this matter? The memmove moves everything back (restores the changes made to the packet on the sending side) so that it is at the "normal" position for an ethernet packet.

Arvid Brodin | Consultant (Linux)
XDIN AB | Knarrarnäsgatan 7 | SE-164 40 Kista | Sweden | xdin.com

Obviously, David is too busy to help me figure out what the problem is (I know he reviews several thousand patches each year, so maybe that's no wonder).

If anyone else has got an idea you are very welcome to chime in, and perhaps we can solve this. I can't fix the problem if I don't understand it.

Maybe he's talking about systems with NET_SKBUFF_DATA_USES_OFFSET here? This means the transport, network and mac headers are all relative to skb->head, if I understand correctly. But at the point in the protocol stack where this code is (the Ethernet protocol handler), the transport and network headers have not been set yet, and the mac header is not moved by the code. And tail is updated by the code. So that should not be a problem?

Arvid Brodin | Consultant (Linux)
XDIN AB | Knarrarnäsgatan 7 | SE-164 40 Kista | Sweden | xdin.com