patch: support long (above 14 bytes) HW addresses in arp_ioctl

PATCH Constantine Gavrilov 3 participants 6 messages Nov 3, 2008

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Constantine Gavrilov

author

2008-11-03 11:11 UTC #0 lore raw

While working with OFED infiniband stack that uses 20 byte long HW addresses for IP over IB, I have paid attention to the following arp_ioctl problem.

The ioctl uses a data structure that limits a length of HW address to 14 bytes. The IP stack and the arp cache code do not have that limitation. This leads to the following problems:

* arp_ioctl cannot be used to set, get, or delete arp entries for those adapters that have HW addresses longer than 14 bytes
* arp_ioctl will corrupt the kernel and user memory when this ioctl is used on the adapters that have HW addresses longer that 14 bytes. This is because when copying the HW address, the arp_ioctl code copies dev->addr_len bytes without checking that addr_len is not above 14 bytes. This is done both for copy_to_user() and memcpy() calls on kernel data structures allocated on stack. The memcpy() call in particular, will corrupt kernel stack.

Attached please find the patch that fixes both problems. In addition, the patch changes the maximal number of bytes for HW address that will be seen in /proc/net/arp from ~10 to ~30. Without the last change, output of /proc/net/arp truncates the the large MAC entries, which makes the arp utility useless.

The patch does not change the existing ABI but extends it. The kernel structure used in arp_ioctl calls is changed to support larger addresses, while the user-space structure is extended by appending extra-space to the end of the structure if ATF_NEWARPCTL -- a new flag -- is set in arp_flags of existing user-space structure. This allows avoiding big changes to the existing code while preserving the ABI compatibility.

Signature · 8 lines

----------------------------------------
Constantine Gavrilov
Kernel Developer
Platform Group
XIV, an IBM global brand 1 Azrieli Center, Tel-Aviv
Phone: +972-3-6074672
Fax:   +972-3-6959749
----------------------------------------

linux/if_arp.h renamed from linux/if_arp.h.orig +17 -0

@@ -99,14 +99,27 @@
 #define	ARPOP_InREPLY	9		/* InARP reply			*/
 #define	ARPOP_NAK	10		/* (ATM)ARP NAK			*/
 
+struct hwaddr {
+  sa_family_t     sa_family;      /* address family, AF_xxx       */
+  char            sa_data[30];    /* 30 bytes of HW address */
+};
 
 /* ARP ioctl request. */
 struct arpreq {
   struct sockaddr	arp_pa;		/* protocol address		*/
+  struct hwaddr		arp_ha;		/* hardware address		*/
+  int			arp_flags;	/* flags			*/
+  struct sockaddr       arp_netmask;    /* netmask (only for proxy arps) */
+  char			arp_dev[16];
+};
+
+struct arpreq_user {
+  struct sockaddr	arp_pa;		/* protocol address		*/
   struct sockaddr	arp_ha;		/* hardware address		*/
   int			arp_flags;	/* flags			*/
   struct sockaddr       arp_netmask;    /* netmask (only for proxy arps) */
   char			arp_dev[16];
+  char			arp_ha_ext[16]; /* extended part of HW address */
 };
 
 struct arpreq_old {

@@ -124,6 +137,10 @@
 #define ATF_NETMASK     0x20            /* want to use a netmask (only
 					   for proxy entries) */
 #define ATF_DONTPUB	0x40		/* don't answer this addresses	*/
+#define ATF_NEWARPCTL	0x80		/* use larger buff for hw address */
+
+#define NOT_VALID_ARP_CTL(__dev, __r) ((__dev->addr_len > sizeof(((struct arpreq_user *)__r)->arp_ha.sa_data)) && \
+	((__dev->addr_len > sizeof(__r->arp_ha.sa_data)) || !(__r->arp_flags & ATF_NEWARPCTL)))
 
 /*
  *	This structure defines an ethernet arp header.

ipv4/arp.c renamed from ipv4/arp.c.orig +53 -6

108 lines not on this page · raw message

Constantine Gavrilov

author

2008-11-03 16:36 UTC #1 lore raw

Updated version of the patch uses MAX_ADDR_LEN from netdevice.h as the maximal length of MAC address.

Quoted 11 lines · Constantine Gavrilov

While working with OFED infiniband stack that uses 20 byte long HW addresses for IP over IB, I have paid attention to the following arp_ioctl problem.
The ioctl uses a data structure that limits a length of HW address to 14 bytes. The IP stack and the arp cache code do not have that limitation. This leads to the following problems:
* arp_ioctl cannot be used to set, get, or delete arp entries for those adapters that have HW addresses longer than 14 bytes
* arp_ioctl will corrupt the kernel and user memory when this ioctl is used on the adapters that have HW addresses longer that 14 bytes. This is because when copying the HW address, the arp_ioctl code copies dev->addr_len bytes without checking that addr_len is not above 14 bytes. This is done both for copy_to_user() and memcpy() calls on kernel data structures allocated on stack. The memcpy() call in particular, will corrupt kernel stack.
Attached please find the patch that fixes both problems. In addition, the patch changes the maximal number of bytes for HW address that will be seen in /proc/net/arp from ~10 to ~30. Without the last change, output of /proc/net/arp truncates the the large MAC entries, which makes the arp utility useless.
The patch does not change the existing ABI but extends it. The kernel structure used in arp_ioctl calls is changed to support larger addresses, while the user-space structure is extended by appending extra-space to the end of the structure if ATF_NEWARPCTL -- a new flag -- is set in arp_flags of existing user-space structure. This allows avoiding big changes to the existing code while preserving the ABI compatibility.

Signature · 8 lines

----------------------------------------
Constantine Gavrilov
Kernel Developer
Platform Group
XIV, an IBM global brand 1 Azrieli Center, Tel-Aviv
Phone: +972-3-6074672
Fax:   +972-3-6959749
----------------------------------------

linux/if_arp.h renamed from linux/if_arp.h.orig +17 -0

@@ -99,14 +99,27 @@
 #define	ARPOP_InREPLY	9		/* InARP reply			*/
 #define	ARPOP_NAK	10		/* (ATM)ARP NAK			*/
 
+struct hwaddr {
+  sa_family_t	sa_family;		/* address family, AF_xxx       */
+  char		sa_data[MAX_ADDR_LEN];	/* 32 bytes of HW address	*/
+};
 
 /* ARP ioctl request. */
 struct arpreq {
   struct sockaddr	arp_pa;		/* protocol address		*/
+  struct hwaddr		arp_ha;		/* hardware address		*/
+  int			arp_flags;	/* flags			*/
+  struct sockaddr       arp_netmask;    /* netmask (only for proxy arps) */
+  char			arp_dev[16];
+};
+
+struct arpreq_user {
+  struct sockaddr	arp_pa;		/* protocol address		*/
   struct sockaddr	arp_ha;		/* hardware address		*/
   int			arp_flags;	/* flags			*/
   struct sockaddr       arp_netmask;    /* netmask (only for proxy arps) */
   char			arp_dev[16];
+  char			arp_ha_ext[MAX_ADDR_LEN-14]; /* extended part of HW address */
 };
 
 struct arpreq_old {

@@ -124,6 +137,10 @@
 #define ATF_NETMASK     0x20            /* want to use a netmask (only
 					   for proxy entries) */
 #define ATF_DONTPUB	0x40		/* don't answer this addresses	*/
+#define ATF_NEWARPCTL	0x80		/* use larger buff for hw address */
+
+#define NOT_VALID_ARP_CTL(__dev, __r) ((__dev->addr_len > sizeof(((struct arpreq_user *)__r)->arp_ha.sa_data)) && \
+	((__dev->addr_len > sizeof(__r->arp_ha.sa_data)) || !(__r->arp_flags & ATF_NEWARPCTL)))
 
 /*
  *	This structure defines an ethernet arp header.

ipv4/arp.c renamed from ipv4/arp.c.orig +53 -6

108 lines not on this page · raw message

Roland Dreier

2008-11-03 17:39 UTC #2 lore raw

In reply to Constantine Gavrilov

Quoted 7 lines

* arp_ioctl will corrupt the kernel and user memory when this ioctl is used on the adapters that have HW addresses longer that 14 bytes. This is because when copying the HW address, the arp_ioctl code copies dev->addr_len bytes without checking that addr_len is not above 14 bytes. This is done both for copy_to_user() and memcpy() calls on kernel data structures allocated on stack. The memcpy() call in particular, will corrupt kernel stack.

It's not obvious to me after a quick glance where this kernel memory corruption occurs, but clearly we should at least fix this bug.

Quoted 7 lines

The patch does not change the existing ABI but extends it. The kernel structure used in arp_ioctl calls is changed to support larger addresses, while the user-space structure is extended by appending extra-space to the end of the structure if ATF_NEWARPCTL -- a new flag -- is set in arp_flags of existing user-space structure. This allows avoiding big changes to the existing code while preserving the ABI compatibility.

However, given that applications need to be changed to use this, wouldn't it make more sense just to change those applications to use rtnetlink, which already supports large hardware addresses? ie is there much point to extending a legacy ABI to add a feature that the preferred modern interface already has?

- R.

Constantine Gavrilov

author

2008-11-03 18:58 UTC #3 lore raw

In arp_req_get() in net/arp.c, there is code:

memcpy(r->arp_ha.sa_data, neigh->ha, dev->addr_len);

dev->addr_len can be larger than size of r->arp_ha.sa_data. Inititally, I thought it would corrupt kernel stack. I was wrong, since r still has enough space not to overflow even for the largest HW address (32 bytes). It would corrupt the data structure though, and that corrupted reply would be propagated to user.

There is a similar situation in arp_req_set(), where a "junk" arp entry will be set if dev->addr_len is larger that 14 bytes.

At the very minimum, both arp_req_set() and arp_req_get() should return error (-EINVAL), and not return junk or set junk. Truncated /proc/net/arp output should also be fixed.

I was not aware that rtnetlink is capable of doing things like arp table or interface manipulation (like netdevice ioctls). My applications needs to be able to manipulate arp cache for large macs, and I do not mind recompiling by adding a flag. I do not mind fixing arp cli to use this either (venerable arp does use arp_ioctl). And there are many many legacy solutions that use arp_ioctl() in programs and arp utility in scripts. Consider porting those to infiniband.

Will rtnetlink work for any net_device (like netdevice ioctls do) for ARP and interface configurations calls or does it require special support in net_device itself? Any possible problems with rtnetlink?

Quoted 27 lines · Roland Dreier

Quoted 7 lines
* arp_ioctl will corrupt the kernel and user memory when this ioctl is used on the adapters that have HW addresses longer that 14 bytes. This is because when copying the HW address, the arp_ioctl code copies dev->addr_len bytes without checking that addr_len is not above 14 bytes. This is done both for copy_to_user() and memcpy() calls on kernel data structures allocated on stack. The memcpy() call in particular, will corrupt kernel stack.
It's not obvious to me after a quick glance where this kernel memory corruption occurs, but clearly we should at least fix this bug.
Quoted 7 lines
The patch does not change the existing ABI but extends it. The kernel structure used in arp_ioctl calls is changed to support larger addresses, while the user-space structure is extended by appending extra-space to the end of the structure if ATF_NEWARPCTL -- a new flag -- is set in arp_flags of existing user-space structure. This allows avoiding big changes to the existing code while preserving the ABI compatibility.
However, given that applications need to be changed to use this, wouldn't it make more sense just to change those applications to use rtnetlink, which already supports large hardware addresses? ie is there much point to extending a legacy ABI to add a feature that the preferred modern interface already has?
- R.

Signature · 8 lines

----------------------------------------
Constantine Gavrilov
Kernel Developer
Platform Group
XIV, an IBM global brand 1 Azrieli Center, Tel-Aviv
Phone: +972-3-6074672
Fax:   +972-3-6959749
----------------------------------------

Stephen Hemminger

2008-11-03 21:58 UTC #4 lore raw

Quoted 26 lines · On Mon, 03 Nov 2008 20:56:44 +0200 Constantine Gavrilov…

In arp_req_get() in net/arp.c, there is code:
memcpy(r->arp_ha.sa_data, neigh->ha, dev->addr_len);
dev->addr_len can be larger than size of r->arp_ha.sa_data. Inititally, I thought it would corrupt kernel stack. I was wrong, since r still has enough space not to overflow even for the largest HW address (32 bytes). It would corrupt the data structure though, and that corrupted reply would be propagated to user.
There is a similar situation in arp_req_set(), where a "junk" arp entry will be set if dev->addr_len is larger that 14 bytes.
At the very minimum, both arp_req_set() and arp_req_get() should return error (-EINVAL), and not return junk or set junk. Truncated /proc/net/arp output should also be fixed.
I was not aware that rtnetlink is capable of doing things like arp table or interface manipulation (like netdevice ioctls). My applications needs to be able to manipulate arp cache for large macs, and I do not mind recompiling by adding a flag. I do not mind fixing arp cli to use this either (venerable arp does use arp_ioctl). And there are many many legacy solutions that use arp_ioctl() in programs and arp utility in scripts. Consider porting those to infiniband.

Quoted 33 lines

Will rtnetlink work for any net_device (like netdevice ioctls do) for ARP and interface configurations calls or does it require special support in net_device itself? Any possible problems with rtnetlink?
Quoted 27 lines · Roland Dreier
Quoted 7 lines
* arp_ioctl will corrupt the kernel and user memory when this ioctl is used on the adapters that have HW addresses longer that 14 bytes. This is because when copying the HW address, the arp_ioctl code copies dev->addr_len bytes without checking that addr_len is not above 14 bytes. This is done both for copy_to_user() and memcpy() calls on kernel data structures allocated on stack. The memcpy() call in particular, will corrupt kernel stack.
It's not obvious to me after a quick glance where this kernel memory corruption occurs, but clearly we should at least fix this bug.
Quoted 7 lines
The patch does not change the existing ABI but extends it. The kernel structure used in arp_ioctl calls is changed to support larger addresses, while the user-space structure is extended by appending extra-space to the end of the structure if ATF_NEWARPCTL -- a new flag -- is set in arp_flags of existing user-space structure. This allows avoiding big changes to the existing code while preserving the ABI compatibility.
However, given that applications need to be changed to use this, wouldn't it make more sense just to change those applications to use rtnetlink, which already supports large hardware addresses? ie is there much point to extending a legacy ABI to add a feature that the preferred modern interface already has?
- R.

Since this is a generic networking issue, please move discussion to netdev mailing list <[email protected]>

Roland Dreier

2008-11-03 23:53 UTC #5 lore raw

In reply to Constantine Gavrilov

[netdev added to cc list]

Quoted 11 lines

In arp_req_get() in net/arp.c, there is code:
memcpy(r->arp_ha.sa_data, neigh->ha, dev->addr_len);
dev->addr_len can be larger than size of r->arp_ha.sa_data. Inititally, I thought it would corrupt kernel stack. I was wrong, since r still has enough space not to overflow even for the largest HW address (32 bytes). It would corrupt the data structure though, and that corrupted reply would be propagated to user.

Quoted 6 lines

There is a similar situation in arp_req_set(), where a "junk" arp entry will be set if dev->addr_len is larger that 14 bytes.
At the very minimum, both arp_req_set() and arp_req_get() should return error (-EINVAL), and not return junk or set junk. Truncated /proc/net/arp output should also be fixed.

The EINVAL return makes sense; I'm not sure /proc/net/arp is important enough to fix. I guess it depends on the impact of the fix.

Quoted 11 lines

I was not aware that rtnetlink is capable of doing things like arp table or interface manipulation (like netdevice ioctls). My applications needs to be able to manipulate arp cache for large macs, and I do not mind recompiling by adding a flag. I do not mind fixing arp cli to use this either (venerable arp does use arp_ioctl). And there are many many legacy solutions that use arp_ioctl() in programs and arp utility in scripts. Consider porting those to infiniband.
Will rtnetlink work for any net_device (like netdevice ioctls do) for ARP and interface configurations calls or does it require special support in net_device itself? Any possible problems with rtnetlink?

rtnetlink is the preferred modern interface between userspace and kernel for networking information. There is also the "iproute2" package that provides a good command line interface that is capable of handling IPoIB addresses. For example:

$ ip addr show dev ib1
5: ib1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
    link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:00:01:65 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.168.145.74/24 brd 192.168.145.255 scope global ib1
    inet6 fe80::202:c903:0:165/64 scope link 
       valid_lft forever preferred_lft forever

$ ip neigh
192.168.145.73 dev ib1 lladdr 80:00:00:48:fe:80:00:00:00:00:00:00:00:02:c9:03:00:00:01:30 STALE
172.29.224.1 dev eth0 lladdr 00:00:0c:07:ac:e0 REACHABLE

and so on.

- R.