IB/IPoIB: Allow setting the device address
PATCH 1/1 IB/IPoIB: Allow setting the device address
From: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>
In IPoIB/rdmacm networks, the device address of an IPoIB interface is used to exchange information between nodes needed for communication.
Currently an IPoIB interface will always be created with a constant device address based on its node GUID.
This change adds the ability to set the device address of an IPoIB interface by value. The mac address ndo is used for that.
The control flow can be divided to two parts:
1) The GID value is already in the GID table,
in this case the interface will be able to set carrier up.
2) The GID value is not yet in the GID table,
in this case the interface won't try to join the multicast group and will wait (listen on GID_CHANGE event) until the GID is inserted.
In order to track those changes, we will add a new flag:
* IPOIB_FLAG_DEV_ADDR_SET.
When this flag is set, it will indicate that the dev_addr is a based on a value in the GID table. The dev_addr change will trigger the bit clear operation.
Per IB specification the port GUID can't be changed if the module is loaded. Port GUID is the basis for GID at index 0 which is the basis for the default device address of a IPoIB interface.
There are devices that don't follow the spec and they change the port GUID while HCA is powered on, so in order do not break existing user space applications, the following flow is suggested.
We will check if the user wants to control the device address and we will assume that he no longer wishes to control it, if he sets the device address back to be based on GID index 0.
In order to track this, we will add an additional flag:
* IPOIB_FLAG_DEV_ADDR_CTRL
When setting the device address, there is lack of validation for 12 upper bytes of the device address (flags, qpn, subnet prefix) as those bytes are not under the control of the user.
---
Available in the "topic/ipoib-device-address" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git
Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/ipoib-device-address
--- drivers/infiniband/ulp/ipoib/ipoib.h | 2 + drivers/infiniband/ulp/ipoib/ipoib_ib.c | 103 ++++++++++++++++++++++++- drivers/infiniband/ulp/ipoib/ipoib_main.c | 38 +++++++++ drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 15 +++- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 3 + drivers/infiniband/ulp/ipoib/ipoib_vlan.c | 1 + 6 files changed, 156 insertions(+), 6 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib.h +2 -0
@@ -92,6 +92,8 @@ enum {IPOIB_FLAG_UMCAST =10 ,IPOIB_STOP_NEIGH_GC =11 ,IPOIB_NEIGH_TBL_FLUSH =12 , +IPOIB_FLAG_DEV_ADDR_SET =13 , +IPOIB_FLAG_DEV_ADDR_CTRL =14 , IPOIB_MAX_BACKOFF_SECONDS = 16,
drivers/infiniband/ulp/ipoib/ipoib_ib.c +101 -2
127 lines not on this page · raw message
drivers/infiniband/ulp/ipoib/ipoib_main.c +38 -0
@@ -99,6 +99,7 @@ static struct net_device *ipoib_get_net_dev_by_params( struct ib_device *dev, u8 port, u16 pkey, const union ib_gid *gid, const struct sockaddr *addr, void *client_data); +static int ipoib_set_mac (struct net_device *dev,void *addr);static struct ib_client ipoib_client = { .name ="ipoib" ,
@@ -1649,6 +1650,7 @@ static const struct net_device_ops ipoib_netdev_ops_pf = { .ndo_get_vf_config = ipoib_get_vf_config, .ndo_get_vf_stats = ipoib_get_vf_stats, .ndo_set_vf_guid = ipoib_set_vf_guid, + .ndo_set_mac_address = ipoib_set_mac, };static const struct net_device_ops ipoib_netdev_ops_vf = {
@@ -1771,6 +1773,41 @@ int ipoib_add_umcast_attr(struct net_device *dev)return device_create_file (&dev->dev, &dev_attr_umcast); } +static void set_base_guid (struct ipoib_dev_priv *priv,union ib_gid *gid, +int nesting) +{ +struct ipoib_dev_priv *child_priv; +struct net_device *netdev = priv->dev; + +netif_addr_lock (netdev); + +if (!nesting) +memcpy (&priv->local_gid.global.interface_id, + &gid->global.interface_id, +sizeof (gid->global.interface_id)); +memcpy (netdev->dev_addr +4 , &priv->local_gid,sizeof (priv->local_gid)); +clear_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags); + +netif_addr_unlock (netdev); + +down_read_nested (&priv->vlan_rwsem, nesting); +list_for_each_entry (child_priv, &priv->child_intfs, list) +set_base_guid (child_priv, gid, nesting +1 ); +up_read (&priv->vlan_rwsem); +} + +static int ipoib_set_mac (struct net_device *dev,void *addr) +{ +struct ipoib_dev_priv *priv =netdev_priv (dev); +struct sockaddr_storage *ss = addr; + +set_base_guid (priv, (union ib_gid *)(ss->__data +4 ),0 ); + +queue_work (ipoib_workqueue, &priv->flush_light); + +return 0 ; +} +static ssize_t create_child (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)
@@ -1894,6 +1931,7 @@ static struct net_device *ipoib_add_port(const char *format, goto device_init_failed; }else memcpy (priv->dev->dev_addr +4 , priv->local_gid.raw,sizeof (union ib_gid )); +set_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags); result =ipoib_dev_init (priv->dev, hca, port);if (result <0 ) {
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +11 -4
@@ -570,11 +570,18 @@ void ipoib_mcast_join_task(struct work_struct *work)return ; } priv->local_lid = port_attr.lid; +netif_addr_lock (dev); -if (ib_query_gid (priv->ca, priv->port,0 , &priv->local_gid,NULL )) -ipoib_warn (priv,"ib_query_gid() failed\n" ); -else -memcpy (priv->dev->dev_addr +4 , priv->local_gid.raw,sizeof (union ib_gid )); +if (test_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags)) { + priv->local_gid.global.subnet_prefix = +cpu_to_be64 (port_attr.subnet_prefix); +memcpy (dev->dev_addr +4 , priv->local_gid.raw, +sizeof (union ib_gid )); +netif_addr_unlock (dev); + }else { +netif_addr_unlock (dev); +return ; + }spin_lock_irq (&priv->lock);if (!test_bit (IPOIB_FLAG_OPER_UP , &priv->flags))
drivers/infiniband/ulp/ipoib/ipoib_verbs.c +3 -0
@@ -307,5 +307,8 @@ void ipoib_event(struct ib_event_handler *handler,queue_work (ipoib_workqueue, &priv->flush_normal); }else if (record->event ==IB_EVENT_PKEY_CHANGE ) {queue_work (ipoib_workqueue, &priv->flush_heavy); + }else if (record->event ==IB_EVENT_GID_CHANGE && + !test_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags)) { +queue_work (ipoib_workqueue, &priv->flush_light); } }
drivers/infiniband/ulp/ipoib/ipoib_vlan.c +1 -0
@@ -68,6 +68,7 @@ int __ipoib_vlan_add(struct ipoib_dev_priv *ppriv, struct ipoib_dev_priv *priv, priv->pkey = pkey;memcpy (priv->dev->dev_addr, ppriv->dev->dev_addr,INFINIBAND_ALEN ); +set_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags); priv->dev->broadcast[8 ] = pkey >>8 ; priv->dev->broadcast[9 ] = pkey &0xff ;
Signature · 6 lines
2.1.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
drivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_umcast_attr(struct net_device *dev) · line 1804
Quoted 6 lines · Leon Romanovsky
drivers/infiniband/ulp/ipoib/ipoib_main.c +6
-0+static int ipoib_set_mac (struct net_device *dev,void *addr) +{ +struct ipoib_dev_priv *priv =netdev_priv (dev); +struct sockaddr_storage *ss = addr; + +set_base_guid (priv, (union ib_gid *)(ss->__data +4 ),0 );
Hurm, this should also check that the QPN and reserved in the provided 20 byte LLADDR match the current state of the device's LLADDR since those cannot be changed.
It is should also be illegal to try and change the GID prefix, so it must also match the current value.
Did these checks from eth_prepare_mac_addr_change get included?
if (!(dev->priv_flags & IFF_LIVE_ADDR_CHANGE) && netif_running(dev))
return -EBUSY;As well as basic sanity checks on the formation of GUID - can't be 0, and must be unicast.
drivers/infiniband/ulp/ipoib/ipoib_multicast.c void ipoib_mcast_join_task(struct work_struct *work) · line 579
Quoted 5 lines
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5
-0+if (test_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags)) { + priv->local_gid.global.subnet_prefix = +cpu_to_be64 (port_attr.subnet_prefix); +memcpy (dev->dev_addr +4 , priv->local_gid.raw, +sizeof (union ib_gid ));
Seems strange - why is the subnet prefix being treated differently in this one case?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi,
Sorry about the delay.
Quoted 21 lines
-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Wednesday, May 04, 2016 9:41 PM
To: Leon Romanovsky <[email protected]>
Cc: [email protected]; [email protected]; Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>; Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/[email protected]>; Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>
Subject: Re: [PATCH rdma-next] IB/IPoIB: Allow setting the device addressdrivers/infiniband/ulp/ipoib/ipoib_main.c int ipoib_add_umcast_attr(struct net_device *dev) · line 1804
Quoted 6 lines · Leon Romanovsky
drivers/infiniband/ulp/ipoib/ipoib_main.c +6
-0+static int ipoib_set_mac (struct net_device *dev,void *addr) +{ +struct ipoib_dev_priv *priv =netdev_priv (dev); +struct sockaddr_storage *ss = addr; + +set_base_guid (priv, (union ib_gid *)(ss->__data +4 ),0 );Hurm, this should also check that the QPN and reserved in the provided 20 byte LLADDR match the current state of the device's LLADDR since those cannot be changed.
Quoted 2 lines
It is should also be illegal to try and change the GID prefix, so it must also match the current value.
I didn't think we need those checks as we are taking only the GUID part, but I'll add them. Better safe than sorry I guess :)
Quoted 6 lines
Did these checks from
eth_prepare_mac_addr_changeget included?if (!(dev->priv_flags & IFF_LIVE_ADDR_CHANGE) && netif_running(dev)) return -EBUSY;
Why do I need to include that?
Quoted 2 lines
As well as basic sanity checks on the formation of GUID - can't be 0, and must be unicast.
I'll add the GUID != 0 check.
The unicast is covered under a previous check, making sure the first 12 bytes are the same. If dev_addr[4] != 0xff -> not multicast, and we start with a none multicast lladdr.
Quoted 8 lines
drivers/infiniband/ulp/ipoib/ipoib_multicast.c void ipoib_mcast_join_task(struct work_struct *work) · line 579
Quoted 5 lines
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5
-0+if (test_bit (IPOIB_FLAG_DEV_ADDR_SET , &priv->flags)) { + priv->local_gid.global.subnet_prefix = +cpu_to_be64 (port_attr.subnet_prefix); +memcpy (dev->dev_addr +4 , priv->local_gid.raw, +sizeof (union ib_gid ));Seems strange - why is the subnet prefix being treated differently in this one case?
Just wanted to make sure we are not trying to join a multicast group with the wrong value. I'll move the copy to: ipoib_dev_addr_changed_valid, this way when a subnet prefix changes It will be copied both to local_gid and dev_addr.
Quoted 1 line
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Mark Bloch
Quoted 6 lines
Did these checks from
eth_prepare_mac_addr_changeget included?if (!(dev->priv_flags & IFF_LIVE_ADDR_CHANGE) && netif_running(dev)) return -EBUSY;Why do I need to include that?
That is standard in all the other implementations - you should be asking instead why IPoIB doesn't need to copy that.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 15 lines
-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Friday, May 13, 2016 7:46 PM
To: Mark Bloch <markb-VPRAkNaXOzVWk0Htik3J/[email protected]>
Cc: Leon Romanovsky <[email protected]>; [email protected]; linux- [email protected]; Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/[email protected]>; Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>
Subject: Re: [PATCH rdma-next] IB/IPoIB: Allow setting the device addressQuoted 2 lines · Mark Bloch
Quoted 2 lines
Did these checks from
eth_prepare_mac_addr_changeget included?
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
if (!(dev->priv_flags & IFF_LIVE_ADDR_CHANGE) && netif_running(dev)) return -EBUSY;Why do I need to include that?
That is standard in all the other implementations - you should be asking instead why IPoIB doesn't need to copy that.
What I've meant is that I don't see why IPoIB won't support live address change. Having said that, it seems IPv6 multicast group membership has issues with live address change. I'll add the if to the next patch version.
Mark.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html