Add port-profile netlink support
Subsystems (2)
v5 cover Add port-profile netlink support
(Resending to fix a bug found in testing get_vf_port_profile).
The following series adds port-profile netlink support and adds an implementation to Cisco's enic netdev driver:
1/3: Adds port-profile netlink RTM_SETLINK/RTM_GETLINK support, and
adds matching netdev ops net_{set|get}_vf_port_profile.
2/3: Adds enic support for net_{set|get}_vf_port_profile for enic
dynamic devices.
3/3: (please don't apply this 3rd patch) Enables SR-IOV support for
enic to illustrate support for port-profile netlink using SR-IOV- compliant devices.
The SETLINK/GETLINK support follows the model for other IFLA_VF_* msgs used for SR-IOV devices where the receipent of the netlink msg is the PF, but the target is the VF.
The intent of this patch set is to cover both definitions of port-profile as defined by Cisco's enic use and as defined by VSI discover protocol (VDP), used in VEPA implemenations. While both definitions are based on pre- standards, the concept of a port-profile to be applied to an external switch port on behalf of a virtual machine interface is common, as well as many of the fields defining the protocols.
PATCH v5 1/3 Add netdev/netlink port-profile support (was iovnl)
From: Scott Feldman <[email protected]>
Add new netdev ops ndo_{set|get}_vf_port_profile to allow setting of port-profile on a netdev interface. Extends netlink socket RTM_SETLINK/ RTM_GETLINK with new sub cmd called IFLA_VF_PORT_PROFILE (added to end of IFLA_cmd list).
A port-profile is used to configure/enable the external switch port backing the netdev interface, not to configure the host-facing side of the netdev. A port-profile is an identifier known to the switch. How port-profiles are installed on the switch or how available port-profiles are made know to the host is outside the scope of this patch.
The general flow is the port-profile is applied to a host netdev interface using RTM_SETLINK, the receiver of the RTM_SETLINK msg (more about that later) communicates with the switch, and the switch port backing the host netdev interface is configured/enabled based on the settings defined by the port- profile. What those settings comprise, and how those settings are managed is again outside the scope of this patch, since this patch only deals with the first step in the flow.
Since we're using netlink sockets, the receiver of the RTM_SETLINK msg can be in kernel- or user-space. For kernel-space recipient, rtnetlink.c, the new ndo_set_vf_port_profile netdev op is called to set the port-profile. User-space recipients can decide how they comminucate the IFLA_VF_PORT_PROFILE to the external switch.
There is a RTM_GETLINK cmd to to return port-profile setting of an interface and to also return the status of the last port-profile.
IFLA_VF_PORT_PROFILE is modeled after the existing IFLA_VF_* cmd where a VF number is passed in to identify the virtual function (VF) of an SR-IOV- capable device. In this case, the target of IFLA_VF_PORT_PROFILE msg is the netdev physical function (PF) device. The PF will apply the port-profile to the VF. IFLA_VF_PORT_PROFILE can also be used for devices that don't adhere to SR-IOV and can apply the port-profile directly to the netdev target. In this case, the VF number is ignored.
Passing in a NULL port-profile is used to delete the port-profile association.
--- include/linux/if_link.h | 25 +++++++++++++++++++++++++ include/linux/netdevice.h | 10 ++++++++++ net/core/rtnetlink.c | 39 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 73 insertions(+), 1 deletions(-)
include/linux/if_link.h +25 -0
@@ -116,6 +116,7 @@ enum {IFLA_VF_TX_RATE ,/* TX Bandwidth Allocation */ IFLA_VFINFO ,IFLA_STATS64 , +IFLA_VF_PORT_PROFILE , __IFLA_MAX };
@@ -259,4 +260,28 @@ struct ifla_vf_info {__u32 qos; __u32 tx_rate; }; + +enum { +IFLA_VF_PORT_PROFILE_STATUS_UNKNOWN , +IFLA_VF_PORT_PROFILE_STATUS_SUCCESS , +IFLA_VF_PORT_PROFILE_STATUS_INPROGRESS , +IFLA_VF_PORT_PROFILE_STATUS_ERROR , +}; + +#define IFLA_VF_PORT_PROFILE_MAX 40 +#define IFLA_VF_UUID_MAX 40 +#define IFLA_VF_CLIENT_NAME_MAX 40 + +struct ifla_vf_port_profile { +__u32 vf; +__u32 flags; +__u32 status; +__u8 port_profile[IFLA_VF_PORT_PROFILE_MAX ]; +__u8 mac[32 ];/* MAX_ADDR_LEN */ +/* UUID e.g. "CEEFD3B1-9E11-11DE-BDFD-000BAB01C0FB" */ +__u8 host_uuid[IFLA_VF_UUID_MAX ]; +__u8 client_uuid[IFLA_VF_UUID_MAX ]; +__u8 client_name[IFLA_VF_CLIENT_NAME_MAX ];/* e.g. "vm0-eth1" */ +}; +#endif /* _LINUX_IF_LINK_H */
include/linux/netdevice.h +10 -0
@@ -696,6 +696,10 @@ struct netdev_rx_queue {* int (*ndo_set_vf_tx_rate)(struct net_device *dev, int vf, int rate); * int (*ndo_get_vf_config)(struct net_device *dev, * int vf, struct ifla_vf_info *ivf); +* int (*ndo_set_vf_port_profile)(struct net_device *dev, int vf, +* struct ifla_vf_port_profile *ivp); +* int (*ndo_get_vf_port_profile)(struct net_device *dev, int vf, +* struct ifla_vf_port_profile *ivp); */ #define HAVE_NET_DEVICE_OPS struct net_device_ops {
@@ -744,6 +748,12 @@ struct net_device_ops {int (*ndo_get_vf_config)(struct net_device *dev,int vf,struct ifla_vf_info *ivf); +int (*ndo_set_vf_port_profile)( +struct net_device *dev,int vf, +struct ifla_vf_port_profile *ivp); +int (*ndo_get_vf_port_profile)( +struct net_device *dev,int vf, +struct ifla_vf_port_profile *ivp);#if defined(CONFIG_FCOE ) || defined(CONFIG_FCOE_MODULE )int (*ndo_fcoe_enable)(struct net_device *dev);int (*ndo_fcoe_disable)(struct net_device *dev);
net/core/rtnetlink.c +38 -1
@@ -747,17 +747,40 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb, struct net_device *dev, goto nla_put_failure;copy_rtnl_link_stats64 (nla_data (attr), stats); +if (dev->dev.parent) +NLA_PUT_U32 (skb,IFLA_NUM_VF ,dev_num_vf (dev->dev.parent)); +if (dev->netdev_ops->ndo_get_vf_config && dev->dev.parent) {int i;struct ifla_vf_info ivi; -NLA_PUT_U32 (skb,IFLA_NUM_VF ,dev_num_vf (dev->dev.parent));for (i =0 ; i <dev_num_vf (dev->dev.parent); i++) {if (dev->netdev_ops->ndo_get_vf_config (dev, i, &ivi))break ;NLA_PUT (skb,IFLA_VFINFO ,sizeof (ivi), &ivi); } } + +if (dev->netdev_ops->ndo_get_vf_port_profile && dev->dev.parent) { +struct ifla_vf_port_profile ivp; + +if (dev_num_vf (dev->dev.parent)) { +int i; + +for (i =0 ; i <dev_num_vf (dev->dev.parent); i++) { +if (dev->netdev_ops->ndo_get_vf_port_profile ( + dev, i, &ivp)) +break ; +NLA_PUT (skb,IFLA_VF_PORT_PROFILE , +sizeof (ivp), &ivp); + } + }else if (!dev->netdev_ops->ndo_get_vf_port_profile (dev, +0 , &ivp)) { +NLA_PUT (skb,IFLA_VF_PORT_PROFILE , +sizeof (ivp), &ivp); + } + } +if (dev->rtnl_link_ops) {if (rtnl_link_fill (skb, dev) <0 ) goto nla_put_failure;
@@ -824,6 +847,8 @@ const struct nla_policy ifla_policy[IFLA_MAX+1] = { .len = sizeof(struct ifla_vf_vlan) }, [IFLA_VF_TX_RATE] = { .type = NLA_BINARY, .len = sizeof(struct ifla_vf_tx_rate) }, + [IFLA_VF_PORT_PROFILE] = { .type = NLA_BINARY, + .len = sizeof(struct ifla_vf_port_profile)}, };EXPORT_SYMBOL (ifla_policy);
@@ -1028,6 +1053,18 @@ static int do_setlink(struct net_device *dev, struct ifinfomsg *ifm, } err =0 ; +if (tb[IFLA_VF_PORT_PROFILE ]) { +struct ifla_vf_port_profile *ivp; + ivp =nla_data (tb[IFLA_VF_PORT_PROFILE ]); + err = -EOPNOTSUPP ; +if (ops->ndo_set_vf_port_profile) + err = ops->ndo_set_vf_port_profile (dev, ivp->vf, ivp); +if (err <0 ) + goto errout; + modified =1 ; + } + err =0 ; + errout:if (err <0 && modified &&net_ratelimit ()) printk(KERN_WARNING "A link change request failed with "
PATCH v5 2/3 Add ndo_{set|get}_vf_port_profile op support for enic dynamic vnics
In reply to Scott FeldmanFrom: Scott Feldman <[email protected]>
Add enic ndo_{set|get}_vf_port_profile ops to support setting/getting port-profile for enic dynamic devices. Enic dynamic devices are just like normal enic eth devices except dynamic enics require an extra configuration step to assign a port-profile identifier to the interface before the interface is useable. Once a port-profile is assigned, link comes up on the interface and is ready for I/O. The port-profile is used to configure the network port assigned to the interface. The network port configuration includes VLAN membership, QoS policies, and port security settings typical of a data center network.
A dynamic enic is assigned a default random mac address. If no mac address parameter is specified in the ndo_set_vf_port_profile op, the default random mac address is used when assigning the port-profile. Otherwise the mac address specified in the op is used.
--- drivers/net/enic/Makefile | 2 drivers/net/enic/enic.h | 3 - drivers/net/enic/enic_main.c | 200 +++++++++++++++++++++++++++++++++++++----- drivers/net/enic/vnic_dev.c | 50 +++++++++++ drivers/net/enic/vnic_dev.h | 3 + drivers/net/enic/vnic_vic.c | 73 +++++++++++++++ drivers/net/enic/vnic_vic.h | 59 ++++++++++++ 7 files changed, 363 insertions(+), 27 deletions(-)
drivers/net/enic/Makefile +1 -1
@@ -1,5 +1,5 @@ obj-$(CONFIG_ENIC) := enic.o enic-y := enic_main.o vnic_cq.o vnic_intr.o vnic_wq.o \ - enic_res.o vnic_dev.o vnic_rq.o + enic_res.o vnic_dev.o vnic_rq.o vnic_vic.o
drivers/net/enic/enic.h +2 -1
@@ -34,7 +34,7 @@#define DRV_NAME "enic"#define DRV_DESCRIPTION "Cisco VIC Ethernet NIC Driver" -#define DRV_VERSION "1.3.1.1" +#define DRV_VERSION "1.3.1.1-pp"#define DRV_COPYRIGHT "Copyright 2008-2009 Cisco Systems, Inc"#define PFX DRV_NAME ": "
@@ -95,6 +95,7 @@ struct enic {u32 port_mtu;u32 rx_coalesce_usecs;u32 tx_coalesce_usecs; +struct ifla_vf_port_profile pp;/* work queue cache line section */ ____cacheline_aligned struct vnic_wq wq[ENIC_WQ_MAX ];
drivers/net/enic/enic_main.c +175 -25
264 lines not on this page · raw message
drivers/net/enic/vnic_dev.c +50 -0
56 lines not on this page · raw message
drivers/net/enic/vnic_dev.h +3 -0
@@ -124,6 +124,9 @@ int vnic_dev_disable(struct vnic_dev *vdev);int vnic_dev_open (struct vnic_dev *vdev,int arg);int vnic_dev_open_done (struct vnic_dev *vdev,int *done);int vnic_dev_init (struct vnic_dev *vdev,int arg); +int vnic_dev_init_done (struct vnic_dev *vdev,int *done,int *err); +int vnic_dev_init_prov (struct vnic_dev *vdev,u8 *buf,u32 len); +int vnic_dev_deinit (struct vnic_dev *vdev);int vnic_dev_soft_reset (struct vnic_dev *vdev,int arg);int vnic_dev_soft_reset_done (struct vnic_dev *vdev,int *done); void vnic_dev_set_intr_mode(struct vnic_dev *vdev,
drivers/net/enic/vnic_vic.c new file +73 -0
73 lines not on this page · raw message
drivers/net/enic/vnic_vic.h new file +59 -0
59 lines not on this page · raw message
Re: [net-next-2.6 V5 PATCH 2/3] Add ndo_{set|get}_vf_port_profile op support for enic dynamic vnics
Quoted 18 lines · Scott Feldman
drivers/net/enic/enic_main.c +8
-3@@ -810,14 +819,24 @@ static void enic_reset_mcaddrs(struct enic *enic)static int enic_set_mac_addr (struct net_device *netdev,char *addr) { -if (!is_valid_ether_addr (addr)) -return -EADDRNOTAVAIL ; +struct enic *enic =netdev_priv (netdev); -memcpy (netdev->dev_addr, addr, netdev->addr_len); +if (enic_is_dynamic (enic)) { +random_ether_addr (netdev->dev_addr); + }else { +if (!is_valid_ether_addr (addr)) +return -EADDRNOTAVAIL ; +memcpy (netdev->dev_addr, addr, netdev->addr_len); + }return 0 ;
drivers/net/enic/enic_main.c static void enic_reset_mcaddrs(struct enic *enic) · line 842
Quoted 10 lines
drivers/net/enic/enic_main.c +5
-0} +static int enic_set_mac_address (struct net_device *netdev,void *p) +{ +return -EOPNOTSUPP ; +} +/* netif_tx_lock held, BHs disabled */ static void enic_set_multicast_list (struct net_device *netdev) {
Thsi looks funny. So you just ignore the address that gets passed to enic_set_mac_addr for dynamic interfaces and instead set a random address?
Arnd
Re: [net-next-2.6 V5 PATCH 2/3] Add ndo_{set|get}_vf_port_profile op support for enic dynamic vnics
Quoted 21 lines · Arnd Bergmann
Quoted 20 lines · Scott Feldman
drivers/net/enic/enic_main.c +8
-3@@ -810,14 +819,24 @@ static void enic_reset_mcaddrs(struct enic *enic)static int enic_set_mac_addr (struct net_device *netdev,char *addr) { -if (!is_valid_ether_addr (addr)) -return -EADDRNOTAVAIL ; +struct enic *enic =netdev_priv (netdev); -memcpy (netdev->dev_addr, addr, netdev->addr_len); +if (enic_is_dynamic (enic)) { +random_ether_addr (netdev->dev_addr); + }else { +if (!is_valid_ether_addr (addr)) +return -EADDRNOTAVAIL ; +memcpy (netdev->dev_addr, addr, netdev->addr_len); + }return 0 ; }
Quoted 12 lines
drivers/net/enic/enic_main.c static void enic_reset_mcaddrs(struct enic *enic) · line 842
Quoted 8 lines
drivers/net/enic/enic_main.c +5
-0+static int enic_set_mac_address (struct net_device *netdev,void *p) +{ +return -EOPNOTSUPP ; +} +/* netif_tx_lock held, BHs disabled */ static void enic_set_multicast_list (struct net_device *netdev) {Thsi looks funny. So you just ignore the address that gets passed to
enic_set_mac_addrfor dynamic interfaces and instead set a random address?
Dynamic enics have all-zero mac address on init, so we assign a random mac addr to the interface. This would seem less funny:
if (enic_is_dynamic(enic) && is_zero_ether_addr(addr))
random_ether_addr(netdev->dev_addr);
else
...I'll make that change and resubmit with your VDP additions if you like.
-scott
Re: [net-next-2.6 V5 PATCH 2/3] Add ndo_{set|get}_vf_port_profile op support for enic dynamic vnics
Quoted 9 lines · Scott Feldman
Dynamic enics have all-zero mac address on init, so we assign a random mac addr to the interface. This would seem less funny:
if (enic_is_dynamic(enic) && is_zero_ether_addr(addr)) random_ether_addr(netdev->dev_addr); else ...I'll make that change and resubmit with your VDP additions if you like.
The change is ok, but what I think would be more helpful is a code comment with your above sentence.
Arnd
PATCH v5 3/3 Add SR-IOV support to enic (please don't apply this patch)
In reply to Scott FeldmanFrom: Scott Feldman <[email protected]>
This patch is to illustrate how port-profiles will be assigned to VFs in a compliant SR-IOV enic device. Here the VF devices are dynamic enics and the PF device is a "static" enic device. Only the PF device resonds to ndo_vf_{set|get}_port_profile to set/get the port-profile on a VF. It's not possible to set a port-profile on a PF since PFs have an immutable port assignment on the external switch, established when the PF was provisioned.
The same driver (enic) is used for both PFs and VFs devices. The PF enables N number of VFs based on a PF configuration parameter assigned when the PF is provisioned.
While this patch is functionally complete, we (Cisco) need to do more testing before we can cliam full SR-IOV support in Linux, so we ask that this patch not be applied at this time. it is provide with this patch set for illustrative purposes only to show how the port-profile netlink API would be used for a SR-IOV compliant device that supports port-profiles.
--- drivers/net/enic/enic.h | 5 +- drivers/net/enic/enic_main.c | 96 +++++++++++++++++++++++++++++++----------- drivers/net/enic/enic_res.c | 3 + drivers/net/enic/vnic_dev.c | 12 +++-- drivers/net/enic/vnic_dev.h | 6 +-- drivers/net/enic/vnic_enet.h | 1 6 files changed, 86 insertions(+), 37 deletions(-)
drivers/net/enic/enic.h +3 -2
@@ -34,7 +34,7 @@#define DRV_NAME "enic"#define DRV_DESCRIPTION "Cisco VIC Ethernet NIC Driver" -#define DRV_VERSION "1.3.1.1-pp" +#define DRV_VERSION "1.3.1.1-sr-iov"#define DRV_COPYRIGHT "Copyright 2008-2009 Cisco Systems, Inc"#define PFX DRV_NAME ": "
@@ -95,7 +95,8 @@ struct enic {u32 port_mtu;u32 rx_coalesce_usecs;u32 tx_coalesce_usecs; -struct ifla_vf_port_profile pp; +struct ifla_vf_port_profile *pp; +unsigned int vf_count;/* work queue cache line section */ ____cacheline_aligned struct vnic_wq wq[ENIC_WQ_MAX ];
drivers/net/enic/enic_main.c +70 -26
205 lines not on this page · raw message
drivers/net/enic/enic_res.c +3 -0
@@ -69,6 +69,7 @@ int enic_get_vnic_config(struct enic *enic)GET_CONFIG (intr_timer_type);GET_CONFIG (intr_mode);GET_CONFIG (intr_timer_usec); +GET_CONFIG (vf_count); c->wq_desc_count = min_t(u32, ENIC_MAX_WQ_DESCS,
@@ -99,6 +100,8 @@ int enic_get_vnic_config(struct enic *enic) c->mtu,ENIC_SETTING (enic,TXCSUM ),ENIC_SETTING (enic,RXCSUM ),ENIC_SETTING (enic,TSO ), ENIC_SETTING(enic, LRO), c->intr_timer_usec); +if (c->vf_count) + printk(KERN_INFO PFX "vNIC SR-IOV VF count %d\n", c->vf_count);return 0 ; }
drivers/net/enic/vnic_dev.c +6 -6
@@ -682,9 +682,9 @@ int vnic_dev_init(struct vnic_dev *vdev, int arg)return r; } -int vnic_dev_init_done (struct vnic_dev *vdev,int *done,int *err) +int vnic_dev_init_done (struct vnic_dev *vdev,u16 vf,int *done,int *err) { -u64 a0 =0 , a1 =0 ; +u64 a0 = vf, a1 =0 ;int wait =1000 ;int ret;
@@ -701,9 +701,9 @@ int vnic_dev_init_done(struct vnic_dev *vdev, int *done, int *err)return 0 ; } -int vnic_dev_init_prov (struct vnic_dev *vdev,u8 *buf,u32 len) +int vnic_dev_init_prov (struct vnic_dev *vdev,u16 vf,u8 *buf,u32 len) { -u64 a0, a1 = len; +u64 a0, a1 = (u64 )len | ((u64 )vf <<32 );int wait =1000 ;u64 prov_pa;void *prov_buf;
@@ -724,9 +724,9 @@ int vnic_dev_init_prov(struct vnic_dev *vdev, u8 *buf, u32 len)return ret; } -int vnic_dev_deinit (struct vnic_dev *vdev) +int vnic_dev_deinit (struct vnic_dev *vdev,u16 vf) { -u64 a0 =0 , a1 =0 ; +u64 a0 = vf, a1 =0 ;int wait =1000 ;return vnic_dev_cmd (vdev,CMD_DEINIT , &a0, &a1, wait);
drivers/net/enic/vnic_dev.h +3 -3
@@ -124,9 +124,9 @@ int vnic_dev_disable(struct vnic_dev *vdev);int vnic_dev_open (struct vnic_dev *vdev,int arg);int vnic_dev_open_done (struct vnic_dev *vdev,int *done);int vnic_dev_init (struct vnic_dev *vdev,int arg); -int vnic_dev_init_done (struct vnic_dev *vdev,int *done,int *err); -int vnic_dev_init_prov (struct vnic_dev *vdev,u8 *buf,u32 len); -int vnic_dev_deinit (struct vnic_dev *vdev); +int vnic_dev_init_done (struct vnic_dev *vdev,u16 vf,int *done,int *err); +int vnic_dev_init_prov (struct vnic_dev *vdev,u16 vf,u8 *buf,u32 len); +int vnic_dev_deinit (struct vnic_dev *vdev,u16 vf);int vnic_dev_soft_reset (struct vnic_dev *vdev,int arg);int vnic_dev_soft_reset_done (struct vnic_dev *vdev,int *done); void vnic_dev_set_intr_mode(struct vnic_dev *vdev,
drivers/net/enic/vnic_enet.h +1 -0
@@ -35,6 +35,7 @@ struct vnic_enet_config {u8 intr_mode;char devname[16 ];u32 intr_timer_usec; +u16 vf_count; };#define VENETF_TSO 0x1/* TSO enabled */
Quoted 6 lines · Scott Feldman
The intent of this patch set is to cover both definitions of port-profile as defined by Cisco's enic use and as defined by VSI discover protocol (VDP), used in VEPA implemenations. While both definitions are based on pre- standards, the concept of a port-profile to be applied to an external switch port on behalf of a virtual machine interface is common, as well as many of the fields defining the protocols.
The description no either no longer matches the patches, or you did not make the changes that were needed based on our last discussion.
What happened to the base-device argument that you were planning to pass?
The fields that I mentioned are needed for VDP (associate/pre-associate/disassociate-flag, VLAN ID, etc) are not there. I assume that means we should use a different data structure for VDP, but then your description above should be updated to state that this is no longer common for the two.
I'll follow up with a draft for VDP based on your definitions.
Arnd
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · Scott Feldman
The intent of this patch set is to cover both definitions of port-profile as defined by Cisco's enic use and as defined by VSI discover protocol (VDP), used in VEPA implemenations. While both definitions are based on pre- standards, the concept of a port-profile to be applied to an external switch port on behalf of a virtual machine interface is common, as well as many of the fields defining the protocols.
Quoted 5 lines
The description no either no longer matches the patches, or you did not make the
changes that were needed based on our last discussion.What happened to the base-device argument that you were planning to pass?
Using the IFLA_VF_* model works better for us where the recipient of the netlink msg is the PF but the msg is to be applied to the VF. The third patch illustrates how this fits nicely with SR-IOV devices. The PF is the base device.
Quoted 7 lines
The fields that I mentioned are needed for VDP (associate/pre-associate/disassociate-flag,
VLAN ID, etc) are not there. I assume that means we should use a different data structure for VDP, but then your description above should be updated to state that this is no longer common for the two.I'll follow up with a draft for VDP based on your definitions.
I tried to accommodate space for VDP, but was hoping you could add the definitions on top of what I had since your more familiar with VDP and can do the testing.
Also, I wasn't sure if you could use the existing IFLA_VF_VLAN msg to apply the VLAN ID or if you wanted VLAN ID also added to IFLA_VF_PORT_PROFILE.
-scott
Quoted 14 lines · Scott Feldman
Quoted 12 lines · Arnd Bergmann
Quoted 6 lines · Scott Feldman
The intent of this patch set is to cover both definitions of port-profile as defined by Cisco's enic use and as defined by VSI discover protocol (VDP), used in VEPA implemenations. While both definitions are based on pre- standards, the concept of a port-profile to be applied to an external switch port on behalf of a virtual machine interface is common, as well as many of the fields defining the protocols.
The description no either no longer matches the patches, or you did not make the
changes that were needed based on our last discussion.
Quoted 6 lines
Quoted 1 line
What happened to the base-device argument that you were planning to pass?
Using the IFLA_VF_* model works better for us where the recipient of the netlink msg is the PF but the msg is to be applied to the VF. The third patch illustrates how this fits nicely with SR-IOV devices. The PF is the base device.
Ah, got it. I did not notice that you had put a vf field in there. It now makes a lot more sense to me, and is more in line with what we need for VDP.
It does however make me wonder how this could be implemented for a software-only implementation of your protocol that does not refer to vf numbers. One way would be to define the 'vf' field as implementation specific and just use the ifindex in this case, which would also work in case of network namespaces. Alternatively, it could use whatever tag you use in your wire protocol (e.g. an S-VID)
Both are a bit of a stretch, but I see no technical problems with them.
Quoted 6 lines
Quoted 6 lines
The fields that I mentioned are needed for VDP (associate/pre-associate/disassociate-flag,
VLAN ID, etc) are not there. I assume that means we should use a different data structure for VDP, but then your description above should be updated to state that this is no longer common for the two.
Quoted 8 lines
Quoted 1 line
I'll follow up with a draft for VDP based on your definitions.
I tried to accommodate space for VDP, but was hoping you could add the definitions on top of what I had since your more familiar with VDP and can do the testing.
Also, I wasn't sure if you could use the existing
IFLA_VF_VLANmsg to apply the VLAN ID or if you wanted VLAN ID also added toIFLA_VF_PORT_PROFILE.
The IFLA_VF_VLAN would not work well here because of the issue we discussed before that I think we need to keep device setup separate from the protocol exchange. IFLA_VF_VLAN configures the VLAN, while we need to tell the switch about the configuration.
One (new) point that came up today is that your protocol is actually much more closely related to the 'CDCP' protocol in 802.1Qbg than to 'VDP'. I'll also try to make sure that we cover this case as well. It should also be possible to do VDP over a dynamic enic VF and have multiple guests using macvtap on that function, and there will probably be adapters that need to use IFLA_VF_PORT_PROFILE (or another set) as the interface between libvirt and the adapter firmware for doing CDCP.
To give some background, CDCP is an LLDP extension that is used to create virtual channels between a physical NIC and the phys bridge on the other side, using S-VLAN tagging. You can either assign one of these channels to a guest directly (similar to what enic does), or use VDP on the channel to connect multiple guests using a bridge device or macvtap in the same way that we also do VDP on the physical device in the absence of CDCP.
Arnd
PATCH 1/1 virtif: initial interface extensions
Building on the work of Scott Feldman, this extends the netlink interface to deal with not only port profiles but also CDCP multichannel and VDP VSI registration.
The protocols are split apart into separate netlink attributes for a cleaner separation. A device can have multiple IFLA_VIRTIF attributes, each pointing to one of the slaves that get registered using one of the protocols. In case of VDP, each IFLA_VIRTIF attribute needs both an IFLA_VIRTIF_VSI and one or more IFLA_VIRTIF_VSI_MAC_VLAN attributes.
The VF number is split out because it only makes sense when the implementation is in the device driver or firmware, but not for software-only implementations like we do for the initial VDP code using LLDPAD. If a IFLA_VIRTIF_VF attribute is given, the kernel takes care of the association through the device driver for that VF, otherwise we do it in user space. This should work for each of the three protocols.
This code is a first rough prototype, completely untested and very likely buggy. This is also the first time I'm trying to deal with netlink in the kernel, so chances are that I've misunderstood something in a major way.
--- include/linux/if_link.h | 81 ++++++++++++++++++++++++++++------ include/linux/netdevice.h | 8 ++-- net/core/rtnetlink.c | 106 ++++++++++++++++++++++++++++++++++----------- 3 files changed, 152 insertions(+), 43 deletions(-)
include/linux/if_link.h +67 -14
@@ -116,7 +116,7 @@ enum {IFLA_VF_TX_RATE ,/* TX Bandwidth Allocation */ IFLA_VFINFO ,IFLA_STATS64 , -IFLA_VF_PORT_PROFILE , +IFLA_VIRTIF , __IFLA_MAX };
@@ -262,26 +262,79 @@ struct ifla_vf_info { };enum { -IFLA_VF_PORT_PROFILE_STATUS_UNKNOWN , -IFLA_VF_PORT_PROFILE_STATUS_SUCCESS , -IFLA_VF_PORT_PROFILE_STATUS_INPROGRESS , -IFLA_VF_PORT_PROFILE_STATUS_ERROR , +IFLA_VIRTIF_UNSPEC , +IFLA_VIRTIF_VF , +IFLA_VIRTIF_PORT_PROFILE ,/* Cisco enic */ +IFLA_VIRTIF_CHANNEL ,/* 802.1Qbg CDCP */ +IFLA_VIRTIF_VSI ,/* 802.1Qbg VDP */ +IFLA_VIRTIF_VSI_MAC_VLAN , + __IFLA_VIRTIF_MAX, }; -#define IFLA_VF_PORT_PROFILE_MAX 40 -#define IFLA_VF_UUID_MAX 40 -#define IFLA_VF_CLIENT_NAME_MAX 40 +#define IFLA_VIRTIF_MAX (__IFLA_VIRTIF_MAX - 1) -struct ifla_vf_port_profile { -__u32 vf; +enum { +VIRTIF_PORT_PROFILE_STATUS_UNKNOWN , +VIRTIF_PORT_PROFILE_STATUS_SUCCESS , +VIRTIF_PORT_PROFILE_STATUS_INPROGRESS , +VIRTIF_PORT_PROFILE_STATUS_ERROR , +}; + +#define VIRTIF_PORT_PROFILE_MAX 40 +#define VIRTIF_UUID_MAX 40 +#define VIRTIF_CLIENT_NAME_MAX 40 + +struct ifla_virtif_port_profile {__u32 flags;__u32 status; -__u8 port_profile[IFLA_VF_PORT_PROFILE_MAX ]; +__u8 port_profile[VIRTIF_PORT_PROFILE_MAX ];__u8 mac[32 ];/* MAX_ADDR_LEN */ /* UUID e.g. "CEEFD3B1-9E11-11DE-BDFD-000BAB01C0FB" */ -__u8 host_uuid[IFLA_VF_UUID_MAX ]; -__u8 client_uuid[IFLA_VF_UUID_MAX ]; -__u8 client_name[IFLA_VF_CLIENT_NAME_MAX ];/* e.g. "vm0-eth1" */ +__u8 host_uuid[VIRTIF_UUID_MAX ]; +__u8 client_uuid[VIRTIF_UUID_MAX ]; +__u8 client_name[VIRTIF_CLIENT_NAME_MAX ];/* e.g. "vm0-eth1" */ +}; + +/* +* CDCP and VDP come from the 802.1Qbg standard, these definitions are taken +* from the respective TLV definitions in there. Adapters implementing CDCP +* or VDP can encapsulate the structures in LLDP/ECP headers and send them +* to the switch. +*/ +struct ifla_virtif_channel { +__u16 scid;/* S-Channel ID */ +__u16 svid;/* S-VLAN ID */ +}; + +enum { +VIRTIF_VDP_REQUEST_PREASSOCIATE =0 , +VIRVIF_VDP_REQUEST_PREASSOCIATE_RR , +VIRVIF_VDP_REQUEST_ASSOCIATE , +VIRVIF_VDP_REQUEST_DISASSOCIATE , +}; + +enum { +VIRTIF_VDP_RESPONSE_SUCCESS =0 , +VIRTIF_VDP_RESPONSE_INVALID_FORMAT , +VIRTIF_VDP_RESPONSE_INSUFFICIENT_RESOURCES , +VIRTIF_VDP_RESPONSE_UNUSED_VTID , +VIRTIF_VDP_RESPONSE_VTID_VIOLATION , +VIRTIF_VDP_RESPONSE_VTID_VERSION_VIOLATION , +VIRTIF_VDP_RESPONSE_OUT_OF_SYNC , +}; + +struct ifla_virtif_vsi { +__u8 mode_request; +__u8 mode_response; +__u8 vsi_mgr_id;/* these three define the policy */ +__u8 vsi_type_id[3 ];/* for the guest */ +__u8 vsi_type_version; +__u8 vsi_instance[16 ];/* identifies the guest */ +}; + +struct ifla_virtif_vsi_mac_vlan { +__u8 mac[6 ]; +__u8 vlan_id[2 ]; };#endif /* _LINUX_IF_LINK_H */
include/linux/netdevice.h +4 -4
@@ -687,9 +687,9 @@ struct netdev_rx_queue {* int (*ndo_get_vf_config)(struct net_device *dev, * int vf, struct ifla_vf_info *ivf); * int (*ndo_set_vf_port_profile)(struct net_device *dev, int vf, -* struct ifla_vf_port_profile *ivp); +* struct virtif_port_profile *ivp); * int (*ndo_get_vf_port_profile)(struct net_device *dev, int vf, -* struct ifla_vf_port_profile *ivp); +* struct virtif_port_profile *ivp); */ #define HAVE_NET_DEVICE_OPS struct net_device_ops {
@@ -741,10 +741,10 @@ struct net_device_ops { struct ifla_vf_info *ivf);int (*ndo_set_vf_port_profile)(struct net_device *dev,int vf, -struct ifla_vf_port_profile *ivp); +struct ifla_virtif_port_profile *ivp);int (*ndo_get_vf_port_profile)(struct net_device *dev,int vf, -struct ifla_vf_port_profile *ivp); +struct ifla_virtif_port_profile *ivp);#if defined(CONFIG_FCOE ) || defined(CONFIG_FCOE_MODULE )int (*ndo_fcoe_enable)(struct net_device *dev);int (*ndo_fcoe_disable)(struct net_device *dev);
net/core/rtnetlink.c +81 -25
148 lines not on this page · raw message
Signature · 1 line
1.6.3.3
Re: [PATCH] virtif: initial interface extensions
Arnd Bergmann <arnd <at> arndb.de> writes:
[...]
net/core/rtnetlink.c static int do_setlink(struct net_device *dev, struct ifinfomsg *ifm, · line 1109
Quoted 12 lines
net/core/rtnetlink.c +12
-0+if (tb[IFLA_VIRTIF ]) { +struct ifla_virtif_port_profile *ivp; +struct nlattr *virtif[IFLA_VIRTIF_MAX +1 ]; +u32 vf; + + err =nla_parse_nested (virtif,IFLA_VIRTIF_MAX , + tb[IFLA_VIRTIF ], ifla_virtif_policy); +if (err <0 ) +return err; + +if (!virtif[IFLA_VIRTIF_VF ] || !virtif[IFLA_VIRTIF_PORT_PROFILE ]) + goto novirtif;/* IFLA_VIRTIF may be directed at user space */
In what case would the IFLA_VIRTIF_PORT_PROFILE be provided? Would libvirt for example need to be aware of whether the Ethernet device can handle the setup protocol via its firmware and in this case provide the port profile parameter and in other cases provide other parameters? Certainly the user or upper layer management software would have to know it when creating the domain XML and in fact different types of parameters were needed. Obviously we should have one common set of (XML) parameters that go into the netlink message and that can be handled by the kernel driver if the firmware knows how to handle it or by LLDPAD. Libvirt would send the parameters via netlink message to trigger the setup protocol and the message may be received by kernel and LLDPAD. From what I can see LLDPAD also may need a way to probe the kernel driver whether it handled the setup protocol via firmware on a given interface, which may or may not be true for all interfaces, but may be necessary to avoid triggering the setup protocol twice.
Stefan
Re: [PATCH] virtif: initial interface extensions
Quoted 25 lines · Stefan Berger
Arnd Bergmann <arnd <at> arndb.de> writes:
[...]
net/core/rtnetlink.c static int do_setlink(struct net_device *dev, struct ifinfomsg *ifm, · line 1109
Quoted 12 lines
+ if (tb[IFLA_VIRTIF]) {
net/core/rtnetlink.c +11
-0+struct ifla_virtif_port_profile *ivp; +struct nlattr *virtif[IFLA_VIRTIF_MAX +1 ]; +u32 vf; + + err =nla_parse_nested (virtif,IFLA_VIRTIF_MAX , + tb[IFLA_VIRTIF ], ifla_virtif_policy); +if (err <0 ) +return err; + +if (!virtif[IFLA_VIRTIF_VF ] || !virtif[IFLA_VIRTIF_PORT_PROFILE ]) + goto novirtif;/* IFLA_VIRTIF may be directed at user space */ In what case would the
IFLA_VIRTIF_PORT_PROFILEbe provided? Would libvirt for example need to be aware of whether the Ethernet device can handle the setup protocol via its firmware and in this case provide the port profile parameter and in other cases provide other parameters? Certainly the user or upper layer management software would have to know it when creating the domain XML and in fact different types of parameters were needed.
Quoted 4 lines
Obviously we should have one
common set of (XML) parameters that go into the netlink message and that can be handled by the kernel driver if the firmware knows how to handle it or by LLDPAD.
With Arnd's latest additions, we have a single netlink msg, but the parameter sets are disjoint between VDP/CDCP and what we need for the kernel driver. So that means the sender (libvirt in this case) needs to know about both setups to send a single netlink msg. An alternative is a have two netlink msgs, one for each setup. That still requires the sender to know about two setups.
Quoted 2 lines
Libvirt would send the parameters via netlink message to trigger the setup protocol and the message may be received by kernel and LLDPAD.
That was the original idea by having libvirt send the netlink msg using multicast.
Re: [PATCH] virtif: initial interface extensions
Quoted 13 lines · On Monday 10 May 2010 20:56:39 Scott Feldman wrote:
Quoted 11 lines · Stefan Berger
In what case would the
IFLA_VIRTIF_PORT_PROFILEbe provided? Would libvirt for example need to be aware of whether the Ethernet device can handle the setup protocol via its firmware and in this case provide the port profile parameter and in other cases provide other parameters? Certainly the user or upper layer management software would have to know it when creating the domain XML and in fact different types of parameters were needed.Obviously we should have one
common set of (XML) parameters that go into the netlink message and that can be handled by the kernel driver if the firmware knows how to handle it or by LLDPAD.
Quoted 6 lines
With Arnd's latest additions, we have a single netlink msg, but the parameter sets are disjoint between VDP/CDCP and what we need for the kernel driver. So that means the sender (libvirt in this case) needs to know about both setups to send a single netlink msg. An alternative is a have two netlink msgs, one for each setup. That still requires the sender to know about two setups.
There are two separate issues here. The first one is whether we're doing the association in the device driver or in user space. The assumption here is that if it's in the device driver, there will be a VF number to identify the channel, while in user space that is not needed.
The other question is which protocol we're using. There are as far as I can tell five options:
1. enic device driver
2. VDP
3. CDCP
4. CDCP + VDP
5. enic + VDP
The first two ones are the most interesting for now, since Linux cannot do S-VLANs yet, and they are required for CDCP. However, each of these options could theoreticall be done in the kernel (plus firmware) or in user space.
If it's done in user space, the VF number is meaningless, because the setup of the software device is also done from software, but instead you need to take care of creating the software device with the correct parameters, e.g. a macvtap device connected to a VLAN interface using the numbers you pass in the VDP protocol.
Right now, we're not planning to do the protocol that enic uses in LLDPAD, because it's not publically released. Similarly, there are no adapters that do VDP in firmware, but both these cases should be covered by the protocol and it would be good if libvirt could handle them.
Stefan, can you just define the XML in a way that matches the netlink definition? What you need is something like
1. VF number (optional, signifies that 2/3 are done in firmware)
2. Lower-level protocol
2.1. CDCP
2.1.1 SVID
2.1.2 SCID 2.2. enic
2.2.1 port profile name
2.2.2 ...
3. VDP
3.1 VSI type/version/provider
3.2 UUID
3.3 MAC/VLAN
You need to have 2. or 3. or both, and 2.1/2.2 are mutually exclusive.
Arnd
Re: [PATCH] virtif: initial interface extensions
Arnd Bergmann <[email protected]> wrote on 05/10/2010 05:46:37 PM:
Quoted 24 lines
The other question is which protocol we're using. There are as far as I can tell five options:
1. enic device driver
2. VDP
3. CDCP
4. CDCP + VDP
5. enic + VDPThe first two ones are the most interesting for now, since Linux cannot do S-VLANs yet, and they are required for CDCP. However, each of these options could theoreticall be done in the kernel (plus firmware) or in user space.
If it's done in user space, the VF number is meaningless, because the setup of the software device is also done from software, but instead you need to take care of creating the software device with the correct parameters, e.g. a macvtap device connected to a VLAN interface using the numbers you pass in the VDP protocol.
Right now, we're not planning to do the protocol that enic uses in LLDPAD, because it's not publically released. Similarly, there are no adapters that do VDP in firmware, but both these cases should be covered by the protocol and it would be good if libvirt could handle them.
Quoted 4 lines
Stefan, can you just define the XML in a way that matches the netlink definition? What you need is something like
1. VF number (optional, signifies that 2/3 are done in firmware)
Shouldn't we be able to query that number via netlink starting with the macvtap device and the following the trail to the root and trying to find a VF number on the way?
Quoted 4 lines
2. Lower-level protocol
2.1. CDCP2.1.1 SVID 2.1.2 SCID
Will the later on be qeueryable via netlink as well but not today??? Vivek tells me svid is vlan, so that could be found out from the kernel.
So if we want to only support 1 and 2 for now, I'd rather skip them for now.
Quoted 2 lines
2.2. enic
2.2.1 port profile name
as proposed on libvirt mailing list
Quoted 3 lines
2.2.2 ...3. VDP
3.1 VSI type/version/provider
as proposed on libvirt mailing list
Quoted 1 line
3.2 UUID
we have a couple of UUIDs, which one?
Quoted 1 line
3.3 MAC/VLAN
MAC: available from libvirt
VLAN: can be found out by querying for every interface for VLAN ID while following the path towards the root device.
Stefan
Re: [PATCH] virtif: initial interface extensions
In reply to Arnd BergmannQuoted 53 lines · Arnd Bergmann
Quoted 6 lines · On Monday 10 May 2010 20:56:39 Scott Feldman wrote:
With Arnd's latest additions, we have a single netlink msg, but the parameter sets are disjoint between VDP/CDCP and what we need for the kernel driver. So that means the sender (libvirt in this case) needs to know about both setups to send a single netlink msg. An alternative is a have two netlink msgs, one for each setup. That still requires the sender to know about two setups.
There are two separate issues here. The first one is whether we're doing the association in the device driver or in user space. The assumption here is that if it's in the device driver, there will be a VF number to identify the channel, while in user space that is not needed.
The other question is which protocol we're using. There are as far as I can tell five options:
1. enic device driver
2. VDP
3. CDCP
4. CDCP + VDP
5. enic + VDPThe first two ones are the most interesting for now, since Linux cannot do S-VLANs yet, and they are required for CDCP. However, each of these options could theoreticall be done in the kernel (plus firmware) or in user space.
If it's done in user space, the VF number is meaningless, because the setup of the software device is also done from software, but instead you need to take care of creating the software device with the correct parameters, e.g. a macvtap device connected to a VLAN interface using the numbers you pass in the VDP protocol.
Right now, we're not planning to do the protocol that enic uses in LLDPAD, because it's not publically released. Similarly, there are no adapters that do VDP in firmware, but both these cases should be covered by the protocol and it would be good if libvirt could handle them.
Stefan, can you just define the XML in a way that matches the netlink definition? What you need is something like
1. VF number (optional, signifies that 2/3 are done in firmware)
2. Lower-level protocol
2.1. CDCP2.1.1 SVID 2.1.2 SCID2.2. enic
2.2.1 port profile name
2.2.2 ...
3. VDP
3.1 VSI type/version/provider
3.2 UUID
3.3 MAC/VLAN
Quoted 1 line
You need to have 2. or 3. or both, and 2.1/2.2 are mutually exclusive.
I'm don't think this is going in the right direction. We're talking a pretty simple concept of a port-profile used to configure the virtual port backing a VM i/f to something that's trying to munge disjoint protocols based on pre-standard work into a single API. It's forcing all those pre-standard protocol details into the API, into the XML, and into the mgmt software (libvirt), and into the admin's lap.
I want the API to pass a port-profile name plus other information associated with the VM i/f to some mgmt object which can setup the virtual port backing the VM i/f. (And unset it). Using netlink for API let's that object be in user- or kernel-space software, hardware or firmware. How the object sets up the virtual port based on the passed port-profile is beyond the scope of the API.
My last port-profile patch is this API. It gives us this:
1) single netlink msg for kernel- and user-space
2) single parameter set from sender's perspective (libvirt)
3) single XML representation of parameters
4) single code path in kernel and libvirt
5) (potential) cross-vendor-switch VM migration
6) admin-friendly port-profile names
7) allows pre-standard (802.1Qbg/bh) details to change without bogging down the API
What I proposed on the libvirt list is to maintain a mapping database from port-profile to vendor-specific or protocol-specific parameters. Using VDP as an example, a port-profile would resolve the VDP tuple:
port-profile: "joes-garage" ---> VSI Manager ID: 15
VSI Type ID: 12345 VSI Type ID Ver: 1 other VSI settings (preassociate)
How the mapping database is maintained is beyond the scope of the API.
The port-profile string is the unifying concept. This is the common ground and the only way to be protocol-independent in the API.
-scott
Re: [PATCH] virtif: initial interface extensions
Quoted 18 lines · Scott Feldman
Quoted 16 lines · Arnd Bergmann
Stefan, can you just define the XML in a way that matches the netlink definition? What you need is something like
1. VF number (optional, signifies that 2/3 are done in firmware)
2. Lower-level protocol
2.1. CDCP2.1.1 SVID 2.1.2 SCID2.2. enic
2.2.1 port profile name
2.2.2 ...
3. VDP
3.1 VSI type/version/provider
3.2 UUID
3.3 MAC/VLAN
Quoted 8 lines
Quoted 1 line
You need to have 2. or 3. or both, and 2.1/2.2 are mutually exclusive.
I'm don't think this is going in the right direction. We're talking a pretty simple concept of a port-profile used to configure the virtual port backing a VM i/f to something that's trying to munge disjoint protocols based on pre-standard work into a single API. It's forcing all those pre-standard protocol details into the API, into the XML, and into the mgmt software (libvirt), and into the admin's lap.
No. I agree that the port-profile concept is simple and that the complexity comes from trying to merge the Linux interface with what we do for VDP. It would be much more sensible IMHO to just unify port-profiles and CDCP in the kernel interface, because they are conceptually more similar and both rather simple (note that there is no S-VLAN implementation in Linux, so CDCP is not there yet either).
If we layer VDP on top of the two and do it always in user space (LLDPAD with lldptool), things are much simpler on the interface side.
The focus of VDP is to manage migration, which is something that enic doesn't do (or doesn't need to do) and most of the complexity in there comes from this.
Quoted 10 lines
I want the API to pass a port-profile name plus other information associated with the VM i/f to some mgmt object which can setup the virtual port backing the VM i/f. (And unset it). Using netlink for API let's that object be in user- or kernel-space software, hardware or firmware. How the object sets up the virtual port based on the passed port-profile is beyond the scope of the API.
My last port-profile patch is this API. It gives us this:
1) single netlink msg for kernel- and user-space
Except for the VF argument, which is kernel- only, and it doesn't work if user space needs additional information that only the firmware has in your case (something like a virtual channel ID).
Quoted 5 lines
2) single parameter set from sender's perspective (libvirt)
3) single XML representation of parameters
4) single code path in kernel and libvirt5) (potential) cross-vendor-switch VM migration
You cannot do live migration between Cisco switches and those implementing 802.1Qbg, because of the differences between port profiles and VSI types, e.g. how they handle VLANs or handover during migration. Migration between 802.1Qbg capable switches is covered by the standard.
Quoted 4 lines
6) admin-friendly port-profile names
7) allows pre-standard (802.1Qbg/bh) details to change without bogging down the API
We're basically nailing down the API right here. As soon as this is supported in Linux, it will be the standard, so the details won't be able to change any more.
Quoted 8 lines
What I proposed on the libvirt list is to maintain a mapping database from port-profile to vendor-specific or protocol-specific parameters. Using VDP as an example, a port-profile would resolve the VDP tuple:
port-profile: "joes-garage" ---> VSI Manager ID: 15
VSI Type ID: 12345 VSI Type ID Ver: 1 other VSI settings (preassociate)
I agree that the port profile name matches the VSI manager/type/version ID to a large degree and that it would be nice to unify these, but I don't think there is a point given all the other differences.
There is no way around the preassociate/associate/disassociate messages being part of the API, because otherwise you cannot do seamless migration across multiple switches.
Also, the primary key on VDP is the VSI UUID, which needs to be the same on the target host after migration, while in your case the switch never identifies the guest itself, only the port profile.
If I have understood you correctly, the primary key identifying a port on enic is something that is only visible to the switch and nic firmware, but never to software, so you identify the guest by its VF number on the user API.
Arnd
Re: [PATCH] virtif: initial interface extensions
In reply to Scott FeldmanQuoted 5 lines
I'm don't think this is going in the right direction. We're talking a pretty simple concept of a port-profile used to configure the virtual port backing a VM i/f to something that's trying to munge disjoint protocols based on pre-standard work into a single API. It's forcing all those
The multi-channel setup and VDP can live, and are designed to live together:
- multiple channels per link (setup by the bridge) -- CDCP
- association of a vsi (virtual interface) to bridge ports -- VDP
However, their implementation can be pursued separately.
Quoted 4 lines
pre-standard protocol details into the API, into the XML, and into the mgmt software (libvirt), and into the admin's lap.
I want the API to pass a port-profile name plus other information associated
The problem with defining a 'name' as the lookup key to another database is that one then requires additional management mechanisms to describe, maintain, and map to the real values needed. It lands in admin's lap anyway by a different path.
Quoted 5 lines
with the VM i/f to some mgmt object which can setup the virtual port backing the VM i/f. (And unset it). Using netlink for API let's that object be in user- or kernel-space software, hardware or firmware. How the object sets up the virtual port based on the passed port-profile is beyond the scope of the API.
Agree. See Stefan's last patch in libvirt - it converges 802.1Qbh/portprofile
and 802.1Qbg quite well. The netlink message can be deciphered by the
recipient and meets the advantages below. (Or, one could use a direct unix domain socket to communicate with a daemon as well).
Quoted 16 lines
My last port-profile patch is this API. It gives us this:
1) single netlink msg for kernel- and user-space
2) single parameter set from sender's perspective (libvirt)
3) single XML representation of parameters
4) single code path in kernel and libvirt
5) (potential) cross-vendor-switch VM migration
6) admin-friendly port-profile names
7) allows pre-standard (802.1Qbg/bh) details to change without bogging down the APIWhat I proposed on the libvirt list is to maintain a mapping database from port-profile to vendor-specific or protocol-specific parameters. Using VDP as an example, a port-profile would resolve the VDP tuple:
Quoted 6 lines
port-profile: "joes-garage" ---> VSI Manager ID: 15
VSI Type ID: 12345 VSI Type ID Ver: 1 other VSI settings (preassociate)How the mapping database is maintained is beyond the scope of the API.
Yes, but that will add another mechanism to set the mappings. The xml posted (on libvirt) defines the vsi values and avoids this additional management. It also allows for port-profile name for 802.1Qbh.
thanks
Vivek
A message that is not in the archive (sent off-list or not yet received): OFFE8F5F70.5C07C656-ON8525771F.00787A71-8525771F.007FCDFC@us.ibm.com
Re: [PATCH] virtif: initial interface extensions
Quoted 10 lines · Stefan Berger
Arnd Bergmann <[email protected]> wrote on 05/10/2010 05:46:37 PM:
Quoted 4 lines
Stefan, can you just define the XML in a way that matches the netlink definition? What you need is something like
1. VF number (optional, signifies that 2/3 are done in firmware)
Shouldn't we be able to query that number via netlink starting with the macvtap device and the following the trail to the root and trying to find a VF number on the way?
No. If we have a macvtap device, there is no VF number. The VF number should be known to libvirt in those cases where instead of creating a macvtap device, it assigns a VF of an SR-IOV adapter to the guest.
Quoted 10 lines
Quoted 4 lines
2. Lower-level protocol
2.1. CDCP2.1.1 SVID 2.1.2 SCIDWill the later on be qeueryable via netlink as well but not today??? Vivek tells me svid is vlan, so that could be found out from the kernel.
So if we want to only support 1 and 2 for now, I'd rather skip them for now.
svid is almost vlan (hence S-VLAN), but slightly different and is not currently supported by the kernel. Again, if the implementation is done in firmware, libvirt needs to set the same S-VLAN ID when setting up the VF and when associating it to the switch.
When it's done in software, we need to create the device (or have it created in advance), so you either know it or can query it as you describe.
You don't need to support it yet in libvirt, but the definition should be done in a way that leaves the option open to add it later.
Quoted 9 lines
Quoted 3 lines
2.2.2 ...3. VDP
3.1 VSI type/version/provideras proposed on libvirt mailing list
Quoted 1 line
3.2 UUID
we have a couple of UUIDs, which one?
This is a UUID that describes the VSI to the switch. It needs to be unique in the migration domain. For a guest that has multiple macvtap interfaces, you either need to have a single UUID and put all MAC/VLAN pairs into the same netlink message with this UUID, or have one UUID per device.
Quoted 5 lines
Quoted 1 line
3.3 MAC/VLAN
MAC: available from libvirt
VLAN: can be found out by querying for every interface for VLAN ID while following the path towards the root device.
Yes, in case of macvtap.
Arnd
A message that is not in the archive (sent off-list or not yet received): OF2E2B37D4.51A81D74-ON85257720.0045FA96-85257720.004C5403@us.ibm.com
Re: [PATCH] virtif: initial interface extensions
Quoted 8 lines · Stefan Berger
Arnd Bergmann <[email protected]> wrote on 05/11/2010 08:25:27 AM:
Quoted 6 lines
netdev, Scott Feldman
Quoted 1 line · Stefan Berger
Arnd Bergmann <[email protected]> wrote on 05/10/2010 05:46:37 PM:
No. If we have a macvtap device, there is no VF number. The VF number should be known to libvirt in those cases where instead of creating a macvtap device, it assigns a VF of an SR-IOV adapter to the guest.
Quoted 8 lines
The only interface type that currently supports the vsi parameters is the 'direct' type of interface which directly maps into macvtap. That's the only one that would currently let you run the setup protocol with the switch. Regular tap devices created through other interface types (bridge, network) do not support these parameters and hence you cannot run the protocol with the switch. I never tried passthrough but I believe libvirt is not aware of what it is passing through nor do we currently support the parameters for passthrough devices.
Ok. I believe we will at least have to add the same kind of setup to bridged devices as well, not just macvtap.
For SR-IOV with device assignment, I'm not sure. This will be more important when adapters show up that actually support VEPA in hardware and don't have their own switch, but even for those with an integrated switch, it would be nice if we could use VDP correctly.
Quoted 7 lines
Quoted 4 lines
svid is almost vlan (hence S-VLAN), but slightly different and is not currently supported by the kernel. Again, if the implementation is done in firmware, libvirt needs to set the same S-VLAN ID when setting up the VF and when associating it to the switch.
The netlink messages go into the kernel and I suppose the driver should be able to find out what the S-VLAN ID is that it needs to use, no?
Possibly yes, but that will depend on how the firmware does this. It may also be possible that adapters implement this similar to what enic does, which does not expose the S-VLAN ID at all and uses the VF number as the identifier.
Maybe we should leave out the CDCP stuff for now, until we start seeing hardware for it.
Quoted 8 lines
Quoted 5 lines
This is a UUID that describes the VSI to the switch. It needs to be unique in the migration domain. For a guest that has multiple macvtap interfaces, you either need to have a single UUID and put all MAC/VLAN pairs into the same netlink message with this UUID, or have one UUID per device.
In that case it's the instanceID as proposed in this XML here:
Quoted 7 lines
<interface type='direct'>
<source dev='static' mode='vepa'/>
<model type='virtio'/>
<vsi managerid='12' typeid='0x123456' typeidversion='1'
instanceid='fa9b7fff-b0a0-4893-8e0e-beef4ff18f8f' />
<filterref filter='clean-traffic'/>
</interface>Yes.
Arnd