virtio-net: help live migrate SR-IOV devices

https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html

Quoted 98 lines · Jason Wang

Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.

Quoted 5 lines

Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?

Can we flip the argument and ask why is the kernel supposed to be responsible for this? It's not like we run DHCP out of the kernel on new interfaces...

Stephen Hemminger

2017-11-30 04:10 UTC #6 lore raw

Quoted 115 lines · On Wed, 29 Nov 2017 19:51:38 -0800 Jakub Kicinski…

Quoted 102 lines · Jason Wang
Quoted 74 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the
virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the
required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device
notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC
to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on
the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Quoted 6 lines
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically. In fact I'm trying to discuss this with NetworkManager folks and Red Hat right now:

Quoted 5 lines

https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
Can we flip the argument and ask why is the kernel supposed to be responsible for this? It's not like we run DHCP out of the kernel on new interfaces...

Although "policy should not be in the kernel" is a a great mantra, it is not practical in the real world.

If you think it can be solved in userspace, then you haven't had to deal with four different network initialization
systems, multiple orchestration systems and customers on ancient Enterprise distributions.
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Stephen Hemminger

2017-11-30 04:10 UTC #7 lore raw

In reply to Stephen Hemminger

Quoted 115 lines · On Wed, 29 Nov 2017 19:51:38 -0800 Jakub Kicinski…

Quoted 102 lines · Jason Wang
Quoted 74 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the
virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the
required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device
notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC
to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on
the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Quoted 6 lines
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically. In fact I'm trying to discuss this with NetworkManager folks and Red Hat right now:

Quoted 5 lines

https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
Can we flip the argument and ask why is the kernel supposed to be responsible for this? It's not like we run DHCP out of the kernel on new interfaces...

Although "policy should not be in the kernel" is a a great mantra, it is not practical in the real world.

Jakub Kicinski

2017-11-30 04:21 UTC #8 lore raw

Quoted 37 lines · Stephen Hemminger

Quoted 30 lines · On Wed, 29 Nov 2017 19:51:38 -0800 Jakub Kicinski wrote:
Quoted 17 lines · Jason Wang
On 2017年11月29日 03:27, Jesse Brandeburg wrote:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC
to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on
the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Quoted 6 lines
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically. In fact I'm trying to discuss this with NetworkManager folks and Red Hat right now:
Quoted 5 lines
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
Can we flip the argument and ask why is the kernel supposed to be responsible for this? It's not like we run DHCP out of the kernel on new interfaces...

Quoted 7 lines

Although "policy should not be in the kernel" is a a great mantra, it is not practical in the real world.
If you think it can be solved in userspace, then you haven't had to deal with four different network initialization
systems, multiple orchestration systems and customers on ancient Enterprise distributions.

I would accept that argument if anyone ever tried to get those Enterprise distros to handle this use case. From conversations I had it seemed like no one ever did, and SR-IOV+virtio bonding is what has been done to solve this since day 1 of SR-IOV networking.

For practical reasons it's easier to push this into the kernel, because vendors rarely employ developers of the user space orchestrations systems. Is that not the real problem here, potentially? :)
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Jakub Kicinski

2017-11-30 04:21 UTC #9 lore raw

Quoted 37 lines · Stephen Hemminger

Quoted 30 lines · On Wed, 29 Nov 2017 19:51:38 -0800 Jakub Kicinski wrote:
Quoted 17 lines · Jason Wang
On 2017年11月29日 03:27, Jesse Brandeburg wrote:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC
to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on
the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Quoted 6 lines
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically. In fact I'm trying to discuss this with NetworkManager folks and Red Hat right now:
Quoted 5 lines
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
Can we flip the argument and ask why is the kernel supposed to be responsible for this? It's not like we run DHCP out of the kernel on new interfaces...

Quoted 7 lines

Although "policy should not be in the kernel" is a a great mantra, it is not practical in the real world.
If you think it can be solved in userspace, then you haven't had to deal with four different network initialization
systems, multiple orchestration systems and customers on ancient Enterprise distributions.

For practical reasons it's easier to push this into the kernel, because vendors rarely employ developers of the user space orchestrations systems. Is that not the real problem here, potentially? :)

achiad shochat

2017-11-30 08:08 UTC #16 lore raw

Quoted 102 lines · Jason Wang

Quoted 71 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]>
Date:   Tue Aug 1 19:58:53 2017 -0700
netvsc: transparent VF management
This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.

Quoted 18 lines

Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
Thanks
Quoted 8 lines
[1]
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/sect-migrating_virtual_machines_between_hosts
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

I do not see why to couple the solution with any specific para-virt technology. Not with netvsc, nor with virt-io.
One may wish to implement the routing between the VMs and the HV without any PV device at all, e.g using VF representors and PCIe loopback, as done with ASAP2-direct.
This method is actually much more efficient in CPU utilization (on the expense of PCIe BW utilization).

So let's try to first specify the problems that need to be resolved in order to support Live Migration with SR-IOV rather than rely on already-done-work (netvsc) without understanding if/why it was right from the beginning.

To my understanding the problems are the following:
1) DMA: with SR-IOV devices write directly into the guests memory which yields dirty guest pages that are not marked as dirty for the host CPU MMU, thus preventing the migration pre-copy phase from starting while the guest is running on the source machine.
2) Guest network interface persistency: VF detachment causes VF driver PCI remove which causes the VF netdev to disappear. If that VF netdev is a guest primary interface (has an IP), sockets using it will break.

Re problem #1:
So far in this mail thread, it was taken for granted that the way to resolve it is to have a PV device as backup for the pre-copy phase. In addition to tying the solution with para-virt being in place (which as already said seems a wrong enforcement to me), it does not really solve the problem, rather partially works around it.
It just mitigates the problem from long service downtime to long service degradation time.
To really stab the problem in its heart we need to just mark the guest DMA written pages as dirty.
Alexander Duyck already initiated patches to address it ~two years ago
(https://groups.google.com/forum/#!topic/linux.kernel/aIQOsh2oJEk) but un-fortunately they were abandoned.
The simplest way I can think of to resolve it is to have the guest VF driver just read-modify-write some word of each DMA page before passing it to the stack by netif_rx().
To limit the performance impact of this operation we can signal the VM to start doing it only upon pre-copy phase start.

Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Another example - when to use PV and when to use VF? One may want to use PV only if VF is gone/down, while others may want to use PV also when VF is up, e.g for multicasting.
I think the right way to address it is to have a new dedicated module for this purpose.
Have it automatically enslave PV and VF netdevs according to user configured policy. Enslave the VF even if there is no PV device at all.

This way we get:
1) Optimal migration performance
2) A PV agnostic (VM may be migrated even from one PV technology to another) and HW device agnostic solution
A dedicated generic module will also enforce a lower common
denominator of guest netdev features, preventing migration dependency on source/guest machine capabilities.
3) Out-of-the box solution yet with generic methods for policy setting

Thanks
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

achiad shochat

2017-11-30 08:08 UTC #17 lore raw

Quoted 102 lines · Jason Wang

Quoted 71 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]>
Date:   Tue Aug 1 19:58:53 2017 -0700
netvsc: transparent VF management
This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.

Quoted 18 lines

Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
Thanks
Quoted 8 lines
[1]
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/virtual_machine_management_guide/sect-migrating_virtual_machines_between_hosts
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Thanks

Michael S. Tsirkin

maintainer

2017-11-30 13:54 UTC #10 lore raw

Quoted 102 lines · Jakub Kicinski

Quoted 101 lines · Jason Wang
Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>

Quoted 6 lines

Quoted 2 lines
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically.

It does have this info (MAC addresses match) but where's the policy here? IMHO the policy has been set by the hypervisor already. From hypervisor POV adding passthrough is a commitment not to migrate until guest stops using the passthrough device.

Within the guest, the bond is required for purely functional reasons - just to maintain a link up since we know SRIOV will will go away. Maintaining an uninterrupted connection is not a policy - it's what networking is about.

Quoted 4 lines

In fact I'm trying to discuss this with NetworkManager
folks and Red Hat right now:
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html

I thought we should do it too, for a while.

But now, I think that the real issue is this: kernel exposes what looks like two network devices to userspace, but in fact it is just one backend device, just exposed by hypervisor in a weird way for compatibility reasons.

For example you will not get a better reliability or throughput by using both of them - the only bonding mode that makes sense is fail over. As another example, if the underlying physical device lost its link, trying to use virtio won't help - it's only useful when the passthrough device is gone for good. As another example, there is no point in not configuring a bond. As a last example, depending on how the backend is configured, virtio might not even work when the pass-through device is active.

So from that point of view, showing two network devices to userspace is a bug that we are asking userspace to work around.

Quoted 2 lines

Can we flip the argument and ask why is the kernel supposed to be responsible for this?

Because if we show a single device to userspace the number of misconfigured guests will go down, and we won't lose any useful flexibility.

Quoted 2 lines

It's not like we run DHCP out of the kernel
on new interfaces...

Because one can set up a static IP, IPv6 doesn't always need DHCP, etc.

Signature · 5 lines

MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Michael S. Tsirkin

maintainer

2017-11-30 13:54 UTC #11 lore raw

Quoted 102 lines · Jakub Kicinski

Quoted 101 lines · Jason Wang
Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>

Quoted 6 lines

Quoted 2 lines
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically.

It does have this info (MAC addresses match) but where's the policy here? IMHO the policy has been set by the hypervisor already.

Quoted 1 line

From hypervisor POV adding passthrough is a commitment not to migrate

until guest stops using the passthrough device.

Quoted 4 lines

In fact I'm trying to discuss this with NetworkManager
folks and Red Hat right now:
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html

I thought we should do it too, for a while.

So from that point of view, showing two network devices to userspace is a bug that we are asking userspace to work around.

Quoted 2 lines

Can we flip the argument and ask why is the kernel supposed to be responsible for this?

Because if we show a single device to userspace the number of misconfigured guests will go down, and we won't lose any useful flexibility.

Quoted 2 lines

It's not like we run DHCP out of the kernel
on new interfaces...

Because one can set up a static IP, IPv6 doesn't always need DHCP, etc.

Signature · 1 line

MST

Michael S. Tsirkin

maintainer

2017-11-30 14:11 UTC #18 lore raw

Quoted 11 lines · achiad shochat

Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.

It's a policy but not guest userspace policy.

The hypervisor certainly knows.

Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".

Quoted 3 lines

Another example - when to use PV and when to use VF? One may want to use PV only if VF is gone/down, while others may want to use PV also when VF is up, e.g for multicasting.

There are a bunch of configurations where these two devices share the same physical backend. In that case there is no point in multicasting through both devices, in fact, PV might not even work at all when passthrough is active.

IMHO these cases are what's worth handling in the kernel.

When there are two separate backends, we are getting into policy and it's best to leave this to userspace (and it's unlikely network manager will automatically do the right thing here, either).

Quoted 8 lines

I think the right way to address it is to have a new dedicated module for this purpose.
Have it automatically enslave PV and VF netdevs according to user configured policy. Enslave the VF even if there is no PV device at all.
This way we get:
1) Optimal migration performance

This remains to be proved.

Quoted 2 lines

2) A PV agnostic (VM may be migrated even from one PV technology to another)

Yes - in theory kvm could expose both a virtio and a hyperv device. But what would be the point? Just do the abstraction in the hypervisor.

Quoted 1 line

and HW device agnostic solution

That's useful but I don't think we need to involve userspace for that. HW abstraction is kernel's job.

Quoted 3 lines

A dedicated generic module will also enforce a lower common
denominator of guest netdev features, preventing migration dependency on source/guest machine capabilities.

If all we are discussing is where this code should live, then I do not really care. Let's implement it in virtio, then if we find we have a lot of common we can factor it out.

Quoted 4 lines

3) Out-of-the box solution yet with generic methods for policy setting
Thanks

There's no real policy that guest can set though. All setting happens on the hypervisor side.

Signature · 1 line

MST

Michael S. Tsirkin

maintainer

2017-11-30 14:11 UTC #19 lore raw

Quoted 11 lines · achiad shochat

Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.

It's a policy but not guest userspace policy.

The hypervisor certainly knows.

Quoted 3 lines

Another example - when to use PV and when to use VF? One may want to use PV only if VF is gone/down, while others may want to use PV also when VF is up, e.g for multicasting.

IMHO these cases are what's worth handling in the kernel.

When there are two separate backends, we are getting into policy and it's best to leave this to userspace (and it's unlikely network manager will automatically do the right thing here, either).

Quoted 8 lines

I think the right way to address it is to have a new dedicated module for this purpose.
Have it automatically enslave PV and VF netdevs according to user configured policy. Enslave the VF even if there is no PV device at all.
This way we get:
1) Optimal migration performance

This remains to be proved.

Quoted 2 lines

2) A PV agnostic (VM may be migrated even from one PV technology to another)

Yes - in theory kvm could expose both a virtio and a hyperv device. But what would be the point? Just do the abstraction in the hypervisor.

Quoted 1 line

and HW device agnostic solution

That's useful but I don't think we need to involve userspace for that. HW abstraction is kernel's job.

Quoted 3 lines

A dedicated generic module will also enforce a lower common
denominator of guest netdev features, preventing migration dependency on source/guest machine capabilities.

If all we are discussing is where this code should live, then I do not really care. Let's implement it in virtio, then if we find we have a lot of common we can factor it out.

Quoted 4 lines

3) Out-of-the box solution yet with generic methods for policy setting
Thanks

There's no real policy that guest can set though. All setting happens on the hypervisor side.

Signature · 1 line

MST

Michael S. Tsirkin

maintainer

2017-11-30 14:14 UTC #47 lore raw

Quoted 2 lines · Jason Wang

If my understanding is correct there's no need to for any extension of virtio spec.

There appears to be a concern that some existing configurations might use same MAC for an unrelated reason. Not sure what that could be, but for sure, we could add a feature flag. That needs to be approved by the virtio TC, but it's just a single line in the spec no big deal, we can help here.

Quoted 1 line

If this is true, maybe you can start to prepare the patch?

Yes, please do. We can add a safeguard of a feature bit on top.

Signature · 1 line

MST

Michael S. Tsirkin

maintainer

2017-11-30 14:14 UTC #48 lore raw

Quoted 2 lines · Jason Wang

If my understanding is correct there's no need to for any extension of virtio spec.

Quoted 1 line

If this is true, maybe you can start to prepare the patch?

Yes, please do. We can add a safeguard of a feature bit on top.

Signature · 1 line

MST

Jakub Kicinski

2017-11-30 20:48 UTC #12 lore raw

Quoted 127 lines · Michael S. Tsirkin

Quoted 102 lines · Jakub Kicinski
Quoted 101 lines · Jason Wang
Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
Quoted 6 lines
Quoted 2 lines
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically.
It does have this info (MAC addresses match) but where's the policy here? IMHO the policy has been set by the hypervisor already. From hypervisor POV adding passthrough is a commitment not to migrate until guest stops using the passthrough device.
Within the guest, the bond is required for purely functional reasons - just to maintain a link up since we know SRIOV will will go away. Maintaining an uninterrupted connection is not a policy - it's what networking is about.
Quoted 4 lines
In fact I'm trying to discuss this with NetworkManager
folks and Red Hat right now:
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
I thought we should do it too, for a while.

Quoted 7 lines

But now, I think that the real issue is this: kernel exposes what looks like two network devices to userspace, but in fact it is just one backend device, just exposed by hypervisor in a weird way for compatibility reasons.
For example you will not get a better reliability or throughput by using both of them - the only bonding mode that makes sense is fail over.

Yes, I'm talking about fail over.

Quoted 9 lines

As another example, if the underlying physical device lost its link, trying to use virtio won't help - it's only useful when the passthrough device is gone for good. As another example, there is no point in not configuring a bond. As a last example, depending on how the backend is configured, virtio might not even work when the pass-through device is active.
So from that point of view, showing two network devices to userspace is a bug that we are asking userspace to work around.

I'm confused by what you're saying here. IIRC the question is whether we expose 2 netdevs or 3. There will always be a virtio netdev and a VF netdev. I assume you're not suggesting hiding the VF netdev. So the question is do we expose a VF netdev and a combo virtio netdev which is also a bond or do we expose a VF netdev a virtio netdev, and a active/passive bond/team which is a well understood and architecturally correct construct.

Quoted 6 lines

Quoted 2 lines
Can we flip the argument and ask why is the kernel supposed to be responsible for this?
Because if we show a single device to userspace the number of misconfigured guests will go down, and we won't lose any useful flexibility.

Again, single device?

Quoted 4 lines

Quoted 2 lines
It's not like we run DHCP out of the kernel
on new interfaces...
Because one can set up a static IP, IPv6 doesn't always need DHCP, etc.

But we don't handle LACP, etc.

Look, as much as I don't like this, I'm not going to argue about this to death. I just find it very dishonest to claim kernel *has to* do it, when no one seem to have made any honest attempts to solve this in user space for the last 10 years :/
_______________________________________________
Virtualization mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Jakub Kicinski

2017-11-30 20:48 UTC #13 lore raw

Quoted 127 lines · Michael S. Tsirkin

Quoted 102 lines · Jakub Kicinski
Quoted 101 lines · Jason Wang
Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
Quoted 6 lines
Quoted 2 lines
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically.
It does have this info (MAC addresses match) but where's the policy here? IMHO the policy has been set by the hypervisor already. From hypervisor POV adding passthrough is a commitment not to migrate until guest stops using the passthrough device.
Within the guest, the bond is required for purely functional reasons - just to maintain a link up since we know SRIOV will will go away. Maintaining an uninterrupted connection is not a policy - it's what networking is about.
Quoted 4 lines
In fact I'm trying to discuss this with NetworkManager
folks and Red Hat right now:
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
I thought we should do it too, for a while.

Quoted 7 lines

But now, I think that the real issue is this: kernel exposes what looks like two network devices to userspace, but in fact it is just one backend device, just exposed by hypervisor in a weird way for compatibility reasons.
For example you will not get a better reliability or throughput by using both of them - the only bonding mode that makes sense is fail over.

Yes, I'm talking about fail over.

Quoted 9 lines

As another example, if the underlying physical device lost its link, trying to use virtio won't help - it's only useful when the passthrough device is gone for good. As another example, there is no point in not configuring a bond. As a last example, depending on how the backend is configured, virtio might not even work when the pass-through device is active.
So from that point of view, showing two network devices to userspace is a bug that we are asking userspace to work around.

Quoted 6 lines

Quoted 2 lines
Can we flip the argument and ask why is the kernel supposed to be responsible for this?
Because if we show a single device to userspace the number of misconfigured guests will go down, and we won't lose any useful flexibility.

Again, single device?

Quoted 4 lines

Quoted 2 lines
It's not like we run DHCP out of the kernel
on new interfaces...
Because one can set up a static IP, IPv6 doesn't always need DHCP, etc.

But we don't handle LACP, etc.

Michael S. Tsirkin

maintainer

2017-12-01 05:13 UTC #14 lore raw

Quoted 145 lines · Jakub Kicinski

Quoted 127 lines · Michael S. Tsirkin
Quoted 102 lines · Jakub Kicinski
Quoted 101 lines · Jason Wang
Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
Quoted 6 lines
Quoted 2 lines
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically.
It does have this info (MAC addresses match) but where's the policy here? IMHO the policy has been set by the hypervisor already. From hypervisor POV adding passthrough is a commitment not to migrate until guest stops using the passthrough device.
Within the guest, the bond is required for purely functional reasons - just to maintain a link up since we know SRIOV will will go away. Maintaining an uninterrupted connection is not a policy - it's what networking is about.
Quoted 4 lines
In fact I'm trying to discuss this with NetworkManager
folks and Red Hat right now:
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
I thought we should do it too, for a while.
Quoted 7 lines
But now, I think that the real issue is this: kernel exposes what looks like two network devices to userspace, but in fact it is just one backend device, just exposed by hypervisor in a weird way for compatibility reasons.
For example you will not get a better reliability or throughput by using both of them - the only bonding mode that makes sense is fail over.
Yes, I'm talking about fail over.
Quoted 7 lines
As another example, if the underlying physical device lost its link, trying to use virtio won't help - it's only useful when the passthrough device is gone for good. As another example, there is no point in not configuring a bond. As a last example, depending on how the backend is configured, virtio might not even work when the pass-through device is active.

Quoted 6 lines

Quoted 2 lines
So from that point of view, showing two network devices to userspace is a bug that we are asking userspace to work around.
I'm confused by what you're saying here. IIRC the question is whether we expose 2 netdevs or 3. There will always be a virtio netdev and a VF netdev. I assume you're not suggesting hiding the VF netdev.

Passthrough is a better term, it does not have to be a VF.

All I am saying is these are not two independent devices.

It's a good point - ideally we would hide it completely. Not sure we can.

Quoted 5 lines

So
the question is do we expose a VF netdev and a combo virtio netdev which is also a bond or do we expose a VF netdev a virtio netdev, and a active/passive bond/team which is a well understood and architecturally correct construct.

It's a well understood construct for bonding but it is not exactly what we are dealing with here. What we have is a single device with two ways to access it.

Quoted 12 lines

Quoted 6 lines
Quoted 2 lines
Can we flip the argument and ask why is the kernel supposed to be responsible for this?
Because if we show a single device to userspace the number of misconfigured guests will go down, and we won't lose any useful flexibility.
Again, single device?
Quoted 3 lines
Quoted 2 lines
It's not like we run DHCP out of the kernel
on new interfaces...

Quoted 6 lines

Quoted 1 line
Because one can set up a static IP, IPv6 doesn't always need DHCP, etc.
But we don't handle LACP, etc.
Look, as much as I don't like this, I'm not going to argue about this to death. I just find it very dishonest to claim kernel *has to* do it,

kernel does not *have* to do it. It might be better to do it in kernel though.

Quoted 2 lines

when no one seem to have made any honest attempts to solve this in user space for the last 10 years :/

Each time I tried to convince userspace maintainers I get the kind of pushback you seem to have encountered. And the reason is that they are used to manage bonding for actual multiple ethernet ports with all the complexity this entails. Maybe it wasn't an honest attempt in that I didn't actually post patches, but there you are.

Signature · 5 lines

MST
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Michael S. Tsirkin

maintainer

2017-12-01 05:13 UTC #15 lore raw

Quoted 145 lines · Jakub Kicinski

Quoted 127 lines · Michael S. Tsirkin
Quoted 102 lines · Jakub Kicinski
Quoted 101 lines · Jason Wang
Quoted 70 lines · Jesse Brandeburg
Hi, I'd like to get some feedback on a proposal to enhance virtio-net to ease configuration of a VM and that would enable live migration of passthrough network SR-IOV devices.
Today we have SR-IOV network devices (VFs) that can be passed into a VM in order to enable high performance networking direct within the VM. The problem I am trying to address is that this configuration is generally difficult to live-migrate. There is documentation [1] indicating that some OS/Hypervisor vendors will support live migration of a system with a direct assigned networking device. The problem I see with these implementations is that the network configuration requirements that are passed on to the owner of the VM are quite complicated. You have to set up bonding, you have to configure it to enslave two interfaces, those interfaces (one is virtio-net, the other is SR-IOV device/driver like ixgbevf) must support MAC address changes requested in the VM, and on and on...
So, on to the proposal:
Modify virtio-net driver to be a single VM network device that enslaves an SR-IOV network device (inside the VM) with the same MAC address. This would cause the virtio-net driver to appear and work like a simplified bonding/team driver. The live migration problem would be solved just like today's bonding solution, but the VM user's networking config would be greatly simplified.
At it's simplest, it would appear something like this in the VM.
==========
= vnet0 =
          =============
(virtio- = |
net) = |
= ==========
= = ixgbef =
========== ==========
(forgive the ASCII art)
The fast path traffic would prefer the ixgbevf or other SR-IOV device path, and fall back to virtio's transmit/receive when migrating.
Compared to today's options this proposal would
1) make virtio-net more sticky, allow fast path traffic at SR-IOV speeds
2) simplify end user configuration in the VM (most if not all of the set up to enable migration would be done in the hypervisor)
3) allow live migration via a simple link down and maybe a PCI hot-unplug of the SR-IOV device, with failover to the virtio-net driver core
4) allow vendor agnostic hardware acceleration, and live migration between vendors if the VM os has driver support for all the required SR-IOV devices.
Runtime operation proposed:
- <in either order> virtio-net driver loads, SR-IOV driver loads
- virtio-net finds other NICs that match it's MAC address by both examining existing interfaces, and sets up a new device notifier
- virtio-net enslaves the first NIC with the same MAC address
- virtio-net brings up the slave, and makes it the "preferred" path
- virtio-net follows the behavior of an active backup bond/team
- virtio-net acts as the interface to the VM
- live migration initiates
- link goes down on SR-IOV, or SR-IOV device is removed
- failover to virtio-net as primary path
- migration continues to new host
- new host is started with virio-net as primary
- if no SR-IOV, virtio-net stays primary
- hypervisor can hot-add SR-IOV NIC, with same MAC addr as virtio
- virtio-net notices new NIC and starts over at enslave step above
Quoted 6 lines
Future ideas (brainstorming):
- Optimize Fast east-west by having special rules to direct east-west traffic through virtio-net traffic path
Thanks for reading!
Jesse
Cc netdev.
Interesting, and this method is actually used by netvsc now:
commit 0c195567a8f6e82ea5535cd9f1d54a1626dd233e
Author: stephen hemminger <[email protected]> Date:   Tue Aug 1 19:58:53 2017 -0700
    netvsc: transparent VF management
    This patch implements transparent fail over from synthetic NIC to SR-IOV virtual function NIC in Hyper-V environment. It is a better alternative to using bonding as is done now. Instead, the receive and transmit fail over is done internally inside the driver.
    Using bonding driver has lots of issues because it depends on the script being run early enough in the boot process and with sufficient information to make the association. This patch moves all that functionality into the kernel.
Signed-off-by Stephen Hemminger <[email protected]>
Signed-off-by David S. Miller <[email protected]>
Quoted 6 lines
Quoted 2 lines
If my understanding is correct there's no need to for any extension of virtio spec. If this is true, maybe you can start to prepare the patch?
IMHO this is as close to policy in the kernel as one can get. User land has all the information it needs to instantiate that bond/team automatically.
It does have this info (MAC addresses match) but where's the policy here? IMHO the policy has been set by the hypervisor already. From hypervisor POV adding passthrough is a commitment not to migrate until guest stops using the passthrough device.
Within the guest, the bond is required for purely functional reasons - just to maintain a link up since we know SRIOV will will go away. Maintaining an uninterrupted connection is not a policy - it's what networking is about.
Quoted 4 lines
In fact I'm trying to discuss this with NetworkManager
folks and Red Hat right now:
https://mail.gnome.org/archives/networkmanager-list/2017-November/msg00038.html
I thought we should do it too, for a while.
Quoted 7 lines
But now, I think that the real issue is this: kernel exposes what looks like two network devices to userspace, but in fact it is just one backend device, just exposed by hypervisor in a weird way for compatibility reasons.
For example you will not get a better reliability or throughput by using both of them - the only bonding mode that makes sense is fail over.
Yes, I'm talking about fail over.
Quoted 7 lines
As another example, if the underlying physical device lost its link, trying to use virtio won't help - it's only useful when the passthrough device is gone for good. As another example, there is no point in not configuring a bond. As a last example, depending on how the backend is configured, virtio might not even work when the pass-through device is active.

Quoted 6 lines

Quoted 2 lines
So from that point of view, showing two network devices to userspace is a bug that we are asking userspace to work around.
I'm confused by what you're saying here. IIRC the question is whether we expose 2 netdevs or 3. There will always be a virtio netdev and a VF netdev. I assume you're not suggesting hiding the VF netdev.

Passthrough is a better term, it does not have to be a VF.

All I am saying is these are not two independent devices.

It's a good point - ideally we would hide it completely. Not sure we can.

Quoted 5 lines

So
the question is do we expose a VF netdev and a combo virtio netdev which is also a bond or do we expose a VF netdev a virtio netdev, and a active/passive bond/team which is a well understood and architecturally correct construct.

It's a well understood construct for bonding but it is not exactly what we are dealing with here. What we have is a single device with two ways to access it.

Quoted 12 lines

Quoted 6 lines
Quoted 2 lines
Can we flip the argument and ask why is the kernel supposed to be responsible for this?
Because if we show a single device to userspace the number of misconfigured guests will go down, and we won't lose any useful flexibility.
Again, single device?
Quoted 3 lines
Quoted 2 lines
It's not like we run DHCP out of the kernel
on new interfaces...

Quoted 6 lines

Quoted 1 line
Because one can set up a static IP, IPv6 doesn't always need DHCP, etc.
But we don't handle LACP, etc.
Look, as much as I don't like this, I'm not going to argue about this to death. I just find it very dishonest to claim kernel *has to* do it,

kernel does not *have* to do it. It might be better to do it in kernel though.

Quoted 2 lines

when no one seem to have made any honest attempts to solve this in user space for the last 10 years :/

Signature · 1 line

MST

Shannon Nelson

2017-12-01 20:08 UTC #20 lore raw

Quoted 13 lines · Michael S. Tsirkin

Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.

Quoted 8 lines

It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".

This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.

The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.

Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.

sln

Shannon Nelson

2017-12-01 20:09 UTC #21 lore raw

Quoted 13 lines · Michael S. Tsirkin

Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.

Quoted 8 lines

It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".

Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.

sln

Michael S. Tsirkin

maintainer

2017-12-03 05:05 UTC #22 lore raw

Quoted 33 lines · Shannon Nelson

Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.

Quoted 5 lines

Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln

I'm not sure what's the feature that is available here.

I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.

However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.

Signature · 1 line

MST

achiad shochat

2017-12-03 09:14 UTC #23 lore raw

Quoted 47 lines · Michael S. Tsirkin

Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.

Quoted 5 lines

However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
--
MST

Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.

I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.

Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.

Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...

Stephen Hemminger

2017-12-03 17:35 UTC #24 lore raw

Quoted 75 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…

Quoted 47 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
Quoted 5 lines
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.

Quoted 4 lines

Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...

There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.
Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.

Stephen Hemminger

2017-12-03 17:35 UTC #25 lore raw

In reply to Stephen Hemminger

Quoted 75 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…

Quoted 47 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
Quoted 5 lines
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.

Quoted 4 lines

Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...

achiad shochat

2017-12-04 09:51 UTC #26 lore raw

Quoted 78 lines · Stephen Hemminger

Quoted 75 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…
Quoted 47 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
Quoted 5 lines
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.

Quoted 8 lines

Quoted 4 lines
Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...
There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.

That's not a given fact.
It's due to the DMA and it should be solve anyway. Please read my first reply in this thread.

Quoted 5 lines

Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.

Right, but that virtual device must not relate to any para-virt specific technology (not netvsc, nor virtio).
Again, it seems you did not read my first reply.

achiad shochat

2017-12-04 09:51 UTC #27 lore raw

Quoted 78 lines · Stephen Hemminger

Quoted 75 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…
Quoted 47 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
Quoted 5 lines
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.

Quoted 8 lines

Quoted 4 lines
Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...
There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.

That's not a given fact.
It's due to the DMA and it should be solve anyway. Please read my first reply in this thread.

Quoted 5 lines

Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.

Right, but that virtual device must not relate to any para-virt specific technology (not netvsc, nor virtio).
Again, it seems you did not read my first reply.

Alexander Duyck

2017-12-04 16:30 UTC #28 lore raw

Quoted 56 lines · achiad shochat

Quoted 54 lines · Stephen Hemminger
Quoted 51 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…
Quoted 50 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.

Migrating without any virtual device is going to be extremely challenging, especially in any kind of virtualization setup where the hosts are not homogeneous. By providing a virtio interface you can guarantee that at least 1 network interface is available on any given host, and then fail over to that as the least common denominator for any migration.

Quoted 11 lines

Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.

This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.

Quoted 10 lines

Quoted 10 lines
Quoted 9 lines
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.
Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...

Quoted 7 lines

Quoted 3 lines
There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.
That's not a given fact.
It's due to the DMA and it should be solve anyway. Please read my first reply in this thread.

For now it is a fact. We would need to do a drastic rewrite of the DMA API in the guest/host/QEMU/IOMMU in order to avoid it for now. So as a first step I would say we should look at using this bonding type solution. Being able to defer the VF eviction could be a next step for all this as it would allow for much better performance, but we still have too many cases where the VF might not be there after a migration.

Quoted 9 lines

Quoted 5 lines
Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.
Right, but that virtual device must not relate to any para-virt specific technology (not netvsc, nor virtio).
Again, it seems you did not read my first reply.

I would agree with the need to make this agnostic. Maybe we could look at the current netvsc solution and find a way to make it generic so it could be applied to any combination of paravirtual interface and PF.

Alexander Duyck

2017-12-04 16:30 UTC #29 lore raw

In reply to Alexander Duyck

Quoted 56 lines · achiad shochat

Quoted 54 lines · Stephen Hemminger
Quoted 51 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…
Quoted 50 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.

Quoted 11 lines

Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.

Quoted 10 lines

Quoted 10 lines
Quoted 9 lines
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.
Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...

Quoted 7 lines

Quoted 3 lines
There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.
That's not a given fact.
It's due to the DMA and it should be solve anyway. Please read my first reply in this thread.

Quoted 9 lines

Quoted 5 lines
Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.
Right, but that virtual device must not relate to any para-virt specific technology (not netvsc, nor virtio).
Again, it seems you did not read my first reply.

achiad shochat

2017-12-05 09:59 UTC #30 lore raw

Quoted 65 lines · Alexander Duyck

Quoted 56 lines · achiad shochat
Quoted 54 lines · Stephen Hemminger
Quoted 51 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…
Quoted 50 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.

Quoted 7 lines

Migrating without any virtual device is going to be extremely challenging, especially in any kind of virtualization setup where the hosts are not homogeneous. By providing a virtio interface you can guarantee that at least 1 network interface is available on any given host, and then fail over to that as the least common denominator for any migration.

I am not sure why you think it is going to be so challenging. Are you referring to preserving the pass-through device driver state (RX/TX rings)?
I do not think we should preserve them, we can simply teardown the whole VF netdev (since we have a parent netdev as application interface).
The downtime impact will be negligible.

Quoted 12 lines

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.

Quoted 9 lines

This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.

I like this approach.

Quoted 18 lines

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.
Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...
Quoted 7 lines
Quoted 3 lines
There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.
That's not a given fact.
It's due to the DMA and it should be solve anyway. Please read my first reply in this thread.

Quoted 7 lines

For now it is a fact. We would need to do a drastic rewrite of the DMA API in the guest/host/QEMU/IOMMU in order to avoid it for now. So as a first step I would say we should look at using this bonding type solution. Being able to defer the VF eviction could be a next step for all this as it would allow for much better performance, but we still have too many cases where the VF might not be there after a migration.

Why would we need such a drastic rewrite?
Why would a simple Read-DontModify-Write (to mark the page as dirty) by the VF driver not do the job?

Anyway, if you have a generic virtual parent netdev handling that can be handled orthogonally.

Quoted 6 lines

Quoted 6 lines
Quoted 5 lines
Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.

Quoted 7 lines

Quoted 3 lines
Right, but that virtual device must not relate to any para-virt specific technology (not netvsc, nor virtio).
Again, it seems you did not read my first reply.
I would agree with the need to make this agnostic. Maybe we could look at the current netvsc solution and find a way to make it generic so it could be applied to any combination of paravirtual interface and PF.

Agree. That's should be the approach IMO.
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.

achiad shochat

2017-12-05 09:59 UTC #31 lore raw

Quoted 65 lines · Alexander Duyck

Quoted 56 lines · achiad shochat
Quoted 54 lines · Stephen Hemminger
Quoted 51 lines · On Sun, 3 Dec 2017 11:14:37 +0200 achiad shochat <[email protected]>…
Quoted 50 lines · Michael S. Tsirkin
Quoted 33 lines · Shannon Nelson
Quoted 13 lines · Michael S. Tsirkin
Quoted 11 lines · achiad shochat
Re. problem #2:
Indeed the best way to address it seems to be to enslave the VF driver netdev under a persistent anchor netdev.
And it's indeed desired to allow (but not enforce) PV netdev and VF netdev to work in conjunction.
And it's indeed desired that this enslavement logic work out-of-the box. But in case of PV+VF some configurable policies must be in place (and they'd better be generic rather than differ per PV technology). For example - based on which characteristics should the PV+VF coupling be done? netvsc uses MAC address, but that might not always be the desire.
Quoted 8 lines
It's a policy but not guest userspace policy.
The hypervisor certainly knows.
Are you concerned that someone might want to create two devices with the same MAC for an unrelated reason? If so, hypervisor could easily set a flag in the virtio device to say "this is a backup, use MAC to find another device".
This is something I was going to suggest: a flag or other configuration on the virtio device to help control how this new feature is used. I can imagine this might be useful to control from either the hypervisor side or the VM side.
The hypervisor might want to (1) disable it (force it off), (2) enable it for VM choice, or (3) force it on for the VM. In case (2), the VM might be able to chose whether it wants to make use of the feature, or stick with the bonding solution.
Quoted 5 lines
Either way, the kernel is making a feature available, and the user (VM or hypervisor) is able to control it by selecting the feature based on the policy desired.
sln
I'm not sure what's the feature that is available here.
I saw this as a flag that says "this device shares backend with another network device which can be found using MAC, and that backend should be preferred". kernel then forces configuration which uses that other backend - as long as it exists.
However, please Cc virtio-dev mailing list if we are doing this since this is a spec extension.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
--
MST
Can someone please explain why assume a virtio device is there at all?? I specified a case where there isn't any.

Quoted 7 lines

Migrating without any virtual device is going to be extremely challenging, especially in any kind of virtualization setup where the hosts are not homogeneous. By providing a virtio interface you can guarantee that at least 1 network interface is available on any given host, and then fail over to that as the least common denominator for any migration.

Quoted 12 lines

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.

Quoted 9 lines

This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.

I like this approach.

Quoted 18 lines

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Re policy -
Indeed the HV can request a policy from the guest but that's not a claim for the virtio device enslaving the pass-through device. Any policy can be queried by the upper enslaving device.
Bottom line - I do not see a single reason to have the virtio netdev (nor netvsc or any other PV netdev) enslave another netdev by itself. If we'd do it right with netvsc from the beginning we wouldn't need this discussion at all...
Quoted 7 lines
Quoted 3 lines
There are several issues with transparent migration.
The first is that the SR-IOV device needs to be shut off for earlier in the migration process.
That's not a given fact.
It's due to the DMA and it should be solve anyway. Please read my first reply in this thread.

Quoted 7 lines

For now it is a fact. We would need to do a drastic rewrite of the DMA API in the guest/host/QEMU/IOMMU in order to avoid it for now. So as a first step I would say we should look at using this bonding type solution. Being able to defer the VF eviction could be a next step for all this as it would allow for much better performance, but we still have too many cases where the VF might not be there after a migration.

Why would we need such a drastic rewrite?
Why would a simple Read-DontModify-Write (to mark the page as dirty) by the VF driver not do the job?

Anyway, if you have a generic virtual parent netdev handling that can be handled orthogonally.

Quoted 6 lines

Quoted 6 lines
Quoted 5 lines
Next, the SR-IOV device in the migrated go guest environment maybe different. It might not exist at all, it might be at a different PCI address, or it could even be a different vendor/speed/model.
Keeping a virtual network device around allows persisting the connectivity, during the process.

Quoted 7 lines

Quoted 3 lines
Right, but that virtual device must not relate to any para-virt specific technology (not netvsc, nor virtio).
Again, it seems you did not read my first reply.
I would agree with the need to make this agnostic. Maybe we could look at the current netvsc solution and find a way to make it generic so it could be applied to any combination of paravirtual interface and PF.

Agree. That's should be the approach IMO.
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.

Michael S. Tsirkin

maintainer

2017-12-05 19:20 UTC #32 lore raw

Quoted 3 lines · achiad shochat

Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.

For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.

But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?

Signature · 1 line

MST

Michael S. Tsirkin

maintainer

2017-12-05 19:20 UTC #33 lore raw

Quoted 3 lines · achiad shochat

Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.

For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.

Signature · 1 line

MST

Jesse Brandeburg

author

2017-12-05 21:52 UTC #34 lore raw

Quoted 8 lines · On Tue, 5 Dec 2017 21:20:07 +0200 "Michael S. Tsirkin" <[email protected]> wrote:

Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.

Thanks, based on this discussion we're going to work on improving virtio-net first, but some of Achiad's points are good. I don't believe it should block the virtio work however.

In particular I'm really interested in figuring out how we can get to the point that virtio is able to make or implement some smart decisions about which NIC to pick for traffic delivery (it's own paravirt path or the passthorugh device path), if Achiad wants to develop the idea into some code, I'd be interested to review it.

Quoted 4 lines

But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?

I personally don't think that will be much of a problem, if a certain device has that issue, can't we just have the virtio-net device pick up the MAC address of the passthrough device? As long as they match things should work OK. It at least is an initial way to do the configuration that has at least some traction as workable, as proved by the Microsoft design.

FWIW, the Intel SR-IOV devices all accept a hypervisor/host provided MAC address.

Jesse Brandeburg

author

2017-12-05 21:52 UTC #35 lore raw

In reply to Jesse Brandeburg

Quoted 8 lines · On Tue, 5 Dec 2017 21:20:07 +0200 "Michael S. Tsirkin" <[email protected]> wrote:

Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.

Thanks, based on this discussion we're going to work on improving virtio-net first, but some of Achiad's points are good. I don't believe it should block the virtio work however.

Quoted 4 lines

But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?

FWIW, the Intel SR-IOV devices all accept a hypervisor/host provided MAC address.

Michael S. Tsirkin

maintainer

2017-12-05 22:05 UTC #36 lore raw

Quoted 22 lines · Jesse Brandeburg

Quoted 8 lines · On Tue, 5 Dec 2017 21:20:07 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.
Thanks, based on this discussion we're going to work on improving virtio-net first, but some of Achiad's points are good. I don't believe it should block the virtio work however.
In particular I'm really interested in figuring out how we can get to the point that virtio is able to make or implement some smart decisions about which NIC to pick for traffic delivery (it's own paravirt path or the passthorugh device path), if Achiad wants to develop the idea into some code, I'd be interested to review it.

Quoted 8 lines

Quoted 4 lines
But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?
I personally don't think that will be much of a problem, if a certain device has that issue, can't we just have the virtio-net device pick up the MAC address of the passthrough device?

Then what do you do after you have migrated to another box? The PT device there likely has a different MAC.

Quoted 4 lines

As long as they match
things should work OK. It at least is an initial way to do the configuration that has at least some traction as workable, as proved by the Microsoft design.

Yes - that design just implements what people have been doing for years using bond so of course it's workable.

Quoted 2 lines

FWIW, the Intel SR-IOV devices all accept a hypervisor/host provided MAC address.

For VFs you often can program the MAC through the PF, but you typically can't do this for PFs. Or as another example consider nested virt with a VF passed through. PF isn't there within L1 guest so can't be used to program the mac of the VF.

Still, we can always start small and require same mac, add other ways to address issues later as we come up with them.

Signature · 1 line

MST

Michael S. Tsirkin

maintainer

2017-12-05 22:05 UTC #37 lore raw

Quoted 22 lines · Jesse Brandeburg

Quoted 8 lines · On Tue, 5 Dec 2017 21:20:07 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.
Thanks, based on this discussion we're going to work on improving virtio-net first, but some of Achiad's points are good. I don't believe it should block the virtio work however.
In particular I'm really interested in figuring out how we can get to the point that virtio is able to make or implement some smart decisions about which NIC to pick for traffic delivery (it's own paravirt path or the passthorugh device path), if Achiad wants to develop the idea into some code, I'd be interested to review it.

Quoted 8 lines

Quoted 4 lines
But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?
I personally don't think that will be much of a problem, if a certain device has that issue, can't we just have the virtio-net device pick up the MAC address of the passthrough device?

Then what do you do after you have migrated to another box? The PT device there likely has a different MAC.

Quoted 4 lines

As long as they match
things should work OK. It at least is an initial way to do the configuration that has at least some traction as workable, as proved by the Microsoft design.

Yes - that design just implements what people have been doing for years using bond so of course it's workable.

Quoted 2 lines

FWIW, the Intel SR-IOV devices all accept a hypervisor/host provided MAC address.

Still, we can always start small and require same mac, add other ways to address issues later as we come up with them.

Signature · 1 line

MST

Jakub Kicinski

2017-12-05 22:29 UTC #43 lore raw

Quoted 21 lines · achiad shochat

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Quoted 8 lines
This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.

Quoted 1 line

I like this approach.

+1 on a separate driver, just enslaving devices to virtio may break existing setups. If people are bonding from user space today, if they update their kernel it may surprise them how things get auto-mangled.

Is what Alex is suggesting a separate PV device that says "I would like to be a bond of those two interfaces"? That would make the HV intent explicit and kernel decisions more understandable.

Jakub Kicinski

2017-12-05 22:29 UTC #44 lore raw

Quoted 21 lines · achiad shochat

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Quoted 8 lines
This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.

Quoted 1 line

I like this approach.

Is what Alex is suggesting a separate PV device that says "I would like to be a bond of those two interfaces"? That would make the HV intent explicit and kernel decisions more understandable.

Stephen Hemminger

2017-12-05 22:41 UTC #45 lore raw

Quoted 24 lines · On Tue, 5 Dec 2017 14:29:28 -0800 Jakub Kicinski <[email protected]> wrote:

Quoted 21 lines · achiad shochat
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Quoted 8 lines
This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.
Quoted 1 line
I like this approach.

Quoted 7 lines

+1 on a separate driver, just enslaving devices to virtio may break existing setups. If people are bonding from user space today, if they update their kernel it may surprise them how things get auto-mangled.
Is what Alex is suggesting a separate PV device that says "I would like to be a bond of those two interfaces"? That would make the HV intent explicit and kernel decisions more understandable.

So far, in my experience it still works.
As long as the kernel slaving happens first, it will work.
The attempt to bond an already slaved device will fail and no scripts seem to check the error return.

Stephen Hemminger

2017-12-05 22:41 UTC #46 lore raw

Quoted 24 lines · On Tue, 5 Dec 2017 14:29:28 -0800 Jakub Kicinski <[email protected]> wrote:

Quoted 21 lines · achiad shochat
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
I second Jacob - having a netdev of one device driver enslave a netdev of another device driver is an awkward a-symmetric model. Regardless of whether they share the same backend device. Only I am not sure the Linux Bond is the right choice.
e.g one may well want to use the virtio device also when the pass-through device is available, e.g for multicasts, east-west traffic, etc.
I'm not sure the Linux Bond fits that functionality.
And, as I hear in this thread, it is hard to make it work out of the box. So I think the right thing would be to write a new dedicated module for this purpose.
Quoted 8 lines
This part I can sort of agree with. What if we were to look at providing a way to somehow advertise that the two devices were meant to be boded for virtualization purposes? For now lets call it a "virt-bond". Basically we could look at providing a means for virtio and VF drivers to advertise that they want this sort of bond. Then it would just be a matter of providing some sort of side channel to indicate where you want things like multicast/broadcast/east-west traffic to go.
Quoted 1 line
I like this approach.

Quoted 7 lines

+1 on a separate driver, just enslaving devices to virtio may break existing setups. If people are bonding from user space today, if they update their kernel it may surprise them how things get auto-mangled.
Is what Alex is suggesting a separate PV device that says "I would like to be a bond of those two interfaces"? That would make the HV intent explicit and kernel decisions more understandable.

achiad shochat

2017-12-07 07:28 UTC #38 lore raw

Quoted 8 lines · Michael S. Tsirkin

Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.

It's not a lot of code either way.
So I fail to understand why not to do it right from the beginning. For the record...

Quoted 8 lines

But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?
--
MST

achiad shochat

2017-12-07 07:28 UTC #39 lore raw

Quoted 8 lines · Michael S. Tsirkin

Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.

It's not a lot of code either way.
So I fail to understand why not to do it right from the beginning. For the record...

Quoted 8 lines

But we do need to have a discussion about how devices are paired. I am not sure using just MAC works. E.g. some passthrough devices don't give host ability to set the MAC.
Are these worth worrying about?
--
MST

Alexander Duyck

2017-12-07 16:45 UTC #40 lore raw

Quoted 6 lines · achiad shochat

Quoted 5 lines · Michael S. Tsirkin
Quoted 3 lines · achiad shochat
Then we'll have a single solution for both netvsc and virtio (and any other PV device).
And we could handle the VF DMA dirt issue agnostically.

Quoted 7 lines

Quoted 3 lines
For the record, I won't block patches adding this kist to virtio on the basis that they must be generic. It's not a lot of code, implementation can come first, prettify later.
It's not a lot of code either way.
So I fail to understand why not to do it right from the beginning. For the record...

What isn't a lot of code? If you are talking about the DMA dirtying then I would have to disagree. The big problem with the DMA is that we have to mark a page as dirty and non-migratable as soon as it is mapped for Rx DMA. It isn't until the driver has either unmapped the page or the device has been disabled that we can then allow the page to be migrated for being dirty. That ends up being the way we have to support this if we don't have the bonding solution.

With the bonding solution we could look at doing a lightweight DMA dirtying which would just require flagging pages as dirty after an unmap or sync call is performed. However it requires that we shut down the driver/device before we can complete the migration which means we have to have the paravirtualized fail-over approach.

As far as indicating that the interfaces are meant to be enslaved I wonder if we couldn't look at tweaking the PCI layout of the guest and use that to indicate that a given set of interfaces are meant to be bonded. For example the VFs are all meant to work as a part of a multi-function device. What if we were to make virtio-net function 0 of a PCI/PCIe device, and then place any direct assigned VFs that are meant to be a part of the bond in functions 1-7 of the device? Then it isn't too far off from the model we have on the host where if the VF goes away we would expect to see the traffic on the PF that is usually occupying function 0 of a given device.

Alexander Duyck

2017-12-07 16:45 UTC #41 lore raw