Skip to contentSkip to conversation

Introduce device migration support commands

PATCH Superseded Parav Pandit 4 participants 341 messages Oct 8, 2023 – Nov 24, 2023 Page 1 of 6

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover Introduce device migration support commands

This series introduces administration commands for member device migration for PCI transport; when needed it can be extended for other transports too.

Use case requirements:
======================
1. A hypervisor system needs to provide a PCI VF as passthrough device to the guest virtual machine and also support live migration of this virtual machine.
2. A virtual machine may have one or more such passthrough virtio devices.
3. A virtual machine may have other PCI passthrough device which may also interact with virtio device.
4. A hypervisor runs a vendor agnostic driver with extension to support device migration.
5. A PCI VF passthrough device needs to support transparent device reset and PCI FLR must while the device migration is ongoing.
6. A owner driver do not involve in device operations mediation for the passthrough device at virtio interface level.
7. Mechanism is generic enough that applies to large family of virtio devices and it does not involve trapping any virtio device interfaces for the passthrough devices.

Overview:
=========
Above usecase requirements is solved by PCI PF group owner driver facilitating the member device migration functionality using administration commands.

There are three major functionalities.

1. Suspend and resume the device operation
2. Read and Write the device context containing all the information that can be transferred from source to destination to migrate to a member device
3. Track pages written by the device during device migration is ongoing

This comprehensive series introduces 4 infrastructure pieces covering PCI transport, peer to peer PCI devices, page tracking (aka dirty page tracking) and generic device context.

1. Device mode get,set (active, stop, freeze)
2. Device context read and write
3. Defines device context
4. Write reporting to track page addresses

This series enables virtio PCI SR-IOV member device to member device migration. It can also be used to/from migrate from PCI SR-IOV member device to software composed PCI device if/when needed which can parse and compose software based PCI virtio device.

Example flow:
=============
Source hypervisor:
1. Instructs device to start tracking pages it is writing
2. Periodically query the addresses of the written pages
3. Suspend the device operation
4. Read the device context and transfer to destination hypervisor

Destination hypervisor:
5. Write the device context received from source
6. Resume the device that has newly written device context

Patch summary:
==============
patch-1: Adds theory of operation for device migration commands
patch-2: Redefine reserved2 to command output field
patch-3: Defines short device context for split virtqueues
patch-4: Adds device migration commands
patch-5: Adds requirements for device migration commands
patch-6: Adds theory of operation for write reporting commands
patch-7: Adds write reporting commands
patch-8: Adds requirements for write reporting commands

It also takes inspiration from the similar idea presented at KVM Forum at [1].

Changelog:
==========
1. Enrich device context to cover device configuration layout, feature bits
2. Fixed alignment of device context fields
3. Added missing Sign-off for the joint work done with Satananda
4. Added link to the github issue

Please review.

[1] https://static.sched.com/hosted_files/kvmforum2022/3a/KVM22-Migratable-Vhost-vDPA.pdf

Parav Pandit (8):
admin: Add theory of operation for device migration
admin: Redefine reserved2 as command specific output
device-context: Define the device context fields for device migration
admin: Add device migration admin commands
admin: Add requirements of device migration commands
admin: Add theory of operation for write recording commands
admin: Add write recording commands
admin: Add requirements of write reporting commands

 admin-cmds-device-migration.tex | 574 ++++++++++++++++++++++++++++++++
 admin.tex                       |  38 ++-
 content.tex                     |   1 +
 device-context.tex              | 142 ++++++++
 4 files changed, 748 insertions(+), 7 deletions(-)
 create mode 100644 admin-cmds-device-migration.tex
 create mode 100644 device-context.tex
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

PATCH 1/8 admin: Add theory of operation for device migration

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

A passthrough PCI VF device is fully owned by the virtual machine device driver. This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset and rest of the virtio device functionality such as control vq, config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely used.

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/vfio.h?h=v6.1.47

---
 admin-cmds-device-migration.tex | 94 +++++++++++++++++++++++++++++++++
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)
 create mode 100644 admin-cmds-device-migration.tex
admin-cmds-device-migration.tex new file +94 -0
new file mode 100644
index 0000000..f839af4
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Device Migration}
+
+In some systems, there is a need to migrate a running virtual machine
+from one to another system. A running virtual machine has one or more
+passthrough virtio member devices attached to it. A passthrough device
+is entirely operated by the guest virtual machine. For example, with
+the SR-IOV group type, group member (VF) may undergo virtio device
+initialization and reset flow and may also undergo PCI function level
+reset(FLR) flow. Such flows must comply to the PCI standard and also
+virtio specification; at the same time such flows must not obstruct
+the device migration flow. In such a scenario, a group owner device
+can provide the administration command interface to facilitate the device
+migration related operations.
+
+When a virtual machine migrates from one hypervisor to another hypervisor,
+these hypervisors are named as source and destination hypervisor respectively.
+In such a scenario, a source hypervisor administers the
+member device to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member device to
+setup a device context and resumes the member device. The source hypervisor
+reads the member device context and the destination hypervisor writes the member
+device context. The method to transfer the member device context from the source
+to the destination hypervisor is outside the scope of this specification.
+
+The member device can be in any of the three migration modes. The owner driver
+sets the member device in one of the following modes during device migration flow.
+
+\begin{tabularx}{\textwidth}{ |l||l|X| }
+\hline
+Value & Name & Description \\
+\hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member device. \\
+\hline
+0x1   & Stop &
+ In this mode, the member device does not send any notifications,
+ and it does not access any driver memory.
+ The member device may receive driver notifications in this mode,
+ the member device context and device configuration space may change. \\
+\hline
+0x2   & Freeze &
+ In this mode, the member device does not accept any driver notifications,
+ it ignores any device configuration space writes,
+ the device do not have any changes in the device context. The
+ member device is not accessed in the system through the virtio interface. \\
+\hline
+\hline
+0x03-0xFF   & -    & reserved for future use \\
+\hline
+\end{tabularx}
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to \field{Stop}. Once the
+device is in the \field{Stop} mode, the device does not initiate any notifications
+or does not access any driver memory. Since the member driver may be still
+active which may send further driver notifications to the device, the device
+context may be updated. When the member driver has stopped accessing the
+device, the owner driver sets the device to \field{Freeze} mode indicating
+to the device that no more driver access occurs. In the \field{Freeze} mode,
+no more changes occur in the device context. At this point, the device ensures
+that there will not be any update to the device context.
+
+The member device has a device context which the owner driver can either
+read or write. The member device context consist of any device specific
+data which is needed by the device to resume its operation when the device mode
+is changed from \field{Stop} to \field{Active} or from \field{Freeze}
+to \field{Active}.
+
+Once the device context is read, it is cleared from the device. Typically, on
+the source hypervisor, the owner driver reads the device context once when
+the device is in \field{Active} or \field{Stop} mode and later once the member
+device is in \field{Freeze} mode.
+
+Typically, the device context is read and written one time on the source and
+the destination hypervisor respectively once the device is in \field{Freeze}
+mode. On the destination hypervisor, after writing the device context,
+when the device mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device operation.
+
+In an alternative flow, on the source hypervisor the owner driver may choose
+to read the device context first time while the device is in \field{Active} mode
+and second time once the device is in \field{Freeze} mode. Similarly, on the
+destination hypervisor writes the device context first time while the device
+is still running in \field{Active} mode on the source hypervisor and writes
+the device context second time while the device is in \field{Freeze} mode.
+This flow may result in very short setup time as the device context likely
+have minimal changes from the previously written device context. This flow may
+reduce the device migration time significantly and may have near constant
+device activation time regardless of number of virtqueues, resources and
+passthough devices in use by the migrating virtual machine.
+
+The owner driver can discard any partially read or written device context when
+any of the device migration flow should be aborted.
admin.tex +1 -0
index 0803c26..6eeef58 100644
@@ -297,6 +297,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 might differ between different group types.
 
 \input{admin-cmds-legacy-interface.tex}
+\input{admin-cmds-device-migration.tex}
 
 \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
 
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#2 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Adding LingShan.

Parav, if you want any specific people to comment, please do cc them.

Quoted 3 lines · Parav Pandit

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Quoted 2 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

Quoted 1 line

and rest of the virtio device functionality such as control vq,

What do you mean by "rest of"? Which part is not controlled and why?

Quoted 3 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely used.

Why is this mentioned here?

Quoted 3 lines

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

Well, if this is specific only to PCI SR-IOV, I'd move it to the PCI transport part. But I guess not.

Quoted 20 lines

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/include/uapi/linux/vfio.h?h=v6.1.47

---
 admin-cmds-device-migration.tex | 94 +++++++++++++++++++++++++++++++++
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)
 create mode 100644 admin-cmds-device-migration.tex
admin-cmds-device-migration.tex new file +0 -0
new file mode 100644
index 0000000..f839af4

admin-cmds-device-migration.tex line 9

Quoted 9 lines
admin-cmds-device-migration.tex new file +9 -0
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Device Migration}
+
+In some systems, there is a need to migrate a running virtual machine
+from one to another system. A running virtual machine has one or more
+passthrough virtio member devices attached to it. A passthrough device
+is entirely operated by the guest virtual machine. For example, with
+the SR-IOV group type, group member (VF) may undergo virtio device
+initialization and reset flow

What do you mean by "reset flow"? It looks not like a terminology defined in the PCI spec. And Google gives me nothing about this.

Quoted 2 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

Quoted 2 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

Quoted 14 lines

at the same time such flows must not obstruct

admin-cmds-device-migration.tex new file +13 -0
+the device migration flow. In such a scenario, a group owner device
+can provide the administration command interface to facilitate the device
+migration related operations.
+
+When a virtual machine migrates from one hypervisor to another hypervisor,
+these hypervisors are named as source and destination hypervisor respectively.
+In such a scenario, a source hypervisor administers the
+member device to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member device to
+setup a device context and resumes the member device. The source hypervisor
+reads the member device context and the destination hypervisor writes the member
+device context. The method to transfer the member device context from the source
+to the destination hypervisor is outside the scope of this specification.

admin-cmds-device-migration.tex line 34

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+
+The member device can be in any of the three migration modes. The owner driver
+sets the member device in one of the following modes during device migration flow.
+
+\begin{tabularx}{\textwidth}{ |l||l|X| }
+\hline
+Value & Name & Description \\
+\hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 4 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any notifications,
+ and it does not access any driver memory.

What's the meaning of "driver memory"?

And stop seems to be a source of inflight buffers.

Quoted 1 line

+ The member device may receive driver notifications in this mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Quoted 1 line

+ the member device context

I don't think we define "device context" anywhere.

Quoted 2 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

Quoted 2 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state machine. For example, what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the two state machines.

Quoted 1 line

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

Quoted 2 lines

+ the device do not have any changes in the device context. The
+ member device is not accessed in the system through the virtio interface. \\

But accessible via PCI interface?

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Quoted 6 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

admin-cmds-device-migration.tex new file +1 -0
+

admin-cmds-device-migration.tex line 62

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to \field{Stop}. Once the
+device is in the \field{Stop} mode, the device does not initiate any notifications
+or does not access any driver memory. Since the member driver may be still
+active which may send further driver notifications to the device, the device
+context may be updated. When the member driver has stopped accessing the
+device, the owner driver sets the device to \field{Freeze} mode indicating
+to the device that no more driver access occurs. In the \field{Freeze} mode,
+no more changes occur in the device context. At this point, the device ensures
+that there will not be any update to the device context.

What is missed here are:

1) it is a virtio specific states or not
2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

admin-cmds-device-migration.tex line 66

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+
+The member device has a device context which the owner driver can either
+read or write. The member device context consist of any device specific
+data which is needed by the device to resume its operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for sure. I'd split it into

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+is changed from \field{Stop} to \field{Active} or from \field{Freeze}
+to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

1) re-try the migration
2) recover from migration failure

admin-cmds-device-migration.tex line 73

Quoted 4 lines

Typically, on

admin-cmds-device-migration.tex new file +3 -0
+the source hypervisor, the owner driver reads the device context once when
+the device is in \field{Active} or \field{Stop} mode and later once the member
+device is in \field{Freeze} mode.

Why need the read while device context could be changed? Or is the dirty page part of the device context?

admin-cmds-device-migration.tex line 79

Quoted 6 lines
admin-cmds-device-migration.tex new file +6 -0
+
+Typically, the device context is read and written one time on the source and
+the destination hypervisor respectively once the device is in \field{Freeze}
+mode. On the destination hypervisor, after writing the device context,
+when the device mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+
+In an alternative flow, on the source hypervisor the owner driver may choose
+to read the device context first time while the device is in \field{Active} mode
+and second time once the device is in \field{Freeze} mode.

Who is going to synchronize the device context with possible configuration from the driver?

admin-cmds-device-migration.tex

Quoted 6 lines

Similarly, on the

admin-cmds-device-migration.tex new file +5 -0
+destination hypervisor writes the device context first time while the device
+is still running in \field{Active} mode on the source hypervisor and writes
+the device context second time while the device is in \field{Freeze} mode.
+This flow may result in very short setup time as the device context likely
+have minimal changes from the previously written device context.

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

admin-cmds-device-migration.tex line 91

Quoted 4 lines

This flow may

admin-cmds-device-migration.tex new file +3 -0
+reduce the device migration time significantly and may have near constant
+device activation time regardless of number of virtqueues, resources and
+passthough devices in use by the migrating virtual machine.

Thanks

Quoted 26 lines
admin-cmds-device-migration.tex new file +3 -0
+
+The owner driver can discard any partially read or written device context when
+any of the device migration flow should be aborted.
admin.tex +1 -1
index 0803c26..6eeef58 100644
@@ -297,6 +297,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 might differ between different group types.
 
 \input{admin-cmds-legacy-interface.tex}
+\input{admin-cmds-device-migration.tex}
 
 \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
 
--

2.34.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 5 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Other than PCI config registers and due to some legacy, msix.
The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific and virtio specific interfaces for modern and future devices.

Quoted 4 lines
Quoted 2 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the guest driver hence, the device migration commands do not interact/depend with FLR flow.

Quoted 5 lines
Quoted 2 lines

and rest of the virtio device functionality such as control vq,

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 6 lines
Quoted 3 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely used.

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on understanding the use case. And you seem opposite, but ok.

As stated, it for the reader to understand the use case and see how proposed commands addresses the use case.

Quoted 6 lines
Quoted 3 lines

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

Well, if this is specific only to PCI SR-IOV, I'd move it to the PCI transport part. But I guess not.

We took the decision to not do so, for other group commands as well. After Michael's suggestion we moved it to group commands. So I will not debate this further.

Quoted 29 lines
Quoted 5 lines

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

Quoted 8 lines

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/
include/uapi/linux/vfio.h?h=v6.1.47

---
 admin-cmds-device-migration.tex | 94

+++++++++++++++++++++++++++++++++

Quoted 15 lines
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)
 create mode 100644 admin-cmds-device-migration.tex
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex new file mode 100644 index 0000000..f839af4

admin-cmds-device-migration.tex new file +5 -0
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a
+Virtio Device / Device groups / Group administration commands /
+Device Migration}
+
+In some systems, there is a need to migrate a running virtual machine
Quoted 9 lines

admin-cmds-device-migration.tex line 10

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+from one to another system. A running virtual machine has one or more
+passthrough virtio member devices attached to it. A passthrough
+device is entirely operated by the guest virtual machine. For
+example, with the SR-IOV group type, group member (VF) may undergo
+virtio device initialization and reset flow

What do you mean by "reset flow"? It looks not like a terminology defined in the PCI spec. And Google gives me nothing about this.

"reset flow" = virtio specification section 2.4 Device Reset flow.

Quoted 5 lines
Quoted 2 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence hypervisor cannot mediate any registers of the VF.

Quoted 6 lines
Quoted 2 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

It is not written in this series, because we believe it must not be synchronized as it is fully controlled by the guest.

Quoted 21 lines

admin-cmds-device-migration.tex

Quoted 8 lines

at the same time such flows must not obstruct

admin-cmds-device-migration.tex new file +6 -0
+the device migration flow. In such a scenario, a group owner device
+can provide the administration command interface to facilitate the
+device migration related operations.
+
+When a virtual machine migrates from one hypervisor to another
+hypervisor, these hypervisors are named as source and destination

hypervisor respectively.

admin-cmds-device-migration.tex

Quoted 7 lines
admin-cmds-device-migration.tex new file +7 -0
+In such a scenario, a source hypervisor administers the member device
+to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member device
+to setup a device context and resumes the member device. The source
+hypervisor reads the member device context and the destination
+hypervisor writes the member device context. The method to transfer
+the member device context from the source to the destination hypervisor is

outside the scope of this specification.

admin-cmds-device-migration.tex

Quoted 3 lines
admin-cmds-device-migration.tex new file +3 -0
+
+The member device can be in any of the three migration modes. The
+owner driver sets the member device in one of the following modes during

device migration flow.

Quoted 8 lines

admin-cmds-device-migration.tex line 34

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name &
+Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member device. \\

I don't think we ever define "instantiation" anywhere.

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec here.

Quoted 7 lines
Quoted 4 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any notifications, +and it does not access any driver memory.

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory allocated by the guest driver?

Quoted 2 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size returned. Stop is not the source of inflight descriptors.

There are inflight descriptors with the device that are not yet returned to the driver, and device wont return them as zero size wrong completions.

Quoted 5 lines
Quoted 1 line

+ The member device may receive driver notifications in this mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may receive as incoming PCIe TLP.

In "stop" mode, the device wont process descriptors.

Quoted 4 lines
Quoted 1 line

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Quoted 5 lines
Quoted 2 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped atomically.
Hence, one device which is in freeze mode, may still receive driver notifications from other pci device, or it may experience a read from the shared memory and get garbage data. And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop themselves, and later when freezed, to not change anything internally.

Quoted 10 lines
Quoted 3 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure.
It is passthrough device, hence hypervisor layer do not get to see sub-state.

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

Quoted 2 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state machine.

First, above is not a state machine.
Second, it is not virtio specific. It is present in leading OS that has fundamental requirement to support P2P devices. Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked again so that this series is complete.

Quoted 5 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the two state machines.

As listed in the definition of the stop mode, the device do not act on the incoming writes, it only keep tracks of its internal device context change as part of this. We would enrich the device context for this, but no need to connects the admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 4 lines
Quoted 1 line

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

As listed, device do not have any changes.
So device configuration change cannot occur.

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level registers regardless of the member device mode.

Quoted 6 lines
Quoted 3 lines

+ the device do not have any changes in the device context. The member
+ device is not accessed in the system through the virtio interface.
+ \\

But accessible via PCI interface?

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

Quoted 18 lines
Quoted 6 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

admin-cmds-device-migration.tex new file +1 -0
+

admin-cmds-device-migration.tex

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+When the owner driver wants to stop the operation of the device, the
+owner driver sets the device mode to \field{Stop}. Once the device is
+in the \field{Stop} mode, the device does not initiate any
+notifications or does not access any driver memory. Since the member
+driver may be still active which may send further driver
+notifications to the device, the device context may be updated. When
+the member driver has stopped accessing the device, the owner driver
+sets the device to \field{Freeze} mode indicating to the device that
+no more driver access occurs. In the \field{Freeze} mode, no more
+changes occur in the device context. At this point, the device ensures that

there will not be any update to the device context.

Quoted 3 lines

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where destination hypervisor knows for sure that there is no other entity accessing the device. And it needs to setup the device context, it received from the source side.
So setting freeze mode can be done directly.

Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any device
+specific data which is needed by the device to resume its operation
+when the device mode

This is too vague. There're states that are not suitable for cmd/queue for sure. I'd split it into

Quoted 3 lines

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

This is theory of operation section. So it capturing such details.
Actual device context definition is outside of theory, and precise states of virtqueue, device specific, etc are in it.

Quoted 10 lines

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+is changed from \field{Stop} to \field{Active} or from \field{Freeze}
+to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?
And which software stack may find this useful?
Is there any existing software that can utilize it?
Why that device context present with the software vanished, in your assumption, if it is?

Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 4 lines

Typically, on

admin-cmds-device-migration.tex new file +3 -0
+the source hypervisor, the owner driver reads the device context once
+when the device is in \field{Active} or \field{Stop} mode and later
+once the member device is in \field{Freeze} mode.

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Quoted 11 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +6 -0
+
+Typically, the device context is read and written one time on the
+source and the destination hypervisor respectively once the device is
+in \field{Freeze} mode. On the destination hypervisor, after writing
+the device context, when the device mode set to \field{Active}, the
+device uses the most recently set device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Can you please what which existing interfaces do you mean here?

Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+
+In an alternative flow, on the source hypervisor the owner driver may
+choose to read the device context first time while the device is in
+\field{Active} mode and second time once the device is in \field{Freeze}

mode.

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that,
When configuration change is done by the guest driver, how does device context change?

If so, device context reading will reflect the new configuration.

Quoted 11 lines

admin-cmds-device-migration.tex

Quoted 4 lines

Similarly, on the

admin-cmds-device-migration.tex new file +3 -0
+destination hypervisor writes the device context first time while the
+device is still running in \field{Active} mode on the source
+hypervisor and writes the device context second time while the device is in

\field{Freeze} mode.

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex new file +2 -0
+This flow may result in very short setup time as the device context
+likely have minimal changes from the previously written device context.

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise. In current proposal, there isn’t a need for hypervisor to do so at all.

The destination side device gets to see the new device context and apply the delta.

#4 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 30 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 4 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Quoted 6 lines
Quoted 3 lines

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Other than PCI config registers and due to some legacy, msix.
The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

Quoted 13 lines
Quoted 6 lines
Quoted 3 lines

and rest of the virtio device functionality such as control vq,

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 2 lines
Quoted 2 lines

config space access, data path descriptors handling.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Additionally, VM live migration using a precopy method is also widely used.

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on understanding the use case. And you seem opposite, but ok.

As stated, it for the reader to understand the use case and see how proposed commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Quoted 49 lines
Quoted 7 lines
Quoted 4 lines

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

Well, if this is specific only to PCI SR-IOV, I'd move it to the PCI transport part. But I guess not.

We took the decision to not do so, for other group commands as well. After Michael's suggestion we moved it to group commands. So I will not debate this further.

Quoted 29 lines
Quoted 5 lines

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

Quoted 8 lines

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/
include/uapi/linux/vfio.h?h=v6.1.47

---
 admin-cmds-device-migration.tex | 94

+++++++++++++++++++++++++++++++++

Quoted 15 lines
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)
 create mode 100644 admin-cmds-device-migration.tex
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex new file mode 100644 index 0000000..f839af4

admin-cmds-device-migration.tex new file +5 -0
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a
+Virtio Device / Device groups / Group administration commands /
+Device Migration}
+
+In some systems, there is a need to migrate a running virtual machine
Quoted 9 lines

admin-cmds-device-migration.tex line 10

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+from one to another system. A running virtual machine has one or more
+passthrough virtio member devices attached to it. A passthrough
+device is entirely operated by the guest virtual machine. For
+example, with the SR-IOV group type, group member (VF) may undergo
+virtio device initialization and reset flow

What do you mean by "reset flow"? It looks not like a terminology defined in the PCI spec. And Google gives me nothing about this.

Quoted 1 line

"reset flow" = virtio specification section 2.4 Device Reset flow.

My git repo show it's still called "device reset" and I see you use "FLR flow" which is also not very clear to me.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

See my above question.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Quoted 6 lines

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Quoted 32 lines

It is not written in this series, because we believe it must not be synchronized as it is fully controlled by the guest.

Quoted 21 lines

admin-cmds-device-migration.tex

Quoted 8 lines

at the same time such flows must not obstruct

admin-cmds-device-migration.tex new file +6 -0
+the device migration flow. In such a scenario, a group owner device
+can provide the administration command interface to facilitate the
+device migration related operations.
+
+When a virtual machine migrates from one hypervisor to another
+hypervisor, these hypervisors are named as source and destination

hypervisor respectively.

admin-cmds-device-migration.tex

Quoted 7 lines
admin-cmds-device-migration.tex new file +7 -0
+In such a scenario, a source hypervisor administers the member device
+to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member device
+to setup a device context and resumes the member device. The source
+hypervisor reads the member device context and the destination
+hypervisor writes the member device context. The method to transfer
+the member device context from the source to the destination hypervisor is

outside the scope of this specification.

admin-cmds-device-migration.tex

Quoted 3 lines
admin-cmds-device-migration.tex new file +3 -0
+
+The member device can be in any of the three migration modes. The
+owner driver sets the member device in one of the following modes during

device migration flow.

Quoted 8 lines

admin-cmds-device-migration.tex line 34

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name &
+Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 2 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec here.

Ok, maybe something like "transport specific instantiation"

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any notifications, +and it does not access any driver memory.

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory allocated by the guest driver?

Virtqueue?

Quoted 7 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size returned. Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 9 lines

There are inflight descriptors with the device that are not yet returned to the driver, and device wont return them as zero size wrong completions.

Quoted 5 lines
Quoted 1 line

+ The member device may receive driver notifications in this mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the queue?

Quoted 2 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive notifications? Or does it really matter if the device can receive or not here?

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

"""
+The member device has a device context which the owner driver can +either read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode
"""

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport (PCI) context at all? If yes, how can it work?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped atomically.
Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage data.

Could you give me an example for this?

Quoted 10 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop themselves, and later when freezed, to not change anything internally.

Quoted 7 lines
Quoted 3 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

Quoted 8 lines
Quoted 3 lines

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure.
It is passthrough device, hence hypervisor layer do not get to see sub-state.

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above? If yes, assuming we are freezing between 2 and 3, what would a device context read gives us for driver_features?

Quoted 4 lines
Quoted 3 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state machine.

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in patch 3.

Quoted 1 line

It is present in leading OS that has fundamental requirement to support P2P devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

Quoted 10 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked again so that this series is complete.

Quoted 5 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the two state machines.

As listed in the definition of the stop mode, the device do not act on the incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

Quoted 8 lines

We would enrich the device context for this, but no need to connects the admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 4 lines
Quoted 1 line

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

Quoted 11 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level registers regardless of the member device mode.

Quoted 4 lines
Quoted 3 lines

+ the device do not have any changes in the device context. The member
+ device is not accessed in the system through the virtio interface.
+ \\

Quoted 8 lines
Quoted 2 lines

But accessible via PCI interface?

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Quoted 42 lines
Quoted 19 lines
Quoted 7 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

admin-cmds-device-migration.tex new file +1 -0
+

admin-cmds-device-migration.tex

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+When the owner driver wants to stop the operation of the device, the
+owner driver sets the device mode to \field{Stop}. Once the device is
+in the \field{Stop} mode, the device does not initiate any
+notifications or does not access any driver memory. Since the member
+driver may be still active which may send further driver
+notifications to the device, the device context may be updated. When
+the member driver has stopped accessing the device, the owner driver
+sets the device to \field{Freeze} mode indicating to the device that
+no more driver access occurs. In the \field{Freeze} mode, no more
+changes occur in the device context. At this point, the device ensures that

there will not be any update to the device context.

Quoted 3 lines

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where destination hypervisor knows for sure that there is no other entity accessing the device. And it needs to setup the device context, it received from the source side.
So setting freeze mode can be done directly.

Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any device
+specific data which is needed by the device to resume its operation
+when the device mode

This is too vague. There're states that are not suitable for cmd/queue for sure. I'd split it into

Quoted 5 lines
Quoted 3 lines

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

This is theory of operation section. So it capturing such details.
Actual device context definition is outside of theory, and precise states of virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +4 -0
+is changed from \field{Stop} to \field{Active} or from \field{Freeze}
+to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

What's the consideration for such clearing?

Quoted 2 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Quoted 10 lines

Why that device context present with the software vanished, in your assumption, if it is?

Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 4 lines

Typically, on

admin-cmds-device-migration.tex new file +3 -0
+the source hypervisor, the owner driver reads the device context once
+when the device is in \field{Active} or \field{Stop} mode and later
+once the member device is in \field{Freeze} mode.

Why need the read while device context could be changed? Or is the dirty page part of the device context?

Quoted 2 lines

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor? If not, let's don't describe such in-mature optimization in the spec.

Quoted 9 lines
Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 7 lines
admin-cmds-device-migration.tex new file +6 -0
+
+Typically, the device context is read and written one time on the
+source and the destination hypervisor respectively once the device is
+in \field{Freeze} mode. On the destination hypervisor, after writing
+the device context, when the device mode set to \field{Active}, the
+device uses the most recently set device context and resumes the device

operation.

Quoted 4 lines
Quoted 3 lines

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

Quoted 7 lines
Quoted 7 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +4 -0
+
+In an alternative flow, on the source hypervisor the owner driver may
+choose to read the device context first time while the device is in
+\field{Active} mode and second time once the device is in \field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that,
When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi)
{
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi);
}

Is it ok to be freezed in the middle of two vp_iowrite()?

Quoted 6 lines
Quoted 6 lines

admin-cmds-device-migration.tex

Quoted 5 lines

Similarly, on the

admin-cmds-device-migration.tex new file +3 -0
+destination hypervisor writes the device context first time while the
+device is still running in \field{Active} mode on the source
+hypervisor and writes the device context second time while the device is in

\field{Freeze} mode.

Quoted 7 lines
Quoted 6 lines

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex new file +2 -0
+This flow may result in very short setup time as the device context
+likely have minimal changes from the previously written device context.

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

Thanks

Quoted 3 lines

In current proposal, there isn’t a need for hypervisor to do so at all.

The destination side device gets to see the new device context and apply the delta.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 38 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 10, 2023 11:21 AM

Quoted 33 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Quoted 8 lines
Quoted 3 lines

Other than PCI config registers and due to some legacy, msix. The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific

and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Virtio device specific means: cvq of crypto device, cvq of net device, flow filter vqs of net device etc. Virtio specific interface: virtio driver notifications, virtio virtqueue and configuration mediation etc.

Quoted 11 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the

guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

Quoted 8 lines

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

In most practical cases #2 followed by #1 should not happen as on the source side the expected is mode change to stop from active. But ok, since we active to freeze mode change is allowed, lets discuss above.

A device is the single synchronization point for any device reset, FLR or admin command operation. So, the migration driver do not need to wait for FLR to complete.
When admin cmd freeze the VF it can expect FLR_completed VF.
Secondly since the FLR is local to the source, intermediate sub state does not migrate.

But I agree, it is worth to have the text capturing this.

Quoted 20 lines
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

and rest of the virtio device functionality such as control vq,

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 3 lines
Quoted 3 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely

used.

Quoted 4 lines
Quoted 3 lines

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on

understanding the use case.

Quoted 9 lines
Quoted 3 lines

And you seem opposite, but ok.

As stated, it for the reader to understand the use case and see how proposed

commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Ok. The use case and requirement of member device passthrough is clear to most reviewers now. So I will remove it from commit log.

Quoted 51 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

Well, if this is specific only to PCI SR-IOV, I'd move it to the PCI transport

part.

Quoted 44 lines
Quoted 1 line

But I guess not.

We took the decision to not do so, for other group commands as well. After Michael's suggestion we moved it to group commands. So I will not debate this further.

Quoted 39 lines
Quoted 10 lines

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/t
ree/
include/uapi/linux/vfio.h?h=v6.1.47

Quoted 4 lines
---
 admin-cmds-device-migration.tex | 94

+++++++++++++++++++++++++++++++++

Quoted 10 lines
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)  create mode 100644

admin-cmds-device-migration.tex

(quoted diff) +0 -0

b/admin-cmds-device-migration.tex new file mode 100644 index 0000000..f839af4

admin-cmds-device-migration.tex line 10

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a
+Virtio Device / Device groups / Group administration commands /
+Device Migration}
+
+In some systems, there is a need to migrate a running virtual
+machine from one to another system. A running virtual machine has
+one or more passthrough virtio member devices attached to it. A
+passthrough device is entirely operated by the guest virtual
+machine. For example, with the SR-IOV group type, group member
+(VF) may undergo virtio device initialization and reset flow

What do you mean by "reset flow"? It looks not like a terminology defined in the PCI spec. And Google gives me nothing about this.

Quoted 5 lines
Quoted 1 line

"reset flow" = virtio specification section 2.4 Device Reset flow.

My git repo show it's still called "device reset" and I see you use "FLR flow" which is also not very clear to me.

Ok. I assume "reset flow" is clear to you now that it points to section 2.4.
This section is not normative section, so using an extra word like "flow" does not confuse anyone. I will link to the section anyway.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

Quoted 8 lines
Quoted 1 line

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device because the member device can undergo device reset and FLR flows owned by the guest. (and hypervisor is not involved in these two flows, hence the admin command interface is designed such that it can fullfil above requirements).

Theory of operation brings out this clarity. Please notice that it is in introductory section with an example. Not normative line.

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Quoted 36 lines
Quoted 2 lines

It is not written in this series, because we believe it must not be synchronized

as it is fully controlled by the guest.

Quoted 33 lines
Quoted 33 lines

admin-cmds-device-migration.tex

Quoted 9 lines

at the same time such flows must not obstruct

admin-cmds-device-migration.tex new file +6 -0
+the device migration flow. In such a scenario, a group owner
+device can provide the administration command interface to
+facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to another
+hypervisor, these hypervisors are named as source and destination

hypervisor respectively.

admin-cmds-device-migration.tex

Quoted 8 lines
admin-cmds-device-migration.tex new file +8 -0
+In such a scenario, a source hypervisor administers the member
+device to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member
+device to setup a device context and resumes the member device.
+The source hypervisor reads the member device context and the
+destination hypervisor writes the member device context. The
+method to transfer the member device context from the source to
+the destination hypervisor is

outside the scope of this specification.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+
+The member device can be in any of the three migration modes. The
+owner driver sets the member device in one of the following modes
+during

device migration flow.

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +6 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name &
+Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member
+device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 6 lines
Quoted 2 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec

here.

Ok, maybe something like "transport specific instantiation"

Ok. that’s a good text. I will change to it.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any notifications, +and it does not access any driver memory.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory

allocated by the guest driver?

Virtqueue?

Virtqueue and any memory referred by the virtqueue.

This is good text, I will change to it.

Quoted 19 lines
Quoted 6 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size

returned.

Quoted 1 line

Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 2 lines

There are inflight descriptors with the device that are not yet returned to the

driver, and device wont return them as zero size wrong completions.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

+ The member device may receive driver notifications in this mode,

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may

receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the queue?

Device cannot process the queue because it does not initiate any read/write towards the virtqueue.

Quoted 4 lines
Quoted 2 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive notifications?

Because notification may still arrive and if the device may update any counters as part of it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device context as part of notifications or access to it.

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Quoted 8 lines

Like this?

"""
+The member device has a device context which the owner driver can +either
read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the theory of operation section so reader can read more detail about it.

Quoted 4 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport (PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout, only the virtio device context is transferred. Which part cannot work?

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped atomically. Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Quoted 6 lines

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage data.

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

Quoted 23 lines
Quoted 2 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

Quoted 8 lines
Quoted 3 lines

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure.
It is passthrough device, hence hypervisor layer do not get to see sub-state.

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

Quoted 7 lines

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the guest will be done. What can happen between #2 and #3, is device mode may change to stop.
And in stop mode, device context would capture #5 or #4, depending where is device at that point.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

Quoted 5 lines
Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in patch 3.

Sure, device context is virtio specific. :)
Device context will reflect if things changed in the stop mode.

Quoted 6 lines
Quoted 1 line

It is present in leading OS that has fundamental requirement to support P2P

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to handle the interaction.

Quoted 9 lines
Quoted 3 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 5 lines
Quoted 5 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

Quoted 9 lines

two state machines.

Quoted 2 lines
Quoted 1 line

As listed in the definition of the stop mode, the device do not act on the

incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net other requests and not do device configuration.

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

Quoted 3 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Quoted 6 lines
Quoted 1 line

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

How about read and the device configuration changes?

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too.
The point is, when in freeze mode, the member device is frozen, hence, device won't initiate those changes.

Quoted 15 lines
Quoted 6 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

+ the device do not have any changes in the device context. The
+ member device is not accessed in the system through the virtio

interface.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Quoted 8 lines
Quoted 6 lines

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

Quoted 47 lines
Quoted 24 lines
Quoted 21 lines
Quoted 9 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

admin-cmds-device-migration.tex new file +2 -0
+
+When the owner driver wants to stop the operation of the device,

admin-cmds-device-migration.tex

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+the owner driver sets the device mode to \field{Stop}. Once the
+device is in the \field{Stop} mode, the device does not initiate
+any notifications or does not access any driver memory. Since the
+member driver may be still active which may send further driver
+notifications to the device, the device context may be updated.
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode indicating to
+the device that no more driver access occurs. In the
+\field{Freeze} mode, no more changes occur in the device context.
+At this point, the device ensures that

there will not be any update to the device context.

Quoted 3 lines

What is missed here are:

1) it is a virtio specific states or not

Quoted 8 lines

It is not.

Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 10 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 7 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any
+device specific data which is needed by the device to resume its
+operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 2 lines
Quoted 2 lines

I'd split it into

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of operation. So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare case scenario.

Quoted 2 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?
2. device context returns incremental value from the previous read. So, it needs to clear it.

Quoted 5 lines
Quoted 2 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

Quoted 11 lines
Quoted 1 line

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 9 lines
Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 5 lines

Typically, on

admin-cmds-device-migration.tex new file +3 -0
+the source hypervisor, the owner driver reads the device context
+once when the device is in \field{Active} or \field{Stop} mode
+and later once the member device is in \field{Freeze} mode.

Why need the read while device context could be changed? Or is the dirty page part of the device context?

Quoted 6 lines
Quoted 2 lines

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

admin-cmds-device-migration.tex

Quoted 9 lines
admin-cmds-device-migration.tex new file +7 -0
+
+Typically, the device context is read and written one time on the
+source and the destination hypervisor respectively once the
+device is in \field{Freeze} mode. On the destination hypervisor,
+after writing the device context, when the device mode set to
+\field{Active}, the device uses the most recently set device
+context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 22 lines
Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +5 -0
+
+In an alternative flow, on the source hypervisor the owner driver
+may choose to read the device context first time while the device
+is in \field{Active} mode and second time once the device is in
+\field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

Quoted 9 lines

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi);
}

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG section captures the partial value.

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 6 lines

Similarly, on the

admin-cmds-device-migration.tex new file +4 -0
+destination hypervisor writes the device context first time while
+the device is still running in \field{Active} mode on the source
+hypervisor and writes the device context second time while the
+device is in

\field{Freeze} mode.

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex new file +2 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in memory"

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has been negotiated".

Is this the only optimization in virtio? No, but we still mention the rationale of why it exists. As long as the rationale do not confuse the reader, and adds the value explaining how things work, it is fine to add. Which is what above few lines did.
So let's keep it.

The easiest is to cut out the whole theory of operation and just write commands like how RSS command did, without even writing a single line about RSS. I think we can do better explanation than that for new things we add.

#6 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 182 lines · Parav Pandit
Quoted 38 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 10, 2023 11:21 AM

Quoted 33 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Quoted 8 lines
Quoted 3 lines

Other than PCI config registers and due to some legacy, msix. The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific

and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Virtio device specific means: cvq of crypto device, cvq of net device, flow filter vqs of net device etc. Virtio specific interface: virtio driver notifications, virtio virtqueue and configuration mediation etc.

Quoted 11 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the

guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

Quoted 8 lines

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

In most practical cases #2 followed by #1 should not happen as on the source side the expected is mode change to stop from active. But ok, since we active to freeze mode change is allowed, lets discuss above.

A device is the single synchronization point for any device reset, FLR or admin command operation. So, the migration driver do not need to wait for FLR to complete.
When admin cmd freeze the VF it can expect FLR_completed VF.
Secondly since the FLR is local to the source, intermediate sub state does not migrate.

But I agree, it is worth to have the text capturing this.

Quoted 20 lines
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

and rest of the virtio device functionality such as control vq,

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 3 lines
Quoted 3 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely

used.

Quoted 4 lines
Quoted 3 lines

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on

understanding the use case.

Quoted 9 lines
Quoted 3 lines

And you seem opposite, but ok.

As stated, it for the reader to understand the use case and see how proposed

commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Ok. The use case and requirement of member device passthrough is clear to most reviewers now. So I will remove it from commit log.

Quoted 51 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

Well, if this is specific only to PCI SR-IOV, I'd move it to the PCI transport

part.

Quoted 44 lines
Quoted 1 line

But I guess not.

We took the decision to not do so, for other group commands as well. After Michael's suggestion we moved it to group commands. So I will not debate this further.

Quoted 39 lines
Quoted 10 lines

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/t
ree/
include/uapi/linux/vfio.h?h=v6.1.47

Quoted 4 lines
---
 admin-cmds-device-migration.tex | 94

+++++++++++++++++++++++++++++++++

Quoted 10 lines
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)  create mode 100644

admin-cmds-device-migration.tex

(quoted diff) +0 -0

b/admin-cmds-device-migration.tex new file mode 100644 index 0000000..f839af4

admin-cmds-device-migration.tex line 10

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a
+Virtio Device / Device groups / Group administration commands /
+Device Migration}
+
+In some systems, there is a need to migrate a running virtual
+machine from one to another system. A running virtual machine has
+one or more passthrough virtio member devices attached to it. A
+passthrough device is entirely operated by the guest virtual
+machine. For example, with the SR-IOV group type, group member
+(VF) may undergo virtio device initialization and reset flow

What do you mean by "reset flow"? It looks not like a terminology defined in the PCI spec. And Google gives me nothing about this.

Quoted 5 lines
Quoted 1 line

"reset flow" = virtio specification section 2.4 Device Reset flow.

My git repo show it's still called "device reset" and I see you use "FLR flow" which is also not very clear to me.

Ok. I assume "reset flow" is clear to you now that it points to section 2.4.
This section is not normative section, so using an extra word like "flow" does not confuse anyone. I will link to the section anyway.

Quoted 12 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

Quoted 7 lines
Quoted 2 lines

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device because the member device can undergo device reset and FLR flows owned by the guest. (and hypervisor is not involved in these two flows, hence the admin command interface is designed such that it can fullfil above requirements).

Theory of operation brings out this clarity. Please notice that it is in introductory section with an example. Not normative line.

All this does beg the question of how is device undergoing flr though and that has to be in a normative statement.

Quoted 128 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Quoted 36 lines
Quoted 2 lines

It is not written in this series, because we believe it must not be synchronized

as it is fully controlled by the guest.

Quoted 33 lines
Quoted 33 lines

admin-cmds-device-migration.tex

Quoted 9 lines

at the same time such flows must not obstruct

admin-cmds-device-migration.tex new file +6 -0
+the device migration flow. In such a scenario, a group owner
+device can provide the administration command interface to
+facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to another
+hypervisor, these hypervisors are named as source and destination

hypervisor respectively.

admin-cmds-device-migration.tex

Quoted 8 lines
admin-cmds-device-migration.tex new file +8 -0
+In such a scenario, a source hypervisor administers the member
+device to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member
+device to setup a device context and resumes the member device.
+The source hypervisor reads the member device context and the
+destination hypervisor writes the member device context. The
+method to transfer the member device context from the source to
+the destination hypervisor is

outside the scope of this specification.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+
+The member device can be in any of the three migration modes. The
+owner driver sets the member device in one of the following modes
+during

device migration flow.

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +6 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name &
+Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member
+device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 6 lines
Quoted 2 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec

here.

Ok, maybe something like "transport specific instantiation"

Ok. that’s a good text. I will change to it.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any notifications, +and it does not access any driver memory.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory

allocated by the guest driver?

Virtqueue?

Virtqueue and any memory referred by the virtqueue.

This is good text, I will change to it.

Quoted 19 lines
Quoted 6 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size

returned.

Quoted 1 line

Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 2 lines

There are inflight descriptors with the device that are not yet returned to the

driver, and device wont return them as zero size wrong completions.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

+ The member device may receive driver notifications in this mode,

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may

receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the queue?

Device cannot process the queue because it does not initiate any read/write towards the virtqueue.

Quoted 4 lines
Quoted 2 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive notifications?

Because notification may still arrive and if the device may update any counters as part of it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device context as part of notifications or access to it.

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

Quoted 8 lines
Quoted 6 lines

"""
+The member device has a device context which the owner driver can +either
read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the theory of operation section so reader can read more detail about it.

mention this in the commit log pls

Quoted 6 lines
Quoted 4 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport (PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout, only the virtio device context is transferred. Which part cannot work?

wait don't we need to transfer pci state too? how is that migrated?

Quoted 95 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped atomically. Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Quoted 6 lines

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage data.

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

Quoted 23 lines
Quoted 2 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

Quoted 8 lines
Quoted 3 lines

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure.
It is passthrough device, hence hypervisor layer do not get to see sub-state.

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

Quoted 7 lines

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the guest will be done. What can happen between #2 and #3, is device mode may change to stop.
And in stop mode, device context would capture #5 or #4, depending where is device at that point.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

Quoted 5 lines
Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in patch 3.

Sure, device context is virtio specific. :)
Device context will reflect if things changed in the stop mode.

Quoted 6 lines
Quoted 1 line

It is present in leading OS that has fundamental requirement to support P2P

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to handle the interaction.

Quoted 15 lines
Quoted 3 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 5 lines
Quoted 5 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

two state machines.

Quoted 2 lines
Quoted 1 line

As listed in the definition of the stop mode, the device do not act on the

incoming writes, it only keep tracks of its internal device context change as part of this.

Quoted 7 lines
Quoted 3 lines

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net other requests and not do device configuration.

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

No it isn't. Pls look at the spec if you need to check that ;)

Quoted 136 lines
Quoted 3 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Quoted 6 lines
Quoted 1 line

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

How about read and the device configuration changes?

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too.
The point is, when in freeze mode, the member device is frozen, hence, device won't initiate those changes.

Quoted 15 lines
Quoted 6 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

+ the device do not have any changes in the device context. The
+ member device is not accessed in the system through the virtio

interface.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Quoted 8 lines
Quoted 6 lines

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

Quoted 47 lines
Quoted 24 lines
Quoted 21 lines
Quoted 9 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

admin-cmds-device-migration.tex new file +2 -0
+
+When the owner driver wants to stop the operation of the device,

admin-cmds-device-migration.tex

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+the owner driver sets the device mode to \field{Stop}. Once the
+device is in the \field{Stop} mode, the device does not initiate
+any notifications or does not access any driver memory. Since the
+member driver may be still active which may send further driver
+notifications to the device, the device context may be updated.
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode indicating to
+the device that no more driver access occurs. In the
+\field{Freeze} mode, no more changes occur in the device context.
+At this point, the device ensures that

there will not be any update to the device context.

Quoted 3 lines

What is missed here are:

1) it is a virtio specific states or not

Quoted 8 lines

It is not.

Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 10 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 7 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any
+device specific data which is needed by the device to resume its
+operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 2 lines
Quoted 2 lines

I'd split it into

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of operation. So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare case scenario.

Quoted 2 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?
2. device context returns incremental value from the previous read. So, it needs to clear it.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

yes

Quoted 101 lines
Quoted 11 lines
Quoted 1 line

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 9 lines
Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 5 lines

Typically, on

admin-cmds-device-migration.tex new file +3 -0
+the source hypervisor, the owner driver reads the device context
+once when the device is in \field{Active} or \field{Stop} mode
+and later once the member device is in \field{Freeze} mode.

Why need the read while device context could be changed? Or is the dirty page part of the device context?

Quoted 6 lines
Quoted 2 lines

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

admin-cmds-device-migration.tex

Quoted 9 lines
admin-cmds-device-migration.tex new file +7 -0
+
+Typically, the device context is read and written one time on the
+source and the destination hypervisor respectively once the
+device is in \field{Freeze} mode. On the destination hypervisor,
+after writing the device context, when the device mode set to
+\field{Active}, the device uses the most recently set device
+context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 22 lines
Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +5 -0
+
+In an alternative flow, on the source hypervisor the owner driver
+may choose to read the device context first time while the device
+is in \field{Active} mode and second time once the device is in
+\field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

Quoted 9 lines

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi);
}

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG section captures the partial value.

Quoted 11 lines
Quoted 9 lines
Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 6 lines

Similarly, on the

admin-cmds-device-migration.tex new file +4 -0
+destination hypervisor writes the device context first time while
+the device is still running in \field{Active} mode on the source
+hypervisor and writes the device context second time while the
+device is in

\field{Freeze} mode.

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex new file +2 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in memory"

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has been negotiated".

Quoted 7 lines

Is this the only optimization in virtio? No, but we still mention the rationale of why it exists. As long as the rationale do not confuse the reader, and adds the value explaining how things work, it is fine to add. Which is what above few lines did.
So let's keep it.

The easiest is to cut out the whole theory of operation and just write commands like how RSS command did, without even writing a single line about RSS. I think we can do better explanation than that for new things we add.

yes i find it useful. of course now we are writing it, we also need it not to be confusing or partial.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#7 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 7 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, October 10, 2023 6:11 PM

All this does beg the question of how is device undergoing flr though and that has to be in a normative statement.

Yes, will add.

Quoted 5 lines
Quoted 1 line

Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

mention this in the commit log pls

Ack. Will add.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Quoted 7 lines
Quoted 1 line

Right. PCI member device is present at source and destination with its layout,

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

wait don't we need to transfer pci state too? how is that migrated?

The hypervisor driver composes the vPCI device. So there isn’t a need to migrate the pci state. Only exception is VIRTIO_PCI_CAP_PCI_CFG, which is covered in this v1.

Quoted 111 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage

data.

Quoted 22 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

Quoted 17 lines
Quoted 3 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 13 lines
Quoted 11 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

Quoted 8 lines
Quoted 7 lines

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

Quoted 2 lines

What can happen between #2 and #3, is device mode may change to stop. And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 11 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

Quoted 4 lines
Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in

patch 3.

Quoted 6 lines
Quoted 1 line

Sure, device context is virtio specific. :) Device context will reflect if things changed in the stop mode.

Quoted 2 lines
Quoted 2 lines

It is present in leading OS that has fundamental requirement to support P2P

Quoted 7 lines
Quoted 5 lines

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to

handle the interaction.

Quoted 23 lines
Quoted 23 lines
Quoted 6 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 6 lines
Quoted 6 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

two state machines.

Quoted 3 lines
Quoted 1 line

As listed in the definition of the stop mode, the device do not act on the

incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Quoted 8 lines
Quoted 1 line

Because for most practical purposes, peer device wants to queue blk, net

other requests and not do device configuration.

Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

No it isn't. Pls look at the spec if you need to check that ;)

Ok. will check. But regardless, it is fine, because when STOP is done, config writes should not occur anyway.

Quoted 151 lines
Quoted 5 lines
Quoted 4 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally

started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too. The point is, when in freeze mode, the member device is frozen, hence,

device won't initiate those changes.

Quoted 30 lines
Quoted 18 lines
Quoted 8 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ the device do not have any changes in the device context.
+ The member device is not accessed in the system through the
+ virtio

interface.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Quoted 8 lines
Quoted 6 lines

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

Quoted 2 lines
Quoted 2 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines

+\end{tabularx}

(quoted diff) +7 -0
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to
+\field{Stop}. Once the device is in the \field{Stop} mode,
+the device does not initiate any notifications or does not
+access any driver memory. Since the member driver may be
+still active which may send further driver notifications to the device,

the device context may be updated.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode
+indicating to the device that no more driver access occurs.
+In the \field{Freeze} mode, no more changes occur in the device

context.

Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+At this point, the device ensures that

there will not be any update to the device context.

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 7 lines
Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 11 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+The member device has a device context which the owner driver
+can either read or write. The member device context consist
+of any device specific data which is needed by the device to
+resume its operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 5 lines
Quoted 5 lines

I'd split it into

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of

operation.

Quoted 18 lines

So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

Quoted 7 lines

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare

case scenario.

Quoted 6 lines
Quoted 3 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?
2. device context returns incremental value from the previous read. So, it

needs to clear it.

Quoted 1 line
Quoted 9 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

yes

Ok. the management sw has access to the context to restore.
Alternatively, it is the incremental context not available as it is read, but the freeze device still has the frozen context. So it can be marked active too.
I will double check if I captured this in the normative or not.

Quoted 115 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 65 lines
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

Review

Quoted 9 lines
(quoted diff) +7 -0
+
+Typically, the device context is read and written one time on
+the source and the destination hypervisor respectively once
+the device is in \field{Freeze} mode. On the destination
+hypervisor, after writing the device context, when the device
+mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 28 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the owner
+driver may choose to read the device context first time while
+the device is in \field{Active} mode and second time once the
+device is in \field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 7 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first time
+while the device is still running in \field{Active} mode on
+the source hypervisor and writes the device context second
+time while the device is in

\field{Freeze} mode.

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously
+written device

context.

Quoted 5 lines
Quoted 4 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

Quoted 7 lines
Quoted 3 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 2 lines

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 2 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

Quoted 1 line

As long as the rationale do not confuse the reader, and adds the value

explaining how things work, it is fine to add.

Quoted 3 lines

Which is what above few lines did.
So let's keep it.

Quoted 7 lines
Quoted 1 line

The easiest is to cut out the whole theory of operation and just write

commands like how RSS command did, without even writing a single line about RSS.

Quoted 1 line

I think we can do better explanation than that for new things we add.

yes i find it useful. of course now we are writing it, we also need it not to be confusing or partial.

Ok. Thanks.

#8 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 22 lines · Parav Pandit
Quoted 7 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, October 10, 2023 6:11 PM

All this does beg the question of how is device undergoing flr though and that has to be in a normative statement.

Yes, will add.

Quoted 5 lines
Quoted 1 line

Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

mention this in the commit log pls

Ack. Will add.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

Quoted 9 lines
Quoted 6 lines

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

wait don't we need to transfer pci state too? how is that migrated?

The hypervisor driver composes the vPCI device. So there isn’t a need to migrate the pci state. Only exception is VIRTIO_PCI_CAP_PCI_CFG, which is covered in this v1.

yes but what seems implicit is that device is in some reasonable state when thing thing happens. e.g. are there no limitations at all e.g. in which order things happen? can you really first configure virtio
then pci config? for sure?

Quoted 112 lines
Quoted 112 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage

data.

Quoted 22 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

Quoted 17 lines
Quoted 3 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 13 lines
Quoted 11 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

Quoted 8 lines
Quoted 7 lines

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

Quoted 2 lines

What can happen between #2 and #3, is device mode may change to stop. And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 11 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

Quoted 4 lines
Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in

patch 3.

Quoted 6 lines
Quoted 1 line

Sure, device context is virtio specific. :) Device context will reflect if things changed in the stop mode.

Quoted 2 lines
Quoted 2 lines

It is present in leading OS that has fundamental requirement to support P2P

Quoted 7 lines
Quoted 5 lines

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to

handle the interaction.

Quoted 24 lines
Quoted 15 lines
Quoted 6 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 6 lines
Quoted 6 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

two state machines.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 2 lines

As listed in the definition of the stop mode, the device do not act on the

incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net

Quoted 8 lines
Quoted 7 lines

other requests and not do device configuration.

Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

No it isn't. Pls look at the spec if you need to check that ;)

Ok. will check. But regardless, it is fine, because when STOP is done, config writes should not occur anyway.

i don't see a statement like this but maybe i missed it.

Quoted 156 lines
Quoted 156 lines
Quoted 5 lines
Quoted 4 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally

started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too. The point is, when in freeze mode, the member device is frozen, hence,

device won't initiate those changes.

Quoted 30 lines
Quoted 18 lines
Quoted 8 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ the device do not have any changes in the device context.
+ The member device is not accessed in the system through the
+ virtio

interface.

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Quoted 8 lines
Quoted 6 lines

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

Quoted 2 lines
Quoted 2 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 8 lines

+\end{tabularx}

(quoted diff) +7 -0
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to
+\field{Stop}. Once the device is in the \field{Stop} mode,
+the device does not initiate any notifications or does not
+access any driver memory. Since the member driver may be
+still active which may send further driver notifications to the device,

the device context may be updated.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode
+indicating to the device that no more driver access occurs.
+In the \field{Freeze} mode, no more changes occur in the device

context.

Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+At this point, the device ensures that

there will not be any update to the device context.

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 7 lines
Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 11 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+The member device has a device context which the owner driver
+can either read or write. The member device context consist
+of any device specific data which is needed by the device to
+resume its operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 5 lines
Quoted 5 lines

I'd split it into

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of

operation.

Quoted 18 lines

So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

Quoted 7 lines

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare

case scenario.

Quoted 6 lines
Quoted 3 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?
2. device context returns incremental value from the previous read. So, it

needs to clear it.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Does libvirt restore on migration failure?

yes

Ok. the management sw has access to the context to restore.
Alternatively, it is the incremental context not available as it is read, but the freeze device still has the frozen context. So it can be marked active too.
I will double check if I captured this in the normative or not.

preferable if context is not erased on the device IMHO. less of a chance of a failure to resume.

Quoted 115 lines
Quoted 115 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 65 lines
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

Review

Quoted 9 lines
(quoted diff) +7 -0
+
+Typically, the device context is read and written one time on
+the source and the destination hypervisor respectively once
+the device is in \field{Freeze} mode. On the destination
+hypervisor, after writing the device context, when the device
+mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 28 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the owner
+driver may choose to read the device context first time while
+the device is in \field{Active} mode and second time once the
+device is in \field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 7 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first time
+while the device is still running in \field{Active} mode on
+the source hypervisor and writes the device context second
+time while the device is in

\field{Freeze} mode.

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously
+written device

context.

Quoted 5 lines
Quoted 4 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

Quoted 7 lines
Quoted 3 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 2 lines

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 2 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

Quoted 1 line

As long as the rationale do not confuse the reader, and adds the value

explaining how things work, it is fine to add.

Quoted 3 lines

Which is what above few lines did.
So let's keep it.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

The easiest is to cut out the whole theory of operation and just write

commands like how RSS command did, without even writing a single line about RSS.

Quoted 1 line

I think we can do better explanation than that for new things we add.

yes i find it useful. of course now we are writing it, we also need it not to be confusing or partial.

Ok. Thanks.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 4 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, October 10, 2023 7:30 PM

Quoted 9 lines
Quoted 1 line

The hypervisor driver composes the vPCI device. So there isn’t a need to

migrate the pci state.

Quoted 2 lines

Only exception is VIRTIO_PCI_CAP_PCI_CFG, which is covered in this v1.

yes but what seems implicit is that device is in some reasonable state when thing thing happens. e.g. are there no limitations at all e.g. in which order things happen? can you really first configure virtio then pci config? for sure?

First pci config is setup, like bus master enable etc. After that point, the device is handed to virtio things.

From device context write perspective, I doubt the order matters.
For example, if pci bus master and msix are enabled after device context restore or before would not matter much. As long as they are done before making the device mode to active.

Quoted 126 lines
Quoted 31 lines
Quoted 31 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 2 lines

or it may experience a read from the shared memory and get garbage

data.

Quoted 14 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

Quoted 9 lines
Quoted 3 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any +driver notifications,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be

migrated easily.

Quoted 89 lines
Quoted 89 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 5 lines
Quoted 3 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

Quoted 3 lines

What can happen between #2 and #3, is device mode may change to stop. And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 12 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

Quoted 5 lines
Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in

patch 3.

Quoted 6 lines
Quoted 1 line

Sure, device context is virtio specific. :) Device context will reflect if things changed in the stop mode.

Quoted 2 lines
Quoted 2 lines

It is present in leading OS that has fundamental requirement to support P2P

Quoted 8 lines
Quoted 5 lines

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to

handle the interaction.

Quoted 17 lines
Quoted 17 lines
Quoted 6 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 6 lines
Quoted 6 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

two state machines.

Quoted 3 lines
Quoted 1 line

As listed in the definition of the stop mode, the device do not act on the

Quoted 8 lines
Quoted 6 lines

incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net

other requests and not do device configuration.

Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

No it isn't. Pls look at the spec if you need to check that ;)

Ok. will check. But regardless, it is fine, because when STOP is done, config

writes should not occur anyway.

i don't see a statement like this but maybe i missed it.

I am missing it, will add.

#10 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 13 lines · Parav Pandit
Quoted 4 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, October 10, 2023 7:30 PM

Quoted 9 lines
Quoted 1 line

The hypervisor driver composes the vPCI device. So there isn’t a need to

migrate the pci state.

Quoted 2 lines

Only exception is VIRTIO_PCI_CAP_PCI_CFG, which is covered in this v1.

yes but what seems implicit is that device is in some reasonable state when thing thing happens. e.g. are there no limitations at all e.g. in which order things happen? can you really first configure virtio then pci config? for sure?

Quoted 6 lines

First pci config is setup, like bus master enable etc. After that point, the device is handed to virtio things.

From device context write perspective, I doubt the order matters.
For example, if pci bus master and msix are enabled after device context restore or before would not matter much. As long as they are done before making the device mode to active.

whatever the requirements, document them.

Quoted 128 lines
Quoted 128 lines
Quoted 31 lines
Quoted 31 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 2 lines

or it may experience a read from the shared memory and get garbage

data.

Quoted 14 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

Quoted 9 lines
Quoted 3 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any +driver notifications,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be

migrated easily.

Quoted 91 lines
Quoted 91 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 5 lines
Quoted 3 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

Quoted 3 lines

What can happen between #2 and #3, is device mode may change to stop. And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 12 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

Quoted 5 lines
Quoted 1 line

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in

patch 3.

Quoted 6 lines
Quoted 1 line

Sure, device context is virtio specific. :) Device context will reflect if things changed in the stop mode.

Quoted 2 lines
Quoted 2 lines

It is present in leading OS that has fundamental requirement to support P2P

Quoted 8 lines
Quoted 5 lines

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to

handle the interaction.

Quoted 17 lines
Quoted 17 lines
Quoted 6 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 6 lines
Quoted 6 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

two state machines.

Quoted 3 lines
Quoted 1 line

As listed in the definition of the stop mode, the device do not act on the

Quoted 8 lines
Quoted 6 lines

incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net

other requests and not do device configuration.

Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

No it isn't. Pls look at the spec if you need to check that ;)

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Ok. will check. But regardless, it is fine, because when STOP is done, config

writes should not occur anyway.

i don't see a statement like this but maybe i missed it.

I am missing it, will add.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#11 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

In reply to Parav Pandit
Quoted 65 lines · Parav Pandit
Quoted 39 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 10, 2023 11:21 AM

Quoted 33 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Quoted 8 lines
Quoted 3 lines

Other than PCI config registers and due to some legacy, msix. The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific

and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Virtio device specific means: cvq of crypto device, cvq of net device, flow filter vqs of net device etc. Virtio specific interface: virtio driver notifications, virtio virtqueue and configuration mediation etc.

Quoted 15 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the

guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

Quoted 5 lines
Quoted 4 lines

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

In most practical cases #2 followed by #1 should not happen as on the source side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Quoted 3 lines

But ok, since we active to freeze mode change is allowed, lets discuss above.

A device is the single synchronization point for any device reset, FLR or admin command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Quoted 1 line

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by FLR.

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

Quoted 1 line

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

Quoted 26 lines

Secondly since the FLR is local to the source, intermediate sub state does not migrate.

But I agree, it is worth to have the text capturing this.

Quoted 22 lines
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

and rest of the virtio device functionality such as control vq,

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 3 lines
Quoted 3 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely

used.

Quoted 4 lines
Quoted 3 lines

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on

understanding the use case.

Quoted 2 lines

And you seem opposite, but ok.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

As stated, it for the reader to understand the use case and see how proposed

commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Ok. The use case and requirement of member device passthrough is clear to most reviewers now.

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

Quoted 53 lines

So I will remove it from commit log.

Quoted 51 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

To support a VM live migration for such passthrough virtio devices, the owner PCI PF device administers the device migration flow.

Well, if this is specific only to PCI SR-IOV, I'd move it to the PCI transport

part.

Quoted 44 lines
Quoted 1 line

But I guess not.

We took the decision to not do so, for other group commands as well. After Michael's suggestion we moved it to group commands. So I will not debate this further.

Quoted 39 lines
Quoted 10 lines

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/t
ree/
include/uapi/linux/vfio.h?h=v6.1.47

Quoted 4 lines
---
 admin-cmds-device-migration.tex | 94

+++++++++++++++++++++++++++++++++

Quoted 10 lines
 admin.tex                       |  1 +
 2 files changed, 95 insertions(+)  create mode 100644

admin-cmds-device-migration.tex

(quoted diff) +0 -0

b/admin-cmds-device-migration.tex new file mode 100644 index 0000000..f839af4

admin-cmds-device-migration.tex line 10

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
@@ -0,0 +1,94 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a
+Virtio Device / Device groups / Group administration commands /
+Device Migration}
+
+In some systems, there is a need to migrate a running virtual
+machine from one to another system. A running virtual machine has
+one or more passthrough virtio member devices attached to it. A
+passthrough device is entirely operated by the guest virtual
+machine. For example, with the SR-IOV group type, group member
+(VF) may undergo virtio device initialization and reset flow

What do you mean by "reset flow"? It looks not like a terminology defined in the PCI spec. And Google gives me nothing about this.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

"reset flow" = virtio specification section 2.4 Device Reset flow.

My git repo show it's still called "device reset" and I see you use "FLR flow" which is also not very clear to me.

Ok. I assume "reset flow" is clear to you now that it points to section 2.4.
This section is not normative section, so using an extra word like "flow" does not confuse anyone. I will link to the section anyway.

Probably, but you mention FLR flow as well.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence

Quoted 8 lines
Quoted 7 lines

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real devices.

Quoted 16 lines

(and hypervisor is not involved in these two flows, hence the admin command interface is designed such that it can fullfil above requirements).

Theory of operation brings out this clarity. Please notice that it is in introductory section with an example. Not normative line.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

For example, how is the power management interaction with the freeze/stop?

Quoted 84 lines
Quoted 37 lines
Quoted 3 lines

It is not written in this series, because we believe it must not be synchronized

as it is fully controlled by the guest.

Quoted 33 lines
Quoted 33 lines

admin-cmds-device-migration.tex

Quoted 9 lines

at the same time such flows must not obstruct

admin-cmds-device-migration.tex new file +6 -0
+the device migration flow. In such a scenario, a group owner
+device can provide the administration command interface to
+facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to another
+hypervisor, these hypervisors are named as source and destination

hypervisor respectively.

admin-cmds-device-migration.tex

Quoted 8 lines
admin-cmds-device-migration.tex new file +8 -0
+In such a scenario, a source hypervisor administers the member
+device to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member
+device to setup a device context and resumes the member device.
+The source hypervisor reads the member device context and the
+destination hypervisor writes the member device context. The
+method to transfer the member device context from the source to
+the destination hypervisor is

outside the scope of this specification.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex new file +4 -0
+
+The member device can be in any of the three migration modes. The
+owner driver sets the member device in one of the following modes
+during

device migration flow.

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +6 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name &
+Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member
+device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 6 lines
Quoted 2 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec

here.

Ok, maybe something like "transport specific instantiation"

Ok. that’s a good text. I will change to it.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any notifications, +and it does not access any driver memory.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory

allocated by the guest driver?

Virtqueue?

Virtqueue and any memory referred by the virtqueue.

This is good text, I will change to it.

Quoted 22 lines
Quoted 6 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size

returned.

Quoted 1 line

Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 2 lines

There are inflight descriptors with the device that are not yet returned to the

driver, and device wont return them as zero size wrong completions.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

+ The member device may receive driver notifications in this mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Receive means, driver can send the notification as PCIe TLP that device may

receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the queue?

Device cannot process the queue because it does not initiate any read/write towards the virtqueue.

Read/Write only results in a driver noticeable behaviour, it doesn't mean the device can't process the buffers. For example, devices can keep processing available buffers and make them as inflight ones.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive notifications?

Because notification may still arrive and if the device may update any counters as part of

Which counters did you mean here?

Quoted 5 lines

it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

Maybe you can give a concrete example.

Quoted 19 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Quoted 8 lines

Like this?

"""
+The member device has a device context which the owner driver can +either
read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the theory of operation section so reader can read more detail about it.

Quoted 6 lines
Quoted 4 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport (PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout, only the virtio device context is transferred. Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be used:

1) Only works for SR-IOV member device like VF
2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented anywhere. And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped atomically. Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Quoted 7 lines
Quoted 6 lines

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage data.

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

Btw, shared regions are tricky for hardware.

Quoted 24 lines
Quoted 24 lines
Quoted 3 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

Quoted 8 lines
Quoted 3 lines

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure.
It is passthrough device, hence hypervisor layer do not get to see sub-state.

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

Quoted 8 lines
Quoted 7 lines

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Quoted 1 line

What can happen between #2 and #3, is device mode may change to stop.

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

Quoted 11 lines

And in stop mode, device context would capture #5 or #4, depending where is device at that point.

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

Quoted 30 lines
Quoted 6 lines
Quoted 2 lines

Second, it is not virtio specific.

It's somehow for sure, for example you said device context need to be preserved. And as far as I see the device context is all virtio specific in patch 3.

Sure, device context is virtio specific. :)
Device context will reflect if things changed in the stop mode.

Quoted 6 lines
Quoted 1 line

It is present in leading OS that has fundamental requirement to support P2P

devices.

If it's PCI specific, instead of trying to do a workaround in virtio, why not invent a mechanism there?

It is not a workaround in virtio.
It is the way pci p2p devices work for which one needs to be receptive to handle the interaction.

Quoted 11 lines
Quoted 3 lines

Third, it is not, interacing with the _actua_ device status.

In "SUSPEND" patch-5, you already asked this question. I assume you asked

again so that this series is complete.

Quoted 5 lines
Quoted 5 lines

For example,
what happens if the driver wants to reset but the device is in stop mode? You told me it is addressed in your series but looks not. Once you try to describe that, you're actually try to connect states between the

two state machines.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

As listed in the definition of the stop mode, the device do not act on the

incoming writes, it only keep tracks of its internal device context change as part of this.

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net other requests and not do device configuration.

You forbid the device to process the queue but only allow the notification. How can the device queue those requests? The device can just do the available buffer check after resume, then it's all fine.

Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

For example, WCE. What's more important, the spec allows config space to be RW, so even if there's no examples before, it doesn't mean we won't have a RW in the future.

Quoted 5 lines
Quoted 4 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what is done in this patch.

Quoted 35 lines
Quoted 7 lines
Quoted 2 lines

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

How about read and the device configuration changes?

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too.
The point is, when in freeze mode, the member device is frozen, hence, device won't initiate those changes.

Quoted 17 lines
Quoted 6 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

+ the device do not have any changes in the device context. The
+ member device is not accessed in the system through the virtio

interface.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Yes, as usual.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

So this implies the freeze needs to wait for FLR otherwise device context may change.

Quoted 82 lines
Quoted 48 lines
Quoted 25 lines
Quoted 22 lines
Quoted 10 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

admin-cmds-device-migration.tex new file +2 -0
+
+When the owner driver wants to stop the operation of the device,

admin-cmds-device-migration.tex

Quoted 10 lines
admin-cmds-device-migration.tex new file +10 -0
+the owner driver sets the device mode to \field{Stop}. Once the
+device is in the \field{Stop} mode, the device does not initiate
+any notifications or does not access any driver memory. Since the
+member driver may be still active which may send further driver
+notifications to the device, the device context may be updated.
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode indicating to
+the device that no more driver access occurs. In the
+\field{Freeze} mode, no more changes occur in the device context.
+At this point, the device ensures that

there will not be any update to the device context.

Quoted 3 lines

What is missed here are:

1) it is a virtio specific states or not

Quoted 8 lines

It is not.

Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 10 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 7 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +5 -0
+
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any
+device specific data which is needed by the device to resume its
+operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 2 lines
Quoted 2 lines

I'd split it into

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of operation. So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex new file +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Quoted 6 lines

Yes, as you say, by restoring the whole context will suffice this corner/rare case scenario.

Quoted 2 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

Quoted 1 line

2. device context returns incremental value from the previous read. So, it needs to clear it.

I don't understand here. This is not the case for most of the devices.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

Yes.

Quoted 12 lines
Quoted 12 lines
Quoted 2 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 9 lines
Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 5 lines

Typically, on

admin-cmds-device-migration.tex new file +3 -0
+the source hypervisor, the owner driver reads the device context
+once when the device is in \field{Active} or \field{Stop} mode
+and later once the member device is in \field{Freeze} mode.

Why need the read while device context could be changed? Or is the dirty page part of the device context?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique to virtio.

Quoted 53 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

admin-cmds-device-migration.tex

Quoted 10 lines
admin-cmds-device-migration.tex new file +7 -0
+
+Typically, the device context is read and written one time on the
+source and the destination hypervisor respectively once the
+device is in \field{Freeze} mode. On the destination hypervisor,
+after writing the device context, when the device mode set to
+\field{Active}, the device uses the most recently set device
+context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 29 lines
Quoted 8 lines
Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex new file +5 -0
+
+In an alternative flow, on the source hypervisor the owner driver
+may choose to read the device context first time while the device
+is in \field{Active} mode and second time once the device is in
+\field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi);
}
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG section captures the partial value.

There's no way for the device to know whether or not it's a partial value or not. No?

Quoted 20 lines
Quoted 12 lines
Quoted 10 lines
Quoted 10 lines

admin-cmds-device-migration.tex

Quoted 7 lines

Similarly, on the

admin-cmds-device-migration.tex new file +4 -0
+destination hypervisor writes the device context first time while
+the device is still running in \field{Active} mode on the source
+hypervisor and writes the device context second time while the
+device is in

\field{Freeze} mode.

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex new file +2 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

Quoted 8 lines

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in memory"

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has been negotiated".

Is this the only optimization in virtio? No, but we still mention the rationale of why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of migration. You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

Thanks

Quoted 6 lines

As long as the rationale do not confuse the reader, and adds the value explaining how things work, it is fine to add. Which is what above few lines did.
So let's keep it.

The easiest is to cut out the whole theory of operation and just write commands like how RSS command did, without even writing a single line about RSS. I think we can do better explanation than that for new things we add.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#12 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 8 lines · Jason Wang
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Quoted 6 lines
Quoted 1 line

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

For example, how is the power management interaction with the freeze/stop?

Right. I think in the same way this allows passthrough BAR access it should allow passthrough config access, that is document what exactly is the requirement. is just atomicity enough? and should we require something from driver too?

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#13 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

In reply to Jason Wang
Quoted 75 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, October 11, 2023 8:44 AM

Quoted 43 lines · Parav Pandit
Quoted 43 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 10, 2023 11:21 AM

Quoted 37 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Other than PCI config registers and due to some legacy, msix. The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific

Quoted 6 lines
Quoted 5 lines

and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Virtio device specific means: cvq of crypto device, cvq of net device, flow filter

vqs of net device etc.

Quoted 1 line

Virtio specific interface: virtio driver notifications, virtio virtqueue and

configuration mediation etc.

Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the

guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and #2.

Quoted 9 lines
Quoted 3 lines

But ok, since we active to freeze mode change is allowed, lets discuss above.

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and other.

Quoted 4 lines
Quoted 1 line

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by FLR.

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

I didn’t say clear the context.
FLR updates the device context.
Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid any race.

Quoted 42 lines
Quoted 1 line

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

Quoted 1 line

Secondly since the FLR is local to the source, intermediate sub state does not

migrate.

Quoted 35 lines

But I agree, it is worth to have the text capturing this.

Quoted 32 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

and rest of the virtio device functionality such as control vq,

Quoted 11 lines
Quoted 2 lines

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 4 lines
Quoted 4 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely

used.

Quoted 4 lines
Quoted 3 lines

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on

understanding the use case.

Quoted 4 lines

And you seem opposite, but ok.

As stated, it for the reader to understand the use case and see how proposed

commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Quoted 6 lines
Quoted 1 line

Ok. The use case and requirement of member device passthrough is clear to

most reviewers now.

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

Quoted 6 lines
Quoted 2 lines

Ok. I assume "reset flow" is clear to you now that it points to section 2.4. This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 1 line

I will link to the section anyway.

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the PCIe transport.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real devices.

That would be some other trap alternative that needs to dissect the device and build infrastructure for such dissection is not desired in the listed use case. Here we are addressing the requirement of passthrough the device.

So your disagreement is fine for non-passthrough devices.

Quoted 21 lines
Quoted 1 line

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 2 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 16 lines

Not normative line.

Quoted 14 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

As author, what we think, I will cover. If you have specific points to add value, please share, I will look into it.

Quoted 2 lines

For example, how is the power management interaction with the freeze/stop?

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

Quoted 97 lines
Quoted 16 lines
Quoted 16 lines
Quoted 4 lines

It is not written in this series, because we believe it must not be synchronized

as it is fully controlled by the guest.

Quoted 11 lines
Quoted 11 lines

Review

Quoted 10 lines

at the same time such flows must not obstruct

(quoted diff) +7 -0
+the device migration flow. In such a scenario, a group owner
+device can provide the administration command interface to
+facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to
+another hypervisor, these hypervisors are named as source and
+destination

hypervisor respectively.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+In such a scenario, a source hypervisor administers the
+member device to suspend the device and preserves the device

context.

Quoted 40 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+Subsequently, a destination hypervisor administers the member
+device to setup a device context and resumes the member device.
+The source hypervisor reads the member device context and the
+destination hypervisor writes the member device context. The
+method to transfer the member device context from the source
+to the destination hypervisor is

outside the scope of this specification.

Review

Quoted 4 lines
(quoted diff) +4 -0
+
+The member device can be in any of the three migration modes.
+The owner driver sets the member device in one of the
+following modes during

device migration flow.

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name
+& Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member
+device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 7 lines
Quoted 3 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec

here.

Ok, maybe something like "transport specific instantiation"

Ok. that’s a good text. I will change to it.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any +notifications, and it does not access any driver memory.

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory

Quoted 38 lines
Quoted 4 lines

allocated by the guest driver?

Virtqueue?

Virtqueue and any memory referred by the virtqueue.

This is good text, I will change to it.

Quoted 30 lines
Quoted 7 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size

returned.

Quoted 1 line

Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 3 lines

There are inflight descriptors with the device that are not yet returned to the

driver, and device wont return them as zero size wrong completions.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

+ The member device may receive driver notifications in this
+ mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may

receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the

Quoted 9 lines

queue?

Quoted 2 lines
Quoted 1 line

Device cannot process the queue because it does not initiate any read/write

towards the virtqueue.

Read/Write only results in a driver noticeable behaviour, it doesn't mean the device can't process the buffers. For example, devices can keep processing available buffers and make them as inflight ones.

The idea is to stop the device and prepare for the migration, so the command to do so. Otherwise just the keep the device in active mode and avoid the complications.

Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to update as result of driver notification. For example caching the posted avail index in the notification.

Quoted 10 lines
Quoted 5 lines

it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

No, it does not.
Device context is updated within the device without accessing the queue memory of the guest.

Quoted 2 lines

Maybe you can give a concrete example.

The above one.

Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

Quoted 8 lines
Quoted 6 lines

"""
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 7 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

Quoted 9 lines

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be used:

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

Quoted 9 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented anywhere. And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

Quoted 14 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

or it may experience a read from the shared memory and get garbage

data.

Quoted 4 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

MMIO read/write can be initiated by the peer device while the device is in stopped state.

Quoted 32 lines

Btw, shared regions are tricky for hardware.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Because hypervisor has stopped the vcpus which are issuing them.

Quoted 5 lines
Quoted 1 line

What can happen between #2 and #3, is device mode may change to stop.

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

On the source hypervisor, the mode changes are active->stop->freeze.
Hence when freeze is done, the hypervisor knows that all inflight has been stopped by now.

Quoted 4 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Quoted 7 lines
Quoted 1 line

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

So question is "how reader knows if a state can go to another state and when"? It is described and listed in the table, when a mode can change.

Quoted 10 lines
Quoted 4 lines
Quoted 3 lines

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net

other requests and not do device configuration.

You forbid the device to process the queue but only allow the notification. How can the device queue those requests? The device can just do the available buffer check after resume, then it's all fine.

Device can always decide to not queue the request and do the available buffer check later. The peer device may read also from MMIO space.

So the intermediate step covers this aspect where device_type specific plumbing is not done. Its generic. A device may choose to omit such doorbells as well as long as it knows it can resume.

Quoted 8 lines
Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

For example, WCE. What's more important, the spec allows config space to be RW, so even if there's no examples before, it doesn't mean we won't have a RW in the future.

Ok.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

Quoted 8 lines
Quoted 1 line

The device would respond to stop/freeze request when it has internally

started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what is done in this patch.

Synchronization point = device.

Quoted 42 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too. The point is, when in freeze mode, the member device is frozen, hence,

device won't initiate those changes.

Quoted 23 lines
Quoted 23 lines
Quoted 8 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ the device do not have any changes in the device context.
+ The member device is not accessed in the system through the
+ virtio

interface.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

So this implies the freeze needs to wait for FLR otherwise device context may change.

Device context can change anytime and reflect what is latest.
I will update the patches to reflect that device is the single synchronization point serving flr, mode changes.

Quoted 91 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 8 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

Review

Quoted 7 lines
(quoted diff) +7 -0
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to
+\field{Stop}. Once the device is in the \field{Stop} mode,
+the device does not initiate any notifications or does not
+access any driver memory. Since the member driver may be
+still active which may send further driver notifications to the device,

the device context may be updated.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode
+indicating to the device that no more driver access occurs.
+In the \field{Freeze} mode, no more changes occur in the device

context.

Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+At this point, the device ensures that

there will not be any update to the device context.

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 7 lines
Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 11 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+The member device has a device context which the owner driver
+can either read or write. The member device context consist
+of any device specific data which is needed by the device to
+resume its operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 5 lines
Quoted 5 lines

I'd split it into

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of

operation.

Quoted 18 lines

So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

Quoted 7 lines

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare

case scenario.

Quoted 1 line
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

If kept, hypervisor may not be able to decide when to change the mode from active->stop. We can opt for a mode where full device context is read in each mode without clearing it. But than it can be very specific to a version of qemu, which we are avoiding it here.

Quoted 5 lines
Quoted 1 line

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Not sure which devices you mean here with "most of the devices".
Device context functions like a write record pages (aka dirty pages).
Whatever is already returned is/should not be repeated in subsequent reads, though device can choose to do so.

Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique to virtio.

It is theory of operation that brings the clarity and rationale. So I will keep it.

Quoted 53 lines
Quoted 53 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

Review

Quoted 10 lines
(quoted diff) +7 -0
+
+Typically, the device context is read and written one time on
+the source and the destination hypervisor respectively once
+the device is in \field{Freeze} mode. On the destination
+hypervisor, after writing the device context, when the device
+mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 28 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the owner
+driver may choose to read the device context first time while
+the device is in \field{Active} mode and second time once the
+device is in \field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or not. No?

Device does not need to know, because when the guest vm and the device is resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 30 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 7 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first time
+while the device is still running in \field{Active} mode on
+the source hypervisor and writes the device context second
+time while the device is in

\field{Freeze} mode.

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously
+written device

context.

Quoted 5 lines
Quoted 4 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

Quoted 7 lines
Quoted 3 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 2 lines

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of migration. You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

It is one of the optimization apart. The comparison is of one_of_example or not.

#14 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 101 lines · Parav Pandit
Quoted 75 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, October 11, 2023 8:44 AM

Quoted 43 lines · Parav Pandit
Quoted 43 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 10, 2023 11:21 AM

Quoted 37 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Other than PCI config registers and due to some legacy, msix. The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific

Quoted 6 lines
Quoted 5 lines

and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Virtio device specific means: cvq of crypto device, cvq of net device, flow filter

vqs of net device etc.

Quoted 1 line

Virtio specific interface: virtio driver notifications, virtio virtqueue and

configuration mediation etc.

Quoted 17 lines
Quoted 17 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the

guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and #2.

Quoted 9 lines
Quoted 3 lines

But ok, since we active to freeze mode change is allowed, lets discuss above.

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and other.

Quoted 4 lines
Quoted 1 line

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by FLR.

Quoted 8 lines

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

I didn’t say clear the context.
FLR updates the device context.
Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid any race.

Heh well but if drivers depend on specific behaviour then we really need to document that in the spec.

Quoted 52 lines
Quoted 42 lines
Quoted 1 line

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

Quoted 1 line

Secondly since the FLR is local to the source, intermediate sub state does not

migrate.

Quoted 35 lines

But I agree, it is worth to have the text capturing this.

Quoted 32 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

and rest of the virtio device functionality such as control vq,

Quoted 11 lines
Quoted 2 lines

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 4 lines
Quoted 4 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely

used.

Quoted 4 lines
Quoted 3 lines

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on

understanding the use case.

Quoted 4 lines

And you seem opposite, but ok.

As stated, it for the reader to understand the use case and see how proposed

commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Quoted 6 lines
Quoted 1 line

Ok. The use case and requirement of member device passthrough is clear to

most reviewers now.

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Ok. I assume "reset flow" is clear to you now that it points to section 2.4. This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 1 line

I will link to the section anyway.

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the PCIe transport.

What I worry about however, is what happens for example if FLR is triggered while an admin command is in progress. This applies to things like legacy admin commands by the way.

Quoted 57 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real devices.

That would be some other trap alternative that needs to dissect the device and build infrastructure for such dissection is not desired in the listed use case. Here we are addressing the requirement of passthrough the device.

So your disagreement is fine for non-passthrough devices.

Quoted 21 lines
Quoted 1 line

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 2 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 16 lines

Not normative line.

Quoted 14 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

Quoted 6 lines

As author, what we think, I will cover. If you have specific points to add value, please share, I will look into it.

Quoted 2 lines

For example, how is the power management interaction with the freeze/stop?

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

I am not sure what exactly all this means though.
Should be clarified, in some way.

Quoted 209 lines
Quoted 97 lines
Quoted 16 lines
Quoted 16 lines
Quoted 4 lines

It is not written in this series, because we believe it must not be synchronized

as it is fully controlled by the guest.

Quoted 11 lines
Quoted 11 lines

Review

Quoted 10 lines

at the same time such flows must not obstruct

(quoted diff) +7 -0
+the device migration flow. In such a scenario, a group owner
+device can provide the administration command interface to
+facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to
+another hypervisor, these hypervisors are named as source and
+destination

hypervisor respectively.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+In such a scenario, a source hypervisor administers the
+member device to suspend the device and preserves the device

context.

Quoted 40 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+Subsequently, a destination hypervisor administers the member
+device to setup a device context and resumes the member device.
+The source hypervisor reads the member device context and the
+destination hypervisor writes the member device context. The
+method to transfer the member device context from the source
+to the destination hypervisor is

outside the scope of this specification.

Review

Quoted 4 lines
(quoted diff) +4 -0
+
+The member device can be in any of the three migration modes.
+The owner driver sets the member device in one of the
+following modes during

device migration flow.

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name
+& Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member
+device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 7 lines
Quoted 3 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec

here.

Ok, maybe something like "transport specific instantiation"

Ok. that’s a good text. I will change to it.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any +notifications, and it does not access any driver memory.

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory

Quoted 38 lines
Quoted 4 lines

allocated by the guest driver?

Virtqueue?

Virtqueue and any memory referred by the virtqueue.

This is good text, I will change to it.

Quoted 30 lines
Quoted 7 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size

returned.

Quoted 1 line

Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 3 lines

There are inflight descriptors with the device that are not yet returned to the

driver, and device wont return them as zero size wrong completions.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

+ The member device may receive driver notifications in this
+ mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may

receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the

Quoted 9 lines

queue?

Quoted 2 lines
Quoted 1 line

Device cannot process the queue because it does not initiate any read/write

towards the virtqueue.

Read/Write only results in a driver noticeable behaviour, it doesn't mean the device can't process the buffers. For example, devices can keep processing available buffers and make them as inflight ones.

The idea is to stop the device and prepare for the migration, so the command to do so. Otherwise just the keep the device in active mode and avoid the complications.

Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to update as result of driver notification. For example caching the posted avail index in the notification.

Quoted 10 lines
Quoted 5 lines

it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

No, it does not.
Device context is updated within the device without accessing the queue memory of the guest.

Quoted 2 lines

Maybe you can give a concrete example.

The above one.

Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

Quoted 8 lines
Quoted 6 lines

"""
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 7 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

Quoted 9 lines

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be used:

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

Quoted 9 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented anywhere. And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage

data.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

MMIO read/write can be initiated by the peer device while the device is in stopped state.

worth mentioning

Quoted 88 lines
Quoted 32 lines

Btw, shared regions are tricky for hardware.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Because hypervisor has stopped the vcpus which are issuing them.

Quoted 5 lines
Quoted 1 line

What can happen between #2 and #3, is device mode may change to stop.

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

On the source hypervisor, the mode changes are active->stop->freeze.
Hence when freeze is done, the hypervisor knows that all inflight has been stopped by now.

Quoted 4 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Quoted 7 lines
Quoted 1 line

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

So question is "how reader knows if a state can go to another state and when"? It is described and listed in the table, when a mode can change.

Quoted 10 lines
Quoted 4 lines
Quoted 3 lines

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net

other requests and not do device configuration.

You forbid the device to process the queue but only allow the notification. How can the device queue those requests? The device can just do the available buffer check after resume, then it's all fine.

Quoted 5 lines

Device can always decide to not queue the request and do the available buffer check later. The peer device may read also from MMIO space.

So the intermediate step covers this aspect where device_type specific plumbing is not done. Its generic. A device may choose to omit such doorbells as well as long as it knows it can resume.

all this is kind of vague ... should be in the spec.

Quoted 15 lines
Quoted 8 lines
Quoted 3 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

For example, WCE. What's more important, the spec allows config space to be RW, so even if there's no examples before, it doesn't mean we won't have a RW in the future.

Ok.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally

Quoted 8 lines
Quoted 7 lines

started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what is done in this patch.

Synchronization point = device.

Then we need to spec device behaviour.

Quoted 302 lines
Quoted 42 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too. The point is, when in freeze mode, the member device is frozen, hence,

device won't initiate those changes.

Quoted 23 lines
Quoted 23 lines
Quoted 8 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ the device do not have any changes in the device context.
+ The member device is not accessed in the system through the
+ virtio

interface.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

So this implies the freeze needs to wait for FLR otherwise device context may change.

Device context can change anytime and reflect what is latest.
I will update the patches to reflect that device is the single synchronization point serving flr, mode changes.

Quoted 91 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 8 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

Review

Quoted 7 lines
(quoted diff) +7 -0
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to
+\field{Stop}. Once the device is in the \field{Stop} mode,
+the device does not initiate any notifications or does not
+access any driver memory. Since the member driver may be
+still active which may send further driver notifications to the device,

the device context may be updated.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode
+indicating to the device that no more driver access occurs.
+In the \field{Freeze} mode, no more changes occur in the device

context.

Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+At this point, the device ensures that

there will not be any update to the device context.

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 7 lines
Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 11 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+The member device has a device context which the owner driver
+can either read or write. The member device context consist
+of any device specific data which is needed by the device to
+resume its operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 5 lines
Quoted 5 lines

I'd split it into

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of

operation.

Quoted 18 lines

So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

Quoted 7 lines

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare

case scenario.

Quoted 1 line
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

What's the consideration for such clearing?

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

If kept, hypervisor may not be able to decide when to change the mode from active->stop. We can opt for a mode where full device context is read in each mode without clearing it. But than it can be very specific to a version of qemu, which we are avoiding it here.

Quoted 5 lines
Quoted 1 line

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Not sure which devices you mean here with "most of the devices".
Device context functions like a write record pages (aka dirty pages).
Whatever is already returned is/should not be repeated in subsequent reads, though device can choose to do so.

Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique to virtio.

It is theory of operation that brings the clarity and rationale. So I will keep it.

Quoted 53 lines
Quoted 53 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

Review

Quoted 10 lines
(quoted diff) +7 -0
+
+Typically, the device context is read and written one time on
+the source and the destination hypervisor respectively once
+the device is in \field{Freeze} mode. On the destination
+hypervisor, after writing the device context, when the device
+mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 28 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the owner
+driver may choose to read the device context first time while
+the device is in \field{Active} mode and second time once the
+device is in \field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or not. No?

Device does not need to know, because when the guest vm and the device is resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 31 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 7 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first time
+while the device is still running in \field{Active} mode on
+the source hypervisor and writes the device context second
+time while the device is in

\field{Freeze} mode.

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously
+written device

context.

Quoted 5 lines
Quoted 4 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

Quoted 7 lines
Quoted 3 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 2 lines

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 2 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

Quoted 9 lines
Quoted 7 lines

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of migration. You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

It is one of the optimization apart. The comparison is of one_of_example or not.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 6 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, October 12, 2023 1:44 AM

On Wed, Oct 11, 2023 at 10:47:23AM +0000, Parav Pandit wrote:

Quoted 7 lines
Quoted 3 lines

FLR updates the device context.
Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid

any race.

Heh well but if drivers depend on specific behaviour then we really need to document that in the spec.

I am working on v2 enrich the spec language for missing items to address comments of yours and Jason.

#16 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

In reply to Parav Pandit
Quoted 80 lines · Parav Pandit
Quoted 80 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, October 11, 2023 8:44 AM

Quoted 43 lines · Parav Pandit
Quoted 43 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 10, 2023 11:21 AM

Quoted 37 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, October 9, 2023 2:19 PM

Adding LingShan.

Thanks for adding him.

Quoted 2 lines

Parav, if you want any specific people to comment, please do cc them.

Sure, will cc them in v2 as now I see there is interest in the review.

Quoted 9 lines · Parav Pandit
Quoted 3 lines

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework such as vfio [1].

Mentioning a specific subsystem in a specific OS may mislead the user to think it can only work in that setup. Let's not do that, virtio is not only used for Linux and VFIO.

Not really. it is an example in the cover letter.
It is not the only use case.
A use case gives a crisp clarity of what UAPI it needs to fulfil. So I will keep it. It is anyway written as one use case.

Quoted 7 lines
Quoted 3 lines

A passthrough PCI VF device is fully owned by the virtual machine device driver.

Is this true? Even VFIO needs to mediate PCI stuff. Or how do you define "passthrough" here?

Other than PCI config registers and due to some legacy, msix. The "device interface" side is not mediated.
The definition of passthrough here is: To not mediate a device type specific

Quoted 6 lines
Quoted 5 lines

and virtio specific interfaces for modern and future devices.

Ok, but what's the difference between "device type specific" and "virtio specific interfaces". Maybe an example for this?

Virtio device specific means: cvq of crypto device, cvq of net device, flow filter

vqs of net device etc.

Quoted 1 line

Virtio specific interface: virtio driver notifications, virtio virtqueue and

configuration mediation etc.

Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset

How about other PCI stuff? Or Why is FLR special?

FLR is special for the readers to get the clarity that FLR is also done by the

guest driver hence, the device migration commands do not interact/depend with FLR flow.

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and #2.

We are discussing the possibility in software/driver side isn't it?

1) is initiated from the guest
2) is initiated from the hypervisor

Both are softwares, and you're saying 2) should not happen after 1) since the device knows what is being done by guests? How can devices control software behaviour?

This only possible thing is to make sure 3) is done before 2) That is what I'm asking but you are saying freeze doesn't need to wait for FLR...

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

But ok, since we active to freeze mode change is allowed, lets discuss above.

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and other.

Great. So we have a perfect race:

1) guest initiates FLR
2) device start FLR
3) hypervisor stop and freeze the device
4) device is freeze
5) hypervisor read device context A
6) migrate device contextA
8) migration is done
9) FLR is done
10) hypervisor read device context B

So we end up with inconsistent device context, no? Dest want B or A+B, but you give A.

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by FLR.

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

I didn’t say clear the context.
FLR updates the device context.

In what sense?

Quoted 2 lines

Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid any race.

You want to leave those details for the vendor to figure out? If devices know everything, why do we need device normative?

I see issues at least for FLR, I'm pretty sure they are others. If a design requires us to audit all the possible conflicts between virtio facilities and transport. It's a strong hint of layer violation and when it happens it for sure may hit a lot of problems that are very hard to find or debug thus we should drop such a design. I suggest using the RFC tag since the next version (if there is one) as I see it is immature in many ways.

What's more, solving races is much easier if the device functionality is self contained. For example, for a self contained device with the transport as the single interface, we can leverage from transport (PCI) for dealing with races, arbitration, ordering, QOS etc which is probably required in the internal channel between the owner and the member. But all of these were missed in your series and even if you can I'm not sure it's worthwhile to reinvent all of them.

For example, for the architecture like owner/member, if the virtio or transport facility could be controlled via device internal channels besides the transport, such a channel may complicate the synchronization a lot. The device needs to be able to handle or synchronize requests from both PCI and owner in parallel. They are just too many possible races and most of my questions so far come from this viewpoint. I wouldn't go further for other stuff since I believe I've spotted sufficient issues and that's why I must stop at this patch before looking at the rest.

Admin commands are fine if it does real administrative jobs such as provisioning since such work is beyond the core virtio functionality.

Again, the goal of virtio spec is to have a device with sufficient guidelines that is easy to implement but not leave the vendors to waste their engineering resources in figuring or fuzzing the corner cases.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

This question is ignored.

Quoted 39 lines
Quoted 39 lines
Quoted 2 lines

Secondly since the FLR is local to the source, intermediate sub state does not

migrate.

Quoted 36 lines

But I agree, it is worth to have the text capturing this.

Quoted 24 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

and rest of the virtio device functionality such as control vq,

Quoted 11 lines
Quoted 2 lines

What do you mean by "rest of"?

As given in the example cvq.

Quoted 1 line

Which part is not controlled and why?

Not controlled because as states, it is passthrough device.

Quoted 4 lines
Quoted 4 lines

config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely

used.

Quoted 4 lines
Quoted 3 lines

Why is this mentioned here?

Huh. You should be positive for bringing clarity to the readers on

understanding the use case.

Quoted 2 lines

And you seem opposite, but ok.

Quoted 8 lines
Quoted 2 lines

As stated, it for the reader to understand the use case and see how proposed

commands addresses the use case.

The problem is that the hardware features should be designed for a general purpose instead of a specific technology if it can. The only missing part for post copy is the page fault.

Ok. The use case and requirement of member device passthrough is clear to

Quoted 8 lines
Quoted 5 lines

most reviewers now.

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

You need to describe this somewhere, no? That's what I'm saying.

And passthrough is misleading here.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Ok. I assume "reset flow" is clear to you now that it points to section 2.4. This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 1 line

I will link to the section anyway.

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the PCIe transport.

Ok, I'm not a native speaker, but I really don't know the difference between "FLR" and "FLR flow".

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the question.

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

See my above question.

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real devices.

That would be some other trap alternative that needs to dissect the device and build infrastructure for such dissection is not desired in the listed use case.

Do you need to trap FLR or not? You're saying the hypervisor is in charge of vPCI, how is this differ to what you proposed? If not, how can vPCI be composed?

I believe you need to document how vpci is supposed to be done, since I believe your proposal can only work with such specific types of PCI composition. This is one of the important things that is missed in this series.

Quoted 1 line

Here we are addressing the requirement of passthrough the device.

I don't think what you proposed here is passthrough, at least the PCI part is not. And whether or not the virto can be passthrough is still questionable to me.

Quoted 27 lines

So your disagreement is fine for non-passthrough devices.

Quoted 24 lines
Quoted 1 line

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 2 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 19 lines

Not normative line.

Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Quoted 6 lines

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

As author, what we think, I will cover. If you have specific points to add value, please share, I will look into it.

I've pointed out sufficient issues. I have a lot of others but I don't want to have a giant thread once again.

Quoted 5 lines
Quoted 3 lines

For example, how is the power management interaction with the freeze/stop?

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

I don't think this is a good answer. I'm asking how the PM interacts with freeze/stop, you answer it works well.

I'm not obliged to design hardware for you but figuring out the bad design for virtio. I'm not convinced with a proposal that misses a lot of obvious critical cases and for sure it's not my job to solve them.

I've demonstrated the possible races with FLR. So did the PM. For example, if VF is in D3cold state, can we still read its device context? If yes, is it a violation of the PCIE spec? If not, why? How about other states? Can the device be freezed in the middle of PM state transitions? If yes, how can it work without migrating PCI states?

Quoted 101 lines
Quoted 101 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines

It is not written in this series, because we believe it must not be synchronized

as it is fully controlled by the guest.

Quoted 11 lines
Quoted 11 lines

Review

Quoted 10 lines

at the same time such flows must not obstruct

(quoted diff) +7 -0
+the device migration flow. In such a scenario, a group owner
+device can provide the administration command interface to
+facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to
+another hypervisor, these hypervisors are named as source and
+destination

hypervisor respectively.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+In such a scenario, a source hypervisor administers the
+member device to suspend the device and preserves the device

context.

Quoted 40 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+Subsequently, a destination hypervisor administers the member
+device to setup a device context and resumes the member device.
+The source hypervisor reads the member device context and the
+destination hypervisor writes the member device context. The
+method to transfer the member device context from the source
+to the destination hypervisor is

outside the scope of this specification.

Review

Quoted 4 lines
(quoted diff) +4 -0
+
+The member device can be in any of the three migration modes.
+The owner driver sets the member device in one of the
+following modes during

device migration flow.

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+\begin{tabularx}{\textwidth}{ |l||l|X| } \hline Value & Name
+& Description \\ \hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member
+device. \\

I don't think we ever define "instantiation" anywhere.

Quoted 7 lines
Quoted 3 lines

Well a transport has implicit definition of the instantiation already. May be a text can be added, but don’t see a value in duplicating PCI spec

here.

Ok, maybe something like "transport specific instantiation"

Ok. that’s a good text. I will change to it.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

+\hline
+0x1 & Stop &
+ In this mode, the member device does not send any +notifications, and it does not access any driver memory.

What's the meaning of "driver memory"?

May be guest memory? Or do you suggest a better naming for the memory

Quoted 38 lines
Quoted 4 lines

allocated by the guest driver?

Virtqueue?

Virtqueue and any memory referred by the virtqueue.

This is good text, I will change to it.

Quoted 30 lines
Quoted 7 lines
Quoted 3 lines

And stop seems to be a source of inflight buffers.

I didn’t follow it.
If you mean without stop there are no inflight buffer, then I don’t agree. We don’t want to violate the spec by having descriptors with zero size

returned.

Quoted 1 line

Stop is not the source of inflight descriptors.

I think not since you forbid access to the used ring here. So even if the buffer were processed by the device it can't be added back to the used ring thus became inflight ones.

Quoted 3 lines

There are inflight descriptors with the device that are not yet returned to the

driver, and device wont return them as zero size wrong completions.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

+ The member device may receive driver notifications in this
+ mode,

What's the meaning of "receive"? For example if the device can still process buffers, "stop" is not accurate.

Receive means, driver can send the notification as PCIe TLP that device may

receive as incoming PCIe TLP.

Ok, so this is the transport level. But the device can keep processing the

queue?

Quoted 2 lines
Quoted 1 line

Device cannot process the queue because it does not initiate any read/write

Quoted 8 lines
Quoted 6 lines

towards the virtqueue.

Read/Write only results in a driver noticeable behaviour, it doesn't mean the device can't process the buffers. For example, devices can keep processing available buffers and make them as inflight ones.

The idea is to stop the device and prepare for the migration, so the command to do so. Otherwise just the keep the device in active mode and avoid the complications.

Well, I meant we need a more precise definition of each state otherwise it could be ambiguous (as I pointed above).

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

Quoted 8 lines
Quoted 6 lines

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to update as result of driver notification. For example caching the posted avail index in the notification.

A link to those proposals? If the device must depend on those cached features to work it's really fragile. If not, we don't need to care about them.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

No, it does not.
Device context is updated within the device without accessing the queue memory of the guest.

This is not documented or explained anywhere?

Quoted 38 lines
Quoted 3 lines

Maybe you can give a concrete example.

The above one.

Quoted 33 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

Quoted 8 lines
Quoted 6 lines

"""
+The member device has a device context which the owner driver can
+either read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 7 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

Quoted 7 lines
Quoted 2 lines

And again, the more I review the more I see how narrow this series can be used:

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

That is exactly what I want to say, it can only work for the owner/member model. It can't work when the virtio device is not structured like that. And you missed that most of the existing virtio devices are not implemented in this model. It means they can't be migrated with a pure virtio specific extension. For you, SR-IOV is all but this is not true for virtio. PCI is not the only transport and SR-IOV is not the only architecture in PCI.

And I'm pretty sure the owner/member is not the only requirement, there are a lot of other assumptions which are missed in this series.

Quoted 11 lines
Quoted 10 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented anywhere. And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

It's not about VFIO at all. It's about to let people know under which case this proposal could work. Otherwise if a vendor develops a BAR/cap which is not at page boundary. How could you make it work with your proposal here?

Quoted 18 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 1 line

or it may experience a read from the shared memory and get garbage

data.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

MMIO read/write can be initiated by the peer device while the device is in stopped state.

Ok, but what I want to say is how it can get the garbage data here?

Quoted 35 lines
Quoted 35 lines

Btw, shared regions are tricky for hardware.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any driver +notifications,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be migrated easily.

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

6) driver read stats and realize FEATURES_OK is being cleared

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Because hypervisor has stopped the vcpus which are issuing them.

MMIO are posted. vCPU is stopped but the transactions are inflight. How could the hypervisor/device know if there's any inflight PCIE transactions here? So I can imagine what happens in fact is the TLP for freezing is ordered with the TLP for posted MMIO. This is probably guaranteed for typical PCIE setup but how about the relaxed ordering?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

What can happen between #2 and #3, is device mode may change to stop.

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

On the source hypervisor, the mode changes are active->stop->freeze.
Hence when freeze is done, the hypervisor knows that all inflight has been stopped by now.

Ok, but how about freezing between 3) and 4). If we allow it, do we need to migrate to this state? If yes, how can it work with your device context? If not, shouldn't we document this?

Quoted 7 lines
Quoted 5 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Ok, it means:

1) there're conditions from stop to freeze, then what are they?
2) how much time at most? E.g FLR takes at most 100ms.
3) If it needs more time, can this time satisfy the downtime requirement?

Quoted 12 lines
Quoted 12 lines
Quoted 3 lines

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

machine.

Quoted 1 line

First, above is not a state machine.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

So question is "how reader knows if a state can go to another state and when"? It is described and listed in the table, when a mode can change.

It's not only "if" but also "when". Your table partially answers the "if '' but not "when". I think you should know now the state transition is conditional. So let's try our best to ease the life of the vendor.

Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines

So only the driver notification is allowed by not config write? What's the consideration for allowing driver notification?

Because for most practical purposes, peer device wants to queue blk, net

other requests and not do device configuration.

You forbid the device to process the queue but only allow the notification. How can the device queue those requests? The device can just do the available buffer check after resume, then it's all fine.

Quoted 5 lines

Device can always decide to not queue the request and do the available buffer check later. The peer device may read also from MMIO space.

So the intermediate step covers this aspect where device_type specific plumbing is not done. Its generic. A device may choose to omit such doorbells as well as long as it knows it can resume.

I'm not sure I will get here, but the device doesn't need to be kicked after resume. That's what I want to say.

Quoted 16 lines
Quoted 9 lines
Quoted 4 lines

Do you know any device configuration space which is RW? For net and blk I recall it as RO?

For example, WCE. What's more important, the spec allows config space to be RW, so even if there's no examples before, it doesn't mean we won't have a RW in the future.

Ok.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Let me ask differently, similar to FLR, what happens if the driver wants a virtio reset but the hypervisor wants to stop or freeze?

The device would respond to stop/freeze request when it has internally

Quoted 8 lines
Quoted 7 lines

started the reset, as device is the single synchronization point which knows how to handle both in parallel.

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what is done in this patch.

Synchronization point = device.

This is obvious as we can't rule stuff outside virtio, and we are talking about devices not drivers here. But the spec needs sufficient guidance/normative for the vendor to implement. It's more than just saying "device is synchronization point".

Quoted 147 lines
Quoted 43 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

We would enrich the device context for this, but no need to connects the

admin mode controlled by the owner device with operational state (device_status) owned by the member device.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ it ignores any device configuration space writes,

How about read and the device configuration changes?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

As listed, device do not have any changes.
So device configuration change cannot occur.

It's not necessarily caused by config write, it could be things like link status or geometry changes that are initiated from the device.

I understand it. Link status was one example, you listed other examples too. The point is, when in freeze mode, the member device is frozen, hence,

device won't initiate those changes.

Quoted 23 lines
Quoted 23 lines
Quoted 8 lines

The device requirements cover this content more explicitly:

For the SR-IOV group type, regardless of the member device mode, all the PCI transport level registers MUST be always accessible and the member device MUST function the same way for all the PCI transport level

registers regardless of the member device mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ the device do not have any changes in the device context.
+ The member device is not accessed in the system through the
+ virtio

interface.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

+ \\

But accessible via PCI interface?

Yes, as usual.

Quoted 3 lines

For example, what happens if we want to freeze during FLR? Does the hypervisor need to wait for the FLR to be completed?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

Hypervisor do not need wait for the FLR to be completed.

So does FLR change device context?

Yes.

So this implies the freeze needs to wait for FLR otherwise device context may change.

Device context can change anytime and reflect what is latest.
I will update the patches to reflect that device is the single synchronization point serving flr, mode changes.

Quoted 93 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 8 lines

+\hline
+\hline
+0x03-0xFF & - & reserved for future use \\
+\hline
+\end{tabularx}

Review

Quoted 7 lines
(quoted diff) +7 -0
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to
+\field{Stop}. Once the device is in the \field{Stop} mode,
+the device does not initiate any notifications or does not
+access any driver memory. Since the member driver may be
+still active which may send further driver notifications to the device,

the device context may be updated.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+When the member driver has stopped accessing the device, the
+owner driver sets the device to \field{Freeze} mode
+indicating to the device that no more driver access occurs.
+In the \field{Freeze} mode, no more changes occur in the device

context.

Quoted 34 lines
Quoted 34 lines
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+At this point, the device ensures that

there will not be any update to the device context.

What is missed here are:

1) it is a virtio specific states or not

It is not.

Quoted 7 lines
Quoted 4 lines

2) if it is a virtio specific state, if or how to synchronize with transport specific interfaces and why
3) can active go directly to freeze and why

Yes. don’t see a reason to not allow it.
Active to freeze mode can change is useful on the destination side, where

destination hypervisor knows for sure that there is no other entity accessing the device.

Quoted 11 lines

And it needs to setup the device context, it received from the source side. So setting freeze mode can be done directly.

Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+The member device has a device context which the owner driver
+can either read or write. The member device context consist
+of any device specific data which is needed by the device to
+resume its operation when the device mode

This is too vague. There're states that are not suitable for cmd/queue for

sure.

Quoted 5 lines
Quoted 5 lines

I'd split it into

1) common states: virtqueue, dirty pages
2) device specific states: defined be each device

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

This is theory of operation section. So it capturing such details. Actual device context definition is outside of theory, and precise states of

virtqueue, device specific, etc are in it.

See my comment above regarding to the device context.

I replied above, device context link is added in the patch-3 in the theory of

operation.

Quoted 18 lines

So reader gets the complete view.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +4 -0
+is changed from \field{Stop} to \field{Active} or from
+\field{Freeze} to \field{Active}.
+
+Once the device context is read, it is cleared from the device.

This is horrible, it means we can't easily

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

1) re-try the migration
2) recover from migration failure

Can you please explain the flow?

When migration fails, management can choose to resume the device(VM) on the source.

Quoted 7 lines

ok. This should be possible as the management which has the device context, it can restore it on the source and move the device mode to active.

Quoted 3 lines

If the state were cleared, it means there's not simple way to resume the device but restoring the whole context.

Yes, as you say, by restoring the whole context will suffice this corner/rare

case scenario.

Quoted 3 lines
Quoted 3 lines

What's the consideration for such clearing?

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

There are two considerations.
1. If one does not clear, till how long should it be kept on the device?

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

If kept, hypervisor may not be able to decide when to change the mode from active->stop.

Why? It is simply done when mgmt requires a migration?

What's more important, PCI allows multiple common_cfgs. So the hypervisor can choose to reserve one common_cfg for live migration. In this case we don't have to read to clear semantics.

Or, are you saying the value read from common_cfg is not device context? Isn't this conflict with your vague definition of device context?

Quoted 10 lines

We can opt for a mode where full device context is read in each mode without clearing it. But than it can be very specific to a version of qemu, which we are avoiding it here.

Quoted 5 lines
Quoted 1 line

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Not sure which devices you mean here with "most of the devices". Device context functions like a write record pages (aka dirty pages).

It's definitely different. We want to migrate dirty pages lively which can consume a lot of bandwidth. So reporting delta makes a lot of sense here since it would have a lot of rounds of syncing and it doesn't result in blockers resuming.

For device context, how many rounds of syncing did you expect, and if we have N rounds, we need to restore N rounds in order to resume? Do you want to live migrating device states? If it's only 1 or 2 rounds, why bother?

And for the delta, how do you know you can easily define deltas for every type of device, especially the ones with complicated internal states? Defining states has already been demonstrated as a complicated task for some devices like virtio-FS and you want to complicate it furtherly?

What is proposed in this series is an ad-hoc optimization for a specific deivce type within a specific subsystem (e.g VFIO) in a specific operating system which is not the general.

As demsonsted many times, starting from something simple and stupid is the most easy way.

Quoted 6 lines

Whatever is already returned is/should not be repeated in subsequent reads, though device can choose to do so.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Libvirt.

Does libvirt restore on migration failure?

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

I disagree, see my above reply.

Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique to virtio.

It is theory of operation that brings the clarity and rationale.

I think it's not. Since it's not something that is unique to virtio.

Quoted 96 lines

So I will keep it.

Quoted 53 lines
Quoted 53 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

Review

Quoted 10 lines
(quoted diff) +7 -0
+
+Typically, the device context is read and written one time on
+the source and the destination hypervisor respectively once
+the device is in \field{Freeze} mode. On the destination
+hypervisor, after writing the device context, when the device
+mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 28 lines
Quoted 8 lines
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the owner
+driver may choose to read the device context first time while
+the device is in \field{Active} mode and second time once the
+device is in \field{Freeze}

mode.

Quoted 7 lines
Quoted 3 lines

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device context change?

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or not. No?

Device does not need to know, because when the guest vm and the device is resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 31 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 7 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first time
+while the device is still running in \field{Active} mode on
+the source hypervisor and writes the device context second
+time while the device is in

\field{Freeze} mode.

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the device
+context likely have minimal changes from the previously
+written device

context.

Quoted 5 lines
Quoted 4 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec wise.

Quoted 7 lines
Quoted 3 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 2 lines

" To help with these optimizations, when VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 2 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

Quoted 8 lines
Quoted 7 lines

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of migration. You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

It is one of the optimization apart. The comparison is of one_of_example or not.

I don't get this.

Thanks

Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#17 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 3 lines

From: Jason Wang <[email protected]>
Sent: Friday, October 13, 2023 6:46 AM

[..]

Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and #2.

Quoted 9 lines

We are discussing the possibility in software/driver side isn't it?

1) is initiated from the guest
2) is initiated from the hypervisor

Both are softwares, and you're saying 2) should not happen after 1) since the device knows what is being done by guests? How can devices control software behaviour?

Device do not control software behavior.
i.e. either hypervisor can initiate device mode change to stop (not freeze) or guest can initiate FLR. Device knows which is initiated first as single recipient of both.
Therefore, device responds accordingly.
For example, in the sequence you described,
A device will delay mode change command response, until the FLR is completed.

Quoted 3 lines

This only possible thing is to make sure 3) is done before 2) That is what I'm asking but you are saying freeze doesn't need to wait for FLR...

I think I responded in previous email further down on synchronization point being fw. I meant to say software do not need to wait for initiation of the freeze mode command. Just the command will complete at right time.

This is anyway very corner case.
On source hypervisor as written in the theory of operation, the sequence is active->stop->freeze. When mode change is done to stop, the vcpus are already suspended.

I agree FLR may have been initiated and driver is waiting now for 100msec.

So yes, device single entity synchronized it.

Quoted 26 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But ok, since we active to freeze mode change is allowed, lets discuss

above.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and other.

Great. So we have a perfect race:

1) guest initiates FLR
2) device start FLR
3) hypervisor stop and freeze the device
4) device is freeze
5) hypervisor read device context A
6) migrate device contextA
8) migration is done
9) FLR is done
10) hypervisor read device context B

Quoted 3 lines

So we end up with inconsistent device context, no? Dest want B or A+B, but you give A.

Since #1 and #2 is done before #3, the device knows to finish the FLR, hence #9 is completed before #4.

Alternatively, in above sequence when destination sees #10, it can immediately finish the FLR as dest device is not under FLR, treating it as no-op.

Both ways to handle are fine. (and rare in practice, but yes, its possible).

I will write both the options in the device requirements.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by FLR.

Quoted 9 lines
Quoted 6 lines

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

I didn’t say clear the context.
FLR updates the device context.

In what sense?

Indicating a new device context indicating a new device context and discard the old one. I am glad you asked this. I wanted to get the basic part captured before adding this optimization. Probably it is good to add it now in the v2 as we crossed this stage now.

Quoted 7 lines
Quoted 2 lines

Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid

any race.

You want to leave those details for the vendor to figure out? If devices know everything, why do we need device normative?

Device knows its implementation.
Implementation guidelines to be in the normative.
I will add it to the normative.

Quoted 7 lines

I see issues at least for FLR, I'm pretty sure they are others. If a design requires us to audit all the possible conflicts between virtio facilities and transport. It's a strong hint of layer violation and when it happens it for sure may hit a lot of problems that are very hard to find or debug thus we should drop such a design. I suggest using the RFC tag since the next version (if there is one) as I see it is immature in many ways.

Technical committee audits the required touch points like rest of the industry committees that I participated. I disagree to your above point.
If you do not want to review, that is fine.
We are reviewing with other members and also contributed by them.

Quoted 8 lines

What's more, solving races is much easier if the device functionality is self contained. For example, for a self contained device with the transport as the single interface, we can leverage from transport
(PCI) for dealing with races, arbitration, ordering, QOS etc which is probably required in the internal channel between the owner and the member. But all of these were missed in your series and even if you can I'm not sure it's worthwhile to reinvent all of them.

At the end there is one physical device serving owner and member devices. So a claim like things are on the VF hence you magically get 200% QoS guarantee is myth.

Quoting "all of these" is also incorrect.

Things added gradually, first functionally with reasonable performance, followed by notion and extension for QoS. By definition of PCI transport for SR-IOV there is internal channel.

It is reasonably well proposal in current form.
There are few race condition that you highlight are extremely rare in nature. Suggestions are welcome to improve.
There were couple of them by Michael too, I am addressing them in the v2.

Quoted 3 lines

For example, for the architecture like owner/member, if the virtio or transport facility could be controlled via device internal channels besides the transport, such a channel may complicate the synchronization a lot.

Two vendors who actually make the hw sriov devices are authoring these and others are also reviewing.
So I am more confident that it is solid enough.
Also, a similar design has been seen with other device for more than a year as GPL integrated with QEMU for a year now and with upstream kernel.

Quoted 6 lines

The device needs to
be able to handle or synchronize requests from both PCI and owner in parallel. They are just too many possible races and most of my questions so far come from this viewpoint. I wouldn't go further for other stuff since I believe I've spotted sufficient issues and that's why I must stop at this patch before looking at the rest.

It is your call to stop or progress.
I find your reviews useful to improve this proposal, so I will fix them.

Quoted 7 lines

Admin commands are fine if it does real administrative jobs such as provisioning since such work is beyond the core virtio functionality.

Again, the goal of virtio spec is to have a device with sufficient guidelines that is easy to implement but not leave the vendors to waste their engineering resources in figuring or fuzzing the corner cases.

I have not seen an industry standard spec or a software that does not have corner cases. The spec proposal is from > 1 device vendors.

I will focus on more practical aspects to progress and improve this spec.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

This question is ignored.

I probably missed. Sorry about it.
No, the driver does not need to wait for FLR to finish to issue resume command, as this typically done on the destination member device which should not be under FLR. I will write up the requirements further.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

You need to describe this somewhere, no? That's what I'm saying.

Mostly not. What is not done is not written.

Quoted 2 lines

And passthrough is misleading here.

Passthrough is mentioned in theory of operation.
It is not present in requirements section.
So, it is fine.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Ok. I assume "reset flow" is clear to you now that it points to section 2.4. This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

I will link to the section anyway.

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the

PCIe transport.

Ok, I'm not a native speaker, but I really don't know the difference between "FLR" and "FLR flow".

Lets keep it simple. I will write it as FLR, as pci transport has it as FLR.

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the

question.

Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

Quoted 7 lines
Quoted 5 lines

It's not about mediation at all, it's about how the device can implement what you want here correctly.

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

Quoted 9 lines
Quoted 3 lines
Quoted 2 lines

I disagree, hypervisors can emulate FLR and never send FLR to real devices.

That would be some other trap alternative that needs to dissect the device

and build infrastructure for such dissection is not desired in the listed use case.

Do you need to trap FLR or not? You're saying the hypervisor is in charge of vPCI, how is this differ to what you proposed? If not, how can vPCI be composed?

Live migration driver do not need to trap FLR.

Quoted 4 lines

I believe you need to document how vpci is supposed to be done, since I believe your proposal can only work with such specific types of PCI composition. This is one of the important things that is missed in this series.

I don’t see a need to describe vpci composition as there may be more than one way to do it. What I think it is worth to describe is the whole pci device is not stored in device context. I will try to add a short description around it.

Quoted 31 lines
Quoted 31 lines

So your disagreement is fine for non-passthrough devices.

Quoted 28 lines
Quoted 2 lines

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 3 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 20 lines

Not normative line.

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

As author, what we think, I will cover. If you have specific points to add value,

please share, I will look into it.

I've pointed out sufficient issues. I have a lot of others but I don't want to have a giant thread once again.

I see following things to improve in the requirements which I will do in v2.

1. Document race around FLR and admin commands for really rare corner case.
2. Some text around not migrating the pci device registers
3. Interaction with PM commands

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

For example, how is the power management interaction with the

freeze/stop?

Quoted 3 lines
Quoted 1 line

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

Quoted 8 lines

I don't think this is a good answer. I'm asking how the PM interacts with freeze/stop, you answer it works well.

I'm not obliged to design hardware for you but figuring out the bad design for virtio. I'm not convinced with a proposal that misses a lot of obvious critical cases and for sure it's not my job to solve them.

I am not asking you to solve.

Quoted 2 lines

I've demonstrated the possible races with FLR. So did the PM. For example, if VF is in D3cold state, can we still read its device context?

I think yes, but I will double check.
If yes, is it a violation of the PCIE spec? If not, why?
No, because device context is owned by the owner device and not the VF. SR-PCIM interface has defined it be outside of scope of PCIe spec.

Quoted 3 lines

How about other states? Can the device be freezed
in the middle of PM state transitions? If yes, how can it work without migrating PCI states?

I will double check, but unlikely, it should be similar to FLR case to keep the device to avoid treating it differently.

Quoted 2 lines

Well, I meant we need a more precise definition of each state otherwise it could be ambiguous (as I pointed above).

Ok. so, few things about read and other messages, I will add.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to

update as result of driver notification.

Quoted 1 line

For example caching the posted avail index in the notification.

A link to those proposals?

[1] https://lists.oasis-open.org/archives/virtio-comment/202310/msg00048.html

Quoted 2 lines

If the device must depend on those cached features to work it's really fragile. If not, we don't need to care about them.

It is not dependent.
It is the infrastructure to enable it.
Same for other shared memory region accesses.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

This is in conflict with what you said above " Device cannot process the queue ..."

No, it does not.
Device context is updated within the device without accessing the queue

memory of the guest.

This is not documented or explained anywhere?

Why should it be explained?
device is not accessing the guest memory -> this is mentioned in stop mode. Hence, there is no need to write above.

Quoted 50 lines
Quoted 42 lines
Quoted 3 lines

Maybe you can give a concrete example.

The above one.

Quoted 37 lines
Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Quoted 9 lines

Like this?

"""
+The member device has a device context which the owner driver
can
+either read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Quoted 3 lines

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 8 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be

used:

Quoted 6 lines
Quoted 1 line

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

Quoted 7 lines

That is exactly what I want to say, it can only work for the owner/member model. It can't work when the virtio device is not structured like that. And you missed that most of the existing virtio devices are not implemented in this model. It means they can't be migrated with a pure virtio specific extension. For you, SR-IOV is all but this is not true for virtio. PCI is not the only transport and SR-IOV is not the only architecture in PCI.

Each transport will have its own way to handle it.
When there is MMIO owner-member relationship arise, one will be able to do so as well. In fact other transports will likely miss out as they have not established such pace.

Quoted 3 lines

And I'm pretty sure the owner/member is not the only requirement, there are a lot of other assumptions which are missed in this series.

One proposal does not do everything.
It is just impractical.

Quoted 11 lines
Quoted 7 lines
Quoted 7 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented

anywhere.

Quoted 3 lines
Quoted 3 lines

And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Quoted 7 lines
Quoted 2 lines

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

It's not about VFIO at all. It's about to let people know under which case this proposal could work. Otherwise if a vendor develops a BAR/cap which is not at page boundary. How could you make it work with your proposal here?

Vendor is a cloud operator which is building the device, so it will always work it has the matching capabilities on source and destination.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 2 lines

or it may experience a read from the shared memory and get garbage

data.

Quoted 3 lines
Quoted 3 lines

Could you give me an example for this?

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

MMIO read/write can be initiated by the peer device while the device is in

stopped state.

Ok, but what I want to say is how it can get the garbage data here?

If the device mode is changed to freeze while it is being read by the peer device, it can get garbage data or last data. Which may not be the one that is expected.
So first all the initiator devices are stopped, ensure that they do not make any requests.

And there are requests, which gets proper answer.

Quoted 45 lines
Quoted 13 lines
Quoted 13 lines

Btw, shared regions are tricky for hardware.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any +driver notifications,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be

migrated easily.

Quoted 26 lines
Quoted 26 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 5 lines
Quoted 3 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Quoted 7 lines
Quoted 1 line

Because hypervisor has stopped the vcpus which are issuing them.

MMIO are posted. vCPU is stopped but the transactions are inflight. How could the hypervisor/device know if there's any inflight PCIE transactions here? So I can imagine what happens in fact is the TLP for freezing is ordered with the TLP for posted MMIO. This is probably guaranteed for typical PCIE setup but how about the relaxed ordering?

Vcpus do not generated relaxed ordering MMIOs.
In pci spec: " If this bit is Set, the Function is permitted to set the Relaxed Ordering bit in the Attributes field of transactions it initiates".

Function initiates RO requests, not the vcpu.
Hence, it is fine.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

What can happen between #2 and #3, is device mode may change to stop.

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

Quoted 8 lines
Quoted 2 lines

On the source hypervisor, the mode changes are active->stop->freeze. Hence when freeze is done, the hypervisor knows that all inflight has been

stopped by now.

Ok, but how about freezing between 3) and 4). If we allow it, do we need to migrate to this state? If yes, how can it work with your device context? If not, shouldn't we document this?

May be, some of these are implementation details. I am not sure it belongs to spec. Like RSS update while packets are received.. such implementation details are not part of the spec.

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Ok, it means:

1) there're conditions from stop to freeze, then what are they?

No, there isn’t condition.
May be I didn’t follow the question.

Quoted 1 line

2) how much time at most? E.g FLR takes at most 100ms.

From the driver side, it is 100msec for device side it can be less too. As soon as FLR is done or enough to record it, is done, stop can continue.

Quoted 2 lines

3) If it needs more time, can this time satisfy the downtime requirement?

Guest VM for all practical purposes is not busy in doing FLR, it is a corner case, yet we have to cover it. And yes, it satisfy the downtime requirements, because VM is already not interested in the packets, it is busy doing the FLR.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 4 lines

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

Quoted 7 lines
Quoted 1 line

So question is "how reader knows if a state can go to another state and

when"?

Quoted 1 line

It is described and listed in the table, when a mode can change.

It's not only "if" but also "when". Your table partially answers the "if '' but not "when". I think you should know now the state transition is conditional. So let's try our best to ease the life of the vendor.

What do you mean when?
I do not understand that "mode change is conditional"? it is not based on the condition. [..]

Quoted 11 lines
Quoted 5 lines
Quoted 4 lines

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what is done in this patch.

Synchronization point = device.

This is obvious as we can't rule stuff outside virtio, and we are talking about devices not drivers here. But the spec needs sufficient guidance/normative for the vendor to implement. It's more than just saying "device is synchronization point".

The requirements are already covering what device needs to do. Some interaction points are missing, as I acked above, I will add them.

[..]

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

If kept, hypervisor may not be able to decide when to change the mode from

active->stop.

Why? It is simply done when mgmt requires a migration?

Mgmt is bit higher level entity. Underneath the software layers may wait until the time is right to migrate. The fundamental point is, the device context is expected to return the incremental value, that is changed content from last time. So once all changed content is read, its empty.

Quoted 3 lines

What's more important, PCI allows multiple common_cfgs. So the hypervisor can choose to reserve one common_cfg for live migration. In this case we don't have to read to clear semantics.

Common_cfg does not serve large device context, nor it serves DMA.

Quoted 2 lines

Or, are you saying the value read from common_cfg is not device context?

The value of common config is part of the device context that represents current common config.

Quoted 2 lines

Isn't this conflict with your vague definition of device context?

You mentioned you stop at this patch, so likely you didn’t read device context patch, hence you quote it vague. So I don’t know what you mean by vague.
Please let me know what you additional thing you want to see in device context after you reach that patch.

Quoted 11 lines
Quoted 1 line

We can opt for a mode where full device context is read in each mode

without clearing it.

Quoted 1 line

But than it can be very specific to a version of qemu, which we are avoiding it

here.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Quoted 8 lines
Quoted 2 lines

Not sure which devices you mean here with "most of the devices". Device context functions like a write record pages (aka dirty pages).

It's definitely different. We want to migrate dirty pages lively which can consume a lot of bandwidth. So reporting delta makes a lot of sense here since it would have a lot of rounds of syncing and it doesn't result in blockers resuming.

Write records are reported as delta from the previous read.

Quoted 4 lines

For device context, how many rounds of syncing did you expect, and if we have N rounds, we need to restore N rounds in order to resume? Do you want to live migrating device states? If it's only 1 or 2 rounds, why bother?

Live migrate the device context. Typically in current software using it, it is 2 rounds. The interface is generic that if needed more rounds are possible.

Even device for most practical purpose will implement 2 rounds.

Quoted 5 lines

And for the delta, how do you know you can easily define deltas for every type of device, especially the ones with complicated internal states? Defining states has already been demonstrated as a complicated task for some devices like virtio-FS and you want to complicate it furtherly?

What is your question? If you say virtio-fs is complicated state, may be it should not have existed itself in the virtio spec as first place. But I differ to think that.
Virtio-fs guest side state wont be changed as part of it.
Virtio-fs is the first device which has considered and listed to migrate the device state.
So it should be possible.

Quoted 4 lines

What is proposed in this series is an ad-hoc optimization for a specific deivce type within a specific subsystem (e.g VFIO) in a specific operating system which is not the general.

Oh now you mention vfio. Not me. :)

I am not going to comment on this. It is not ad-hoc.
It uses similar dirty page tracking like technique present in cpu hw and other devices.

Quoted 13 lines

As demsonsted many times, starting from something simple and stupid is the most easy way.

Quoted 1 line

Whatever is already returned is/should not be repeated in subsequent reads,

though device can choose to do so.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Does libvirt restore on migration failure?

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

I disagree, see my above reply.

I replied above.

Quoted 141 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique to virtio.

It is theory of operation that brings the clarity and rationale.

I think it's not. Since it's not something that is unique to virtio.

Quoted 42 lines

So I will keep it.

Quoted 40 lines
Quoted 40 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Review

Quoted 11 lines
(quoted diff) +8 -0
+
+Typically, the device context is read and written one
+time on the source and the destination hypervisor
+respectively once the device is in \field{Freeze} mode.
+On the destination hypervisor, after writing the device
+context, when the device mode set to \field{Active}, the
+device uses the most recently set device context and
+resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 14 lines
Quoted 14 lines
Quoted 11 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the
+owner driver may choose to read the device context first
+time while the device is in \field{Active} mode and
+second time once the device is in \field{Freeze}

mode.

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device

context change?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or

not.

Quoted 3 lines
Quoted 2 lines

No?

Device does not need to know, because when the guest vm and the device is

resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines

Review

Quoted 8 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first
+time while the device is still running in \field{Active}
+mode on the source hypervisor and writes the device
+context second time while the device is in

\field{Freeze} mode.

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the
+device context likely have minimal changes from the
+previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec

wise.

Quoted 13 lines
Quoted 13 lines
Quoted 9 lines
Quoted 4 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 3 lines

" To help with these optimizations, when
VIRTIO_F_NOTIFICATION_DATA has

Quoted 8 lines
Quoted 8 lines

been negotiated".

Quoted 3 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of

Quoted 8 lines

migration.

Quoted 4 lines
Quoted 3 lines

You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

It is one of the optimization apart. The comparison is of one_of_example or

not.

I don't get this.

Theory of operation is describing a flow how things are done and how the constructs are helpful to achieve it. And it is not the end of the list.
That does not mean one should not write those.

#18 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 33 lines · Parav Pandit
Quoted 4 lines

From: Jason Wang <[email protected]>
Sent: Friday, October 13, 2023 6:46 AM

[..]

Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and #2.

Quoted 9 lines

We are discussing the possibility in software/driver side isn't it?

1) is initiated from the guest
2) is initiated from the hypervisor

Both are softwares, and you're saying 2) should not happen after 1) since the device knows what is being done by guests? How can devices control software behaviour?

Quoted 6 lines

Device do not control software behavior.
i.e. either hypervisor can initiate device mode change to stop (not freeze) or guest can initiate FLR. Device knows which is initiated first as single recipient of both.
Therefore, device responds accordingly.
For example, in the sequence you described,
A device will delay mode change command response, until the FLR is completed.

Finally but ok.

Quoted 7 lines
Quoted 5 lines

This only possible thing is to make sure 3) is done before 2) That is what I'm asking but you are saying freeze doesn't need to wait for FLR...

I think I responded in previous email further down on synchronization point being fw. I meant to say software do not need to wait for initiation of the freeze mode command.

For software, did you mean the hypervisor?

Quoted 5 lines

Just the command will complete at right time.

This is anyway very corner case.
On source hypervisor as written in the theory of operation, the sequence is active->stop->freeze. When mode change is done to stop, the vcpus are already suspended.

The problem here is not the vcpu but the when FLR is being done since it may change the device context.

Quoted 2 lines

I agree FLR may have been initiated and driver is waiting now for 100msec.

For driver, did you mean the driver in the guest?

Quoted 29 lines

So yes, device single entity synchronized it.

Quoted 26 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

But ok, since we active to freeze mode change is allowed, lets discuss

above.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and other.

Great. So we have a perfect race:

1) guest initiates FLR
2) device start FLR
3) hypervisor stop and freeze the device
4) device is freeze
5) hypervisor read device context A
6) migrate device contextA
8) migration is done
9) FLR is done
10) hypervisor read device context B

Quoted 4 lines
Quoted 3 lines

So we end up with inconsistent device context, no? Dest want B or A+B, but you give A.

Since #1 and #2 is done before #3, the device knows to finish the FLR, hence #9 is completed before #4.

Ok, that's my understanding and that's why I'm asking, but you said freeze/stop doesn't need to wait for FLR before.

Quoted 16 lines

Alternatively, in above sequence when destination sees #10, it can immediately finish the FLR as dest device is not under FLR, treating it as no-op.

Both ways to handle are fine. (and rare in practice, but yes, its possible).

I will write both the options in the device requirements.

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by FLR.

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I didn’t say clear the context.
FLR updates the device context.

In what sense?

Indicating a new device context indicating a new device context and discard the old one.

For example, what will queue_address have after an FLR?

Quoted 1 line

I am glad you asked this. I wanted to get the basic part captured before adding this optimization.

Ok.

Quoted 20 lines

Probably it is good to add it now in the v2 as we crossed this stage now.

Quoted 7 lines
Quoted 2 lines

Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid

any race.

You want to leave those details for the vendor to figure out? If devices know everything, why do we need device normative?

Device knows its implementation.
Implementation guidelines to be in the normative.
I will add it to the normative.

Quoted 7 lines

I see issues at least for FLR, I'm pretty sure they are others. If a design requires us to audit all the possible conflicts between virtio facilities and transport. It's a strong hint of layer violation and when it happens it for sure may hit a lot of problems that are very hard to find or debug thus we should drop such a design. I suggest using the RFC tag since the next version (if there is one) as I see it is immature in many ways.

Quoted 3 lines

Technical committee audits the required touch points like rest of the industry committees that I participated. I disagree to your above point.
If you do not want to review, that is fine.

I don't want to hold my breath if I see something that is obviously wrong. Using RFC may help people to know that it is a draft that has something to be improved before it can be merged.

Quoted 11 lines

We are reviewing with other members and also contributed by them.

Quoted 8 lines

What's more, solving races is much easier if the device functionality is self contained. For example, for a self contained device with the transport as the single interface, we can leverage from transport
(PCI) for dealing with races, arbitration, ordering, QOS etc which is probably required in the internal channel between the owner and the member. But all of these were missed in your series and even if you can I'm not sure it's worthwhile to reinvent all of them.

At the end there is one physical device serving owner and member devices.

This doesn't happen yet. For example, a VF with adminq that can be isolated with PASID makes some sense.

Quoted 1 line

So a claim like things are on the VF hence you magically get 200% QoS guarantee is myth.

That's not my point, I'm saying VF could benefit from the e.g QOS support in PCIE. I'm not saying it's perfect.

Quoted 8 lines

Quoting "all of these" is also incorrect.

Things added gradually, first functionally with reasonable performance, followed by notion and extension for QoS. By definition of PCI transport for SR-IOV there is internal channel.

It is reasonably well proposal in current form.
There are few race condition that you highlight are extremely rare in nature.

It's not rare since there's no way to know what the guest is doing. It's actually the critical part for live migration to be correct. You are proposing migration so it must cover all those cases to make sure there is no case to make your proposal a dead end.

Quoted 1 line

Suggestions are welcome to improve.

I have given some and I will give more.

Quoted 9 lines

There were couple of them by Michael too, I am addressing them in the v2.

Quoted 3 lines

For example, for the architecture like owner/member, if the virtio or transport facility could be controlled via device internal channels besides the transport, such a channel may complicate the synchronization a lot.

Two vendors who actually make the hw sriov devices are authoring these and others are also reviewing.
So I am more confident that it is solid enough.
Also, a similar design has been seen with other device for more than a year as GPL integrated with QEMU for a year now and with upstream kernel.

Quoted 8 lines
Quoted 6 lines

The device needs to
be able to handle or synchronize requests from both PCI and owner in parallel. They are just too many possible races and most of my questions so far come from this viewpoint. I wouldn't go further for other stuff since I believe I've spotted sufficient issues and that's why I must stop at this patch before looking at the rest.

It is your call to stop or progress.
I find your reviews useful to improve this proposal, so I will fix them.

My point is to make the theory correct before looking at the others as I had a lot of questions (as demonstrated in this thread). I think it's not hard to understand as the rest of the series are based on the theory.

Quoted 9 lines
Quoted 8 lines

Admin commands are fine if it does real administrative jobs such as provisioning since such work is beyond the core virtio functionality.

Again, the goal of virtio spec is to have a device with sufficient guidelines that is easy to implement but not leave the vendors to waste their engineering resources in figuring or fuzzing the corner cases.

I have not seen an industry standard spec or a software that does not have corner cases.

Corner cases are probably not accurate. I meant, for you, it's probably a corner case, but for me it's kind of obvious.

Quoted 1 line

The spec proposal is from > 1 device vendors.

That's good but it doesn't mean it doesn't have any (major) issues. E.g vendors may choose to just implement part of the PCIE capabilities so they don't do audits for the rest.

Quoted 6 lines

I will focus on more practical aspects to progress and improve this spec.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

When admin cmd freeze the VF it can expect FLR_completed VF.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

This question is ignored.

I probably missed. Sorry about it.
No, the driver does not need to wait for FLR to finish to issue resume command,

Good but I want to know if stop/freeze->active requires to wait for the completion of FLR. I guess the answer is yes.

Quoted 12 lines

as this typically done on the destination member device which should not be under FLR. I will write up the requirements further.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

You need to describe this somewhere, no? That's what I'm saying.

Quoted 7 lines

Mostly not. What is not done is not written.

Quoted 2 lines

And passthrough is misleading here.

Passthrough is mentioned in theory of operation.
It is not present in requirements section.
So, it is fine.

I suggest documenting or defining the "passthrough" methodology somewhere. Michael tries to define it in another thread, if it's ok, let's use that. We can't require people to read VFIO code in order to know what happens in the virtio spec.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

Ok. I assume "reset flow" is clear to you now that it points to section 2.4. This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 1 line

I will link to the section anyway.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the

PCIe transport.

Ok, I'm not a native speaker, but I really don't know the difference between "FLR" and "FLR flow".

Lets keep it simple. I will write it as FLR, as pci transport has it as FLR.

Ok.

Quoted 27 lines
Quoted 27 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the

question.

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

Quoted 7 lines
Quoted 5 lines

It's not about mediation at all, it's about how the device can implement what you want here correctly.

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real devices.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

That would be some other trap alternative that needs to dissect the device

and build infrastructure for such dissection is not desired in the listed use case.

Do you need to trap FLR or not? You're saying the hypervisor is in charge of vPCI, how is this differ to what you proposed? If not, how can vPCI be composed?

Live migration driver do not need to trap FLR.

Maybe I misunderstood your vPCI composition, but it's really helpful to document how it is expected to be done.

Quoted 6 lines
Quoted 5 lines

I believe you need to document how vpci is supposed to be done, since I believe your proposal can only work with such specific types of PCI composition. This is one of the important things that is missed in this series.

I don’t see a need to describe vpci composition as there may be more than one way to do it.

More than one way for sure, but this contradicts what you say: you said you don't trap FLR ...

People like me may wonder for example why FLR is mentioned, as FLR can be trapped and emulated.

Another example, when a device can be saved and restored, the hypervisor may schedule the device among multiple VMs, in that case, trapping FLR is a must.

Quoted 60 lines

What I think it is worth to describe is the whole pci device is not stored in device context. I will try to add a short description around it.

Quoted 31 lines
Quoted 31 lines

So your disagreement is fine for non-passthrough devices.

Quoted 28 lines
Quoted 2 lines

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 3 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 20 lines

Not normative line.

Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it doesn't need, why?

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

As author, what we think, I will cover. If you have specific points to add value,

please share, I will look into it.

I've pointed out sufficient issues. I have a lot of others but I don't want to have a giant thread once again.

I see following things to improve in the requirements which I will do in v2.

1. Document race around FLR and admin commands for really rare corner case.
2. Some text around not migrating the pci device registers
3. Interaction with PM commands

Quoted 11 lines
Quoted 2 lines
Quoted 2 lines

For example, how is the power management interaction with the

freeze/stop?

Quoted 3 lines
Quoted 1 line

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

I don't think this is a good answer. I'm asking how the PM interacts with freeze/stop, you answer it works well.

Quoted 5 lines
Quoted 4 lines

I'm not obliged to design hardware for you but figuring out the bad design for virtio. I'm not convinced with a proposal that misses a lot of obvious critical cases and for sure it's not my job to solve them.

I am not asking you to solve.

My point is that, it's better for you to have some investigation on the PM instead of me.

Quoted 5 lines
Quoted 3 lines

I've demonstrated the possible races with FLR. So did the PM. For example, if VF is in D3cold state, can we still read its device context?

I think yes, but I will double check.
If yes, is it a violation of the PCIE spec? If not, why?

So you are emulating the state instead of a real suspension?

Quoted 6 lines

No, because device context is owned by the owner device and not the VF. SR-PCIM interface has defined it be outside of scope of PCIe spec.

Quoted 3 lines

How about other states? Can the device be freezed
in the middle of PM state transitions? If yes, how can it work without migrating PCI states?

I will double check, but unlikely, it should be similar to FLR case to keep the device to avoid treating it differently.

The reason why I see it is different from FLR is that

1) D3cold requires the VF to be off the power
2) State transition might takes more than what FLR did, PCI seems only cover the minimum delay but not maximum which may have implications for downtime

Quoted 17 lines
Quoted 3 lines

Well, I meant we need a more precise definition of each state otherwise it could be ambiguous (as I pointed above).

Ok. so, few things about read and other messages, I will add.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to

update as result of driver notification.

Quoted 1 line

For example caching the posted avail index in the notification.

A link to those proposals?

[1] https://lists.oasis-open.org/archives/virtio-comment/202310/msg00048.html

I don't see how this is related to "posted avail index" etc.

Quoted 19 lines
Quoted 2 lines

If the device must depend on those cached features to work it's really fragile. If not, we don't need to care about them.

It is not dependent.
It is the infrastructure to enable it.
Same for other shared memory region accesses.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 8 lines

it which needs to be migrated or store the received notification.

Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

No, it does not.
Device context is updated within the device without accessing the queue

memory of the guest.

This is not documented or explained anywhere?

Why should it be explained?
device is not accessing the guest memory -> this is mentioned in stop mode.

Isn't it hard to see the difference between the following two?

1) In stop mode, device is not accessing guest memory
2) device context is updated without accessing the queue memory of the guest

1) is to define the stop mode, 2) is to define the behaviour of device context

Or are you saying device context can only be fetched after the device is stopped?

Quoted 63 lines

Hence, there is no need to write above.

Quoted 50 lines
Quoted 42 lines
Quoted 3 lines

Maybe you can give a concrete example.

The above one.

Quoted 37 lines
Quoted 17 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Quoted 9 lines

Like this?

"""
+The member device has a device context which the owner driver
can
+either read or write. The member device context consist of any device +specific data which is needed by the device to resume its operation +when the device mode """

Quoted 3 lines

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 8 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be

used:

Quoted 6 lines
Quoted 1 line

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

Quoted 7 lines

That is exactly what I want to say, it can only work for the owner/member model. It can't work when the virtio device is not structured like that. And you missed that most of the existing virtio devices are not implemented in this model. It means they can't be migrated with a pure virtio specific extension. For you, SR-IOV is all but this is not true for virtio. PCI is not the only transport and SR-IOV is not the only architecture in PCI.

Each transport will have its own way to handle it.
When there is MMIO owner-member relationship arise, one will be able to do so as well. In fact other transports will likely miss out as they have not established such pace.

Quoted 5 lines
Quoted 3 lines

And I'm pretty sure the owner/member is not the only requirement, there are a lot of other assumptions which are missed in this series.

One proposal does not do everything.
It is just impractical.

For other assumptions, I meant:

1) how vpci is composed, if it can be composed as vhost, why do we need to mention "passthrough"
2) the cap/bar layout, for example if a cap shares BARs with others, it can't be "passthrough", no?

Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
Quoted 8 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented

anywhere.

Quoted 3 lines
Quoted 3 lines

And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

It's not about VFIO at all. It's about to let people know under which case this proposal could work. Otherwise if a vendor develops a BAR/cap which is not at page boundary. How could you make it work with your proposal here?

Vendor is a cloud operator which is building the device, so it will always work it has the matching capabilities on source and destination.

I meant, for example, if common_cfg shares a BAR with others but doesn't own a page exclusively, you need to trap, no?

Quoted 30 lines
Quoted 30 lines
Quoted 21 lines
Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 2 lines

or it may experience a read from the shared memory and get garbage

data.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

MMIO read/write can be initiated by the peer device while the device is in

stopped state.

Quoted 7 lines
Quoted 2 lines

Ok, but what I want to say is how it can get the garbage data here?

If the device mode is changed to freeze while it is being read by the peer device, it can get garbage data or last data. Which may not be the one that is expected.
So first all the initiator devices are stopped, ensure that they do not make any requests.

And there are requests, which gets proper answer.

Ok.

Quoted 54 lines
Quoted 46 lines
Quoted 14 lines
Quoted 14 lines

Btw, shared regions are tricky for hardware.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything internally.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any +driver notifications,

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be

migrated easily.

Quoted 26 lines
Quoted 26 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 5 lines
Quoted 3 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Quoted 7 lines
Quoted 1 line

Because hypervisor has stopped the vcpus which are issuing them.

MMIO are posted. vCPU is stopped but the transactions are inflight. How could the hypervisor/device know if there's any inflight PCIE transactions here? So I can imagine what happens in fact is the TLP for freezing is ordered with the TLP for posted MMIO. This is probably guaranteed for typical PCIE setup but how about the relaxed ordering?

Quoted 7 lines

Vcpus do not generated relaxed ordering MMIOs.
In pci spec: " If this bit is Set, the Function is permitted to set the Relaxed Ordering bit in the Attributes field of transactions it initiates".

Function initiates RO requests, not the vcpu.
Hence, it is fine.

Ok.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

What can happen between #2 and #3, is device mode may change to stop.

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

On the source hypervisor, the mode changes are active->stop->freeze. Hence when freeze is done, the hypervisor knows that all inflight has been

Quoted 7 lines
Quoted 6 lines

stopped by now.

Ok, but how about freezing between 3) and 4). If we allow it, do we need to migrate to this state? If yes, how can it work with your device context? If not, shouldn't we document this?

May be, some of these are implementation details. I am not sure it belongs to spec.

The point is to make sure that your deivce context covers this case. If it can't be covered, it's a design defect.

Quoted 1 line

Like RSS update while packets are received.. such implementation details are not part of the spec.

This is definitely different, the driver can choose to synchronize or the end user can tolerate the possible out of order packets in this case.

This is not the case here, if freezing between 3) and 4) is allowed, your current device context can't cover this case and guests can't tolerate such kinds of errors after migration for sure.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Ok, it means:

1) there're conditions from stop to freeze, then what are they?

No, there isn’t condition.
May be I didn’t follow the question.

E.g under which condition could the device change the status from active to stop etc. That's something I keep asking with a concrete example (e.g FLR).

Quoted 7 lines
Quoted 1 line

2) how much time at most? E.g FLR takes at most 100ms.

From the driver side, it is 100msec for device side it can be less too. As soon as FLR is done or enough to record it, is done, stop can continue.

Quoted 2 lines

3) If it needs more time, can this time satisfy the downtime requirement?

Guest VM for all practical purposes is not busy in doing FLR, it is a corner case, yet we have to cover it.

Corner case in what sense? A loop in a simple shell script can trigger this easily.

Quoted 1 line

And yes, it satisfy the downtime requirements, because VM is already not interested in the packets, it is busy doing the FLR.

Well, it has subtle differences. VM may have more than one interface, just one of the interfaces is doing FLR.

Quoted 22 lines
Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

machine.

Quoted 1 line

First, above is not a state machine.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

So question is "how reader knows if a state can go to another state and

when"?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

It is described and listed in the table, when a mode can change.

It's not only "if" but also "when". Your table partially answers the "if '' but not "when". I think you should know now the state transition is conditional. So let's try our best to ease the life of the vendor.

What do you mean when?
I do not understand that "mode change is conditional"? it is not based on the condition. [..]

See above.

Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what is done in this patch.

Quoted 7 lines
Quoted 1 line

Synchronization point = device.

This is obvious as we can't rule stuff outside virtio, and we are talking about devices not drivers here. But the spec needs sufficient guidance/normative for the vendor to implement. It's more than just saying "device is synchronization point".

The requirements are already covering what device needs to do. Some interaction points are missing, as I acked above, I will add them.

[..]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

If kept, hypervisor may not be able to decide when to change the mode from

active->stop.

Why? It is simply done when mgmt requires a migration?

Mgmt is bit higher level entity. Underneath the software layers may wait until the time is right to migrate.

I don't understand, anyhow the migration request could not be sent to the device directly without the assistance in hypervisor.

Quoted 2 lines

The fundamental point is, the device context is expected to return the incremental value, that is changed content from last time. So once all changed content is read, its empty.

You can't easily define an incremental value for all types of states or structures:

1) device with complicated states like RAM or other
2) the device state has complicated data structures

Quoted 5 lines
Quoted 4 lines

What's more important, PCI allows multiple common_cfgs. So the hypervisor can choose to reserve one common_cfg for live migration. In this case we don't have to read to clear semantics.

Common_cfg does not serve large device context, nor it serves DMA.

Well, I'd think e.g the address of the descriptor table is part of the device context, and it can be read some common_cfg.

Quoted 8 lines
Quoted 3 lines

Or, are you saying the value read from common_cfg is not device context?

The value of common config is part of the device context that represents current common config.

Quoted 2 lines

Isn't this conflict with your vague definition of device context?

You mentioned you stop at this patch,

Stop means stopping comment.

Quoted 2 lines

so likely you didn’t read device context patch, hence you quote it vague. So I don’t know what you mean by vague.

So in this patch you define device context as:

"The member device context consist of any device specific data which is needed by the device to resume its operation"

So the address of the descriptor table satisfy this definition? If not, why?

Quoted 22 lines

Please let me know what you additional thing you want to see in device context after you reach that patch.

Quoted 11 lines
Quoted 1 line

We can opt for a mode where full device context is read in each mode

without clearing it.

Quoted 1 line

But than it can be very specific to a version of qemu, which we are avoiding it

here.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Quoted 8 lines
Quoted 2 lines

Not sure which devices you mean here with "most of the devices". Device context functions like a write record pages (aka dirty pages).

It's definitely different. We want to migrate dirty pages lively which can consume a lot of bandwidth. So reporting delta makes a lot of sense here since it would have a lot of rounds of syncing and it doesn't result in blockers resuming.

Quoted 7 lines

Write records are reported as delta from the previous read.

Quoted 4 lines

For device context, how many rounds of syncing did you expect, and if we have N rounds, we need to restore N rounds in order to resume? Do you want to live migrating device states? If it's only 1 or 2 rounds, why bother?

Live migrate the device context. Typically in current software using it, it is 2 rounds.

If it's just 2 rounds, why bother for delta? It is only helpful is we want to live migrate some device with giant states with sevreal rounds, and in that can we should leave it as a device specific state.

Quoted 10 lines

The interface is generic that if needed more rounds are possible.

Even device for most practical purpose will implement 2 rounds.

Quoted 5 lines

And for the delta, how do you know you can easily define deltas for every type of device, especially the ones with complicated internal states? Defining states has already been demonstrated as a complicated task for some devices like virtio-FS and you want to complicate it furtherly?

What is your question? If you say virtio-fs is complicated state, may be it should not have existed itself in the virtio spec as first place.

We have just more than FS that can't work for live migration. Crypto and GPU are two other examples, and I'm pretty sure we have more.

Until we figure out how they can, we can't say a device context work for all types. No?

1) Trying to define a format that works for all types of devices
2) Leavce the states to be defined by individual device types

Which method is esay?

Quoted 4 lines

But I differ to think that.
Virtio-fs guest side state wont be changed as part of it.
Virtio-fs is the first device which has considered and listed to migrate the device state. So it should be possible.

I wouldn't repeart the discussion of virtio-FS migration here, you can serach the archives for more details.

But the point is obvious, it's really hard to say a simple device context can work for all type of devices. We should allow a device specific states definition. This seems to be agreed by Michale and LingShan.

Quoted 8 lines
Quoted 5 lines

What is proposed in this series is an ad-hoc optimization for a specific deivce type within a specific subsystem (e.g VFIO) in a specific operating system which is not the general.

Oh now you mention vfio. Not me. :)

I am not going to comment on this. It is not ad-hoc.

You need to justify how it is not. Based on the current discussion, you have demonstreated a lot of asusmptions in order to make your proposal to work.

Quoted 167 lines

It uses similar dirty page tracking like technique present in cpu hw and other devices.

Quoted 13 lines

As demsonsted many times, starting from something simple and stupid is the most easy way.

Quoted 1 line

Whatever is already returned is/should not be repeated in subsequent reads,

though device can choose to do so.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Does libvirt restore on migration failure?

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

I disagree, see my above reply.

I replied above.

Quoted 142 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

Why that device context present with the software vanished, in your

assumption, if it is?

Quoted 5 lines
Quoted 5 lines

Review

Quoted 5 lines

Typically, on

(quoted diff) +3 -0
+the source hypervisor, the owner driver reads the device
+context once when the device is in \field{Active} or
+\field{Stop} mode and later once the member device is in

\field{Freeze} mode.

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique to virtio.

It is theory of operation that brings the clarity and rationale.

I think it's not. Since it's not something that is unique to virtio.

Quoted 42 lines

So I will keep it.

Quoted 40 lines
Quoted 40 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Review

Quoted 11 lines
(quoted diff) +8 -0
+
+Typically, the device context is read and written one
+time on the source and the destination hypervisor
+respectively once the device is in \field{Freeze} mode.
+On the destination hypervisor, after writing the device
+context, when the device mode set to \field{Active}, the
+device uses the most recently set device context and
+resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 14 lines
Quoted 14 lines
Quoted 11 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the
+owner driver may choose to read the device context first
+time while the device is in \field{Active} mode and
+second time once the device is in \field{Freeze}

mode.

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device

context change?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or

not.

Quoted 3 lines
Quoted 2 lines

No?

Device does not need to know, because when the guest vm and the device is

resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
Quoted 9 lines

Review

Quoted 8 lines

Similarly, on the

(quoted diff) +4 -0
+destination hypervisor writes the device context first
+time while the device is still running in \field{Active}
+mode on the source hypervisor and writes the device
+context second time while the device is in

\field{Freeze} mode.

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the
+device context likely have minimal changes from the
+previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec

wise.

Quoted 13 lines
Quoted 13 lines
Quoted 9 lines
Quoted 4 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 3 lines

" To help with these optimizations, when
VIRTIO_F_NOTIFICATION_DATA has

Quoted 8 lines
Quoted 8 lines

been negotiated".

Quoted 3 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of

migration.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

It is one of the optimization apart. The comparison is of one_of_example or

not.

I don't get this.

Theory of operation is describing a flow how things are done and how the constructs are helpful to achieve it.

Immature optimzation doesn't belong to theory for sure. I see your delta reporting immature in many ways. That's the point.

Thanks

Quoted 2 lines

And it is not the end of the list.
That does not mean one should not write those.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#19 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 54 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 17, 2023 7:12 AM

Quoted 16 lines · Parav Pandit
Quoted 4 lines

From: Jason Wang <[email protected]>
Sent: Friday, October 13, 2023 6:46 AM

[..]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and

#2.

Quoted 6 lines
Quoted 6 lines

We are discussing the possibility in software/driver side isn't it?

1) is initiated from the guest
2) is initiated from the hypervisor

Quoted 6 lines
Quoted 4 lines

Both are softwares, and you're saying 2) should not happen after 1) since the device knows what is being done by guests? How can devices control software behaviour?

Device do not control software behavior.
i.e. either hypervisor can initiate device mode change to stop (not freeze) or

guest can initiate FLR.

Quoted 4 lines

Device knows which is initiated first as single recipient of both. Therefore, device responds accordingly.
For example, in the sequence you described, A device will delay mode change command response, until the FLR is completed.

Finally but ok.

Quoted 5 lines
Quoted 5 lines

This only possible thing is to make sure 3) is done before 2) That is what I'm asking but you are saying freeze doesn't need to wait for FLR...

Quoted 7 lines
Quoted 1 line

I think I responded in previous email further down on synchronization point

being fw.

Quoted 1 line

I meant to say software do not need to wait for initiation of the freeze mode

command.

For software, did you mean the hypervisor?

Yes.

Quoted 10 lines
Quoted 4 lines

Just the command will complete at right time.

This is anyway very corner case.
On source hypervisor as written in the theory of operation, the sequence is

active->stop->freeze.

Quoted 1 line

When mode change is done to stop, the vcpus are already suspended.

The problem here is not the vcpu but the when FLR is being done since it may change the device context.

Once the freeze mode transition is completed, the hypervisor sw reads the final device context to migrate.

Quoted 5 lines
Quoted 2 lines

I agree FLR may have been initiated and driver is waiting now for 100msec.

For driver, did you mean the driver in the guest?

Yes for 100msec wait time, it is the guest driver.

Quoted 31 lines
Quoted 17 lines

So yes, device single entity synchronized it.

Quoted 14 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

But ok, since we active to freeze mode change is allowed, lets discuss

above.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and

other.

Quoted 13 lines
Quoted 13 lines

Great. So we have a perfect race:

1) guest initiates FLR
2) device start FLR
3) hypervisor stop and freeze the device
4) device is freeze
5) hypervisor read device context A
6) migrate device contextA
8) migration is done
9) FLR is done
10) hypervisor read device context B

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

So we end up with inconsistent device context, no? Dest want B or A+B, but you give A.

Since #1 and #2 is done before #3, the device knows to finish the FLR, hence

#9 is completed before #4.

Ok, that's my understanding and that's why I'm asking, but you said freeze/stop doesn't need to wait for FLR before.

Hypervisor side does not need to wait to issue the freeze/stop command. It just completes later from the device if FLR in this corner case was ongoing. I covered this in v2 now.

Quoted 21 lines
Quoted 2 lines

Alternatively, in above sequence when destination sees #10, it can

immediately finish the FLR as dest device is not under FLR, treating it as no-op.

Quoted 9 lines

Both ways to handle are fine. (and rare in practice, but yes, its possible).

I will write both the options in the device requirements.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by

FLR.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

I didn’t say clear the context.
FLR updates the device context.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

In what sense?

Indicating a new device context indicating a new device context and discard

the old one.

For example, what will queue_address have after an FLR?

All zeros.
It probably does not matter because device context to capture this FLR notion. device_status shows that device is under reset after the FLR.

Quoted 47 lines
Quoted 1 line

I am glad you asked this. I wanted to get the basic part captured before

adding this optimization.

Ok.

Quoted 5 lines

Probably it is good to add it now in the v2 as we crossed this stage now.

Quoted 3 lines
Quoted 3 lines

Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid

Quoted 13 lines
Quoted 5 lines

any race.

You want to leave those details for the vendor to figure out? If devices know everything, why do we need device normative?

Device knows its implementation.
Implementation guidelines to be in the normative.
I will add it to the normative.

Quoted 4 lines

I see issues at least for FLR, I'm pretty sure they are others. If a design requires us to audit all the possible conflicts between virtio facilities and transport. It's a strong hint of layer violation and when it happens it for sure may hit a lot of problems that are

very hard to find or debug thus we should drop such a design.

Quoted 4 lines
Quoted 3 lines

I suggest using the RFC tag since the next version (if there is one) as I see it is immature in many ways.

Technical committee audits the required touch points like rest of the industry

committees that I participated.

Quoted 2 lines

I disagree to your above point.
If you do not want to review, that is fine.

I don't want to hold my breath if I see something that is obviously wrong. Using RFC may help people to know that it is a draft that has something to be improved before it can be merged.

Quoted 11 lines

We are reviewing with other members and also contributed by them.

Quoted 9 lines

What's more, solving races is much easier if the device functionality is self contained. For example, for a self contained device with the transport as the single interface, we can leverage from transport
(PCI) for dealing with races, arbitration, ordering, QOS etc which is probably required in the internal channel between the owner and the member. But all of these were missed in your series and even if you can I'm not sure it's worthwhile to reinvent all of them.

Quoted 4 lines
Quoted 1 line

At the end there is one physical device serving owner and member devices.

This doesn't happen yet. For example, a VF with adminq that can be isolated with PASID makes some sense.

PASID is for the process of user space.
Kernel space do not consume a PASID.
Depending on the PASID can work for mediation approach only.
Last I heard is the cpu or kernel took the support out of it as some special instruction did not work for that cpu.

Quoted 7 lines
Quoted 2 lines

So a claim like things are on the VF hence you magically get 200% QoS

guarantee is myth.

That's not my point, I'm saying VF could benefit from the e.g QOS support in PCIE. I'm not saying it's perfect.

It is not the QOS support in PCIE.
It is the restrictions to respond in N usec and burden on the device to put something in always available memory for rarely used element is wasteful.

Quoted 11 lines
Quoted 4 lines

Quoting "all of these" is also incorrect.

Things added gradually, first functionally with reasonable performance,

followed by notion and extension for QoS.

Quoted 4 lines

By definition of PCI transport for SR-IOV there is internal channel.

It is reasonably well proposal in current form.
There are few race condition that you highlight are extremely rare in nature.

It's not rare since there's no way to know what the guest is doing.

I will be practical. It is rare, because a production environment guest is interested in running traffic and not constantly engaging in device reset. And if it does, taking longer time to migrate is also fine, because guest is driving the device for production apps.

Quoted 3 lines

It's actually the critical part for live migration to be correct. You are proposing migration so it must cover all those cases to make sure there is no case to make your proposal a dead end.

Yes, it should be correct.

Quoted 5 lines
Quoted 2 lines

Suggestions are welcome to improve.

I have given some and I will give more.

Sure, that is very helpful as usual.

Quoted 55 lines
Quoted 5 lines

There were couple of them by Michael too, I am addressing them in the v2.

Quoted 3 lines

For example, for the architecture like owner/member, if the virtio or transport facility could be controlled via device internal channels besides the transport, such a channel may complicate the

synchronization a lot.

Quoted 1 line

Two vendors who actually make the hw sriov devices are authoring these and

others are also reviewing.

Quoted 2 lines

So I am more confident that it is solid enough.
Also, a similar design has been seen with other device for more than a year

as GPL integrated with QEMU for a year now and with upstream kernel.

Quoted 3 lines
Quoted 3 lines

The device needs to
be able to handle or synchronize requests from both PCI and owner in

parallel.

Quoted 6 lines
Quoted 4 lines

They are just too many possible races and most of my questions so far come from this viewpoint. I wouldn't go further for other stuff since I believe I've spotted sufficient issues and that's why I must stop at this patch before looking at the rest.

It is your call to stop or progress.
I find your reviews useful to improve this proposal, so I will fix them.

My point is to make the theory correct before looking at the others as I had a lot of questions (as demonstrated in this thread). I think it's not hard to understand as the rest of the series are based on the theory.

Quoted 9 lines
Quoted 8 lines

Admin commands are fine if it does real administrative jobs such as provisioning since such work is beyond the core virtio functionality.

Again, the goal of virtio spec is to have a device with sufficient guidelines that is easy to implement but not leave the vendors to waste their engineering resources in figuring or fuzzing the corner cases.

I have not seen an industry standard spec or a software that does not have

corner cases.

Corner cases are probably not accurate. I meant, for you, it's probably a corner case, but for me it's kind of obvious.

Quoted 1 line

The spec proposal is from > 1 device vendors.

That's good but it doesn't mean it doesn't have any (major) issues. E.g vendors may choose to just implement part of the PCIE capabilities so they don't do audits for the rest.

Quoted 9 lines

I will focus on more practical aspects to progress and improve this spec.

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

This question is ignored.

I probably missed. Sorry about it.
No, the driver does not need to wait for FLR to finish to issue resume command,

Good but I want to know if stop/freeze->active requires to wait for the completion of FLR. I guess the answer is yes.

Yes.

Quoted 17 lines
Quoted 1 line

as this typically done on the destination member device which should not be

under FLR.

Quoted 15 lines

I will write up the requirements further.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

You need to describe this somewhere, no? That's what I'm saying.

Mostly not. What is not done is not written.

Quoted 2 lines

And passthrough is misleading here.

Quoted 9 lines
Quoted 3 lines

Passthrough is mentioned in theory of operation.
It is not present in requirements section.
So, it is fine.

I suggest documenting or defining the "passthrough" methodology somewhere. Michael tries to define it in another thread, if it's ok, let's use that. We can't require people to read VFIO code in order to know what happens in the virtio spec.

Yes. I captured in v2 in assumptions section.

Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Ok. I assume "reset flow" is clear to you now that it points to section

2.4.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 1 line

I will link to the section anyway.

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

PCIe transport.

Ok, I'm not a native speaker, but I really don't know the difference between "FLR" and "FLR flow".

Lets keep it simple. I will write it as FLR, as pci transport has it as FLR.

Ok.

Done in v2.

Quoted 34 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the

question.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real

devices.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

That would be some other trap alternative that needs to dissect the device

and build infrastructure for such dissection is not desired in the listed use

case.

Quoted 1 line
Quoted 1 line
Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

Do you need to trap FLR or not? You're saying the hypervisor is in charge of vPCI, how is this differ to what you proposed? If not, how can vPCI be composed?

Live migration driver do not need to trap FLR.

Maybe I misunderstood your vPCI composition, but it's really helpful to document how it is expected to be done.

We can probably have it in the cover letter as hypervisors may evolve and do more passthrough work than done today.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

I believe you need to document how vpci is supposed to be done, since I believe your proposal can only work with such specific types of PCI composition. This is one of the important things that is missed in this

Quoted 8 lines

series.

Quoted 2 lines
Quoted 1 line

I don’t see a need to describe vpci composition as there may be more than

one way to do it.

More than one way for sure, but this contradicts what you say: you said you don't trap FLR ...

Don’t trap the FLR in the live migration driver.

Quoted 2 lines

People like me may wonder for example why FLR is mentioned, as FLR can be trapped and emulated.

It can be, that emulation requires the knowledge of device specific things. This may be fine in such stack which is composing the virtio device using non virtio devices.

Quoted 4 lines

Another example, when a device can be saved and restored, the hypervisor may schedule the device among multiple VMs, in that case, trapping FLR is a must.

I think you meant sharing when you say scheduling.
Such can work when the data path is also trapped.
May be one can do using PASID and queue assignment.
But than it is not passthrough.
So two different use cases, in that case the whole PCI config space is fully replicated and hence, FLR never reaches to the real device.

Quoted 64 lines
Quoted 1 line

What I think it is worth to describe is the whole pci device is not stored in

device context.

Quoted 25 lines

I will try to add a short description around it.

Quoted 23 lines
Quoted 23 lines

So your disagreement is fine for non-passthrough devices.

Quoted 20 lines
Quoted 2 lines

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 3 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 13 lines

Not normative line.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it

doesn't need, why?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

As author, what we think, I will cover. If you have specific points to add value,

Quoted 8 lines
Quoted 5 lines

please share, I will look into it.

I've pointed out sufficient issues. I have a lot of others but I don't want to have a giant thread once again.

I see following things to improve in the requirements which I will do in v2.

1. Document race around FLR and admin commands for really rare corner

case.

Quoted 14 lines

2. Some text around not migrating the pci device registers 3. Interaction with PM commands

Quoted 11 lines
Quoted 2 lines
Quoted 2 lines

For example, how is the power management interaction with the

freeze/stop?

Quoted 3 lines
Quoted 1 line

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

I don't think this is a good answer. I'm asking how the PM interacts with freeze/stop, you answer it works well.

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

I'm not obliged to design hardware for you but figuring out the bad design for virtio. I'm not convinced with a proposal that misses a lot of obvious critical cases and for sure it's not my job to solve them.

I am not asking you to solve.

My point is that, it's better for you to have some investigation on the PM instead of me.

Yes, I updated the device requirement in v2.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I've demonstrated the possible races with FLR. So did the PM. For example, if VF is in D3cold state, can we still read its device context?

I think yes, but I will double check.
If yes, is it a violation of the PCIE spec? If not, why?

So you are emulating the state instead of a real suspension?

No. not emulating the state. It is present in the device at PCI level under guest and member device control.
When controlling function (owner PF) administrate the member device to freeze, it put the whole device in freeze so D3->D0 cannot happen.

Quoted 7 lines
Quoted 1 line

No, because device context is owned by the owner device and not the VF. SR-

PCIM interface has defined it be outside of scope of PCIe spec.

Quoted 5 lines
Quoted 4 lines

How about other states? Can the device be freezed in the middle of PM state transitions? If yes, how can it work without migrating PCI states?

I will double check, but unlikely, it should be similar to FLR case to keep the

Quoted 9 lines

device to avoid treating it differently.

The reason why I see it is different from FLR is that

1) D3cold requires the VF to be off the power
2) State transition might takes more than what FLR did, PCI seems only cover the minimum delay but not maximum which may have implications for downtime

D3cold is not controlled by the guest driver.
PM register can change D0 to D3hot.

Quoted 24 lines
Quoted 24 lines
Quoted 3 lines

Well, I meant we need a more precise definition of each state otherwise it could be ambiguous (as I pointed above).

Ok. so, few things about read and other messages, I will add.

Quoted 19 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

Quoted 8 lines
Quoted 6 lines

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to

update as result of driver notification.

Quoted 1 line

For example caching the posted avail index in the notification.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

A link to those proposals?

[1]
https://lists.oasis-open.org/archives/virtio-comment/202310/msg00048.h
tml

I don't see how this is related to "posted avail index" etc.

Driver notification counters are updated.

Quoted 23 lines
Quoted 23 lines
Quoted 3 lines

If the device must depend on those cached features to work it's really fragile. If not, we don't need to care about them.

It is not dependent.
It is the infrastructure to enable it.
Same for other shared memory region accesses.

Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines

it which needs to be migrated or store the received notification.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

No, it does not.
Device context is updated within the device without accessing the queue

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

memory of the guest.

This is not documented or explained anywhere?

Why should it be explained?
device is not accessing the guest memory -> this is mentioned in stop mode.

Isn't it hard to see the difference between the following two?

I am not following your question.

Quoted 3 lines

1) In stop mode, device is not accessing guest memory
2) device context is updated without accessing the queue memory of the guest

Device context is read/written by the owner PF, so it does not touch the guest memory.

Quoted 5 lines

1) is to define the stop mode, 2) is to define the behaviour of device context

Or are you saying device context can only be fetched after the device is stopped?

No device context can be fetched in all 3 modes.
It is weird for the hypervisor to fetch the device context while mode transition in progress.

Quoted 71 lines
Quoted 60 lines

Hence, there is no need to write above.

Quoted 58 lines
Quoted 44 lines
Quoted 3 lines

Maybe you can give a concrete example.

The above one.

Quoted 39 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

Quoted 11 lines
Quoted 8 lines

"""
+The member device has a device context which the owner
driver can

(quoted diff) +2 -0
+either read or write. The member device context consist of
+any

device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 8 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be

used:

Quoted 6 lines
Quoted 1 line

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

That is exactly what I want to say, it can only work for the owner/member model. It can't work when the virtio device is not structured like that. And you missed that most of the existing virtio devices are not implemented in this model. It means they can't be migrated with a pure virtio specific extension. For you, SR-IOV is all but this is not true for virtio. PCI is not the only transport and

SR-IOV is not the only architecture in PCI.

Quoted 3 lines
Quoted 1 line

Each transport will have its own way to handle it.
When there is MMIO owner-member relationship arise, one will be able to do

so as well.

Quoted 1 line

In fact other transports will likely miss out as they have not established such

pace.

Quoted 4 lines
Quoted 4 lines

And I'm pretty sure the owner/member is not the only requirement, there are a lot of other assumptions which are missed in this series.

Quoted 7 lines
Quoted 2 lines

One proposal does not do everything.
It is just impractical.

For other assumptions, I meant:

1) how vpci is composed, if it can be composed as vhost, why do we need to mention "passthrough"

I am lost, above you said you want to capture how vpci is composed. Here you say why to mention passthrough.

Quoted 2 lines

2) the cap/bar layout, for example if a cap shares BARs with others, it can't be "passthrough", no?

Why not, the cap exposes all the things to the guest.

Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
Quoted 10 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented

anywhere.

Quoted 3 lines
Quoted 3 lines

And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

It's not about VFIO at all. It's about to let people know under which case this proposal could work. Otherwise if a vendor develops a BAR/cap which is not at page boundary. How could you make it work with

Quoted 8 lines

your proposal here?

Quoted 2 lines
Quoted 1 line

Vendor is a cloud operator which is building the device, so it will always work

it has the matching capabilities on source and destination.

I meant, for example, if common_cfg shares a BAR with others but doesn't own a page exclusively, you need to trap, no?

For passthrough device why such restriction?

Quoted 130 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 2 lines

or it may experience a read from the shared memory and get garbage

data.

Quoted 4 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

MMIO read/write can be initiated by the peer device while the device is in

stopped state.

Ok, but what I want to say is how it can get the garbage data here?

If the device mode is changed to freeze while it is being read by the peer

device, it can get garbage data or last data.

Quoted 2 lines

Which may not be the one that is expected.
So first all the initiator devices are stopped, ensure that they do not make any

requests.

Quoted 2 lines

And there are requests, which gets proper answer.

Ok.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Btw, shared regions are tricky for hardware.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything

internally.

Quoted 39 lines
Quoted 39 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any +driver notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be

migrated easily.

Quoted 27 lines
Quoted 27 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 6 lines
Quoted 4 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Because hypervisor has stopped the vcpus which are issuing them.

MMIO are posted. vCPU is stopped but the transactions are inflight. How could the hypervisor/device know if there's any inflight PCIE transactions here? So I can imagine what happens in fact is the TLP for freezing is ordered with the TLP for posted MMIO. This is probably guaranteed for typical PCIE setup but how about the relaxed

ordering?

Quoted 8 lines

Vcpus do not generated relaxed ordering MMIOs.
In pci spec: " If this bit is Set, the Function is permitted to set the Relaxed Ordering bit in the Attributes field of transactions it initiates".

Function initiates RO requests, not the vcpu.
Hence, it is fine.

Ok.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

What can happen between #2 and #3, is device mode may change to

stop.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

On the source hypervisor, the mode changes are active->stop->freeze. Hence when freeze is done, the hypervisor knows that all inflight has been

Quoted 7 lines
Quoted 6 lines

stopped by now.

Ok, but how about freezing between 3) and 4). If we allow it, do we need to migrate to this state? If yes, how can it work with your device context? If not, shouldn't we document this?

May be, some of these are implementation details. I am not sure it belongs to

spec.

Quoted 9 lines

The point is to make sure that your deivce context covers this case. If it can't be covered, it's a design defect.

Quoted 1 line

Like RSS update while packets are received.. such implementation details are

not part of the spec.

This is definitely different, the driver can choose to synchronize or the end user can tolerate the possible out of order packets in this case.

Right but it is not defined in the spec.

Quoted 4 lines

This is not the case here, if freezing between 3) and 4) is allowed, your current device context can't cover this case and guests can't tolerate such kinds of errors after migration for sure.

Ok. I will add the text around this in v3.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

Ok, it means:

1) there're conditions from stop to freeze, then what are they?

No, there isn’t condition.
May be I didn’t follow the question.

E.g under which condition could the device change the status from active to stop etc. That's something I keep asking with a concrete example (e.g FLR).

Device mode is changed by the driver from active to stop. This is the admin mode.
FLR do not change the mode to stop/freeze because it is guest driver controlled operational state of the device.

Quoted 11 lines
Quoted 7 lines
Quoted 1 line

2) how much time at most? E.g FLR takes at most 100ms.

From the driver side, it is 100msec for device side it can be less too. As soon as FLR is done or enough to record it, is done, stop can continue.

Quoted 2 lines

3) If it needs more time, can this time satisfy the downtime requirement?

Guest VM for all practical purposes is not busy in doing FLR, it is a corner

case, yet we have to cover it.

Corner case in what sense? A loop in a simple shell script can trigger this easily.

Sure.
Which is not the practical application of the guest VM. Hence, it is corner case.

Quoted 6 lines
Quoted 1 line

And yes, it satisfy the downtime requirements, because VM is already not

interested in the packets, it is busy doing the FLR.

Well, it has subtle differences. VM may have more than one interface, just one of the interfaces is doing FLR.

Sure, it can do. But in that case the time of that VF stop is not critical. VM is busy in non-critical work.

Quoted 68 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 5 lines

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

So question is "how reader knows if a state can go to another state and

Quoted 8 lines
Quoted 6 lines

when"?

Quoted 1 line

It is described and listed in the table, when a mode can change.

It's not only "if" but also "when". Your table partially answers the "if '' but not "when". I think you should know now the state transition is conditional. So let's try our best to ease the life of the vendor.

What do you mean when?
I do not understand that "mode change is conditional"? it is not based on the

condition.

Quoted 1 line

[..]

See above.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what

is done in this patch.

Quoted 16 lines
Quoted 8 lines
Quoted 2 lines
Quoted 1 line

Synchronization point = device.

This is obvious as we can't rule stuff outside virtio, and we are talking about devices not drivers here. But the spec needs sufficient guidance/normative for the vendor to implement. It's more than just saying "device is synchronization point".

The requirements are already covering what device needs to do. Some interaction points are missing, as I acked above, I will add them.

[..]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

If kept, hypervisor may not be able to decide when to change the mode from

active->stop.

Why? It is simply done when mgmt requires a migration?

Mgmt is bit higher level entity. Underneath the software layers may wait until

the time is right to migrate.

I don't understand, anyhow the migration request could not be sent to the device directly without the assistance in hypervisor.

Quoted 9 lines
Quoted 1 line

The fundamental point is, the device context is expected to return the

incremental value, that is changed content from last time.

Quoted 1 line

So once all changed content is read, its empty.

You can't easily define an incremental value for all types of states or structures:

1) device with complicated states like RAM or other
2) the device state has complicated data structures

What I parse is, that device context is complicated structure. So it will be defined incrementally as it becomes more mature.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

What's more important, PCI allows multiple common_cfgs. So the hypervisor can choose to reserve one common_cfg for live migration. In this case we don't have to read to clear semantics.

Common_cfg does not serve large device context, nor it serves DMA.

Well, I'd think e.g the address of the descriptor table is part of the device context, and it can be read some common_cfg.

It can be read but we are talking about not saving 64 VQs tables, and RSS, flow filters, statistics all in some common config registers.

Quoted 16 lines
Quoted 5 lines
Quoted 4 lines

Or, are you saying the value read from common_cfg is not device context?

The value of common config is part of the device context that represents

current common config.

Quoted 4 lines
Quoted 3 lines

Isn't this conflict with your vague definition of device context?

You mentioned you stop at this patch,

Stop means stopping comment.

Quoted 2 lines

so likely you didn’t read device context patch, hence you quote it vague. So I don’t know what you mean by vague.

Quoted 7 lines

So in this patch you define device context as:

"The member device context consist of any device specific data which is needed by the device to resume its operation"

So the address of the descriptor table satisfy this definition? If not, why?

Address of the descriptor table is part of device context.

Quoted 30 lines
Quoted 1 line

Please let me know what you additional thing you want to see in device

context after you reach that patch.

Quoted 18 lines
Quoted 18 lines
Quoted 4 lines

We can opt for a mode where full device context is read in each mode

without clearing it.

Quoted 2 lines

But than it can be very specific to a version of qemu, which we are avoiding it

here.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Not sure which devices you mean here with "most of the devices". Device context functions like a write record pages (aka dirty pages).

Quoted 10 lines
Quoted 5 lines

It's definitely different. We want to migrate dirty pages lively which can consume a lot of bandwidth. So reporting delta makes a lot of sense here since it would have a lot of rounds of syncing and it doesn't result in blockers resuming.

Write records are reported as delta from the previous read.

Quoted 3 lines

For device context, how many rounds of syncing did you expect, and if we have N rounds, we need to restore N rounds in order to resume? Do you want to live migrating device states? If it's only 1 or 2 rounds, why

Quoted 9 lines

bother?

Quoted 2 lines
Quoted 1 line

Live migrate the device context. Typically in current software using it, it is 2

rounds.

If it's just 2 rounds, why bother for delta? It is only helpful is we want to live migrate some device with giant states with sevreal rounds, and in that can we should leave it as a device specific state.

The number of rounds matter less. The number of things a device needs to setup is a lot.
And a VM may have many devices so just like pre-copy of dirty pages, it is an extension for the device state (context) to pre-copy.

Quoted 8 lines
Quoted 8 lines

The interface is generic that if needed more rounds are possible.

Even device for most practical purpose will implement 2 rounds.

Quoted 4 lines

And for the delta, how do you know you can easily define deltas for every type of device, especially the ones with complicated internal states? Defining states has already been demonstrated as a complicated task for some devices like virtio-FS and you want to complicate

Quoted 8 lines

it furtherly?

Quoted 2 lines
Quoted 1 line

What is your question? If you say virtio-fs is complicated state, may be it

should not have existed itself in the virtio spec as first place.

We have just more than FS that can't work for live migration. Crypto and GPU are two other examples, and I'm pretty sure we have more.

Industry have migrated gpu, rdma, nvme, virtio-net, virtio-blk, devices, and susupend/resumed gpu devices too. So I can imagine it will happen as wider devices adopt the device context.

Quoted 3 lines

Until we figure out how they can, we can't say a device context work for all types. No?

No.

Quoted 5 lines

1) Trying to define a format that works for all types of devices
2) Leavce the states to be defined by individual device types

Which method is esay?

Best of both. i.e. generic fields for generic virtio items like vq config, common config, device config area. And device specific context fields like rss, flow filters, counters.

Quoted 4 lines
Quoted 3 lines

But I differ to think that.
Virtio-fs guest side state wont be changed as part of it. Virtio-fs is the first device which has considered and listed to migrate the

device state.

Quoted 9 lines
Quoted 1 line

So it should be possible.

I wouldn't repeart the discussion of virtio-FS migration here, you can serach the archives for more details.

But the point is obvious, it's really hard to say a simple device context can work for all type of devices. We should allow a device specific states definition. This seems to be agreed by Michale and LingShan.

Sure. I covered this in v2 at [2].
Device specific state definitions will be able to grow.

[2] https://lists.oasis-open.org/archives/virtio-comment/202310/msg00190.html

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

What is proposed in this series is an ad-hoc optimization for a specific deivce type within a specific subsystem (e.g VFIO) in a specific operating system which is not the general.

Quoted 7 lines
Quoted 3 lines

Oh now you mention vfio. Not me. :)

I am not going to comment on this. It is not ad-hoc.

You need to justify how it is not. Based on the current discussion, you have demonstreated a lot of asusmptions in order to make your proposal to work.

Listed in v2 now.

Quoted 196 lines
Quoted 1 line

It uses similar dirty page tracking like technique present in cpu hw and other

devices.

Quoted 17 lines
Quoted 17 lines

As demsonsted many times, starting from something simple and stupid is the most easy way.

Quoted 2 lines

Whatever is already returned is/should not be repeated in subsequent reads,

though device can choose to do so.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

Quoted 36 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

I disagree, see my above reply.

I replied above.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Why that device context present with the software
vanished, in your

Quoted 7 lines
Quoted 7 lines

assumption, if it is?

Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines

Typically, on

(quoted diff) +4 -0
+the source hypervisor, the owner driver reads the
+device context once when the device is in
+\field{Active} or \field{Stop} mode and later once
+the member device is in

\field{Freeze} mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique

to virtio.

Quoted 63 lines
Quoted 63 lines
Quoted 2 lines
Quoted 1 line

It is theory of operation that brings the clarity and rationale.

I think it's not. Since it's not something that is unique to virtio.

Quoted 43 lines

So I will keep it.

Quoted 41 lines
Quoted 41 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Review

Quoted 11 lines
(quoted diff) +8 -0
+
+Typically, the device context is read and written one
+time on the source and the destination hypervisor
+respectively once the device is in \field{Freeze} mode.
+On the destination hypervisor, after writing the
+device context, when the device mode set to
+\field{Active}, the device uses the most recently set
+device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 15 lines
Quoted 15 lines
Quoted 11 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the
+owner driver may choose to read the device context
+first time while the device is in \field{Active} mode
+and second time once the device is in \field{Freeze}

mode.

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device

context change?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context

VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

Quoted 58 lines
Quoted 58 lines
Quoted 4 lines
Quoted 4 lines

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or

not.

Quoted 4 lines
Quoted 2 lines

No?

Device does not need to know, because when the guest vm and the device is

resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 10 lines

Review

Quoted 9 lines

Similarly, on the

(quoted diff) +5 -0
+destination hypervisor writes the device context
+first time while the device is still running in
+\field{Active} mode on the source hypervisor and
+writes the device context second time while the
+device is in

\field{Freeze} mode.

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the
+device context likely have minimal changes from the
+previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec

wise.

Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 3 lines

" To help with these optimizations, when
VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of

migration.

Quoted 3 lines
Quoted 3 lines

You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

It is one of the optimization apart. The comparison is of one_of_example or

not.

I don't get this.

Theory of operation is describing a flow how things are done and how the

constructs are helpful to achieve it.

Immature optimzation doesn't belong to theory for sure. I see your delta reporting immature in many ways. That's the point.

Thanks

Quoted 2 lines

And it is not the end of the list.
That does not mean one should not write those.

#20 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 4 lines · Parav Pandit
Quoted 2 lines

This doesn't happen yet. For example, a VF with adminq that can be isolated with PASID makes some sense.

PASID is for the process of user space.
Kernel space do not consume a PASID.

It could if we need it to.

Quoted 2 lines

Depending on the PASID can work for mediation approach only.
Last I heard is the cpu or kernel took the support out of it as some special instruction did not work for that cpu.

Are you talking about ENQCMD things? That's nice but I don't think it's necessary for isolation here.

...

Quoted 9 lines
Quoted 7 lines

The reason why I see it is different from FLR is that

1) D3cold requires the VF to be off the power
2) State transition might takes more than what FLR did, PCI seems only cover the minimum delay but not maximum which may have implications for downtime

D3cold is not controlled by the guest driver.
PM register can change D0 to D3hot.

It likely makes sense to document how all this interacts with things like PM and other pci config. maybe you did in v2 didn't look at it yet.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#21 lore raw

RE: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, October 18, 2023 3:50 PM

Quoted 4 lines · Parav Pandit
Quoted 2 lines

This doesn't happen yet. For example, a VF with adminq that can be isolated with PASID makes some sense.

PASID is for the process of user space.
Kernel space do not consume a PASID.

Quoted 9 lines

It could if we need it to.

Quoted 2 lines

Depending on the PASID can work for mediation approach only. Last I heard is the cpu or kernel took the support out of it as some special

instruction did not work for that cpu.

Are you talking about ENQCMD things? That's nice but I don't think it's necessary for isolation here.

Yes. I believe SVA is disabled.

Quoted 9 lines

...

Quoted 7 lines
Quoted 7 lines

The reason why I see it is different from FLR is that

1) D3cold requires the VF to be off the power
2) State transition might takes more than what FLR did, PCI seems only cover the minimum delay but not maximum which may have implications for downtime

Quoted 6 lines
Quoted 2 lines

D3cold is not controlled by the guest driver.
PM register can change D0 to D3hot.

It likely makes sense to document how all this interacts with things like PM and other pci config. maybe you did in v2 didn't look at it yet.

Yeah, v2 covered it. Please see.
I didn't repeat what pci already covers for D3hot and D3 cold.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#22 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

In reply to Parav Pandit
Quoted 206 lines · Parav Pandit
Quoted 55 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 17, 2023 7:12 AM

Quoted 16 lines · Parav Pandit
Quoted 4 lines

From: Jason Wang <[email protected]>
Sent: Friday, October 13, 2023 6:46 AM

[..]

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

It's still not clear to me how this is done.

1) guest starts FLR
2) adminq freeze the VF
3) FLR is done

If the freezing doesn't wait for the FLR, does it mean we need to migrate to a state like FLR is pending? If yes, do we need to migrate the other sub states like this? If not, why?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

In most practical cases #2 followed by #1 should not happen as on the source

side the expected is mode change to stop from active.

How does the hypervisor know if a guest is doing what without trapping?

Hypervisor does not know. The device knows being the recipient of #1 and

#2.

Quoted 6 lines
Quoted 6 lines

We are discussing the possibility in software/driver side isn't it?

1) is initiated from the guest
2) is initiated from the hypervisor

Quoted 6 lines
Quoted 4 lines

Both are softwares, and you're saying 2) should not happen after 1) since the device knows what is being done by guests? How can devices control software behaviour?

Device do not control software behavior.
i.e. either hypervisor can initiate device mode change to stop (not freeze) or

guest can initiate FLR.

Quoted 4 lines

Device knows which is initiated first as single recipient of both. Therefore, device responds accordingly.
For example, in the sequence you described, A device will delay mode change command response, until the FLR is completed.

Finally but ok.

Quoted 5 lines
Quoted 5 lines

This only possible thing is to make sure 3) is done before 2) That is what I'm asking but you are saying freeze doesn't need to wait for FLR...

Quoted 7 lines
Quoted 1 line

I think I responded in previous email further down on synchronization point

being fw.

Quoted 1 line

I meant to say software do not need to wait for initiation of the freeze mode

command.

For software, did you mean the hypervisor?

Yes.

Quoted 10 lines
Quoted 4 lines

Just the command will complete at right time.

This is anyway very corner case.
On source hypervisor as written in the theory of operation, the sequence is

active->stop->freeze.

Quoted 1 line

When mode change is done to stop, the vcpus are already suspended.

The problem here is not the vcpu but the when FLR is being done since it may change the device context.

Once the freeze mode transition is completed, the hypervisor sw reads the final device context to migrate.

Quoted 5 lines
Quoted 2 lines

I agree FLR may have been initiated and driver is waiting now for 100msec.

For driver, did you mean the driver in the guest?

Yes for 100msec wait time, it is the guest driver.

Quoted 31 lines
Quoted 17 lines

So yes, device single entity synchronized it.

Quoted 14 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

But ok, since we active to freeze mode change is allowed, lets discuss

above.

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

A device is the single synchronization point for any device reset, FLR or admin

command operation.

So you agree we need synchronization? And I'm not sure I get the meaning of synchronization point, do you mean the synchronization between freeze/stop and virtio facilities?

Synchronization means, handling two events in parallel such as FLR and

other.

Quoted 13 lines
Quoted 13 lines

Great. So we have a perfect race:

1) guest initiates FLR
2) device start FLR
3) hypervisor stop and freeze the device
4) device is freeze
5) hypervisor read device context A
6) migrate device contextA
8) migration is done
9) FLR is done
10) hypervisor read device context B

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

So we end up with inconsistent device context, no? Dest want B or A+B, but you give A.

Since #1 and #2 is done before #3, the device knows to finish the FLR, hence

#9 is completed before #4.

Ok, that's my understanding and that's why I'm asking, but you said freeze/stop doesn't need to wait for FLR before.

Hypervisor side does not need to wait to issue the freeze/stop command. It just completes later from the device if FLR in this corner case was ongoing. I covered this in v2 now.

Quoted 21 lines
Quoted 2 lines

Alternatively, in above sequence when destination sees #10, it can

immediately finish the FLR as dest device is not under FLR, treating it as no-op.

Quoted 9 lines

Both ways to handle are fine. (and rare in practice, but yes, its possible).

I will write both the options in the device requirements.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

So, the migration driver do not need to wait for FLR to complete.

I'm confused, you said below that device context could be changed by

FLR.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Yes.

Quoted 3 lines

If FLR needs to clear device context, we can have a race where device context is cleared when we are trying to read it?

I didn’t say clear the context.
FLR updates the device context.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

In what sense?

Indicating a new device context indicating a new device context and discard

the old one.

For example, what will queue_address have after an FLR?

All zeros.
It probably does not matter because device context to capture this FLR notion. device_status shows that device is under reset after the FLR.

Quoted 47 lines
Quoted 1 line

I am glad you asked this. I wanted to get the basic part captured before

adding this optimization.

Ok.

Quoted 5 lines

Probably it is good to add it now in the v2 as we crossed this stage now.

Quoted 3 lines
Quoted 3 lines

Device is serving the device context read write commands, serving FLR, answering mode change command, So device knows the best how to avoid

Quoted 13 lines
Quoted 5 lines

any race.

You want to leave those details for the vendor to figure out? If devices know everything, why do we need device normative?

Device knows its implementation.
Implementation guidelines to be in the normative.
I will add it to the normative.

Quoted 4 lines

I see issues at least for FLR, I'm pretty sure they are others. If a design requires us to audit all the possible conflicts between virtio facilities and transport. It's a strong hint of layer violation and when it happens it for sure may hit a lot of problems that are

very hard to find or debug thus we should drop such a design.

Quoted 4 lines
Quoted 3 lines

I suggest using the RFC tag since the next version (if there is one) as I see it is immature in many ways.

Technical committee audits the required touch points like rest of the industry

committees that I participated.

Quoted 2 lines

I disagree to your above point.
If you do not want to review, that is fine.

I don't want to hold my breath if I see something that is obviously wrong. Using RFC may help people to know that it is a draft that has something to be improved before it can be merged.

Quoted 11 lines

We are reviewing with other members and also contributed by them.

Quoted 9 lines

What's more, solving races is much easier if the device functionality is self contained. For example, for a self contained device with the transport as the single interface, we can leverage from transport
(PCI) for dealing with races, arbitration, ordering, QOS etc which is probably required in the internal channel between the owner and the member. But all of these were missed in your series and even if you can I'm not sure it's worthwhile to reinvent all of them.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

At the end there is one physical device serving owner and member devices.

This doesn't happen yet. For example, a VF with adminq that can be isolated with PASID makes some sense.

PASID is for the process of user space.
Kernel space do not consume a PASID.
Depending on the PASID can work for mediation approach only.

Why? If your proposal is justified why can't it be used with the PASID?

Quoted 1 line

Last I heard is the cpu or kernel took the support out of it as some special instruction did not work for that cpu.

This is not what I heard from the CPU vendor.

Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

So a claim like things are on the VF hence you magically get 200% QoS

guarantee is myth.

That's not my point, I'm saying VF could benefit from the e.g QOS support in PCIE. I'm not saying it's perfect.

It is not the QOS support in PCIE.
It is the restrictions to respond in N usec and burden on the device to put something in always available memory for rarely used element is wasteful.

Let's leave the QOS topic aside in the future, I'm pretty sure we're talking about different things.

Quoted 192 lines
Quoted 4 lines
Quoted 4 lines

Quoting "all of these" is also incorrect.

Quoted 8 lines
Quoted 1 line

Things added gradually, first functionally with reasonable performance,

followed by notion and extension for QoS.

Quoted 4 lines

By definition of PCI transport for SR-IOV there is internal channel.

It is reasonably well proposal in current form.
There are few race condition that you highlight are extremely rare in nature.

It's not rare since there's no way to know what the guest is doing.

I will be practical. It is rare, because a production environment guest is interested in running traffic and not constantly engaging in device reset. And if it does, taking longer time to migrate is also fine, because guest is driving the device for production apps.

Quoted 3 lines

It's actually the critical part for live migration to be correct. You are proposing migration so it must cover all those cases to make sure there is no case to make your proposal a dead end.

Yes, it should be correct.

Quoted 5 lines
Quoted 2 lines

Suggestions are welcome to improve.

I have given some and I will give more.

Sure, that is very helpful as usual.

Quoted 55 lines
Quoted 5 lines

There were couple of them by Michael too, I am addressing them in the v2.

Quoted 3 lines

For example, for the architecture like owner/member, if the virtio or transport facility could be controlled via device internal channels besides the transport, such a channel may complicate the

synchronization a lot.

Quoted 1 line

Two vendors who actually make the hw sriov devices are authoring these and

others are also reviewing.

Quoted 2 lines

So I am more confident that it is solid enough.
Also, a similar design has been seen with other device for more than a year

as GPL integrated with QEMU for a year now and with upstream kernel.

Quoted 3 lines
Quoted 3 lines

The device needs to
be able to handle or synchronize requests from both PCI and owner in

parallel.

Quoted 6 lines
Quoted 4 lines

They are just too many possible races and most of my questions so far come from this viewpoint. I wouldn't go further for other stuff since I believe I've spotted sufficient issues and that's why I must stop at this patch before looking at the rest.

It is your call to stop or progress.
I find your reviews useful to improve this proposal, so I will fix them.

My point is to make the theory correct before looking at the others as I had a lot of questions (as demonstrated in this thread). I think it's not hard to understand as the rest of the series are based on the theory.

Quoted 9 lines
Quoted 8 lines

Admin commands are fine if it does real administrative jobs such as provisioning since such work is beyond the core virtio functionality.

Again, the goal of virtio spec is to have a device with sufficient guidelines that is easy to implement but not leave the vendors to waste their engineering resources in figuring or fuzzing the corner cases.

I have not seen an industry standard spec or a software that does not have

corner cases.

Corner cases are probably not accurate. I meant, for you, it's probably a corner case, but for me it's kind of obvious.

Quoted 1 line

The spec proposal is from > 1 device vendors.

That's good but it doesn't mean it doesn't have any (major) issues. E.g vendors may choose to just implement part of the PCIE capabilities so they don't do audits for the rest.

Quoted 9 lines

I will focus on more practical aspects to progress and improve this spec.

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

When admin cmd freeze the VF it can expect FLR_completed VF.

We need to explain why and how about the resume? For example, is resuming required to wait for the completion of FLR, if not, why?

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

This question is ignored.

I probably missed. Sorry about it.
No, the driver does not need to wait for FLR to finish to issue resume command,

Good but I want to know if stop/freeze->active requires to wait for the completion of FLR. I guess the answer is yes.

Yes.

Quoted 17 lines
Quoted 1 line

as this typically done on the destination member device which should not be

under FLR.

Quoted 15 lines

I will write up the requirements further.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

In another thread you are saying that the PCI composition is done by hypervisor, so passthrough is really confusing at least for me.

I explained there what vPCI composition is done there. PCI config space and msix side of composition is done. The whole virtio interface is not composed.

You need to describe this somewhere, no? That's what I'm saying.

Mostly not. What is not done is not written.

Quoted 2 lines

And passthrough is misleading here.

Quoted 9 lines
Quoted 3 lines

Passthrough is mentioned in theory of operation.
It is not present in requirements section.
So, it is fine.

I suggest documenting or defining the "passthrough" methodology somewhere. Michael tries to define it in another thread, if it's ok, let's use that. We can't require people to read VFIO code in order to know what happens in the virtio spec.

Yes. I captured in v2 in assumptions section.

Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Ok. I assume "reset flow" is clear to you now that it points to section

2.4.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

This section is not normative section, so using an extra word like "flow" does

not confuse anyone.

Quoted 1 line

I will link to the section anyway.

Probably, but you mention FLR flow as well.

As I said, not repeating the PCIe spec here. The reader knows what FLR of the

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

PCIe transport.

Ok, I'm not a native speaker, but I really don't know the difference between "FLR" and "FLR flow".

Lets keep it simple. I will write it as FLR, as pci transport has it as FLR.

Ok.

Done in v2.

Quoted 34 lines
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

and may also undergo PCI function level
+reset(FLR) flow.

Why is only FLR special here? I've asked FRS but you ignore the

question.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

FLR is special to bring clarity that guest owns the VF doing FLR, hence

hypervisor cannot mediate any registers of the VF.

It's not about mediation at all, it's about how the device can implement what you want here correctly.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

See my above question.

Ok. it is clear that live migration commands cannot stay on the member device

because the member device can undergo device reset and FLR flows owned by the guest.

I disagree, hypervisors can emulate FLR and never send FLR to real

devices.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

That would be some other trap alternative that needs to dissect the device

and build infrastructure for such dissection is not desired in the listed use

case.

Quoted 1 line
Quoted 1 line
Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

Do you need to trap FLR or not? You're saying the hypervisor is in charge of vPCI, how is this differ to what you proposed? If not, how can vPCI be composed?

Live migration driver do not need to trap FLR.

Maybe I misunderstood your vPCI composition, but it's really helpful to document how it is expected to be done.

We can probably have it in the cover letter as hypervisors may evolve and do more passthrough work than done today.

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

I believe you need to document how vpci is supposed to be done, since I believe your proposal can only work with such specific types of PCI composition. This is one of the important things that is missed in this

series.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

I don’t see a need to describe vpci composition as there may be more than

one way to do it.

More than one way for sure, but this contradicts what you say: you said you don't trap FLR ...

Don’t trap the FLR in the live migration driver.

But trapped by other?

Quoted 6 lines
Quoted 3 lines

People like me may wonder for example why FLR is mentioned, as FLR can be trapped and emulated.

It can be, that emulation requires the knowledge of device specific things. This may be fine in such stack which is composing the virtio device using non virtio devices.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Another example, when a device can be saved and restored, the hypervisor may schedule the device among multiple VMs, in that case, trapping FLR is a must.

I think you meant sharing when you say scheduling.
Such can work when the data path is also trapped.
May be one can do using PASID and queue assignment.
But than it is not passthrough.
So two different use cases, in that case the whole PCI config space is fully replicated and hence, FLR never reaches to the real device.

It's pretty common, e.g it works as system reset from VCPU never goes to CPU.

Quoted 80 lines
Quoted 66 lines
Quoted 3 lines

What I think it is worth to describe is the whole pci device is not stored in

device context.

Quoted 25 lines

I will try to add a short description around it.

Quoted 23 lines
Quoted 23 lines

So your disagreement is fine for non-passthrough devices.

Quoted 20 lines
Quoted 2 lines

(and hypervisor is not involved in these two flows, hence the admin command

interface is designed such that it can fullfil above requirements).

Quoted 3 lines

Theory of operation brings out this clarity. Please notice that it is in

introductory section with an example.

Quoted 13 lines

Not normative line.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Such flows must comply to the PCI standard and also +virtio specification;

This seems unnecessary and obvious as it applies to all other PCI and virtio functionality.

Great. But your comment is contradicts.

Quoted 2 lines

What's more, for the things that need to be synchronized, I don't see any descriptions in this patch. And if it

doesn't need, why?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

With which operation should it be synchronized and why? Can you please be specific?

See my above question regarding FLR. And it may have others which I haven't had time to audit.

Ok. when you get chance to audit, lets discuss that time.

Well, I'm not the author of this series, it should be your job otherwise it would be too late.

As author, what we think, I will cover. If you have specific points to add value,

Quoted 8 lines
Quoted 5 lines

please share, I will look into it.

I've pointed out sufficient issues. I have a lot of others but I don't want to have a giant thread once again.

I see following things to improve in the requirements which I will do in v2.

1. Document race around FLR and admin commands for really rare corner

case.

Quoted 14 lines

2. Some text around not migrating the pci device registers 3. Interaction with PM commands

Quoted 11 lines
Quoted 2 lines
Quoted 2 lines

For example, how is the power management interaction with the

freeze/stop?

Quoted 3 lines
Quoted 1 line

Power management is owned by the guest, like any other virtio interface. So freeze/stop do not interfere with it.

I don't think this is a good answer. I'm asking how the PM interacts with freeze/stop, you answer it works well.

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

I'm not obliged to design hardware for you but figuring out the bad design for virtio. I'm not convinced with a proposal that misses a lot of obvious critical cases and for sure it's not my job to solve them.

I am not asking you to solve.

My point is that, it's better for you to have some investigation on the PM instead of me.

Yes, I updated the device requirement in v2.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I've demonstrated the possible races with FLR. So did the PM. For example, if VF is in D3cold state, can we still read its device context?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I think yes, but I will double check.
If yes, is it a violation of the PCIE spec? If not, why?

So you are emulating the state instead of a real suspension?

No. not emulating the state. It is present in the device at PCI level under guest and member device control.
When controlling function (owner PF) administrate the member device to freeze, it put the whole device in freeze so D3->D0 cannot happen.

What I want to ask is if the PM is exposed to guest drivers, not whether the PF can do that.

If the PM can't, we lose a very important functionality in the guest. If yes, how is the e.g D3state synchronized with the free/stop/device context read, or if it can't and why.

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

No, because device context is owned by the owner device and not the VF. SR-

PCIM interface has defined it be outside of scope of PCIe spec.

Quoted 5 lines
Quoted 4 lines

How about other states? Can the device be freezed in the middle of PM state transitions? If yes, how can it work without migrating PCI states?

I will double check, but unlikely, it should be similar to FLR case to keep the

device to avoid treating it differently.

Quoted 8 lines
Quoted 7 lines

The reason why I see it is different from FLR is that

1) D3cold requires the VF to be off the power
2) State transition might takes more than what FLR did, PCI seems only cover the minimum delay but not maximum which may have implications for downtime

D3cold is not controlled by the guest driver.

How do you know that?

Spec said:

"All Functions must support the D0 and D3 states (both D3Hot and D3Cold). The D1 and D2 states are optional."

So the function is available in the VF. And actually VFIO has started to support that:

commit cc2742fe3660cc6500021d3da8f937d326392dbd

Author: Abhishek Sahu <[email protected]>
Date:   Mon Aug 29 17:18:49 2022 +0530

vfio/pci: Implement VFIO_DEVICE_FEATURE_LOW_POWER_ENTRY/EXIT

And even for D3hot, it has very subtle things that needs special care:

Spec said:

"If the No_Soft_Reset bit is Clear, functional context is not required to be maintained by the Function in the D3Hot state"

So it looks like your proposal can only work with the No_Soft_Reset set. Or you need at least document the context is maintained in this case. Since the context might be lost in it is clear. So you can't migrate the device within the D3hot state.

I believe you should have much more knowledge than me in PCI. I just read the spec in minutes and spot the above. I'm pretty sure it's not hard for you to find more issues and try to think of a way to solve them or not. It would be more efficient for you instead of waiting for a person like me to point out the issue. No?

Again, it's really awkward that we need to go through everywhere in the PCIE spec in order to prove your proposal. That's really a layer violation in the design where we know for sure there will be more blockers in the future.

Quoted 26 lines

PM register can change D0 to D3hot.

Quoted 24 lines
Quoted 24 lines
Quoted 3 lines

Well, I meant we need a more precise definition of each state otherwise it could be ambiguous (as I pointed above).

Ok. so, few things about read and other messages, I will add.

Quoted 19 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In "stop" mode, the device wont process descriptors.

If the device won't process descriptors, why still allow it to receive

Quoted 8 lines
Quoted 6 lines

notifications?

Quoted 2 lines

Because notification may still arrive and if the device may update any counters as part of

Which counters did you mean here?

The counter that Xuan is adding and any other state that device may have to

update as result of driver notification.

Quoted 1 line

For example caching the posted avail index in the notification.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

A link to those proposals?

[1]
https://lists.oasis-open.org/archives/virtio-comment/202310/msg00048.h
tml

I don't see how this is related to "posted avail index" etc.

Driver notification counters are updated.

There's no guarantee that the counters are exactly accurate for any case. E.g counter could be updated in the middle of a read. I don't think there would be a software that depends on the precise value of a statistics counter.

Quoted 39 lines
Quoted 24 lines
Quoted 24 lines
Quoted 4 lines

If the device must depend on those cached features to work it's really fragile. If not, we don't need to care about them.

It is not dependent.
It is the infrastructure to enable it.
Same for other shared memory region accesses.

Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines

it which needs to be migrated or store the received notification.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Or does it really matter if the device can receive or not here?

From device point of view, the device is given the chance to update its device

context as part of notifications or access to it.

This is in conflict with what you said above " Device cannot process the queue ..."

No, it does not.
Device context is updated within the device without accessing the queue

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

memory of the guest.

This is not documented or explained anywhere?

Why should it be explained?
device is not accessing the guest memory -> this is mentioned in stop mode.

Isn't it hard to see the difference between the following two?

I am not following your question.

Quoted 3 lines

1) In stop mode, device is not accessing guest memory
2) device context is updated without accessing the queue memory of the guest

Device context is read/written by the owner PF, so it does not touch the guest memory.

Quoted 7 lines
Quoted 5 lines

1) is to define the stop mode, 2) is to define the behaviour of device context

Or are you saying device context can only be fetched after the device is stopped?

No device context can be fetched in all 3 modes.
It is weird for the hypervisor to fetch the device context while mode transition in progress.

Ok.

Quoted 75 lines
Quoted 75 lines
Quoted 61 lines

Hence, there is no need to write above.

Quoted 58 lines
Quoted 44 lines
Quoted 3 lines

Maybe you can give a concrete example.

The above one.

Quoted 39 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

+ the member device context

I don't think we define "device context" anywhere.

It is defined further in the description.

Like this?

Quoted 11 lines
Quoted 8 lines

"""
+The member device has a device context which the owner
driver can

(quoted diff) +2 -0
+either read or write. The member device context consist of
+any

device +specific data which is needed by the device to resume its operation +when the device mode """

Yes.
Further patch-3 adds the device context and also add the link to it in the

theory of operation section so reader can read more detail about it.

Quoted 8 lines
Quoted 6 lines

"Any" is probably too hard for vendors to implement. And in patch 3 I only see virtio device context. Does this mean we don't need transport
(PCI) context at all? If yes, how can it work?

Right. PCI member device is present at source and destination with its layout,

only the virtio device context is transferred.

Quoted 1 line

Which part cannot work?

It is explained in another thread where you are saying the PCI requires mediation. I think any author should not ignore such important assumptions in both the change log and the patch.

And again, the more I review the more I see how narrow this series can be

used:

Quoted 6 lines
Quoted 1 line

I explained this before and also covered in the cover letter.

Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

That is exactly what I want to say, it can only work for the owner/member model. It can't work when the virtio device is not structured like that. And you missed that most of the existing virtio devices are not implemented in this model. It means they can't be migrated with a pure virtio specific extension. For you, SR-IOV is all but this is not true for virtio. PCI is not the only transport and

SR-IOV is not the only architecture in PCI.

Quoted 3 lines
Quoted 1 line

Each transport will have its own way to handle it.
When there is MMIO owner-member relationship arise, one will be able to do

so as well.

Quoted 1 line

In fact other transports will likely miss out as they have not established such

pace.

Quoted 6 lines
Quoted 4 lines

And I'm pretty sure the owner/member is not the only requirement, there are a lot of other assumptions which are missed in this series.

One proposal does not do everything.
It is just impractical.

Quoted 6 lines
Quoted 4 lines

For other assumptions, I meant:

1) how vpci is composed, if it can be composed as vhost, why do we need to mention "passthrough"

I am lost, above you said you want to capture how vpci is composed. Here you say why to mention passthrough.

I mean you need to document how vpci is composed, then people can see if it makes sense or not.

Quoted 4 lines
Quoted 3 lines

2) the cap/bar layout, for example if a cap shares BARs with others, it can't be "passthrough", no?

Why not, the cap exposes all the things to the guest.

What if a cap shared the same page with MSI-X? You can't let guests program MSI-X directly, no?

Quoted 23 lines
Quoted 23 lines
Quoted 15 lines
Quoted 15 lines
Quoted 11 lines
Quoted 11 lines

2) Mediate PCI but not virtio which is tricky
3) Can only work for a specific BAR/capability register layout

Only 1) is described in the change log.

The other important assumptions like 2) and 3) are not documented

anywhere.

Quoted 3 lines
Quoted 3 lines

And this patch never explains why 2) and 3) is needed or why it can be used for subsystems other than VFIO/Linux.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Since I am not mentioning vfio now, I will refrain from mentioning others as well. :)

It's not about VFIO at all. It's about to let people know under which case this proposal could work. Otherwise if a vendor develops a BAR/cap which is not at page boundary. How could you make it work with

your proposal here?

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Vendor is a cloud operator which is building the device, so it will always work

it has the matching capabilities on source and destination.

I meant, for example, if common_cfg shares a BAR with others but doesn't own a page exclusively, you need to trap, no?

For passthrough device why such restriction?

See above.

Quoted 175 lines
Quoted 131 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

and device configuration space may change. \\
+\hline

I still don't get why we need a "stop" state in the middle.

All pci devices which belong to a single guest VM are not stopped

atomically.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Hence, one device which is in freeze mode, may still receive driver notifications from other pci device,

Device may choose to ignore those notifications, no?

Quoted 2 lines

or it may experience a read from the shared memory and get garbage

data.

Quoted 4 lines
Quoted 3 lines

Could you give me an example for this?

Section 2.10 Shared Memory Regions.

How can it experience a read in this case?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

MMIO read/write can be initiated by the peer device while the device is in

stopped state.

Ok, but what I want to say is how it can get the garbage data here?

If the device mode is changed to freeze while it is being read by the peer

device, it can get garbage data or last data.

Quoted 2 lines

Which may not be the one that is expected.
So first all the initiator devices are stopped, ensure that they do not make any

requests.

Quoted 2 lines

And there are requests, which gets proper answer.

Ok.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Btw, shared regions are tricky for hardware.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

And things can break.
Hence the stop mode, ensures that all the devices get enough chance to stop

themselves, and later when freezed, to not change anything

internally.

Quoted 39 lines
Quoted 39 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

+0x2 & Freeze &
+ In this mode, the member device does not accept any +driver notifications,

This is too vague. Is the device allowed to be freezed in the middle of any virtio or PCI operations?

For example, in the middle of feature negotiation etc. It may cause implementation specific sub-states which can't be

migrated easily.

Quoted 27 lines
Quoted 27 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Yes. it is allowed in middle of feature negotiation, for sure. It is passthrough device, hence hypervisor layer do not get to see sub-

state.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Not sure why you comment, why it cannot be migrated easily. The device context already covers this sub-state.

1) driver writes driver_features
2) driver sets FEAUTRES_OK

3) device receive driver_features
4) device validating driver_features
5) device clears FEATURES_OK

Quoted 6 lines
Quoted 4 lines

6) driver read stats and realize FEATURES_OK is being cleared

Is it valid to be frozen of the above?

No. device mode is frozen when hypervisor is sure that no more access by the

guest will be done.

How, you don't trap so 1) and 2) are posted, how can hypervisor know if there's inflight transactions to any registers?

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Because hypervisor has stopped the vcpus which are issuing them.

MMIO are posted. vCPU is stopped but the transactions are inflight. How could the hypervisor/device know if there's any inflight PCIE transactions here? So I can imagine what happens in fact is the TLP for freezing is ordered with the TLP for posted MMIO. This is probably guaranteed for typical PCIE setup but how about the relaxed

ordering?

Quoted 8 lines

Vcpus do not generated relaxed ordering MMIOs.
In pci spec: " If this bit is Set, the Function is permitted to set the Relaxed Ordering bit in the Attributes field of transactions it initiates".

Function initiates RO requests, not the vcpu.
Hence, it is fine.

Ok.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

What can happen between #2 and #3, is device mode may change to

stop.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Why can't be freezed in this case? It's really hard to deduce why it can't just from your above descriptions.

On the source hypervisor, the mode changes are active->stop->freeze. Hence when freeze is done, the hypervisor knows that all inflight has been

Quoted 7 lines
Quoted 6 lines

stopped by now.

Ok, but how about freezing between 3) and 4). If we allow it, do we need to migrate to this state? If yes, how can it work with your device context? If not, shouldn't we document this?

May be, some of these are implementation details. I am not sure it belongs to

spec.

Quoted 9 lines

The point is to make sure that your deivce context covers this case. If it can't be covered, it's a design defect.

Quoted 1 line

Like RSS update while packets are received.. such implementation details are

not part of the spec.

This is definitely different, the driver can choose to synchronize or the end user can tolerate the possible out of order packets in this case.

Right but it is not defined in the spec.

Quoted 4 lines

This is not the case here, if freezing between 3) and 4) is allowed, your current device context can't cover this case and guests can't tolerate such kinds of errors after migration for sure.

Ok. I will add the text around this in v3.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 6 lines

Even if it had, is it even possible to list all the places where freezing is prohibited? We don't want to end up with a spec that is hard to implement or leave the vendor to figure out those tricky parts.

The general idea is not prohibiting the freeze/stop mode. If the device needs more time, let device take time to do it.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

Ok, it means:

1) there're conditions from stop to freeze, then what are they?

No, there isn’t condition.
May be I didn’t follow the question.

E.g under which condition could the device change the status from active to stop etc. That's something I keep asking with a concrete example (e.g FLR).

Device mode is changed by the driver from active to stop. This is the admin mode.
FLR do not change the mode to stop/freeze because it is guest driver controlled operational state of the device.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

2) how much time at most? E.g FLR takes at most 100ms.

From the driver side, it is 100msec for device side it can be less too. As soon as FLR is done or enough to record it, is done, stop can continue.

Quoted 2 lines

3) If it needs more time, can this time satisfy the downtime requirement?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Guest VM for all practical purposes is not busy in doing FLR, it is a corner

case, yet we have to cover it.

Corner case in what sense? A loop in a simple shell script can trigger this easily.

Sure.
Which is not the practical application of the guest VM. Hence, it is corner case.

Such thought is wrong. You can't assume the behaviour happnes in guest, that's the basic assumption for virtualization and security. Or you will end up with endless bugs that is hard to fix.

Quoted 82 lines
Quoted 7 lines
Quoted 2 lines

And yes, it satisfy the downtime requirements, because VM is already not

interested in the packets, it is busy doing the FLR.

Well, it has subtle differences. VM may have more than one interface, just one of the interfaces is doing FLR.

Sure, it can do. But in that case the time of that VF stop is not critical. VM is busy in non-critical work.

Quoted 72 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 5 lines

And in stop mode, device context would capture #5 or #4, depending where is

device at that point.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

And what's more, the above state machine seems to be virtio specific, but you don't explain the interaction with the device status state

Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

machine.

Quoted 1 line

First, above is not a state machine.

So how do readers know if a state can go to another state and when?

Not sure what you mean by reader. Can you please explain.

The people who read virtio spec.

So question is "how reader knows if a state can go to another state and

Quoted 8 lines
Quoted 6 lines

when"?

Quoted 1 line

It is described and listed in the table, when a mode can change.

It's not only "if" but also "when". Your table partially answers the "if '' but not "when". I think you should know now the state transition is conditional. So let's try our best to ease the life of the vendor.

What do you mean when?
I do not understand that "mode change is conditional"? it is not based on the

condition.

Quoted 1 line

[..]

See above.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Let's define the synchronization point first. And it demonstrates at least devices need to synchronize between the free/stop and virtio device status machine which is not as easy as what

is done in this patch.

Quoted 16 lines
Quoted 8 lines
Quoted 2 lines
Quoted 1 line

Synchronization point = device.

This is obvious as we can't rule stuff outside virtio, and we are talking about devices not drivers here. But the spec needs sufficient guidance/normative for the vendor to implement. It's more than just saying "device is synchronization point".

The requirements are already covering what device needs to do. Some interaction points are missing, as I acked above, I will add them.

[..]

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Until virtio reset, this is how virtio works now. I've pointed out that it may cause extra troubles when trying to resume, but you don't tell me what's wrong to keep that?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

If kept, hypervisor may not be able to decide when to change the mode from

active->stop.

Why? It is simply done when mgmt requires a migration?

Mgmt is bit higher level entity. Underneath the software layers may wait until

the time is right to migrate.

I don't understand, anyhow the migration request could not be sent to the device directly without the assistance in hypervisor.

Quoted 1 line

The fundamental point is, the device context is expected to return the

incremental value, that is changed content from last time.

Quoted 1 line

So once all changed content is read, its empty.

Quoted 7 lines
Quoted 5 lines

You can't easily define an incremental value for all types of states or structures:

1) device with complicated states like RAM or other
2) the device state has complicated data structures

What I parse is, that device context is complicated structure. So it will be defined incrementally as it becomes more mature.

Increament only makes sense when

1) The state is giant, like RAM
2) There's a requirement of live migrating the state, dirty pages, blocks from local devices
3) Giant state can converge after limited rounds

All of the above seems to be device type speicifc not the case for a general device context.

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 5 lines

What's more important, PCI allows multiple common_cfgs. So the hypervisor can choose to reserve one common_cfg for live migration. In this case we don't have to read to clear semantics.

Common_cfg does not serve large device context, nor it serves DMA.

Well, I'd think e.g the address of the descriptor table is part of the device context, and it can be read some common_cfg.

It can be read but we are talking about not saving 64 VQs tables, and RSS, flow filters, statistics all in some common config registers.

The problem is, what you define as the incresmental device context conflict with what has already been existed.

You say device context is incresmental but I give you one example that it doesn't.

Quoted 17 lines
Quoted 17 lines
Quoted 6 lines
Quoted 5 lines

Or, are you saying the value read from common_cfg is not device context?

The value of common config is part of the device context that represents

current common config.

Quoted 4 lines
Quoted 3 lines

Isn't this conflict with your vague definition of device context?

You mentioned you stop at this patch,

Stop means stopping comment.

Quoted 2 lines

so likely you didn’t read device context patch, hence you quote it vague. So I don’t know what you mean by vague.

Quoted 8 lines
Quoted 7 lines

So in this patch you define device context as:

"The member device context consist of any device specific data which is needed by the device to resume its operation"

So the address of the descriptor table satisfy this definition? If not, why?

Address of the descriptor table is part of device context.

Ok, see my above reply.

Quoted 33 lines
Quoted 33 lines
Quoted 2 lines

Please let me know what you additional thing you want to see in device

context after you reach that patch.

Quoted 18 lines
Quoted 18 lines
Quoted 4 lines

We can opt for a mode where full device context is read in each mode

without clearing it.

Quoted 2 lines

But than it can be very specific to a version of qemu, which we are avoiding it

here.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

2. device context returns incremental value from the previous read. So, it

needs to clear it.

I don't understand here. This is not the case for most of the devices.

Not sure which devices you mean here with "most of the devices". Device context functions like a write record pages (aka dirty pages).

Quoted 10 lines
Quoted 5 lines

It's definitely different. We want to migrate dirty pages lively which can consume a lot of bandwidth. So reporting delta makes a lot of sense here since it would have a lot of rounds of syncing and it doesn't result in blockers resuming.

Write records are reported as delta from the previous read.

Quoted 3 lines

For device context, how many rounds of syncing did you expect, and if we have N rounds, we need to restore N rounds in order to resume? Do you want to live migrating device states? If it's only 1 or 2 rounds, why

bother?

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Live migrate the device context. Typically in current software using it, it is 2

rounds.

If it's just 2 rounds, why bother for delta? It is only helpful is we want to live migrate some device with giant states with sevreal rounds, and in that can we should leave it as a device specific state.

The number of rounds matter less. The number of things a device needs to setup is a lot.

How many context in bytes do you exepct for e.g the virtio-net?

Quoted 1 line

And a VM may have many devices so just like pre-copy of dirty pages, it is an extension for the device state (context) to pre-copy.

Hypervisor can choose to do things like pre-heat but even in this case, delta is not a must for device unless it have giant states that need to be live migrated.

Quoted 11 lines
Quoted 11 lines
Quoted 9 lines

The interface is generic that if needed more rounds are possible.

Even device for most practical purpose will implement 2 rounds.

Quoted 4 lines

And for the delta, how do you know you can easily define deltas for every type of device, especially the ones with complicated internal states? Defining states has already been demonstrated as a complicated task for some devices like virtio-FS and you want to complicate

it furtherly?

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

What is your question? If you say virtio-fs is complicated state, may be it

should not have existed itself in the virtio spec as first place.

We have just more than FS that can't work for live migration. Crypto and GPU are two other examples, and I'm pretty sure we have more.

Industry have migrated gpu, rdma, nvme, virtio-net, virtio-blk, devices, and susupend/resumed gpu devices too. So I can imagine it will happen as wider devices adopt the device context.

How many device you mentioned above have a open standard? Did you know how many type of devices that virtio has supported? But we are talking about different things. Unless you implement virtio-(gp)gpu/rdma/nvme, then we can talk here.

What I want to say here is, from the virtio point of view, it's not promising if you can just migrate net/blk. You know, we only need very little extension then we can migrate net and block.

Quoted 5 lines
Quoted 4 lines

Until we figure out how they can, we can't say a device context work for all types. No?

No.

Why?

Quoted 7 lines
Quoted 5 lines

1) Trying to define a format that works for all types of devices
2) Leavce the states to be defined by individual device types

Which method is esay?

Best of both. i.e. generic fields for generic virtio items like vq config, common config, device config area. And device specific context fields like rss, flow filters, counters.

This is conflict with your definition of device context and that is the source of the misunderstanding.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

But I differ to think that.
Virtio-fs guest side state wont be changed as part of it. Virtio-fs is the first device which has considered and listed to migrate the

device state.

Quoted 1 line

So it should be possible.

I wouldn't repeart the discussion of virtio-FS migration here, you can serach the archives for more details.

Quoted 6 lines
Quoted 4 lines

But the point is obvious, it's really hard to say a simple device context can work for all type of devices. We should allow a device specific states definition. This seems to be agreed by Michale and LingShan.

Sure. I covered this in v2 at [2].
Device specific state definitions will be able to grow.

Great.

Thanks

Quoted 8 lines

[2] https://lists.oasis-open.org/archives/virtio-comment/202310/msg00190.html

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

What is proposed in this series is an ad-hoc optimization for a specific deivce type within a specific subsystem (e.g VFIO) in a specific operating system which is not the general.

Quoted 206 lines
Quoted 7 lines
Quoted 3 lines

Oh now you mention vfio. Not me. :)

I am not going to comment on this. It is not ad-hoc.

You need to justify how it is not. Based on the current discussion, you have demonstreated a lot of asusmptions in order to make your proposal to work.

Listed in v2 now.

Quoted 196 lines
Quoted 1 line

It uses similar dirty page tracking like technique present in cpu hw and other

devices.

Quoted 17 lines
Quoted 17 lines

As demsonsted many times, starting from something simple and stupid is the most easy way.

Quoted 2 lines

Whatever is already returned is/should not be repeated in subsequent reads,

though device can choose to do so.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

And which software stack may find this useful? Is there any existing software that can utilize it?

Libvirt.

Does libvirt restore on migration failure?

Quoted 36 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Yes.

Ok. the device will be able to resume when it is marked active. The device context returned is the incremental delta as explained above.

I disagree, see my above reply.

I replied above.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Why that device context present with the software
vanished, in your

Quoted 7 lines
Quoted 7 lines

assumption, if it is?

Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines

Typically, on

(quoted diff) +4 -0
+the source hypervisor, the owner driver reads the
+device context once when the device is in
+\field{Active} or \field{Stop} mode and later once
+the member device is in

\field{Freeze} mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Why need the read while device context could be changed? Or is the dirty page part of the device context?

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

It is not part of the dirty page.
It needs to read in the active/stop mode, so that it can be shared with

destination hypervisor, which will pre-setup the complex context of the device, while it is still running on the source side.

Is such a method used by any hypervisor?

Yes. qemu which uses vfio interface uses it.

Ok, such software technology could be used for all types of devices, I don't see any advantages to mention it here unless it's unique

to virtio.

Quoted 63 lines
Quoted 63 lines
Quoted 2 lines
Quoted 1 line

It is theory of operation that brings the clarity and rationale.

I think it's not. Since it's not something that is unique to virtio.

Quoted 43 lines

So I will keep it.

Quoted 41 lines
Quoted 41 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Review

Quoted 11 lines
(quoted diff) +8 -0
+
+Typically, the device context is read and written one
+time on the source and the destination hypervisor
+respectively once the device is in \field{Freeze} mode.
+On the destination hypervisor, after writing the
+device context, when the device mode set to
+\field{Active}, the device uses the most recently set
+device context and resumes the device

operation.

There's no context sequence, so this is obvious. It's the semantic of all other existing interfaces.

Quoted 7 lines
Quoted 1 line

Can you please what which existing interfaces do you mean here?

For any common cfg member. E.g queue_addr.

The driver wrote 100 different values to queue_addr and the device used the value written last time.

o.k. I don’t see any problem in stating what is done, which is less vague. 😊

Quoted 15 lines
Quoted 15 lines
Quoted 11 lines

Review

Quoted 6 lines
(quoted diff) +5 -0
+
+In an alternative flow, on the source hypervisor the
+owner driver may choose to read the device context
+first time while the device is in \field{Active} mode
+and second time once the device is in \field{Freeze}

mode.

Who is going to synchronize the device context with possible configuration from the driver?

Not sure I understand the question.
If I understand you right, do you mean that, When configuration change is done by the guest driver, how does device

context change?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 1 line

Yes.

Quoted 1 line

If so, device context reading will reflect the new configuration.

How do you do that? For example:

static inline void vp_iowrite64_twopart(u64 val,

                                        __le32 __iomem *lo,
                                        __le32 __iomem *hi) {
        vp_iowrite32((u32)val, lo);
        vp_iowrite32(val >> 32, hi); }
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Is it ok to be freezed in the middle of two vp_iowrite()?

Yes. the device context

VIRTIO_DEV_CTX_PCI_COMMON_RUNTIME_CFG

Quoted 58 lines
Quoted 58 lines
Quoted 4 lines
Quoted 4 lines

section captures the partial value.

There's no way for the device to know whether or not it's a partial value or

not.

Quoted 4 lines
Quoted 2 lines

No?

Device does not need to know, because when the guest vm and the device is

resumed on the destination, it the guest vm will continue with writing the 2nd part.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 10 lines

Review

Quoted 9 lines

Similarly, on the

(quoted diff) +5 -0
+destination hypervisor writes the device context
+first time while the device is still running in
+\field{Active} mode on the source hypervisor and
+writes the device context second time while the
+device is in

\field{Freeze} mode.

Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+This flow may result in very short setup time as the
+device context likely have minimal changes from the
+previously written device

context.

Quoted 1 line
Quoted 1 line
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Is the hypervisor who is in charge of doing the comparison and writing only the delta?

The spec commands allow to do so. So possibility exists from spec

wise.

Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines

There are various optimizations for migration for sure, I don't think mentioning any specific one is good.

The text is informative text similar to,

" However, some devices benefit from the ability to find out the amount of available data in the queue without accessing the virtqueue in

memory"

Quoted 3 lines

" To help with these optimizations, when
VIRTIO_F_NOTIFICATION_DATA has

been negotiated".

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Is this the only optimization in virtio? No, but we still mention the rationale of

why it exists.

The above is a good example as it explain VIRTIO_F_NOTIFICATION_DATA is the only way without accessing the virtqueue. But this is not the case of

migration.

Quoted 3 lines
Quoted 3 lines

You said it's just a possibility but not a must which is not the case for VIRTIO_F_NOTIFICATION_DATA.

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

It is one of the optimization apart. The comparison is of one_of_example or

not.

I don't get this.

Theory of operation is describing a flow how things are done and how the

constructs are helpful to achieve it.

Immature optimzation doesn't belong to theory for sure. I see your delta reporting immature in many ways. That's the point.

Thanks

Quoted 2 lines

And it is not the end of the list.
That does not mean one should not write those.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#23 lore raw

Re: [virtio-comment] [PATCH v1 1/8] admin: Add theory of operation for device migration

In reply to Jason Wang
Quoted 11 lines · Jason Wang
Quoted 3 lines
Quoted 1 line

1) Only works for SR-IOV member device like VF

It can be extended to SIOV member device in future. Today these are the only type of member device virtio has.

That is exactly what I want to say, it can only work for the owner/member model. It can't work when the virtio device is not structured like that. And you missed that most of the existing virtio devices are not implemented in this model. It means they can't be migrated with a pure virtio specific extension. For you, SR-IOV is all but this is not true for virtio. PCI is not the only transport and SR-IOV is not the only architecture in PCI.

Original version of admin command work supported another group type including just the device itself. In the end these commands are just a level of indirection.
Using them for migration was explicitly listed as a use case when they were introduced.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/