Skip to contentSkip to conversation

Introduce device migration support commands

PATCH Superseded Parav Pandit 5 participants 157 messages Oct 30, 2023 – Nov 29, 2023 Page 1 of 3

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Introduce device migration support commands

This series introduces administration commands for member device migration for PCI transport; when needed it can be extended for other transports too.

It takes inspiration from the similar idea presented at KVM Forum at [1].

Use case requirements:
======================
1. A hypervisor system needs to provide a PCI VF as passthrough device to the guest virtual machine and also support live migration of this virtual machine.
A passthrough device has typically only PCI configuration space and MSI-X table emulated by hypervisor. No virtio native interface offered by the virtio member device is trapped and/or emulated. This includes utilizing member device's native virtio common and device config region, device specific cvq, data vqs without any VMEXIT from the guest virtual machine and without any device type specific code in hypervisor; this is because it is already present in the owner and member device natively as unified interface for guest virtual machines, containers and may be more use cases.
2. A virtual machine may have one or more such passthrough virtio devices.
3. A virtual machine may have other PCI passthrough device which may also interact with virtio device.
4. A hypervisor runs a generic device type agnostic driver with extension to support device migration.
5. A PCI VF passthrough device needs to support transparent device reset and PCI FLR while the device migration is ongoing.
6. A owner driver do not involve in device operations mediation for the passthrough device at virtio interface level.
7. Mechanism is generic enough that applies to large family of virtio devices and it does not involve trapping any virtio device interfaces for the passthrough devices.

Overview:
=========
Above usecase requirements is solved by PCI PF group owner driver facilitating the member device migration functionality using administration commands.

There are three major functionalities.

1. Suspend and resume the device operation
2. Read and Write the device context containing all the information that can be transferred from source to destination to migrate to a member device
3. Track pages written by the device during device migration is ongoing

This comprehensive series introduces 4 infrastructure pieces covering PCI transport, peer to peer PCI devices, page write tracking (aka dirty page tracking) and generic virtio device context.

1. Device mode get,set (active, stop, freeze)
2. Device context read and write
3. Defines device context and compatibility command
4. Write reporting to track page addresses

This series enables virtio PCI SR-IOV member device to member device migration. It can also be used to/from migrate from PCI SR-IOV member device to software composed PCI device if/when needed which can parse and compose software based PCI virtio device.

This can also be useful for accessing member devices using some variant of only data path acceleration instead of of major passthrough functionality.

In future, for nested environment may be able to utilize the same infrastructure with VF capable of supporting nested VF with SR-IOV capability.

Example flow:
=============
Source hypervisor:
1. Instructs device to start tracking pages it is writing
2. Periodically query the addresses of the written pages
3. Suspend the device operation
4. Read the device context and transfer to destination hypervisor

Destination hypervisor:
5. Write the device context received from source
6. Resume the device that has newly written device context

Patch summary:
==============
patch-1: Adds theory of operation for device migration commands
patch-2: Redefine reserved2 to command output field
patch-3: Defines short device context for split virtqueues
patch-4: Adds device migration commands
patch-5: Adds requirements for device migration commands
patch-6: Adds theory of operation for write reporting commands
patch-7: Adds write reporting commands
patch-8: Adds requirements for write reporting commands

Please review.

Changelog:
==========
v2->v3:
- updated cover letter to reflect the use for passthrough and for only data path acceleration
- updated cover letter to utilize same infra for nested
- Addressed comments from Michael
- updated read context command to not depend on returned data length for closing the read context stream, instead depend on explicit read response with zero length
- fixed copy paste errors in write context command for fields description
- added device and driver normatives for {_START,_END}_MARKER fields
- wrote member VF device instead of VF device
v1->v2:
- Addressed comments from Michael and Jason
- replaced iova to page/physical address range in write recording commands
- several device specific requirements added to clarify, interaction of device reset, FLR, PCI PM and admin commands
- added device context fields query command to learn compatibility
- split device context field type range into generic and device specific
- added device context extension section to maintain backward and future compatibility
- several rewording in theory of operation
- added requirements to cover config space read/write interaction with device context commands
- added assumption about pci config space and msix table not present in device context, which can be added when hypervisor need them
v0->v1:
- enrich device context to cover device configuration layout, feature bits
- fixed alignment of device context fields
- added missing Sign-off for the joint work done with Satananda
- added link to the github issue

[1] https://static.sched.com/hosted_files/kvmforum2022/3a/KVM22-Migratable-Vhost-vDPA.pdf

Parav Pandit (8):
admin: Add theory of operation for device migration
admin: Redefine reserved2 as command specific output
device-context: Define the device context fields for device migration
admin: Add device migration admin commands
admin: Add requirements of device migration commands
admin: Add theory of operation for write recording commands
admin: Add write recording commands
admin: Add requirements of write reporting commands

 admin-cmds-device-migration.tex | 672 ++++++++++++++++++++++++++++++++
 admin.tex                       |  40 +-
 content.tex                     |   1 +
 device-context.tex              | 241 ++++++++++++
 4 files changed, 947 insertions(+), 7 deletions(-)
 create mode 100644 admin-cmds-device-migration.tex
 create mode 100644 device-context.tex
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

PATCH v3 1/8 admin: Add theory of operation for device migration

One or more passthrough PCI VF devices are ubiquitous for virtual machines usage using generic kernel framework.

A passthrough PCI VF device is fully owned by the virtual machine device driver. This passthrough device controls its own device reset flow, basic functionality as PCI VF function level reset and rest of the virtio device functionality such as control vq, config space access, data path descriptors handling.

Additionally, VM live migration using a precopy method is also widely used.

To support a VM live migration for such passthrough virtio member devices, the owner PCI PF device administers the device migration flow.

This patch introduces the basic theory of operation which describes the flow and supporting administration commands.

---
changelog:
v0->v1:
- addressed comments from Jason
- simplified commit log to remove wording of flow
- added link to the device reset section
- addressed comments from Michael

---
 admin-cmds-device-migration.tex | 95 +++++++++++++++++++++++++++++++++
 admin.tex                       |  1 +
 2 files changed, 96 insertions(+)
 create mode 100644 admin-cmds-device-migration.tex
admin-cmds-device-migration.tex new file +95 -0
new file mode 100644
index 0000000..d172130
@@ -0,0 +1,95 @@
+\subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Device Migration}
+
+In some systems, there is a need to migrate a running virtual machine
+from one to another system. A running virtual machine has one or more
+passthrough virtio member devices attached to it. A passthrough device
+is entirely operated by the guest virtual machine. For example, with
+the SR-IOV group type, group member device VF undergos device reset
+\ref{sec:Basic Facilities of a Virtio Device / Device Reset}
+and may also undergo PCI function level reset(FLR). Such operations
+are in control of the guest virtual machine which must comply to the
+device reset requirements and the PCI standard; at the same time those
+operations must not obstruct the device migration. In such a scenario,
+a group owner device can provide the administration command interface
+to facilitate the device migration related operations.
+
+When a virtual machine migrates from one hypervisor to another hypervisor,
+these hypervisors are named as source and destination hypervisor respectively.
+In such a scenario, a source hypervisor administers the
+member device to suspend the device and preserves the device context.
+Subsequently, a destination hypervisor administers the member device to
+setup a device context and resumes the member device. The source hypervisor
+reads the member device context and the destination hypervisor writes the member
+device context. The method to transfer the member device context from the source
+to the destination hypervisor is outside the scope of this specification.
+
+The member device can be in any of the three migration modes. The owner driver
+sets the member device in one of the following modes during device migration flow.
+
+\begin{tabularx}{\textwidth}{ |l||l|X| }
+\hline
+Value & Name & Description \\
+\hline \hline
+0x0   & Active &
+  It is the default mode after instantiation of the member device. \\
+\hline
+0x1   & Stop &
+ In this mode, the member device does not send any notifications,
+ and it does not access any driver memory.
+ The member device may receive driver notifications in this mode,
+ the member device context and device configuration space may change. \\
+\hline
+0x2   & Freeze &
+ In this mode, the member device does not accept any driver notifications,
+ it ignores any device configuration space writes,
+ the device do not have any changes in the device context. The
+ member device is not accessed in the system through the virtio interface. \\
+\hline
+\hline
+0x03-0xFF   & -    & reserved for future use \\
+\hline
+\end{tabularx}
+
+When the owner driver wants to stop the operation of the
+device, the owner driver sets the device mode to \field{Stop}. Once the
+device is in the \field{Stop} mode, the device does not initiate any notifications
+or does not access any driver memory. Since the member driver may be still
+active which may send further driver notifications to the device, the device
+context may be updated. When the member driver has stopped accessing the
+device, the owner driver sets the device to \field{Freeze} mode indicating
+to the device that no more driver access occurs. In the \field{Freeze} mode,
+no more changes occur in the device context. At this point, the device ensures
+that there will not be any update to the device context.
+
+The member device has a device context which the owner driver can either
+read or write. The member device context consist of any device specific
+data which is needed by the device to resume its operation when the device mode
+is changed from \field{Stop} to \field{Active} or from \field{Freeze}
+to \field{Active}.
+
+Once the device context is read, it is cleared from the device. Typically, on
+the source hypervisor, the owner driver reads the device context once when
+the device is in \field{Active} or \field{Stop} mode and later once the member
+device is in \field{Freeze} mode.
+
+Typically, the device context is read and written one time on the source and
+the destination hypervisor respectively once the device is in \field{Freeze}
+mode. On the destination hypervisor, after writing the device context,
+when the device mode set to \field{Active}, the device uses the most recently
+set device context and resumes the device operation.
+
+In an alternative flow, on the source hypervisor the owner driver may choose
+to read the device context first time while the device is in \field{Active} mode
+and second time once the device is in \field{Freeze} mode. Similarly, on the
+destination hypervisor writes the device context first time while the device
+is still running in \field{Active} mode on the source hypervisor and writes
+the device context second time while the device is in \field{Freeze} mode.
+This flow may result in very short setup time as the device context likely
+have minimal changes from the previously written device context. This flow may
+reduce the device migration time significantly and may have near constant
+device activation time regardless of number of virtqueues, resources and
+passthough devices in use by the migrating virtual machine.
+
+The owner driver can discard any partially read or written device context when
+any of the device migration flow should be aborted.
admin.tex +1 -0
index 0803c26..6eeef58 100644
@@ -297,6 +297,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 might differ between different group types.
 
 \input{admin-cmds-legacy-interface.tex}
+\input{admin-cmds-device-migration.tex}
 
 \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
 
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#2 lore raw

PATCH v3 2/8 admin: Redefine reserved2 as command specific output

In reply to Parav Pandit

Currently when a command wants to get two distinct types of data in the result, such as one consumed by the driver, other to be zero copied to some user buffers, the driver needs to prepare an extra descriptor for driver consumed field. When such a field is <= 4 bytes, extra descriptor is an overhead.

virtio_admin_command already has 4B of reserved for the device writable area. Utilize it to define as device writable output.

---
 admin.tex | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
admin.tex +9 -6
index 6eeef58..c86813d 100644
@@ -90,8 +90,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
         /* Device-writable part */
         le16 status;
         le16 status_qualifier;
-        /* unused, reserved for future extensions */
-        u8 reserved2[4];
+        u8 command_specific_output[4];
         u8 command_specific_result[];
 };
 \end{lstlisting}
@@ -192,11 +191,15 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline
 \end{tabularx}
 
-Each command uses a different \field{command_specific_data} and
-\field{command_specific_result} structures and the length of
+Each command uses a different \field{command_specific_data},
+\field{command_specific_output} and
+\field{command_specific_result} fields. The length of
 \field{command_specific_data} and \field{command_specific_result}
-depends on these structures and is described separately or is
-implicit in the structure description.
+depends on the command and is described separately or is
+implicit in the structure description. The \field{command_specific_output}
+describes any command specific output which is up to 4 bytes size. The
+\field{command_specific_output} contain one or more command specific
+fields.
 
 Before sending any group administration commands to the device, the driver
 needs to communicate to the device which commands it is going to
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#3 lore raw

PATCH v3 3/8 device-context: Define the device context fields for device migration

In reply to Parav Pandit

Define the device context and its fields for purpose of device migration. The device context is read and written by the owner driver on source and destination hypervisor respectively.

Device context fields will experience a rapid growth post this initial version to cover many details of the device.

---
changelog:
v2->v3:
- drop fields_count to support dynamic size device context
- instead, define start and end marker context fields
- added split virtqueue's used ring fields at they are read only fields for the device, so the device may not be able to read them after device is moved to active mode
- added device context invalidate tlv to speed up context migration under device reset flow
- split context types in 3 categories, marker, device common and device specific
v1->v2:
- addressed comments from Michael
- dropped layout from the enums and definition
- defined more practical fields type range of 16-bit
- split the range to generic and device type range
- added assumptions and device context extension sections for future proofing
v0->v1:
- enrich device context to cover feature bits, device configuration fields
- corrected alignment of device context fields

---
 content.tex        |   1 +
 device-context.tex | 241 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 242 insertions(+)
 create mode 100644 device-context.tex
content.tex +1 -0
index 0a62dce..2698931 100644
@@ -503,6 +503,7 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
 \input{admin.tex}
+\input{device-context.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
device-context.tex new file +241 -0

241 lines not on this page · raw message

Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#4 lore raw

PATCH v3 4/8 admin: Add device migration admin commands

In reply to Parav Pandit

A passthrough device is mapped to the guest VM. A passthrough device accessed by the driver can undergo its own device reset and for PCI transport it can undergo its PCI FLR while the guest VM migration is ongoing.
The passhtrough device may not have any direct channel through which device migration related administrative tasks can be done, and even if it may have such adminstative task must not be interrupted by the device reset or VF FLR flow initiated by the passthrough device.

Hence, the owner driver which administers the member devices, facilitate the device migration flow.

Add device migration administration commands that owner driver can use for the passthrough device.

Subsequent patch defines the device context in detail.

---
changelog:
v2->v3:
- updated read context command to not depend on returned data length to close the read context stream
- fixed copy paste errors in write context command for fields description
v1->v2:
- addressed comments from Michael
- updated commit log to refer to device context in later patch
- moved admin command table opcode to this (right) patch
- added command to query supported fields of the device context

---
 admin-cmds-device-migration.tex | 232 +++++++++++++++++++++++++++++++-
 admin.tex                       |  16 ++-
 device-context.tex              |   6 +-
 3 files changed, 249 insertions(+), 5 deletions(-)
admin-cmds-device-migration.tex +231 -1
index d172130..c5030d2 100644
@@ -66,7 +66,8 @@ \subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device /
 read or write. The member device context consist of any device specific
 data which is needed by the device to resume its operation when the device mode
 is changed from \field{Stop} to \field{Active} or from \field{Freeze}
-to \field{Active}.
+to \field{Active}. The device context is described in section
+\ref{sec:Basic Facilities of a Virtio Device / Device Context}.
 
 Once the device context is read, it is cleared from the device. Typically, on
 the source hypervisor, the owner driver reads the device context once when
@@ -93,3 +94,232 @@ \subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device /
 
 The owner driver can discard any partially read or written device context when
 any of the device migration flow should be aborted.
+
+The owner driver uses following device migration group administration commands.
+
+\begin{enumerate}
+\item Device Mode Get Command
+\item Device Mode Set Command
+\item Device Context Size Get Command
+\item Device Context Read Command
+\item Device Context Write Command
+\item Device Context Supported Fields Query Command
+\item Device Context Discard Command
+\end{enumerate}
+
+These commands are currently only defined for the SR-IOV group type.
+
+\paragraph{Device Mode Get Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Mode Get Command}
+
+This command reads the mode of the device.
+For the command VIRTIO_ADMIN_CMD_DEV_MODE_GET, \field{opcode}
+is set to 0x7.
+The \field{group_member_id} refers to the member device to be accessed.
+This command has no command specific data.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_mode_get_result {
+        u8 mode;
+};
+\end{lstlisting}
+
+When the command completes successfully, \field{command_specific_result}
+is in the format \field{struct virtio_admin_cmd_dev_mode_get_result}
+returned by the device where the device returns the \field{mode} value to
+either \field{Active} or \field{Stop} or \field{Freeze}.
+
+\paragraph{Device Mode Set Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Mode Set Command}
+
+This command sets the mode of the device.
+The \field{command_specific_data} is in the format
+\field{struct virtio_admin_cmd_dev_mode_set_data} describing the new device mode.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_mode_set_data {
+        u8 mode;
+};
+\end{lstlisting}
+
+For the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, \field{opcode} is set to 0x8.
+The \field{group_member_id} refers to the member device to be accessed.
+The \field{mode} is set to either \field{Active} or \field{Stop} or
+\field{Freeze}.
+
+This command has no command specific result. When the command completes
+successfully, device is set in the new \field{mode}. When the command fails
+the device stays in the previous mode.
+
+\paragraph{Device Context Size Get Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Context Size Get Command}
+
+This command returns the remaining estimated device context size. The 
+driver can query the remaining estimated device context size
+for the current mode or for the \field{Freeze} mode. While
+reading the device context using VIRTIO_ADMIN_CMD_DEV_CTX_READ command, the
+actual device context size may differ than what is being returned by
+this command. After reading the device context using command
+VIRTIO_ADMIN_CMD_DEV_CTX_READ, the remaining estimated context size
+usually reduces by amount of device context read by the driver using
+VIRTIO_ADMIN_CMD_DEV_CTX_READ command. If the device context is updated
+rapidly the remaining estimated context size may also increase even after
+reading the device context using VIRTIO_ADMIN_CMD_DEV_CTX_READ command.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_SIZE_GET, \field{opcode} is set to 0x9.
+The \field{group_member_id} refers to the member device to be accessed.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_ctx_size_get_data {
+        u8 freeze_mode;
+};
+\end{lstlisting}
+
+The \field{command_specific_data} is in the format
+\field{struct virtio_admin_cmd_dev_ctx_size_get_data}.
+When \field{freeze_mode} is set to 1, the device returns the estimated
+device context size when the device will be in \field{Freeze} mode.
+As the device context is read from the device, the remaining estimated
+context size may decrease. For example, member device mode is
+\field{Stop}, the device has estimated total device context size
+as 12KB; the device would return 12KB for the first
+VIRTIO_ADMIN_CMD_DEV_CTX_SIZE_GET command, once the driver has
+already read 8KB of device context data using
+VIRTIO_ADMIN_CMD_DEV_CTX_READ command, and the remaining data is
+4KB, hence the device returns 4KB in the subsequent
+VIRTIO_ADMIN_CMD_DEV_CTX_SIZE_GET command.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_ctx_size_get_result {
+        le64 size;
+};
+\end{lstlisting}
+
+When the command completes successfully, \field{command_specific_result} is in
+the format \field{struct virtio_admin_cmd_dev_ctx_size_get_result}.
+
+Once the device context is fully read, this command returns zero for
+\field{size} until the new device context is generated.
+
+\paragraph{Device Context Read Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Context Read Command}
+
+This command reads the current device context.
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_READ, \field{opcode} is set to 0xa.
+The \field{group_member_id} refers to the member device to be accessed.
+
+This command has no command specific data.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_ctx_rd_len {
+        le32 context_len;
+};
+
+struct virtio_admin_cmd_dev_ctx_rd_result {
+        u8 data[];
+};
+\end{lstlisting}
+
+When the command completes successfully, \field{command_specific_result}
+is in the format \field{struct virtio_admin_cmd_dev_ctx_rd_result}
+returned by the device containing the device context data and
+\field{command_specific_output} is in format of
+\field{struct virtio_admin_cmd_dev_ctx_rd_len} containing length of
+context data returned by the device in the command response. When the length
+returned is zero the device do not have any device context data left that
+the device can report, at this point the device context stream ends.
+
+The driver can read the whole device context data using one or multiple
+commands. When the device context does not fit in the
+\field{command_specific_result}, driver reads the subsequent remaining
+bytes using one or more subsequent commands.
+
+\paragraph{Device Context Write Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Context Write Command}
+
+This command writes the device context data. The device context can be written
+only when the device mode is \field{Freeze}.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_WRITE, \field{opcode}
+is set to 0xb.
+The \field{group_member_id} refers to the member device to be accessed.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_ctx_wr_data {
+        u8 data[];
+};
+\end{lstlisting}
+
+The \field{command_specific_data} is in the format
+\field{struct virtio_admin_cmd_dev_ctx_wr_data} describing
+the access to be performed.
+
+This command has no command specific result.
+The device fails the command when command is executed when the device mode
+is other than \field{Freeze}.
+
+The written device context is effective when the device mode is changed
+from \field{Freeze} to \field{Stop} or from \field{Freeze} to \field{Active}.
+
+The driver can write the whole device context using one or multiple
+commands. When the device context does not fit in one command data
+\field{struct virtio_admin_cmd_dev_ctx_wr_data}, the driver writes the
+subsequent remaining bytes using one or more subsequent commands.
+
+\paragraph{Device Context Supported Fields Query Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Context Supported Fields Query Command}
+
+This command reads supported fields in the device context.
+Each listed \field{type} of the device context of
+\ref{sec:Basic Facilities of a Virtio Device / Device Context} is represented
+as one entry the command response. When the device support a given \field{type} for the member
+device, corresponding entry is set in the command response.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_FIELDS_QUERY, \field{opcode} is set to 0xc.
+The \field{group_member_id} refers to the member device to be accessed.
+
+This command has no command specific data.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_dev_ctx_supported_field {
+        le16 type;
+        u8 reserved[6];
+        le64 length;
+};
+
+struct virtio_admin_cmd_dev_ctx_supported_fields_result {
+        struct virtio_admin_cmd_dev_ctx_supported_field fields[];
+};
+\end{lstlisting}
+
+When the command completes successfully, \field{command_specific_result}
+is in the format \field{struct virtio_admin_cmd_dev_ctx_supported_fields_result}.
+Each entry in array \field{fields} represents the supported \field{type}
+and its length as described in \ref{tab:Device Context Fields}.
+
+\paragraph{Device Context Discard Command}
+\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration / Device Context Discard Command}
+
+This command discards any partial device context that is yet to be read
+by the driver and it also discards any device context that is partially written.
+This command can be used by the driver to abort any device context migration
+flow when there may have been any partial context read or write operations
+have occurred.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_DISCARD, \field{opcode}
+is set to 0xd.
+The \field{group_member_id} refers to the member device to be accessed.
+
+This command has no command specific data.
+This command has no command specific result.
+
+Once this command completes successfully, the device context is
+discarded. If the device context that is discarded was part of the write
+operation, once this command completes, the device functions as if the device
+context was never written. If the device context that is discarded was part
+of the read operation, once this command completes, the device functions as if
+the device context was never read in the given device mode. Once the device
+context is discarded, in subsequent VIRTIO_ADMIN_CMD_DEV_CTX_READ command,
+the device returns new device context entry. Once the device context is
+discarded, subsequent VIRTIO_ADMIN_CMD_DEV_CTX_WRITE command writes a new device
+context.
admin.tex +15 -1
index c86813d..142692c 100644
@@ -126,7 +126,21 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline
 0x0006 & VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO & Query the notification region information \\
 \hline
-0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
+0x0007 & VIRTIO_ADMIN_CMD_DEV_MODE_GET & Query the device mode \\
+\hline
+0x0008 & VIRTIO_ADMIN_CMD_DEV_MODE_SET & Set the device mode \\
+\hline
+0x0009 & VIRTIO_ADMIN_CMD_DEV_CTX_SIZE_GET & Query the device context size \\
+\hline
+0x000a & VIRTIO_ADMIN_CMD_DEV_CTX_READ & Read the device context data \\
+\hline
+0x000b & VIRTIO_ADMIN_CMD_DEV_CTX_WRITE & Write the device context data \\
+\hline
+0x000c & VIRTIO_ADMIN_CMD_DEV_CTX_FIELDS_QUERY & Query Supported fields of device context \\
+\hline
+0x000d & VIRTIO_ADMIN_CMD_DEV_CTX_DISCARD & Clear the device context data \\
+\hline
+0x000e - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
 \hline
device-context.tex +3 -3
index 06ed43d..1eb71f7 100644
@@ -133,9 +133,9 @@ \subsubsection{PCI Virtqueue Configuration Context}
         le16 queue_size;
         le16 queue_msix_vector;
         le16 reserved;
-        le64 queue_desc;
-        le64 queue_driver;
-        le64 queue_device;
+        le64 queue_desc;
+        le64 queue_driver;
+        le64 queue_device;
 };
 \end{lstlisting}
 
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#5 lore raw

PATCH v3 5/8 admin: Add requirements of device migration commands

In reply to Parav Pandit

Add device and driver side requirements for the device migration commands.

---
changelog:
v2->v3:
- added device and driver normatives for {_START,_END}_MARKER fields
- wrote member VF device instead of VF device
v1->v2:
- fixed spelling from membe to member
- removed device requirement line of FLR making the device active as it was incorrectly written to mix operational and admin state
- added requirements to clarify flr, device reset, pm and admin commands
- group sriov requirements
- added description for device config space access in stop mode
- removed stale requirement around pci ids
- made device context write command requirements more robust for future and backward compatibility

---
 admin-cmds-device-migration.tex | 173 ++++++++++++++++++++++++++++++++
 1 file changed, 173 insertions(+)
admin-cmds-device-migration.tex +173 -0
index c5030d2..ed911e4 100644
@@ -323,3 +323,176 @@ \subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device /
 the device returns new device context entry. Once the device context is
 discarded, subsequent VIRTIO_ADMIN_CMD_DEV_CTX_WRITE command writes a new device
 context.
+
+\devicenormative{\paragraph}{Device Migration}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration}
+
+A device MUST either support all of, or none of
+VIRTIO_ADMIN_CMD_DEV_MODE_GET,
+VIRTIO_ADMIN_CMD_DEV_MODE_SET,
+VIRTIO_ADMIN_CMD_DEV_CTX_SIZE_GET,
+VIRTIO_ADMIN_CMD_DEV_READ,
+VIRTIO_ADMIN_CMD_DEV_WRITE and
+VIRTIO_ADMIN_CMD_DEV_CTX_DISCARD commands.
+
+When the device \field{mode} supplied in the command
+VIRTIO_ADMIN_CMD_DEV_MODE_SET is same as what the mode in the device, the device
+MUST complete the command successfully.
+
+The device MUST fail the command VIRTIO_ADMIN_CMD_DEV_MODE_SET when the \field{mode}
+is other than \field{Active} or \field{Stop} or \field{Freeze}.
+
+When changing the device mode using the command VIRTIO_ADMIN_CMD_DEV_MODE_SET,
+if the command fails, the device MUST retain the current device mode.
+
+The device MUST fail VIRTIO_ADMIN_CMD_DEV_MODE_SET command when \field{mode}
+is set to \field{Active} or \field{Stop} and if the device context is
+partially read or written using VIRTIO_ADMIN_CMD_DEV_CTX_READ and
+VIRTIO_ADMIN_CMD_DEV_CTX_WRITE commands respectively.
+
+When VIRTIO_ADMIN_CMD_DEV_CTX_READ command is received multiple times
+in a given mode, and when the complete device context is already read by the
+driver, on subsequent reception of command VIRTIO_ADMIN_CMD_DEV_CTX_READ,
+the device MUST complete the command successfully with
+\field{context_len} set to zero.
+
+The device MUST support reading the device context when the device is
+in any mode \field{Active} or \field{Stop} or \field{Freeze} using command
+VIRTIO_ADMIN_CMD_DEV_CTX_READ.
+
+When the device is in any of the mode, and if the device context is read
+partially using VIRTIO_ADMIN_CMD_DEV_CTX_READ command, the device MUST discard
+the device context when VIRTIO_ADMIN_CMD_DEV_CTX_DISCARD command is executed;
+In subsequent execution of VIRTIO_ADMIN_CMD_DEV_CTX_READ and
+VIRTIO_ADMIN_CMD_DEV_CTX_SIZE_GET, the device MUST return the remaining
+estimated device context size and the device context respectively for the
+current mode as if VIRTIO_ADMIN_CMD_DEV_CTX_READ was never received by the
+device for the current device mode.
+
+The device MUST set VIRTIO_DEV_CTX_START_MARKER, VIRTIO_DEV_CTX_END_MARKER
+and VIRTIO_DEV_CTX_DISCARD in VIRTIO_ADMIN_CMD_DEV_CTX_FIELDS_QUERY
+command result.
+
+The device MUST support writing the complete device context multiple times
+by the command VIRTIO_ADMIN_CMD_DEV_CTX_WRITE.
+
+The device MUST fail VIRTIO_ADMIN_CMD_DEV_CTX_WRITE command when the device
+mode is not \field{Freeze}.
+
+For the SR-IOV group type,
+\begin{itemize}
+\item the device MUST not initiate any PCI transaction
+      when the device mode is not \field{Active}.
+\item the device MUST finish all the outstanding PCI transactions before completing
+      the command VIRTIO_ADMIN_CMD_DEV_MODE_SET.
+\item when the device mode is \field{Stop}, the device MUST accept driver
+       notifications and the device MAY update any fields of the device context.
+\item the device MUST respond with valid values for PCI read requests when
+      the device mode is \field{Stop}.
+\item the device MUST function same for the PCI architected interfaces
+      regardless of the device mode.
+\item the device MUST not generate any PCI PME when the device is
+      not in \field{Active} state.
+\item the device MUST NOT update any fields of the device context when the
+      device is in \field{Freeze} mode, the device MAY update fields of the
+      device context when the device transitions from \field{Stop} to
+      \field{Freeze} mode.
+\end{itemize}
+
+When the device mode is not \field{Active},
+\begin{itemize}
+\item the device MUST not access any virtqueue memory or any memory referred
+      by the virtqueue when the device mode is not \field{Active}.
+
+\item the device MUST not generate any configuration change notification. 
+\end{itemize}
+
+When the device is in \field{Freeze} mode, and if any device context is
+written partially by VIRTIO_ADMIN_CMD_DEV_CTX_WRITE, the device MUST discard
+the device context when VIRTIO_ADMIN_CMD_DEV_CTX_DISCARD
+command is executed, i.e. the device functions as if the command
+VIRTIO_ADMIN_CMD_DEV_CTX_WRITE was never received.
+
+For the SR-IOV group type,
+\begin{itemize}
+\item when the device is in \field{Freeze} mode, any
+write access to virtio configuration space MUST not update any fields and any
+configuration space read MAY return any value.
+
+\item for the VIRTIO_PCI_CAP_PCI_CFG capability area,
+the device MUST ignore writes when the device mode is set to \field{Freeze}
+and on receiving the reads, the device MUST function same regardless of the
+device mode is \field{Active} or \field{Stop} or \field{Freeze}.
+
+\item the member VF device MUST respond to commands
+VIRTIO_ADMIN_CMD_DEV_MODE_SET, VIRTIO_ADMIN_CMD_DEV_CTX_WRITE and
+VIRTIO_ADMIN_CMD_DEV_CTX_READ after the member VF device FLR completes in the
+device, if the member VF device FLR is in progress when the device receives
+any of these commands.
+
+\item the member device MUST respond to commands
+VIRTIO_ADMIN_CMD_DEV_MODE_SET, VIRTIO_ADMIN_CMD_DEV_CTX_WRITE and
+VIRTIO_ADMIN_CMD_DEV_CTX_READ after the device reset completes in the device, if the
+device reset is in progress when the device receives any of these commands.
+
+\item the member device MUST respond to commands
+VIRTIO_ADMIN_CMD_DEV_MODE_SET, VIRTIO_ADMIN_CMD_DEV_CTX_WRITE and
+VIRTIO_ADMIN_CMD_DEV_CTX_READ after the device power management state transition completes
+in the device, if the power management state transition is in progress
+when the device receives any of these commands.
+\end{itemize}
+
+The device MUST respond with an error for the command
+VIRTIO_ADMIN_CMD_DEV_CTX_WRITE, if there is a mismatch between the
+device context field length supplied in the
+VIRTIO_ADMIN_CMD_DEV_CTX_WRITE data and the length of the field
+in the device.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_READ,
+\begin{itemize}
+\item when there is valid context to respond at the start, the device MUST respond
+\field{field[0]} in \field{struct virtio_dev_ctx} of type
+VIRTIO_DEV_CTX_START_MARKER.
+
+\item the device MUST end the device context \field{struct virtio_dev_ctx},
+with last entry of \field{fields[]} with type VIRTIO_DEV_CTX_END_MARKER.
+
+\item the device MUST NOT respond VIRTIO_DEV_CTX_END_MARKER followed by
+VIRTIO_DEV_CTX_START_MARKER in single command response.
+
+\item the device MAY respond VIRTIO_DEV_CTX_START_MARKER followed by
+VIRTIO_DEV_CTX_END_MARKER in two different command responses.
+
+\item the device MAY respond VIRTIO_DEV_CTX_START_MARKER followed by
+VIRTIO_DEV_CTX_END_MARKER in single command response.
+\end{itemize}
+
+\drivernormative{\paragraph}{Device Migration}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device Migration}
+
+The driver SHOULD read the complete device context using one or multiple
+VIRTIO_ADMIN_CMD_DEV_CTX_READ commands.
+
+The driver MAY write the device context before changing the device mode from
+\field{Freeze} to \field{Stop} or from \field{Freeze} to \field{Active};
+the driver MUST write a complete device context using one or multiple
+VIRTIO_ADMIN_CMD_DEV_CTX_WRITE commands.
+
+The driver MUST NOT change the device mode to \field{Stop} or \field{Active}
+in the command VIRTIO_ADMIN_CMD_DEV_MODE_SET when device context is
+partially written.
+
+For the SR-IOV group type, the driver SHOULD NOT access device configuration
+space described in section
+\ref{sec:Basic Facilities of a Virtio Device / Device Configuration Space}
+when the device mode is set to \field{Freeze} or \field{Stop}.
+
+For the SR-IOV group type, the driver MUST NOT write into the
+VIRTIO_PCI_CAP_PCI_CFG capability area when the device mode is set to
+\field{Freeze}.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_WRITE, at start, the driver MUST
+set \field{field[0]} in \field{struct virtio_dev_ctx} of type
+VIRTIO_DEV_CTX_START_MARKER.
+
+For the command VIRTIO_ADMIN_CMD_DEV_CTX_WRITE, the driver MUST
+end the device context \field{struct virtio_dev_ctx}, with last
+entry of \field{fields[]} with type VIRTIO_DEV_CTX_END_MARKER.
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#6 lore raw

PATCH v3 6/8 admin: Add theory of operation for write recording commands

In reply to Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
admin-cmds-device-migration.tex +15 -0
index ed911e4..2e32f2c 100644
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device /
 The owner driver can discard any partially read or written device context when
 any of the device migration flow should be aborted.
 
+During the device migration flow, a passthrough device may write data to the
+guest virtual machine's memory, a source hypervisor needs to keep track of these
+written memory to migrate such memory to destination hypervisor.
+Some systems may not be able to keep track of such memory write addresses at
+hypervisor level. In such a scenario, a device records and reports these
+written memory addresses to the owner device. The owner driver enables write
+recording for one or more physical address ranges per device during device
+migration flow. The owner driver periodically queries these written physical
+address records from the device. As the driver reads the written address records,
+the device clears those records from the device.
+Once the device reports zero or small number of written address records, the device
+mode is set to \field{Stop} or \field{Freeze}. Once the device is set to \field{Stop}
+or \field{Freeze} mode, and once all the IOVA records are read, the driver stops
+the write recording in the device.
+
 The owner driver uses following device migration group administration commands.
 
 \begin{enumerate}
Signature · 18 lines
2.34.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#7 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 30 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
admin-cmds-device-migration.tex +0 -0
index ed911e4..2e32f2c 100644
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device /
 The owner driver can discard any partially read or written device context when
 any of the device migration flow should be aborted.

admin-cmds-device-migration.tex \subsubsection{Device Migration}\label{sec:Basic Facilities of a Virtio Device / · line 106

Quoted 10 lines
admin-cmds-device-migration.tex +9 -0
 
+During the device migration flow, a passthrough device may write data to the
+guest virtual machine's memory, a source hypervisor needs to keep track of these
+written memory to migrate such memory to destination hypervisor.
+Some systems may not be able to keep track of such memory write addresses at
+hypervisor level. In such a scenario, a device records and reports these
+written memory addresses to the owner device. The owner driver enables write
+recording for one or more physical address ranges per device during device
+migration flow. The owner driver periodically queries these written physical
+address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 42 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 28 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
Quoted 2 lines

Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex +3 -0
+During the device migration flow, a passthrough device may write data
+to the guest virtual machine's memory, a source hypervisor needs to
+keep track of these written memory to migrate such memory to destination

hypervisor.

Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device records
+and reports these written memory addresses to the owner device. The
+owner driver enables write recording for one or more physical address
+ranges per device during device migration flow. The owner driver
+periodically queries these written physical address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.

#9 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 51 lines · Parav Pandit
Quoted 42 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 28 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
Quoted 2 lines

Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex +3 -0
+During the device migration flow, a passthrough device may write data
+to the guest virtual machine's memory, a source hypervisor needs to
+keep track of these written memory to migrate such memory to destination

hypervisor.

Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device records
+and reports these written memory addresses to the owner device. The
+owner driver enables write recording for one or more physical address
+ranges per device during device migration flow. The owner driver
+periodically queries these written physical address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Quoted 1 line

Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.

Yes the spec calls the address accessed by the device "physical address". Granted, this is pointless - there is only one type of address device can access. We can if we want to replace that with just "address" or "memory address". I don't think this ever caused confusion though and worth the churn.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#10 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 48 lines · Michael S. Tsirkin
Quoted 47 lines · Parav Pandit
Quoted 47 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 27 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
  admin-cmds-device-migration.tex | 15 +++++++++++++++
  1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
Quoted 2 lines

Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 3 lines
admin-cmds-device-migration.tex +3 -0
+During the device migration flow, a passthrough device may write data
+to the guest virtual machine's memory, a source hypervisor needs to
+keep track of these written memory to migrate such memory to destination

hypervisor.

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device records
+and reports these written memory addresses to the owner device. The
+owner driver enables write recording for one or more physical address
+ranges per device during device migration flow. The owner driver
+periodically queries these written physical address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Quoted 8 lines
Quoted 1 line

Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.

Yes the spec calls the address accessed by the device "physical address". Granted, this is pointless - there is only one type of address device can access. We can if we want to replace that with just "address" or "memory address". I don't think this ever caused confusion though and worth the churn.

But you know PA means disable IOMMU and always enable ATS on both the device and host

Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#11 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 53 lines · Lingshan
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Parav Pandit
Quoted 39 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 27 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
  admin-cmds-device-migration.tex | 15 +++++++++++++++
  1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
Quoted 2 lines

Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 3 lines
admin-cmds-device-migration.tex +3 -0
+During the device migration flow, a passthrough device may write data
+to the guest virtual machine's memory, a source hypervisor needs to
+keep track of these written memory to migrate such memory to destination

hypervisor.

Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device records
+and reports these written memory addresses to the owner device. The
+owner driver enables write recording for one or more physical address
+ranges per device during device migration flow. The owner driver
+periodically queries these written physical address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.

Quoted 8 lines
Quoted 6 lines

Yes the spec calls the address accessed by the device "physical address". Granted, this is pointless - there is only one type of address device can access. We can if we want to replace that with just "address" or "memory address". I don't think this ever caused confusion though and worth the churn.

But you know PA means disable IOMMU and always enable ATS on both the device and host

That is not how virtio uses the term. Just grep "physical address" in the spec. This patch should be consistent.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#12 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 57 lines · Michael S. Tsirkin
Quoted 56 lines · Lingshan
Quoted 54 lines · Michael S. Tsirkin
Quoted 46 lines · Parav Pandit
Quoted 38 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 27 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
   admin-cmds-device-migration.tex | 15 +++++++++++++++
   1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
Quoted 2 lines

Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 3 lines
admin-cmds-device-migration.tex +3 -0
+During the device migration flow, a passthrough device may write data
+to the guest virtual machine's memory, a source hypervisor needs to
+keep track of these written memory to migrate such memory to destination

hypervisor.

Quoted 8 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device records
+and reports these written memory addresses to the owner device. The
+owner driver enables write recording for one or more physical address
+ranges per device during device migration flow. The owner driver
+periodically queries these written physical address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Quoted 1 line

Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.

Yes the spec calls the address accessed by the device "physical address". Granted, this is pointless - there is only one type of address device can access. We can if we want to replace that with just "address" or "memory address". I don't think this ever caused confusion though and worth the churn.

Quoted 4 lines
Quoted 2 lines

But you know PA means disable IOMMU and always enable ATS on both the device and host

That is not how virtio uses the term. Just grep "physical address" in the spec. This patch should be consistent.

OK, I use other terms in my series, let's see whether it is better.

Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#13 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Parav Pandit
Quoted 52 lines · Parav Pandit
Quoted 43 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 28 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device Migration}\label{sec:Basic
Quoted 2 lines

Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 4 lines
admin-cmds-device-migration.tex +3 -0
+During the device migration flow, a passthrough device may write data
+to the guest virtual machine's memory, a source hypervisor needs to
+keep track of these written memory to migrate such memory to destination

hypervisor.

Quoted 9 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device records
+and reports these written memory addresses to the owner device. The
+owner driver enables write recording for one or more physical address
+ranges per device during device migration flow. The owner driver
+periodically queries these written physical address records from the device.

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Quoted 1 line

Michael suggested to keep the language uniform as PA as this is ultimately what the guest driver is supplying during vq creation and in posting buffers as physical address.

This seems to need some work. And, can you show me how it can work?

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?
3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer? I can't see how it can compete with the functionality that is provided by the platform. And what's more, we can't assume virtio is the only device that is used by the guest.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#14 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 65 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, November 1, 2023 6:00 AM

Quoted 45 lines · Parav Pandit
Quoted 45 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 28 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device
Quoted 3 lines

Migration}\label{sec:Basic Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +4 -0
+During the device migration flow, a passthrough device may write
+data to the guest virtual machine's memory, a source hypervisor
+needs to keep track of these written memory to migrate such
+memory to destination

hypervisor.

Quoted 6 lines
Quoted 6 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device
+records and reports these written memory addresses to the owner
+device. The owner driver enables write recording for one or more
+physical address ranges per device during device migration flow.
+The owner driver periodically queries these written physical address

records from the device.

Quoted 4 lines
Quoted 3 lines

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately

what the guest driver is supplying during vq creation and in posting buffers as physical address.

Quoted 6 lines

This seems to need some work. And, can you show me how it can work?

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?

No when dirty page logging starts, only at that time, device will reserve enough resources.

Quoted 2 lines

3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer?

He he, you have funny comment.
When an abstract facility is added to virtio you say to do in transport. When one does something in transport, you say, this is transport specific, do some generic.

Here the device is being tracked is virtio device.
PCI-SIG has told already that PCIM interface is outside the scope of it. Hence, this is done in virtio layer here in abstract way.

Quoted 4 lines

I can't see how it can compete with the functionality
that is provided by the platform. And what's more, we can't assume virtio is the only device that is used by the guest.

You raised this before and it was answered.
Not all platform support dirty page tracking effectively.
This is optional facility that speed up the migration down time significantly. So until platform supports it, it is supported by virtio.

Quoted 1 line

Thanks

#15 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 66 lines · Parav Pandit
Quoted 66 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, November 1, 2023 6:00 AM

Quoted 45 lines · Parav Pandit
Quoted 45 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

Quoted 28 lines · Parav Pandit

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these written pages. These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device
Quoted 3 lines

Migration}\label{sec:Basic Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +4 -0
+During the device migration flow, a passthrough device may write
+data to the guest virtual machine's memory, a source hypervisor
+needs to keep track of these written memory to migrate such
+memory to destination

hypervisor.

Quoted 6 lines
Quoted 6 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory write
+addresses at hypervisor level. In such a scenario, a device
+records and reports these written memory addresses to the owner
+device. The owner driver enables write recording for one or more
+physical address ranges per device during device migration flow.
+The owner driver periodically queries these written physical address

records from the device.

Quoted 4 lines
Quoted 3 lines

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately

what the guest driver is supplying during vq creation and in posting buffers as physical address.

Quoted 7 lines
Quoted 6 lines

This seems to need some work. And, can you show me how it can work?

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?

No when dirty page logging starts, only at that time, device will reserve enough resources.

GAW is 48bit, how large would it have then? What happens if we're trying to migrate more than 1 device?

Quoted 5 lines
Quoted 3 lines

3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer?

He he, you have funny comment.
When an abstract facility is added to virtio you say to do in transport.

So it's not done in the general facility but tied to the admin part. And we all know dirty page tracking is a challenge and Eugenio has a good summary of pros/cons. A revisit of those docs make me think virtio is not the good place for doing that for may reasons:

1) as stated, platform will evolve to be able to tracking dirty pages, actually, it has been supported by a lot of major IOMMU vendors
2) you can't assume virtio is the only device that can be used by the guest, having dirty pages tracking to be implemented in each type of device is unrealistic
3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform
4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 5 lines

When one does something in transport, you say, this is transport specific, do some generic.

Here the device is being tracked is virtio device.
PCI-SIG has told already that PCIM interface is outside the scope of it. Hence, this is done in virtio layer here in abstract way.

You will end up with a competition with the platform/transport one that will fail.

Quoted 8 lines
Quoted 5 lines

I can't see how it can compete with the functionality
that is provided by the platform. And what's more, we can't assume virtio is the only device that is used by the guest.

You raised this before and it was answered.
Not all platform support dirty page tracking effectively.
This is optional facility that speed up the migration down time significantly.

I can hardly believe the downtime is determined by the speed of logging dirty pages...

Quoted 1 line

So until platform supports it, it is supported by virtio.

Some platforms already support that.

Thanks

Quoted 3 lines
Quoted 2 lines

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#16 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 79 lines

From: Jason Wang <[email protected]>
Sent: Thursday, November 2, 2023 9:54 AM

Quoted 14 lines · Parav Pandit
Quoted 14 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, November 1, 2023 6:00 AM

Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

On Mon, Oct 30, 2023 at 9:21 PM Parav Pandit <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these

written pages.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 25 lines

These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device
Quoted 3 lines

Migration}\label{sec:Basic Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +4 -0
+During the device migration flow, a passthrough device may
+write data to the guest virtual machine's memory, a source
+hypervisor needs to keep track of these written memory to
+migrate such memory to destination

hypervisor.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +5 -0
+Some systems may not be able to keep track of such memory
+write addresses at hypervisor level. In such a scenario, a
+device records and reports these written memory addresses to
+the owner device. The owner driver enables write recording
+for one or more physical address ranges per device during device

migration flow.

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines
Quoted 2 lines

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex +2 -0
+The owner driver periodically queries these written physical
+address

records from the device.

Quoted 5 lines
Quoted 3 lines

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately

what the guest driver is supplying during vq creation and in posting buffers as physical address.

This seems to need some work. And, can you show me how it can work?

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?

No when dirty page logging starts, only at that time, device will reserve

enough resources.

GAW is 48bit, how large would it have then?

Dirty page tracking is not dependent on the size of the GAW.
It is function of address ranges for the amount of guest memory regardless of GAW.

Quoted 2 lines

What happens if we're trying to migrate more than 1 device?

That is perfectly fine.
Each device is updating its log of pages it wrote. The hypervisor is collecting their sum.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer?

He he, you have funny comment.
When an abstract facility is added to virtio you say to do in transport.

Quoted 7 lines

So it's not done in the general facility but tied to the admin part. And we all know dirty page tracking is a challenge and Eugenio has a good summary of pros/cons. A revisit of those docs make me think virtio is not the good place for doing that for may reasons:

1) as stated, platform will evolve to be able to tracking dirty pages, actually, it has been supported by a lot of major IOMMU vendors

This is optional facility in virtio.
Can you please point to the references? I don’t see it in the common Linux kernel support for it. Instead Linux kernel choose to extend to the devices.
At least not seen to arrive this in any near term in start of 2024 which is where users must use this.

Quoted 3 lines

2) you can't assume virtio is the only device that can be used by the guest, having dirty pages tracking to be implemented in each type of device is unrealistic

Of course, there is no such assumption made. Where did you see a text that made such assumption? Each virtio and non virtio devices who wants to report their dirty page report, will do their way.

Quoted 4 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Can you bring the cpu vendors and committement to virtio tc with timelines so that virtio TC can omit?
i.e. in first year of 2024?
If not, we are better off to offer this, and when/if platform support is, sure, this feature can be disabled/not used/not enabled.

Quoted 2 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate done by the cpu. It simply does not even pass a simple 10Gbps test.
There is no requirement for mandating PRI either.
So it is unusable.

Quoted 11 lines
Quoted 2 lines

When one does something in transport, you say, this is transport specific, do

some generic.

Quoted 4 lines

Here the device is being tracked is virtio device.
PCI-SIG has told already that PCIM interface is outside the scope of it. Hence, this is done in virtio layer here in abstract way.

You will end up with a competition with the platform/transport one that will fail.

I don’t see a reason. There is no competition.
Platform always have a choice to not use device side page tracking when it is supported.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I can't see how it can compete with the functionality that is provided by the platform. And what's more, we can't assume virtio is the only device that is used by the guest.

Quoted 7 lines
Quoted 3 lines

You raised this before and it was answered.
Not all platform support dirty page tracking effectively.
This is optional facility that speed up the migration down time significantly.

I can hardly believe the downtime is determined by the speed of logging dirty pages...

Without dirty page tracking, in pre-copy, all the memory must be migrated again, which takes significantly longer time.

Quoted 4 lines
Quoted 1 line

So until platform supports it, it is supported by virtio.

Some platforms already support that.

And some done.
Hence whichever wants to use by platform, will use by platform. Whichever prefer to use from device will use by the device.

#17 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 84 lines · Parav Pandit
Quoted 84 lines

From: Jason Wang <[email protected]>
Sent: Thursday, November 2, 2023 9:54 AM

Quoted 14 lines · Parav Pandit
Quoted 14 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, November 1, 2023 6:00 AM

Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

On Mon, Oct 30, 2023 at 9:21 PM Parav Pandit <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these

written pages.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 25 lines

These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device
Quoted 3 lines

Migration}\label{sec:Basic Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +4 -0
+During the device migration flow, a passthrough device may
+write data to the guest virtual machine's memory, a source
+hypervisor needs to keep track of these written memory to
+migrate such memory to destination

hypervisor.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +5 -0
+Some systems may not be able to keep track of such memory
+write addresses at hypervisor level. In such a scenario, a
+device records and reports these written memory addresses to
+the owner device. The owner driver enables write recording
+for one or more physical address ranges per device during device

migration flow.

Quoted 17 lines
Quoted 17 lines
Quoted 2 lines
Quoted 2 lines

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex +2 -0
+The owner driver periodically queries these written physical
+address

records from the device.

Quoted 5 lines
Quoted 3 lines

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately

what the guest driver is supplying during vq creation and in posting buffers as physical address.

This seems to need some work. And, can you show me how it can work?

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

No when dirty page logging starts, only at that time, device will reserve

enough resources.

GAW is 48bit, how large would it have then?

Dirty page tracking is not dependent on the size of the GAW.
It is function of address ranges for the amount of guest memory regardless of GAW.

The problem is, e.g when vIOMMU is enabled, you can't know which IOVA is actually used by guests. And even for the case when vIOMMU is not enabled, the guest may have several TBs. Is it easy to reserve sufficient resources by the device itself?

Host should always have more resources than device, in that sense there could be several methods that tries to utilize host memory instead of the one in the device. I think we've discussed this when going through the doc prepared by Eugenio.

Quoted 6 lines
Quoted 3 lines

What happens if we're trying to migrate more than 1 device?

That is perfectly fine.
Each device is updating its log of pages it wrote. The hypervisor is collecting their sum.

See above.

Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines

3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer?

He he, you have funny comment.
When an abstract facility is added to virtio you say to do in transport.

So it's not done in the general facility but tied to the admin part. And we all know dirty page tracking is a challenge and Eugenio has a good summary of pros/cons. A revisit of those docs make me think virtio is not the good place for doing that for may reasons:

Quoted 5 lines
Quoted 2 lines

1) as stated, platform will evolve to be able to tracking dirty pages, actually, it has been supported by a lot of major IOMMU vendors

This is optional facility in virtio.
Can you please point to the references? I don’t see it in the common Linux kernel support for it.

Note that when IOMMUFD is being proposed, dirty page tracking is one of the major considerations.

This is one recent proposal:

https://www.spinics.net/lists/kvm/msg330894.html

Quoted 1 line

Instead Linux kernel choose to extend to the devices.

Well, as I stated, tracking dirty pages is challenging if you want to do it on a device, and you can't simply invent dirty page tracking for each type of the devices.

Quoted 6 lines

At least not seen to arrive this in any near term in start of 2024 which is where users must use this.

Quoted 3 lines

2) you can't assume virtio is the only device that can be used by the guest, having dirty pages tracking to be implemented in each type of device is unrealistic

Of course, there is no such assumption made. Where did you see a text that made such assumption?

So what happens if you have a guest with virtio and other devices assigned?

Quoted 7 lines

Each virtio and non virtio devices who wants to report their dirty page report, will do their way.

Quoted 4 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Can you bring the cpu vendors and committement to virtio tc with timelines so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

Quoted 1 line

i.e. in first year of 2024?

Why does it matter in 2024?

Quoted 7 lines

If not, we are better off to offer this, and when/if platform support is, sure, this feature can be disabled/not used/not enabled.

Quoted 2 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate done by the cpu. It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Quoted 2 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

When one does something in transport, you say, this is transport specific, do

some generic.

Quoted 4 lines

Here the device is being tracked is virtio device.
PCI-SIG has told already that PCIM interface is outside the scope of it. Hence, this is done in virtio layer here in abstract way.

Quoted 5 lines
Quoted 3 lines

You will end up with a competition with the platform/transport one that will fail.

I don’t see a reason. There is no competition.
Platform always have a choice to not use device side page tracking when it is supported.

Platform provides a lot of other functionalities for dirty logging:
e.g per PASID, granular, etc. So you want to duplicate them again in the virtio? If not, why choose this way?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I can't see how it can compete with the functionality that is provided by the platform. And what's more, we can't assume virtio is the only device that is used by the guest.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

You raised this before and it was answered.
Not all platform support dirty page tracking effectively.
This is optional facility that speed up the migration down time significantly.

I can hardly believe the downtime is determined by the speed of logging dirty pages...

Without dirty page tracking, in pre-copy, all the memory must be migrated again, which takes significantly longer time.

It's about w/ and w/o dirty page tracking, it's not about the speed of dirty page tracking.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

So until platform supports it, it is supported by virtio.

Some platforms already support that.

And some done.
Hence whichever wants to use by platform, will use by platform. Whichever prefer to use from device will use by the device.

I don't think so, for example virtio has a hard time when it doesn't rely on the platform (e.g IOMMU). We don't want to repeat that tragedy.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#18 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 98 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 6, 2023 12:04 PM

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Jason Wang <[email protected]>
Sent: Thursday, November 2, 2023 9:54 AM

Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, November 1, 2023 6:00 AM

On Tue, Oct 31, 2023 at 11:27 AM Parav Pandit <[email protected]>

Quoted 79 lines · wrote:
Quoted 79 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

On Mon, Oct 30, 2023 at 9:21 PM Parav Pandit
<[email protected]>

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these

written pages.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 25 lines

These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device
Quoted 3 lines

Migration}\label{sec:Basic Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +4 -0
+During the device migration flow, a passthrough device
+may write data to the guest virtual machine's memory, a
+source hypervisor needs to keep track of these written
+memory to migrate such memory to destination

hypervisor.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory
+write addresses at hypervisor level. In such a scenario,
+a device records and reports these written memory
+addresses to the owner device. The owner driver enables
+write recording for one or more physical address ranges
+per device during device

migration flow.

Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Quoted 2 lines

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex +2 -0
+The owner driver periodically queries these written
+physical address

records from the device.

Quoted 5 lines
Quoted 3 lines

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately

what the guest driver is supplying during vq creation and in posting buffers as physical address.

Quoted 8 lines
Quoted 6 lines

This seems to need some work. And, can you show me how it can work?

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?

No when dirty page logging starts, only at that time, device will reserve

enough resources.

GAW is 48bit, how large would it have then?

Quoted 9 lines
Quoted 2 lines

Dirty page tracking is not dependent on the size of the GAW.
It is function of address ranges for the amount of guest memory regardless of

GAW.

The problem is, e.g when vIOMMU is enabled, you can't know which IOVA is actually used by guests. And even for the case when vIOMMU is not enabled, the guest may have several TBs. Is it easy to reserve sufficient resources by the device itself?

When page tracking is enabled per device, it knows about the range and it can reserve certain resource.

Quoted 32 lines

Host should always have more resources than device, in that sense there could be several methods that tries to utilize host memory instead of the one in the device. I think we've discussed this when going through the doc prepared by Eugenio.

Quoted 6 lines
Quoted 3 lines

What happens if we're trying to migrate more than 1 device?

That is perfectly fine.
Each device is updating its log of pages it wrote. The hypervisor is collecting their sum.

See above.

Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines

3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer?

He he, you have funny comment.
When an abstract facility is added to virtio you say to do in transport.

So it's not done in the general facility but tied to the admin part. And we all know dirty page tracking is a challenge and Eugenio has a good summary of pros/cons. A revisit of those docs make me think virtio is not the good place for doing that for may reasons:

Quoted 6 lines
Quoted 3 lines

1) as stated, platform will evolve to be able to tracking dirty pages, actually, it has been supported by a lot of major IOMMU vendors

This is optional facility in virtio.
Can you please point to the references? I don’t see it in the common Linux

Quoted 9 lines

kernel support for it.

Note that when IOMMUFD is being proposed, dirty page tracking is one of the major considerations.

This is one recent proposal:

https://www.spinics.net/lists/kvm/msg330894.html

Sure, so if platform supports it. it can be used from the platform. If it does not, the device supplies it.

Quoted 6 lines
Quoted 1 line

Instead Linux kernel choose to extend to the devices.

Well, as I stated, tracking dirty pages is challenging if you want to do it on a device, and you can't simply invent dirty page tracking for each type of the devices.

It is not invented.
It is generic framework for all virtio device types as proposed here. Keep in mind, that it is optional already in v3 series.

Quoted 11 lines
Quoted 1 line

At least not seen to arrive this in any near term in start of 2024 which is

where users must use this.

Quoted 5 lines
Quoted 4 lines

2) you can't assume virtio is the only device that can be used by the guest, having dirty pages tracking to be implemented in each type of device is unrealistic

Of course, there is no such assumption made. Where did you see a text that

made such assumption?

So what happens if you have a guest with virtio and other devices assigned?

What happens? Each device type would do its own dirty page tracking.
And if all devices does not have support, hypervisor knows to fall back to platform iommu or its own.

Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking. And the work seems to have started for some platforms.
Without such platform commitment, virtio also skipping it would not work.

Quoted 3 lines
Quoted 1 line

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 4 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?
In the internal tests we don’t see this happening.

Quoted 6 lines
Quoted 3 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.
In the current state, it is mandating.
And if you think PRI is the only way, than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines

When one does something in transport, you say, this is transport specific, do

some generic.

Quoted 4 lines

Here the device is being tracked is virtio device.
PCI-SIG has told already that PCIM interface is outside the scope of it. Hence, this is done in virtio layer here in abstract way.

You will end up with a competition with the platform/transport one that will fail.

Quoted 8 lines
Quoted 2 lines

I don’t see a reason. There is no competition.
Platform always have a choice to not use device side page tracking when it is

supported.

Platform provides a lot of other functionalities for dirty logging:
e.g per PASID, granular, etc. So you want to duplicate them again in the virtio? If not, why choose this way?

It is optional for the platforms where platform do not have it.

Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

I can't see how it can compete with the functionality that is provided by the platform. And what's more, we can't assume virtio is the only device that is used by the guest.

You raised this before and it was answered.
Not all platform support dirty page tracking effectively. This is optional facility that speed up the migration down time

significantly.

Quoted 5 lines
Quoted 4 lines

I can hardly believe the downtime is determined by the speed of logging dirty pages...

Without dirty page tracking, in pre-copy, all the memory must be migrated

again, which takes significantly longer time.

It's about w/ and w/o dirty page tracking, it's not about the speed of dirty page tracking.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

So until platform supports it, it is supported by virtio.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Some platforms already support that.

And some done.
Hence whichever wants to use by platform, will use by platform. Whichever prefer to use from device will use by the device.

I don't think so, for example virtio has a hard time when it doesn't rely on the platform (e.g IOMMU). We don't want to repeat that tragedy.

To general statement that I don’t think is applicable here.

But lets see the support from platform.

#19 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 101 lines · Parav Pandit
Quoted 101 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 6, 2023 12:04 PM

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Jason Wang <[email protected]>
Sent: Thursday, November 2, 2023 9:54 AM

Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Wednesday, November 1, 2023 6:00 AM

On Tue, Oct 31, 2023 at 11:27 AM Parav Pandit <[email protected]>

Quoted 81 lines · wrote:
Quoted 74 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, October 31, 2023 7:13 AM

On Mon, Oct 30, 2023 at 9:21 PM Parav Pandit
<[email protected]>

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

During a device migration flow (typically in a precopy phase of the live migration), a device may write to the guest memory. Some iommu/hypervisor may not be able to track these

written pages.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 25 lines

These pages to be migrated from source to destination hypervisor.

A device which writes to these pages, provides the page address record of the to the owner device. The owner device starts write recording for the device and queries all the page addresses written by the device.

---
changelog:
v1->v2:
- addressed comments from Michael
- replaced iova with physical address

---
 admin-cmds-device-migration.tex | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
(quoted diff) +0 -0

b/admin-cmds-device-migration.tex index ed911e4..2e32f2c 100644

admin-cmds-device-migration.tex +0 -0
@@ -95,6 +95,21 @@ \subsubsection{Device
Quoted 3 lines

Migration}\label{sec:Basic Facilities of a Virtio Device / The owner driver can discard any partially read or written device context when any of the device migration flow

should be aborted.

admin-cmds-device-migration.tex

Quoted 5 lines
admin-cmds-device-migration.tex +4 -0
+During the device migration flow, a passthrough device
+may write data to the guest virtual machine's memory, a
+source hypervisor needs to keep track of these written
+memory to migrate such memory to destination

hypervisor.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

admin-cmds-device-migration.tex

Quoted 6 lines
admin-cmds-device-migration.tex +6 -0
+Some systems may not be able to keep track of such memory
+write addresses at hypervisor level. In such a scenario,
+a device records and reports these written memory
+addresses to the owner device. The owner driver enables
+write recording for one or more physical address ranges
+per device during device

migration flow.

Quoted 17 lines
Quoted 17 lines
Quoted 2 lines
Quoted 2 lines

admin-cmds-device-migration.tex

Quoted 2 lines
admin-cmds-device-migration.tex +2 -0
+The owner driver periodically queries these written
+physical address

records from the device.

Quoted 5 lines
Quoted 3 lines

I wonder how PA works in this case. Device uses untranslated requests so it can only see IOVA. We can't mandate ATS anyhow.

Michael suggested to keep the language uniform as PA as this is ultimately

what the guest driver is supplying during vq creation and in posting buffers as physical address.

This seems to need some work. And, can you show me how it can work?

1) e.g if GAW is 48 bit, is the hypervisor expected to do a bisection of the whole range?
2) does the device need to reserve sufficient internal resources for logging the dirty page and why (not)?

Quoted 5 lines
Quoted 2 lines

No when dirty page logging starts, only at that time, device will reserve

enough resources.

GAW is 48bit, how large would it have then?

Dirty page tracking is not dependent on the size of the GAW.
It is function of address ranges for the amount of guest memory regardless of

Quoted 8 lines
Quoted 7 lines

GAW.

The problem is, e.g when vIOMMU is enabled, you can't know which IOVA is actually used by guests. And even for the case when vIOMMU is not enabled, the guest may have several TBs. Is it easy to reserve sufficient resources by the device itself?

When page tracking is enabled per device, it knows about the range and it can reserve certain resource.

I didn't see such an interface in this series. Anything I miss?

Btw, the IOVA is allocated by the guest actually, how can we know the range? (or using the host range?)

Quoted 76 lines
Quoted 33 lines

Host should always have more resources than device, in that sense there could be several methods that tries to utilize host memory instead of the one in the device. I think we've discussed this when going through the doc prepared by Eugenio.

Quoted 6 lines
Quoted 3 lines

What happens if we're trying to migrate more than 1 device?

That is perfectly fine.
Each device is updating its log of pages it wrote. The hypervisor is collecting their sum.

See above.

Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
Quoted 4 lines

3) DMA is part of the transport, it's natural to do logging there, why duplicate efforts in the virtio layer?

He he, you have funny comment.
When an abstract facility is added to virtio you say to do in transport.

So it's not done in the general facility but tied to the admin part. And we all know dirty page tracking is a challenge and Eugenio has a good summary of pros/cons. A revisit of those docs make me think virtio is not the good place for doing that for may reasons:

Quoted 6 lines
Quoted 3 lines

1) as stated, platform will evolve to be able to tracking dirty pages, actually, it has been supported by a lot of major IOMMU vendors

This is optional facility in virtio.
Can you please point to the references? I don’t see it in the common Linux

Quoted 9 lines

kernel support for it.

Note that when IOMMUFD is being proposed, dirty page tracking is one of the major considerations.

This is one recent proposal:

https://www.spinics.net/lists/kvm/msg330894.html

Sure, so if platform supports it. it can be used from the platform. If it does not, the device supplies it.

Quoted 6 lines
Quoted 1 line

Instead Linux kernel choose to extend to the devices.

Well, as I stated, tracking dirty pages is challenging if you want to do it on a device, and you can't simply invent dirty page tracking for each type of the devices.

It is not invented.
It is generic framework for all virtio device types as proposed here. Keep in mind, that it is optional already in v3 series.

Quoted 11 lines
Quoted 1 line

At least not seen to arrive this in any near term in start of 2024 which is

where users must use this.

Quoted 5 lines
Quoted 4 lines

2) you can't assume virtio is the only device that can be used by the guest, having dirty pages tracking to be implemented in each type of device is unrealistic

Of course, there is no such assumption made. Where did you see a text that

made such assumption?

So what happens if you have a guest with virtio and other devices assigned?

What happens? Each device type would do its own dirty page tracking.
And if all devices does not have support, hypervisor knows to fall back to platform iommu or its own.

Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 2 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 6 lines
Quoted 6 lines

When one does something in transport, you say, this is transport specific, do

Quoted 9 lines

some generic.

Quoted 4 lines

Here the device is being tracked is virtio device.
PCI-SIG has told already that PCIM interface is outside the scope of it. Hence, this is done in virtio layer here in abstract way.

You will end up with a competition with the platform/transport one that will fail.

I don’t see a reason. There is no competition.
Platform always have a choice to not use device side page tracking when it is

Quoted 7 lines
Quoted 6 lines

supported.

Platform provides a lot of other functionalities for dirty logging:
e.g per PASID, granular, etc. So you want to duplicate them again in the virtio? If not, why choose this way?

It is optional for the platforms where platform do not have it.

We are developing new virtio functionalities that are targeted for future platforms. Otherwise we would end up with a feature with a very narrow use case.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#20 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 28 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 4 lines
Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads. Definitely KVM did not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM, what Lingshan proposed is analogous to bit per page - problem unfortunately is
you can't easily set a bit by DMA.

So I think this dirty tracking is a good option to have.

Quoted 25 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

Quoted 7 lines

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

There is no requirement for mandating PRI either.
So it is unusable.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

Quoted 11 lines
Quoted 2 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection? virtio is here in the first place because emulating devices didn't work well.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#21 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 30 lines · Michael S. Tsirkin
Quoted 28 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Quoted 3 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.
2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled. And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

We wouldn't need to care about all of them if it is done at platform IOMMU level.

Quoted 4 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 31 lines

So I think this dirty tracking is a good option to have.

Quoted 27 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them. Missing functions in platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

Quoted 4 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#22 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 33 lines · Jason Wang
Quoted 32 lines · Michael S. Tsirkin
Quoted 30 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

Quoted 7 lines
Quoted 3 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

Quoted 6 lines

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 6 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking? Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Quoted 5 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 3 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Quoted 2 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

Quoted 45 lines
Quoted 4 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 37 lines

So I think this dirty tracking is a good option to have.

Quoted 32 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Quoted 7 lines
Quoted 2 lines

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

Quoted 4 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 8 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Quoted 8 lines
Quoted 8 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#23 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, November 8, 2023 1:47 PM

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Rx is the main part to test for the dirty page tracking. Tx only is around ring updates, so not very interesting test.

When page tracking is started, there is impact on Rx as natural throttle by slowing down rx packet rate. And devices have implementation choice to covering the spectrum on amount of drop rate. And has implementations evolve, it will improve as well.

PRI for sure is out of question. Most workload won't use it at all given other limitations of virtio on PCI front and existing PRI behaviors.

There is also a test with platform IOMMU dirty page tracking done as well but it is very early stage to comment in this forum.

#24 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 9 lines · Parav Pandit
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, November 8, 2023 1:47 PM

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Rx is the main part to test for the dirty page tracking. Tx only is around ring updates, so not very interesting test.

When page tracking is started, there is impact on Rx as natural throttle by slowing down rx packet rate.

Wait a sec sounds kind of like what we see with SVQ as well?

Quoted 1 line

And devices have implementation choice to covering the spectrum on amount of drop rate.

What does this mean practically?

Quoted 1 line

And has implementations evolve, it will improve as well.

Maybe, or maybe not.

Quoted 1 line

PRI for sure is out of question. Most workload won't use it at all given other limitations of virtio on PCI front and existing PRI behaviors.

Out of curiousity what do you refer to?

Quoted 1 line

There is also a test with platform IOMMU dirty page tracking done as well but it is very early stage to comment in this forum.

Interesting. That's a very important test I'd say - if platform based dirty page tracking is just as good we can avoid adding it to virtio.

And I'm rather confused at this point - I was under the impression you already have a prototype which shows negligeable performance impact with on device tracking.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#25 lore raw

RE: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 15 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: Wednesday, November 8, 2023 10:46 PM

Quoted 9 lines · Parav Pandit
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, November 8, 2023 1:47 PM

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Rx is the main part to test for the dirty page tracking. Tx only is around ring updates, so not very interesting test.

When page tracking is started, there is impact on Rx as natural throttle by

Quoted 8 lines

slowing down rx packet rate.

Wait a sec sounds kind of like what we see with SVQ as well?

Quoted 1 line

And devices have implementation choice to covering the spectrum on amount

of drop rate.

What does this mean practically?

How to implement dirty page tracking is implementation specific detail.
So for example, what does extra things device does in its rx path to track pages, which data structures etc is used, depends on device implementation. For example, practically, the device under test by us cannot do at 800Gbps in current generation, but possibly in future.

Quoted 10 lines
Quoted 2 lines

And has implementations evolve, it will improve as well.

Maybe, or maybe not.

Quoted 1 line

PRI for sure is out of question. Most workload won't use it at all given other

limitations of virtio on PCI front and existing PRI behaviors.

Out of curiousity what do you refer to?

The whole scheme to notify page faults from device to iommu and serving them back is not as efficient and scalable as the way cpu can page fault.

Quoted 5 lines
Quoted 1 line

There is also a test with platform IOMMU dirty page tracking done as well but

it is very early stage to comment in this forum.

Interesting. That's a very important test I'd say - if platform based dirty page tracking is just as good we can avoid adding it to virtio.

I agree. I don't see all platforms have it. There is ongoing work.

Quoted 5 lines

And I'm rather confused at this point - I was under the impression you already have a prototype which shows negligeable performance impact with on device tracking.

Compared to no tracking by platform, vs device tracking, there is significant gain. Compared to platform tracking and device tracking, hard to say.
Given all platforms may not have it, it is not very interesting from perf point of view.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#26 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Michael S. Tsirkin
Quoted 39 lines · Michael S. Tsirkin
Quoted 37 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 32 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Quoted 12 lines
Quoted 12 lines
Quoted 4 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

Quoted 7 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Quoted 11 lines
Quoted 8 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Quoted 8 lines
Quoted 6 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Quoted 7 lines
Quoted 4 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

Quoted 7 lines
Quoted 4 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level easily.

Quoted 50 lines
Quoted 50 lines
Quoted 6 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 39 lines

So I think this dirty tracking is a good option to have.

Quoted 25 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

Quoted 7 lines

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Quoted 7 lines
Quoted 4 lines

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 8 lines
Quoted 5 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Quoted 44 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 7 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Quoted 5 lines
Quoted 1 line

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#27 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 40 lines · Jason Wang
Quoted 39 lines · Michael S. Tsirkin
Quoted 37 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 32 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

Quoted 19 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Quoted 8 lines

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Quoted 10 lines
Quoted 7 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally. Problem of course is that it might overflow and cause things like packet drops.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

Quoted 5 lines
Quoted 2 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

There's a way to have hardware find dirty PTEs for you quickly? I don't know how it's done. Do tell.

Quoted 6 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Quoted 8 lines

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in the spec too.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

Quoted 5 lines
Quoted 1 line

SVA being what here?

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

what does pasid have to do with it? anyway, just give driver control over page size.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level easily.

there's no "easily" about live migration ever.
For example on-device iommus are a thing.

Quoted 55 lines
Quoted 55 lines
Quoted 54 lines
Quoted 6 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 31 lines

So I think this dirty tracking is a good option to have.

Quoted 27 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

There is no requirement for mandating PRI either.
So it is unusable.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really. So I'd like Parav to publish some experiment results and/or some estimates.

Quoted 46 lines
Quoted 46 lines
Quoted 23 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 8 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

Quoted 8 lines
Quoted 3 lines

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#28 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 8 lines
Quoted 1 line

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 6 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 6 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in the spec too.

As you listed the flush is needed for vhost or device-based DPT.
The necessary plumbing is already covered for this in the query (read and clear) command of this v3 proposal. It is listed in Device Write Records Read Command.

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not documenting those details. May be such read is needed in some other path too depending on how hypervisor implemented.

#29 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 15 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 8 lines
Quoted 1 line

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 6 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in the spec too.

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

Quoted 1 line

The necessary plumbing is already covered for this in the query (read and clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Quoted 1 line

It is listed in Device Write Records Read Command.

Please explain how your proposal can solve the above race.

Quoted 3 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Quoted 1 line

May be such read is needed in some other path too depending on how hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#30 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 27 lines · Jason Wang
Quoted 15 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 8 lines
Quoted 1 line

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 6 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in the spec too.

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Quoted 6 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

Devices with IOMMU exist.
So if it's for device IOMMU that's fine, as long as it's well isolated.

Quoted 15 lines
Quoted 1 line

The necessary plumbing is already covered for this in the query (read and clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Quoted 1 line

It is listed in Device Write Records Read Command.

Please explain how your proposal can solve the above race.

Quoted 3 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Quoted 6 lines
Quoted 1 line

May be such read is needed in some other path too depending on how hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

Thanks

It seems like a subtle enough race that it really should be documented.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#31 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 12 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 7 lines
Quoted 1 line

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 5 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in the spec too.

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Quoted 8 lines
Quoted 6 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

Devices with IOMMU exist.
So if it's for device IOMMU that's fine, as long as it's well isolated.

device side IOMMU is not a must and I hope virito features don't depend on it.

Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

The necessary plumbing is already covered for this in the query (read and clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Quoted 1 line

It is listed in Device Write Records Read Command.

Please explain how your proposal can solve the above race.

Quoted 2 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

May be such read is needed in some other path too depending on how hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

Thanks

It seems like a subtle enough race that it really should be documented.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#32 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 27 lines · Lingshan
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Jason Wang
Quoted 12 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 7 lines
Quoted 1 line

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 5 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in the spec too.

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

Devices with IOMMU exist.

It only exists for very few vendors. Whether or not virtio needs a device IOMMU is a separate topic.

Again, it's far easier to leverage what platform can give us than re-inventing stuff in virtio.

Quoted 3 lines
Quoted 1 line

So if it's for device IOMMU that's fine, as long as it's well isolated.

device side IOMMU is not a must and I hope virito features don't depend on it.

Exactly, and I didn't see any description in this series to:

1) claim the feature depends on device IOMMU
2) invent interfaces for device IOMMU

Thanks

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

The necessary plumbing is already covered for this in the query (read and clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Quoted 1 line

It is listed in Device Write Records Read Command.

Please explain how your proposal can solve the above race.

Quoted 2 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

May be such read is needed in some other path too depending on how hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

Thanks

It seems like a subtle enough race that it really should be documented.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#33 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Jason Wang
Quoted 26 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:11 AM

Quoted 21 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 14 lines
Quoted 2 lines

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in

Quoted 7 lines

the spec too.

Quoted 2 lines
Quoted 1 line

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

Quoted 2 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

The dependency is on the iommu which would have the mapping of GIOVA to GPA like any sw implementation. No dependency on ATS.

Quoted 4 lines

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

You should stop calling this specific vendor thing.
One can equally say that suspend bit proposal is for the sw_vendor device who is forcing virtio hw device to only implement ioqueues + PASID + non_unified interface for PF, VF, SIOVs + non_TDISP based devices.

Quoted 5 lines
Quoted 1 line

The necessary plumbing is already covered for this in the query (read and

clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Read and clear helps that ensures that all the dirty pages are reported, hence there is no mapping/unmapping race. As everything is reported.

Quoted 4 lines
Quoted 1 line

It is listed in Device Write Records Read Command.

Please explain how your proposal can solve the above race.

In below manner.
1. guest has GIOVA to GPA_1 mapping
2. RX packets occurred to GIOVA
3. device reported dirty page log for GIOVA (hypervisor is yet to read)
4. guest requested mapping change from GIOVA to GPA_2
4.1 During this IOTLB is invalidated and dirty page report is queried ensuring, it can change the mapping

Quoted 8 lines
Quoted 3 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not

documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Likely yes, but it is not the scope of the spec to list those paths etc.

Quoted 5 lines
Quoted 1 line

May be such read is needed in some other path too depending on how

hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

No. I meant hypervisor may have more operations that map/unmap/flush where it may need to implement it. Some one may call it set_map(), some may say dma_map()...

#34 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 38 lines · Parav Pandit
Quoted 27 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:11 AM

Quoted 21 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 14 lines
Quoted 2 lines

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in

Quoted 7 lines

the spec too.

Quoted 2 lines
Quoted 1 line

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

Quoted 2 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

Quoted 8 lines

The dependency is on the iommu which would have the mapping of GIOVA to GPA like any sw implementation. No dependency on ATS.

Quoted 4 lines

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

You should stop calling this specific vendor thing.

Well, as you have explained, the confusion came from "DPT" ...

Quoted 8 lines

One can equally say that suspend bit proposal is for the sw_vendor device who is forcing virtio hw device to only implement ioqueues + PASID + non_unified interface for PF, VF, SIOVs + non_TDISP based devices.

Quoted 5 lines
Quoted 1 line

The necessary plumbing is already covered for this in the query (read and

clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Read and clear helps that ensures that all the dirty pages are reported, hence there is no mapping/unmapping race.

Reported as IOVA ...

Quoted 4 lines

As everything is reported.

Quoted 2 lines
Quoted 1 line

It is listed in Device Write Records Read Command.

Quoted 8 lines
Quoted 2 lines

Please explain how your proposal can solve the above race.

In below manner.
1. guest has GIOVA to GPA_1 mapping
2. RX packets occurred to GIOVA
3. device reported dirty page log for GIOVA (hypervisor is yet to read)
4. guest requested mapping change from GIOVA to GPA_2
4.1 During this IOTLB is invalidated and dirty page report is queried ensuring, it can change the mapping

It requires

1) hypervisor traps IOTLB invalidation, which doesn't work when nesting could be offloaded (IOMMUFD has started the work to support nesting)
2) query the device about the dirty page on each IOTLB invalidation which:
2.1) A huge round trip: guest IOTLB invalidation -> trapped by hypervisor -> start the query from the device -> device return -> hypervisor reports IOTLB invalidation is done -> let guest run. Have you benchmarked the RTT in this case? There are just too many places that cause the delay in the middle.
2.2) Guest triggerable behaviour, malicious guest can simply do endless IOTLB invalidation to DOS the e.g admin virtqueue

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not

documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Likely yes, but it is not the scope of the spec to list those paths etc.

Fine, but as a reviewer I need to know if it can work with a hypervisor well.

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

May be such read is needed in some other path too depending on how

hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

No. I meant hypervisor may have more operations that map/unmap/flush where it may need to implement it. Some one may call it set_map(), some may say dma_map()...

Ok.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#35 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 76 lines · Jason Wang
Quoted 38 lines · Parav Pandit
Quoted 27 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:11 AM

Quoted 21 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 14 lines
Quoted 2 lines

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in

Quoted 7 lines

the spec too.

Quoted 2 lines
Quoted 1 line

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

Quoted 2 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

Quoted 8 lines

The dependency is on the iommu which would have the mapping of GIOVA to GPA like any sw implementation. No dependency on ATS.

Quoted 4 lines

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

You should stop calling this specific vendor thing.

Well, as you have explained, the confusion came from "DPT" ...

Quoted 8 lines

One can equally say that suspend bit proposal is for the sw_vendor device who is forcing virtio hw device to only implement ioqueues + PASID + non_unified interface for PF, VF, SIOVs + non_TDISP based devices.

Quoted 5 lines
Quoted 1 line

The necessary plumbing is already covered for this in the query (read and

clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Read and clear helps that ensures that all the dirty pages are reported, hence there is no mapping/unmapping race.

Reported as IOVA ...

Quoted 4 lines

As everything is reported.

Quoted 2 lines
Quoted 1 line

It is listed in Device Write Records Read Command.

Quoted 8 lines
Quoted 2 lines

Please explain how your proposal can solve the above race.

In below manner.
1. guest has GIOVA to GPA_1 mapping
2. RX packets occurred to GIOVA
3. device reported dirty page log for GIOVA (hypervisor is yet to read)
4. guest requested mapping change from GIOVA to GPA_2
4.1 During this IOTLB is invalidated and dirty page report is queried ensuring, it can change the mapping

It requires

Quoted 9 lines

1) hypervisor traps IOTLB invalidation, which doesn't work when nesting could be offloaded (IOMMUFD has started the work to support nesting)
2) query the device about the dirty page on each IOTLB invalidation which:
2.1) A huge round trip: guest IOTLB invalidation -> trapped by hypervisor -> start the query from the device -> device return -> hypervisor reports IOTLB invalidation is done -> let guest run. Have you benchmarked the RTT in this case? There are just too many places that cause the delay in the middle.

To be fair invalidations are already expensive e.g. with vhost iotlb it requires a slow system call.
This will make them *even more* expensive.

Problem for some but not all workloads. Again I agree motivation, tradeoffs and comparison with both dirty tracking by iommu and shadow vq approaches really should be included.

Quoted 2 lines

2.2) Guest triggerable behaviour, malicious guest can simply do endless IOTLB invalidation to DOS the e.g admin virtqueue

I'm not sure how much to worry about it - just don't allow more than one in flight per VM.

Quoted 20 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not

documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Likely yes, but it is not the scope of the spec to list those paths etc.

Fine, but as a reviewer I need to know if it can work with a hypervisor well.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

May be such read is needed in some other path too depending on how

hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

Quoted 6 lines
Quoted 2 lines

No. I meant hypervisor may have more operations that map/unmap/flush where it may need to implement it. Some one may call it set_map(), some may say dma_map()...

Ok.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#36 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 88 lines · Michael S. Tsirkin
Quoted 76 lines · Jason Wang
Quoted 38 lines · Parav Pandit
Quoted 27 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:11 AM

Quoted 21 lines · Parav Pandit

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 14 lines
Quoted 2 lines

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in

Quoted 7 lines

the spec too.

Quoted 2 lines
Quoted 1 line

As you listed the flush is needed for vhost or device-based DPT.

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

Quoted 2 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

Quoted 8 lines

The dependency is on the iommu which would have the mapping of GIOVA to GPA like any sw implementation. No dependency on ATS.

Quoted 4 lines

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

You should stop calling this specific vendor thing.

Well, as you have explained, the confusion came from "DPT" ...

Quoted 8 lines

One can equally say that suspend bit proposal is for the sw_vendor device who is forcing virtio hw device to only implement ioqueues + PASID + non_unified interface for PF, VF, SIOVs + non_TDISP based devices.

Quoted 5 lines
Quoted 1 line

The necessary plumbing is already covered for this in the query (read and

clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can help.

Read and clear helps that ensures that all the dirty pages are reported, hence there is no mapping/unmapping race.

Reported as IOVA ...

Quoted 4 lines

As everything is reported.

Quoted 2 lines
Quoted 1 line

It is listed in Device Write Records Read Command.

Quoted 8 lines
Quoted 2 lines

Please explain how your proposal can solve the above race.

In below manner.
1. guest has GIOVA to GPA_1 mapping
2. RX packets occurred to GIOVA
3. device reported dirty page log for GIOVA (hypervisor is yet to read)
4. guest requested mapping change from GIOVA to GPA_2
4.1 During this IOTLB is invalidated and dirty page report is queried ensuring, it can change the mapping

It requires

Quoted 9 lines

1) hypervisor traps IOTLB invalidation, which doesn't work when nesting could be offloaded (IOMMUFD has started the work to support nesting)
2) query the device about the dirty page on each IOTLB invalidation which:
2.1) A huge round trip: guest IOTLB invalidation -> trapped by hypervisor -> start the query from the device -> device return -> hypervisor reports IOTLB invalidation is done -> let guest run. Have you benchmarked the RTT in this case? There are just too many places that cause the delay in the middle.

Quoted 3 lines

To be fair invalidations are already expensive e.g. with vhost iotlb it requires a slow system call.
This will make them *even more* expensive.

Yes, a slow syscall plus a virtqueue query RTT.

Need some benchmark. It looks to me currently the invalidation is done via a queued based interface in vtd. So guests may need to spin where it may trigger a lockup in the guest.

Quoted 4 lines

Problem for some but not all workloads. Again I agree motivation, tradeoffs and comparison with both dirty tracking by iommu and shadow vq approaches really should be included.

+1

Quoted 7 lines
Quoted 4 lines

2.2) Guest triggerable behaviour, malicious guest can simply do endless IOTLB invalidation to DOS the e.g admin virtqueue

I'm not sure how much to worry about it - just don't allow more than one in flight per VM.

That's fine but it may need a note.

Thanks

Quoted 23 lines
Quoted 23 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

When the page write record is fully read, it is flushed.
How/when to use, I think its hypervisor specific, so we probably better off not

documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Likely yes, but it is not the scope of the spec to list those paths etc.

Fine, but as a reviewer I need to know if it can work with a hypervisor well.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

May be such read is needed in some other path too depending on how

hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

No. I meant hypervisor may have more operations that map/unmap/flush where it may need to implement it. Some one may call it set_map(), some may say dma_map()...

Ok.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#37 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 103 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, November 21, 2023 9:52 AM

Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Thu, Nov 16, 2023 at 1:37 AM Parav Pandit <[email protected]>

Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:11 AM

On Fri, Nov 10, 2023 at 2:46 PM Parav Pandit <[email protected]>

Quoted 26 lines · wrote:
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 22 lines

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 15 lines
Quoted 2 lines

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in

the spec too.

Quoted 2 lines
Quoted 1 line

As you listed the flush is needed for vhost or device-based DPT.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

Quoted 2 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

The dependency is on the iommu which would have the mapping of

GIOVA to GPA like any sw implementation.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines

No dependency on ATS.

Quoted 4 lines

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

You should stop calling this specific vendor thing.

Well, as you have explained, the confusion came from "DPT" ...

Quoted 1 line

One can equally say that suspend bit proposal is for the sw_vendor

device who is forcing virtio hw device to only implement ioqueues + PASID + non_unified interface for PF, VF, SIOVs + non_TDISP based devices.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

The necessary plumbing is already covered for this in the query (read and

clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can

help.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Read and clear helps that ensures that all the dirty pages are reported,

hence there is no mapping/unmapping race.

Quoted 28 lines
Quoted 27 lines

Reported as IOVA ...

Quoted 4 lines

As everything is reported.

Quoted 2 lines
Quoted 1 line

It is listed in Device Write Records Read Command.

Quoted 8 lines
Quoted 2 lines

Please explain how your proposal can solve the above race.

In below manner.
1. guest has GIOVA to GPA_1 mapping 2. RX packets occurred to GIOVA 3. device reported dirty page log for GIOVA (hypervisor is yet to read) 4. guest requested mapping change from GIOVA to GPA_2 4.1 During this IOTLB is invalidated and dirty page report is queried ensuring, it can change the mapping

It requires

1) hypervisor traps IOTLB invalidation, which doesn't work when nesting could be offloaded (IOMMUFD has started the work to support nesting)
2) query the device about the dirty page on each IOTLB invalidation which:
2.1) A huge round trip: guest IOTLB invalidation -> trapped by hypervisor -> start the query from the device -> device return -> hypervisor reports IOTLB invalidation is done -> let guest run. Have you benchmarked the RTT in this case? There are just too many places that cause the delay in the middle.

Quoted 6 lines
Quoted 3 lines

To be fair invalidations are already expensive e.g. with vhost iotlb it requires a slow system call.
This will make them *even more* expensive.

Yes, a slow syscall plus a virtqueue query RTT.

Only during viommu case.
Without this is not applicable.

Quoted 10 lines

Need some benchmark. It looks to me currently the invalidation is done via a queued based interface in vtd. So guests may need to spin where it may trigger a lockup in the guest.

Quoted 4 lines

Problem for some but not all workloads. Again I agree motivation, tradeoffs and comparison with both dirty tracking by iommu and shadow vq approaches really should be included.

Dirty tracking is iommu to be considered.
Shadow vq is not in my scope and it does not fit the basic requirements as explained before. So it is different discussion.

Quoted 48 lines

+1

Quoted 7 lines
Quoted 4 lines

2.2) Guest triggerable behaviour, malicious guest can simply do endless IOTLB invalidation to DOS the e.g admin virtqueue

I'm not sure how much to worry about it - just don't allow more than one in flight per VM.

That's fine but it may need a note.

Thanks

Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

When the page write record is fully read, it is flushed. How/when to use, I think its hypervisor specific, so we probably better off not

Quoted 6 lines
Quoted 5 lines

documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Likely yes, but it is not the scope of the spec to list those paths etc.

Fine, but as a reviewer I need to know if it can work with a hypervisor well.

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

May be such read is needed in some other path too depending on how

hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

No. I meant hypervisor may have more operations that

map/unmap/flush where it may need to implement it.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Some one may call it set_map(), some may say dma_map()...

Ok.

Thanks

#38 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 104 lines · Parav Pandit
Quoted 104 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, November 21, 2023 9:52 AM

Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Thu, Nov 16, 2023 at 1:37 AM Parav Pandit <[email protected]>

Quoted 6 lines · wrote:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:11 AM

On Fri, Nov 10, 2023 at 2:46 PM Parav Pandit <[email protected]>

Quoted 26 lines · wrote:
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 22 lines

Hi Michael,

Quoted 2 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, November 9, 2023 1:29 PM

[..]

Quoted 15 lines
Quoted 2 lines

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point. And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow. Parav what's the plan for this? Should be addressed in

the spec too.

Quoted 2 lines
Quoted 1 line

As you listed the flush is needed for vhost or device-based DPT.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

Quoted 2 lines

We have discussed it many times. You can't just depend on ATS or reinventing wheels in virtio.

The dependency is on the iommu which would have the mapping of

GIOVA to GPA like any sw implementation.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines

No dependency on ATS.

Quoted 4 lines

What's more, please try not to give me the impression that the proposal is optimized for a specific vendor (like device IOMMU stuffs).

You should stop calling this specific vendor thing.

Well, as you have explained, the confusion came from "DPT" ...

Quoted 1 line

One can equally say that suspend bit proposal is for the sw_vendor

device who is forcing virtio hw device to only implement ioqueues + PASID + non_unified interface for PF, VF, SIOVs + non_TDISP based devices.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

The necessary plumbing is already covered for this in the query (read and

clear) command of this v3 proposal.

The issue is logging via IOVA ... I don't see how "read and clear" can

help.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Read and clear helps that ensures that all the dirty pages are reported,

hence there is no mapping/unmapping race.

Quoted 28 lines
Quoted 27 lines

Reported as IOVA ...

Quoted 4 lines

As everything is reported.

Quoted 2 lines
Quoted 1 line

It is listed in Device Write Records Read Command.

Quoted 8 lines
Quoted 2 lines

Please explain how your proposal can solve the above race.

In below manner.
1. guest has GIOVA to GPA_1 mapping 2. RX packets occurred to GIOVA 3. device reported dirty page log for GIOVA (hypervisor is yet to read) 4. guest requested mapping change from GIOVA to GPA_2 4.1 During this IOTLB is invalidated and dirty page report is queried ensuring, it can change the mapping

It requires

1) hypervisor traps IOTLB invalidation, which doesn't work when nesting could be offloaded (IOMMUFD has started the work to support nesting)
2) query the device about the dirty page on each IOTLB invalidation which:
2.1) A huge round trip: guest IOTLB invalidation -> trapped by hypervisor -> start the query from the device -> device return -> hypervisor reports IOTLB invalidation is done -> let guest run. Have you benchmarked the RTT in this case? There are just too many places that cause the delay in the middle.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

To be fair invalidations are already expensive e.g. with vhost iotlb it requires a slow system call.
This will make them *even more* expensive.

Yes, a slow syscall plus a virtqueue query RTT.

Only during viommu case.

What's worse, modern IOMMU drivers tend to batch the invalidation via a per domain invalidation. I can easily imagine how slow it is.

For vhost, it's still one syscall. But for your proposal, it needs to do a query of all the possible IOVA ranges which is horribly slow as it needs several queries requests or PCI transactions where guest lockup would not be rare.

Quoted 1 line

Without this is not applicable.

List is a place to discuss the possible issue, no?

Quoted 7 lines
Quoted 7 lines

Need some benchmark. It looks to me currently the invalidation is done via a queued based interface in vtd. So guests may need to spin where it may trigger a lockup in the guest.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

Problem for some but not all workloads. Again I agree motivation, tradeoffs and comparison with both dirty tracking by iommu and shadow vq approaches really should be included.

Dirty tracking is iommu to be considered.
Shadow vq is not in my scope and it does not fit the basic requirements as explained before.

I don't see a good explanation other than "I meet a bug in Linus tree, it's unstable so I wouldn't test anymore".

Thanks

Quoted 51 lines

So it is different discussion.

Quoted 48 lines

+1

Quoted 7 lines
Quoted 4 lines

2.2) Guest triggerable behaviour, malicious guest can simply do endless IOTLB invalidation to DOS the e.g admin virtqueue

I'm not sure how much to worry about it - just don't allow more than one in flight per VM.

That's fine but it may need a note.

Thanks

Quoted 18 lines
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

When the page write record is fully read, it is flushed. How/when to use, I think its hypervisor specific, so we probably better off not

Quoted 6 lines
Quoted 5 lines

documenting those details.

Well, as the author of this proposal, at least you need to know how a hypervisor can work with your proposal, no?

Likely yes, but it is not the scope of the spec to list those paths etc.

Fine, but as a reviewer I need to know if it can work with a hypervisor well.

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

May be such read is needed in some other path too depending on how

hypervisor implemented.

What do you mean by "May be ... some other path" here? You're inventing a mechanism that you don't know how a hypervisor can use?

No. I meant hypervisor may have more operations that

map/unmap/flush where it may need to implement it.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

Some one may call it set_map(), some may say dma_map()...

Ok.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#39 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Parav Pandit
Quoted 4 lines · Parav Pandit
Quoted 3 lines

What does DPT mean? Device Page Table? Let's not invent terminology which is not known by others please.

Sorry for using the acronym. I meant dirty page tracking.

:)
Yea don't make up new ones please.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Acked-by Michael S. Tsirkin
#40 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Michael S. Tsirkin
Quoted 44 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 41 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 37 lines · Michael S. Tsirkin
Quoted 28 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 4 lines
Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 22 lines
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

Quoted 8 lines
Quoted 6 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 4 lines
Quoted 2 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Quoted 11 lines

I don't know how it's done. Do tell.

Quoted 8 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

Quoted 8 lines
Quoted 6 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 3 lines

And by the way e.g. vhost has the same
issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Quoted 3 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level easily.

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 55 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 31 lines

So I think this dirty tracking is a good option to have.

Quoted 27 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

Quoted 3 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 8 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Quoted 7 lines
Quoted 4 lines

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

Quoted 4 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#41 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 50 lines · Jason Wang
Quoted 49 lines · Michael S. Tsirkin
Quoted 47 lines · Jason Wang
Quoted 45 lines · Michael S. Tsirkin
Quoted 33 lines · Jason Wang
Quoted 32 lines · Michael S. Tsirkin
Quoted 30 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

What I would like to see is effort to switch shadow on/off not keep it on at all times. That's only good enough for a PoC. And to work on top of virtio that will require effort in the spec. If I see spec patches that do that I personally would support that. It needs to be reasonably generic though, a single 16 bit RW number is not going to be enough. I think it's likely admin commands is a good interface for this. If it's a hack making vendor specific assumptions, just keep it in vdpa.

Quoted 52 lines
Quoted 22 lines
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

Quoted 8 lines
Quoted 6 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 4 lines
Quoted 2 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

Quoted 7 lines
Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

People buy hardware to improve performance. Apparently there are people who want to build this hardware. It is not our role to tell either of the groups "this should not be a general goal".

Quoted 41 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Quoted 16 lines

I don't know how it's done. Do tell.

Quoted 13 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

And that's perfect for migration.

Quoted 33 lines
Quoted 3 lines

And by the way e.g. vhost has the same
issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Quoted 8 lines
Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Any IOMMU flushes IOTLB on translation changes. If vhost doesn't then it's highly likely to be a bug.

Quoted 21 lines
Quoted 3 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 15 lines
Quoted 14 lines
Quoted 10 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Quoted 7 lines
Quoted 3 lines

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

virtio must always log PA.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 6 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

Exactly it's a mess. Instead of making everything 10x more complex, let's just keep talking about PA and leave translation to IOMMU.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

what does pasid have to do with it? anyway, just give driver control over page size.

Quoted 8 lines

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

1) translated/untranslated request
2) request w/ and w/o PASID

Can't say I understand. All the talk about IOVA is just confusing - what we care about for logging is which page to resend.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

True, but my question is where or not it can be done at the device level easily.

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Code walks though.

Quoted 4 lines
Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

What do you refer to? People buy hardware and use it for years with no chance to add features.

Quoted 62 lines
Quoted 62 lines
Quoted 61 lines
Quoted 51 lines
Quoted 51 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 39 lines

So I think this dirty tracking is a good option to have.

Quoted 25 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

Quoted 7 lines

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Quoted 7 lines
Quoted 4 lines

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

Quoted 6 lines
Quoted 2 lines

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

My point is PRI causes rx ring underruns and throttling CPU makes it worse not better. And I believe people actually tried, nvidia have a pri implementation in hardware. If they come and say virtio help is needed for performance I tend to believe them.

Quoted 31 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

Quoted 5 lines

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Dirty tracking in the IOMMU is annoying enough that I am not sure it's usable. Go ahead but I want to see patches then.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 9 lines
Quoted 6 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Quoted 7 lines
Quoted 2 lines

Not true platforms have things like NICs on board and have for many years. It's about performance really.

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS).

Maybe. This kind of statement is just vague enough not to be falsifiable.

Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

Quoted 63 lines
Quoted 3 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 56 lines
Quoted 55 lines
Quoted 45 lines
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 7 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Quoted 5 lines
Quoted 1 line

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

Quoted 20 lines
Quoted 2 lines

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

Quoted 4 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#42 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 379 lines · Michael S. Tsirkin
Quoted 37 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Jason Wang
Quoted 34 lines · Michael S. Tsirkin
Quoted 33 lines · Jason Wang
Quoted 26 lines · Michael S. Tsirkin
Quoted 25 lines · Jason Wang
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 4 lines
Quoted 4 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

What I would like to see is effort to switch shadow on/off not keep it on at all times. That's only good enough for a PoC. And to work on top of virtio that will require effort in the spec. If I see spec patches that do that I personally would support that. It needs to be reasonably generic though, a single 16 bit RW number is not going to be enough. I think it's likely admin commands is a good interface for this. If it's a hack making vendor specific assumptions, just keep it in vdpa.

Quoted 40 lines
Quoted 17 lines
Quoted 17 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 5 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

Quoted 7 lines
Quoted 6 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

Quoted 10 lines
Quoted 9 lines
Quoted 7 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

Quoted 6 lines
Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

People buy hardware to improve performance. Apparently there are people who want to build this hardware. It is not our role to tell either of the groups "this should not be a general goal".

Quoted 29 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Quoted 10 lines

I don't know how it's done. Do tell.

Quoted 7 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

Quoted 8 lines
Quoted 7 lines
Quoted 6 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

And that's perfect for migration.

Quoted 32 lines
Quoted 3 lines

And by the way e.g. vhost has the same
issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Quoted 8 lines
Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Any IOMMU flushes IOTLB on translation changes. If vhost doesn't then it's highly likely to be a bug.

Quoted 15 lines
Quoted 3 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 6 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Quoted 6 lines
Quoted 3 lines

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

virtio must always log PA.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 6 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

Exactly it's a mess. Instead of making everything 10x more complex, let's just keep talking about PA and leave translation to IOMMU.

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

what does pasid have to do with it? anyway, just give driver control over page size.

Quoted 8 lines

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

1) translated/untranslated request
2) request w/ and w/o PASID

Can't say I understand. All the talk about IOVA is just confusing - what we care about for logging is which page to resend.

Quoted 11 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level easily.

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Code walks though.

Quoted 3 lines
Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

What do you refer to? People buy hardware and use it for years with no chance to add features.

Quoted 45 lines
Quoted 45 lines
Quoted 45 lines
Quoted 45 lines
Quoted 37 lines
Quoted 5 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 29 lines

So I think this dirty tracking is a good option to have.

Quoted 25 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 2 lines
Quoted 1 line

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Quoted 6 lines
Quoted 2 lines

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

My point is PRI causes rx ring underruns and throttling CPU makes it worse not better. And I believe people actually tried, nvidia have a pri implementation in hardware. If they come and say virtio help is needed for performance I tend to believe them.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 8 lines
Quoted 3 lines

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Dirty tracking in the IOMMU is annoying enough that I am not sure it's usable. Go ahead but I want to see patches then.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS).

Maybe. This kind of statement is just vague enough not to be falsifiable.

Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

Quoted 6 lines

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation, https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies, and the spec is not a FIFO, not like a early bird can catch all the worm.

Quoted 66 lines
Quoted 66 lines
Quoted 6 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 17 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 7 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Quoted 6 lines
Quoted 4 lines

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#43 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 20 lines · Lingshan
Quoted 16 lines · Michael S. Tsirkin
Quoted 15 lines · Jason Wang
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Jason Wang
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Jason Wang
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Jason Wang
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 4 lines

3) inventing it in the virtio layer will be deprecated in the future
for sure, as platform will provide much rich features for logging
e.g it can do it per PASID etc, I don't see any reason virtio need
to compete with the features that will be provided by the platform

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

[...]

Quoted 12 lines
Quoted 6 lines

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 4 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This is my understanding as well.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#44 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Zhu, Lingshan
Quoted 9 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

Quoted 5 lines

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Quoted 4 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#45 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

Quoted 7 lines
Quoted 5 lines

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

Quoted 5 lines
Quoted 5 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#46 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 18 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#47 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 18 lines · Michael S. Tsirkin
Quoted 17 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight descriptor tracking facility.

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#48 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 26 lines · Lingshan
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Lingshan
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

Quoted 7 lines
Quoted 5 lines

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

Quoted 4 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#49 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 26 lines · Michael S. Tsirkin
Quoted 25 lines · Lingshan
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Lingshan
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

Quoted 7 lines
Quoted 5 lines

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

we need to record the on-device avail_idx and used_idx, or how can the destination side know the device internal values.

Quoted 11 lines
Quoted 5 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

On a second thought, shadow vqs are hypervisor behaviors, maybe should not be described in this device spec.

Since SUSPEND is in device status, so for now I see every type of device implements device_status should support SUSPEND. This should be a general facility.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

But I still believe we are here try our best to work out an industrial spec with better quality, to serve broad interest. This is not competition between companies,
and the spec is not a FIFO, not like a early bird can catch all the worm.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#50 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 34 lines · Lingshan
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Lingshan
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

Quoted 5 lines
Quoted 3 lines

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

we need to record the on-device avail_idx and used_idx, or how can the destination side know the device internal values.

Again you never documented what state the device is in so I can't really say for sure. But generally whenever a buffer is used the internal values are written out to memory.

Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

Quoted 7 lines

On a second thought, shadow vqs are hypervisor behaviors, maybe should not be
described in this device spec.

Since SUSPEND is in device status, so for now I see every type of device implements
device_status should support SUSPEND. This should be a general facility.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#51 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Lingshan
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Lingshan
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

we need to record the on-device avail_idx and used_idx, or how can the destination side know the device internal values.

Again you never documented what state the device is in so I can't really say for sure. But generally whenever a buffer is used the internal values are written out to memory.

This is the state of a virtqueue, in my series I have defined what is vq state in [PATCH V2 1/6] virtio: introduce virtqueue state, and give an example of PCI implementation.

In short, for split vq it is last_avail_idx and in-flight descriptors.

I humbly request an explicit list of missing gaps, so that I can improve my V3

Thanks

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

Quoted 8 lines
Quoted 7 lines

On a second thought, shadow vqs are hypervisor behaviors, maybe should not be
described in this device spec.

Since SUSPEND is in device status, so for now I see every type of device implements
device_status should support SUSPEND. This should be a general facility.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#52 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 48 lines · Lingshan
Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Lingshan
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Lingshan
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

we need to record the on-device avail_idx and used_idx, or how can the destination side know the device internal values.

Again you never documented what state the device is in so I can't really say for sure. But generally whenever a buffer is used the internal values are written out to memory.

This is the state of a virtqueue, in my series I have defined what is vq state in [PATCH V2 1/6] virtio: introduce virtqueue state, and give an example of PCI implementation.

Quoted 6 lines

In short, for split vq it is last_avail_idx and in-flight descriptors.

I humbly request an explicit list of missing gaps, so that I can improve my V3

Thanks

I don't know how to help you without resorting to writing it instead of you, I sent 3 messages in response to that one patch alone. Your patch just adds some bits here and there without much in the way of documentation. Patch needs to explain what these things are and how do they interact with VQ state in memory.

But besides, Parav needs to do exactly the same too. So why don't you let Parav do the work on this and then later just add a small interface to send admin commands through VF itself? Looks like this will be good enough for VDPA. Meanwhile I feel your energy would be better spend working on transport vq which no one else is working on.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

On a second thought, shadow vqs are hypervisor behaviors, maybe should not be
described in this device spec.

Since SUSPEND is in device status, so for now I see every type of device implements
device_status should support SUSPEND. This should be a general facility.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#53 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 52 lines · Michael S. Tsirkin
Quoted 51 lines · Lingshan
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Lingshan
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Lingshan
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

we need to record the on-device avail_idx and used_idx, or how can the destination side know the device internal values.

Again you never documented what state the device is in so I can't really say for sure. But generally whenever a buffer is used the internal values are written out to memory.

This is the state of a virtqueue, in my series I have defined what is vq state in [PATCH V2 1/6] virtio: introduce virtqueue state, and give an example of PCI implementation.

In short, for split vq it is last_avail_idx and in-flight descriptors.

I humbly request an explicit list of missing gaps, so that I can improve my V3

Quoted 6 lines
Quoted 1 line

Thanks

I don't know how to help you without resorting to writing it instead of you, I sent 3 messages in response to that one patch alone. Your patch just adds some bits here and there without much in the way of documentation. Patch needs to explain what these things are and how do they interact with VQ state in memory.

Please allow me quote the series:
+The available state field is two bytes of virtqueue state that is used by +the device to read the next available buffer. It is presented in the following format:
+
+\begin{lstlisting}
+le16 last_avail_idx;
+\end{lstlisting}
+
+The \field{last_avail_idx} field is the free-running available ring +index where the device will read the next available head of a +descriptor chain.

+When SUSPEND is set, the device MUST record the Available State of every enabled splited virtqueue +in \field{Available State} field,
+and correspondingly restore the Available State of every enabled splited virtqueue +from \field{Available State} field when DRIVER_OK is set.
+
+The device SHOULD reset \field{Available State} field upon a device reset

I will add these contents in next series:
1) vq states refer to the device internal states, so have to record and restore them
2) in-flight descriprots tracking.

Not sure whether I should describe how in-guest-memory vq config migrated, because they migrated with guest memory.

Quoted 5 lines

But besides, Parav needs to do exactly the same too. So why don't you let Parav do the work on this and then later just add a small interface to send admin commands through VF itself? Looks like this will be good

I think this is the topic: shall we process admin cmds in config space.....

Quoted 2 lines

enough for VDPA. Meanwhile I feel your energy would be better spend working on transport vq which no one else is working on.

I can start rework on transport vq next week.

Thanks

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 8 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

On a second thought, shadow vqs are hypervisor behaviors, maybe should not be
described in this device spec.

Since SUSPEND is in device status, so for now I see every type of device implements
device_status should support SUSPEND. This should be a general facility.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#54 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 65 lines · Lingshan
Quoted 52 lines · Michael S. Tsirkin
Quoted 51 lines · Lingshan
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Lingshan
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Lingshan
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Lingshan
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Lingshan
Quoted 9 lines
Quoted 3 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using
shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost. Personally if this is coming from a hardware vendor, I am inclined to trust them wrt PCI transactions. But anyway, discussing this at a high level theoretically is pointless - whoever bothers with actual prototyping for performance testing wins, if no one does I'd expect a back of a napkin estimate to be included.

if so, Intel has released productions implementing these interfaces years ago,
see live migration in 4.1. IFCVF vDPA Implementation,
https://doc.dpdk.org/guides-21.11/vdpadevs/ifc.html
and

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

That one is based on shadow queue, right? Which I think this shows is worth supporting.

Yes, it is shadow virtqueue, I assume this is already mostly done, do you see any gaps we need to address in our series that we should work on?

Thanks

There were a ton of comments posted on your series.

Hope I didn't miss anything. I see your latest comments are about vq states, as replied before, I think we can record the states by two le16 and the in-flight
descriptor tracking facility.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I don't know why you need the le16. in-flight tracking should be enough. And given it needs DMA I would try really hard to actually use admin commands for this.

we need to record the on-device avail_idx and used_idx, or how can the destination side know the device internal values.

Again you never documented what state the device is in so I can't really say for sure. But generally whenever a buffer is used the internal values are written out to memory.

This is the state of a virtqueue, in my series I have defined what is vq state in [PATCH V2 1/6] virtio: introduce virtqueue state, and give an example of PCI implementation.

In short, for split vq it is last_avail_idx and in-flight descriptors.

I humbly request an explicit list of missing gaps, so that I can improve my V3

Quoted 6 lines
Quoted 1 line

Thanks

I don't know how to help you without resorting to writing it instead of you, I sent 3 messages in response to that one patch alone. Your patch just adds some bits here and there without much in the way of documentation. Patch needs to explain what these things are and how do they interact with VQ state in memory.

Please allow me quote the series:

(quoted diff) +2 -0
+The available state field is two bytes of virtqueue state that is used by
+the device to read the next available buffer. It is presented in the

following format:

Review

Quoted 8 lines
(quoted diff) +8 -0
+
+\begin{lstlisting}
+le16 last_avail_idx;
+\end{lstlisting}
+
+The \field{last_avail_idx} field is the free-running available ring
+index where the device will read the next available head of a
+descriptor chain.

Next *after what*? The last used buffer? This is exactly the used index.

Quoted 22 lines

+When SUSPEND is set, the device MUST record the Available State of every enabled splited virtqueue

(quoted diff) +2 -0
+in \field{Available State} field,
+and correspondingly restore the Available State of every enabled splited

virtqueue

(quoted diff) +3 -0
+from \field{Available State} field when DRIVER_OK is set.
+
+The device SHOULD reset \field{Available State} field upon a device reset

I will add these contents in next series:
1) vq states refer to the device internal states, so have to record and restore them
2) in-flight descriprots tracking.

Not sure whether I should describe how in-guest-memory vq config migrated, because they migrated with guest memory.

Quoted 5 lines

But besides, Parav needs to do exactly the same too. So why don't you let Parav do the work on this and then later just add a small interface to send admin commands through VF itself? Looks like this will be good

Quoted 27 lines

I think this is the topic: shall we process admin cmds in config space.....

Quoted 2 lines

enough for VDPA. Meanwhile I feel your energy would be better spend working on transport vq which no one else is working on.

I can start rework on transport vq next week.

Thanks

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines

For this shadow virtqueue, do you think I should address this in my V4? Like saying: acknowledged control commands through the control virtqueue should be recorded, and we want to use shadow virtqueue to track dirty pages.

Quoted 6 lines

What you need to do is actually describe what do you expect the device to do when it enters this suspend state. since you mention control virtqueue then it seems that there needs to be a device type specific text explaining the behaviour. If so this implies there needs to be a list of device types that support suspend until someone looks at each type and documents what it does.

On a second thought, shadow vqs are hypervisor behaviors, maybe should not be
described in this device spec.

Since SUSPEND is in device status, so for now I see every type of device implements
device_status should support SUSPEND. This should be a general facility.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#55 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Michael S. Tsirkin
Quoted 56 lines · Michael S. Tsirkin
Quoted 54 lines · Jason Wang
Quoted 52 lines · Michael S. Tsirkin
Quoted 40 lines · Jason Wang
Quoted 39 lines · Michael S. Tsirkin
Quoted 37 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 32 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Repeating myself - I'm not sure that will work well for all workloads.

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

Quoted 8 lines
Quoted 4 lines

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

What I would like to see is effort to switch shadow on/off not keep it on at all times. That's only good enough for a PoC. And to work on top of virtio that will require effort in the spec.

Well, there're various approaches. If we just care about the shadow vq on/off. Virtqueue indexes plus inflight should be sufficient.

Talking about the future, since vDPA allows to conditionally trap a virtqueue via ASID. I expect virtio can do the same if PASID is supported (and there used to be a proposal for this in the past).

Quoted 3 lines

If I see spec patches
that do that I personally would support that. It needs to be reasonably generic though, a single 16 bit RW number is not going to be enough.

It's really device specific, vDPA has demonstrated that it's sufficient for networking devices.

Quoted 3 lines

I
think it's likely admin commands is a good interface for this. If it's a hack making vendor specific assumptions, just keep it in vdpa.

This part I don't understand. Most of the virtqueue states were accessed via common_cfg, I don't see the advantages of separating the others in other places unless there's a new transport.

Quoted 56 lines
Quoted 56 lines
Quoted 23 lines
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

Quoted 8 lines
Quoted 6 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 4 lines
Quoted 2 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Quoted 7 lines
Quoted 4 lines

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

People buy hardware to improve performance. Apparently there are people who want to build this hardware.

We are talking about different things. What I'm saying is that sticking to wire speed somehow conflicts with the goal of downtime. If mgmt/guest doesn't allow to increase the downtime, it's very hard to stick the wirespeed during live dirty page tracking. This doesn't prevent people from building and using faster hardware, the hardware might just run slower when doing live migration. If I was wrong, please explain why.

Quoted 2 lines

It is not our role to tell either
of the groups "this should not be a general goal".

Well, the downtime has been well studied and used for years, and I describe the assumptions:

"
It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.
"

KVM and Qemu have a lot of mechanisms to throttle as well.

Quoted 43 lines
Quoted 43 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Quoted 16 lines

I don't know how it's done. Do tell.

Quoted 13 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

And that's perfect for migration.

Right.

Quoted 41 lines
Quoted 41 lines
Quoted 4 lines

And by the way e.g. vhost has the same
issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>
Quoted 4 lines
Quoted 1 line

All of the above is not what virtio did right now.

Any IOMMU flushes IOTLB on translation changes. If vhost doesn't then it's highly likely to be a bug.

It is exactly what vhost did.

Quoted 27 lines
Quoted 27 lines
Quoted 5 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

Quoted 5 lines
Quoted 1 line

SVA being what here?

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

Quoted 5 lines
Quoted 3 lines

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

virtio must always log PA.

How? Without ATS, the device can't see PA since it can only use untranslated requests ...

Quoted 22 lines
Quoted 14 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 6 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

Quoted 2 lines

Exactly it's a mess. Instead of making everything 10x more complex, let's just keep talking about PA and leave translation to IOMMU.

For many reasons, the device can't see PA.

Even with PA, it's still problematic, is it GPA or HPA? GPA may only work if the device is abstracted as two dimension I/O page tables like IOMMU. For HPA, we can't just report it to the userspace which requires a software translation again. What's more, as stated above, there's no way for the device to know if the PA is valid or not (unless there's an ATS), logging an invalid PA is dangerous and may have security implications.

Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

Quoted 5 lines
Quoted 2 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Can't say I understand. All the talk about IOVA is just confusing - what we care about for logging is which page to resend.

See above.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level easily.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Code walks though.

There's even no code work from Parav to describe how it can work for a hypervisor.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

What do you refer to? People buy hardware and use it for years with no chance to add features.

IOMMU evolves quickly, duplicating its functionality looks like a re-inventing of the wheels.

Again, I think we don't want to suffer from the hard times in bypassing the platform IOMMU again like in the past.

Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
Quoted 58 lines
Quoted 58 lines
Quoted 57 lines
Quoted 9 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 31 lines

So I think this dirty tracking is a good option to have.

Quoted 27 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

Quoted 8 lines
Quoted 3 lines

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

My point is PRI causes rx ring underruns and throttling CPU makes it worse not better. And I believe people actually tried, nvidia have a pri implementation in hardware. If they come and say virtio help is needed for performance I tend to believe them.

I'm not saying I'm not trusting NV. It's not about trust at all, I'm
saying: if they fail with PRI,

1) if there's any fault in virtio that damages the performance of PRI, let's fix it in virtio
2) if it's not the fault of virtio in the context of PRI, it doesn't necessarily mean logging via virtio is the only way to go, we can seek support from others which fit better

Unfortunately, they didn't explain why they chose to do it in virtio until I pointed out the issues.

Quoted 34 lines
Quoted 34 lines
Quoted 33 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

There is no requirement for mandating PRI either.
So it is unusable.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

Quoted 7 lines
Quoted 5 lines

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Dirty tracking in the IOMMU is annoying enough that I am not

What issue did you see? We can report them to platform vendors anyhow.

Quoted 1 line

sure it's usable. Go ahead but I want to see patches then.

If we agree to log via IOMMU what kind of patches did you expect to see?

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

Quoted 6 lines
Quoted 4 lines

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS).

Maybe. This kind of statement is just vague enough not to be falsifiable.

I don't think so. It could be falsifiable if some vendor comes with real numbers:

1) demonstrate the possibility of converging a migration when virito is running at wire speed
2) demonstrate logging dirty pages in one VF doesn't damage the performance of other

with reasonable explanations. It's not hard to test the above two simple cases.

Quoted 6 lines
Quoted 4 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost.

This is the way current vhost is working. We know the pros/cons. And there are many ways to limit the bandwidth/QOS of a software based dirty tracking.

Quoted 3 lines

Personally if this is
coming from a hardware vendor, I am inclined to trust them wrt PCI transactions.

The point is not about trust. I think Parav has said in another thread that RX performance is throttled by the dirty tracking.

Quoted 2 lines

But anyway, discussing this at a high level theoretically is pointless -

As a reviewer, the most important thing for me is to make sure the proposal is theoretically correct before I can go through the details.

Quoted 2 lines

whoever bothers with actual prototyping for performance testing wins,

This part I don't understand.

LingShan has given you the proof that Intel has done it several years ago. And shadow virtqueue is inspired by those works as well. LingShan's proposal is based on those experiences and that's why LingShan's proposal does not come with dirty page tracking.

My understanding is, being an open device standard, the spec needs to seek the best way to go instead of just one of the possible ways to go. We never claim "we are the first so let's go with my way".

Quoted 2 lines

if no one does I'd expect a back of a napkin estimate to be included.

I'd expect any huge feature like this needs to be prototyped before they can be discussed or it needs to be tagged as RFC.

Thanks

Quoted 83 lines
Quoted 83 lines
Quoted 6 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 8 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Quoted 7 lines
Quoted 4 lines

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

Quoted 1 line
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#56 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 62 lines · Jason Wang
Quoted 61 lines · Michael S. Tsirkin
Quoted 58 lines · Jason Wang
Quoted 44 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 41 lines · Michael S. Tsirkin
Quoted 39 lines · Jason Wang
Quoted 37 lines · Michael S. Tsirkin
Quoted 28 lines · Jason Wang
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 5 lines
Quoted 5 lines

3) inventing it in the virtio layer will be deprecated in the future
for sure, as platform will provide much rich features for logging
e.g it can do it per PASID etc, I don't see any reason virtio need
to compete with the features that will be provided by the platform

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's no need to duplicate their job. Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to support dirty page tracking.

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support.
"""

Quoted 4 lines
Quoted 1 line

Without such platform commitment, virtio also skipping it would not work.

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 6 lines
Quoted 3 lines

What I would like to see is effort to switch shadow on/off not keep it on at all times. That's only good enough for a PoC. And to work on top of virtio that will require effort in the spec.

Well, there're various approaches. If we just care about the shadow vq on/off. Virtqueue indexes plus inflight should be sufficient.

I'm not sure what "inflight" is and what "indexes" are but yes, you need information about buffers that have been made available to device and have not been consumed yet.

Quoted 3 lines

Talking about the future, since vDPA allows to conditionally trap a virtqueue via ASID. I expect virtio can do the same if PASID is supported (and there used to be a proposal for this in the past).

I don't know what does "trap" mean in this sentence.

Quoted 6 lines
Quoted 3 lines

If I see spec patches
that do that I personally would support that. It needs to be reasonably generic though, a single 16 bit RW number is not going to be enough.

It's really device specific, vDPA has demonstrated that it's sufficient for networking devices.

I think that existing vdpa devices are just silently in order. If device is in order, and given it's networking so there's no processing as such - just DMA - then I think the state of the ring is fully described by the available and used index in memory. Maybe I'm missing something obvious.

Quoted 7 lines
Quoted 3 lines

I
think it's likely admin commands is a good interface for this. If it's a hack making vendor specific assumptions, just keep it in vdpa.

This part I don't understand. Most of the virtqueue states were accessed via common_cfg, I don't see the advantages of separating the others in other places unless there's a new transport.

A ring has up to 64k buffers available and not used. I'm not sure how much info is necessary for each but even with a byte per buffer, and multiplied by 32k queues we are pushing a gigabyte. Reading this out through a register mapped interface from the hypervisor, with an exit per dword is going to be unreasonably slow.

So you are going to do DMA, and pass some commands back and forth. Why not reuse the admin command structure for this? The admin command header is 16 bytes for write portion and 8 bytes for read portion. And that is overkill? Saving 24 bytes of DMA on slow path is worth inventing a custom format for? Color me unimpressed.

Yes, in order is simpler and you might get away without this. I am not very excited about a feature so limited, but hey - make the dependency explicit, we can discuss.

Quoted 64 lines
Quoted 56 lines
Quoted 56 lines
Quoted 23 lines
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM (earlier).

Quoted 8 lines
Quoted 6 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 4 lines
Quoted 2 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Quoted 7 lines
Quoted 4 lines

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

People buy hardware to improve performance. Apparently there are people who want to build this hardware.

Quoted 7 lines

We are talking about different things. What I'm saying is that sticking to wire speed somehow conflicts with the goal of downtime. If mgmt/guest doesn't allow to increase the downtime, it's very hard to stick the wirespeed during live dirty page tracking. This doesn't prevent people from building and using faster hardware, the hardware might just run slower when doing live migration. If I was wrong, please explain why.

Which wire? Think about it. If your "wire speed" is saturating the pci link then extra traffic on that link is going to mean you go slower. This does not immediately mean you can just ignore speed completely either btw. Are all devices and all work-loads always saturating pci? I doubt it. For example, latency matters for a lot of people. You don't saturate pci but you don't want your hypervisor to be on the data path. That's a problem for shadow and for PRI.

Quoted 6 lines
Quoted 2 lines

It is not our role to tell either
of the groups "this should not be a general goal".

Well, the downtime has been well studied and used for years, and I describe the assumptions:

Quoted 6 lines

"
It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.
"

KVM and Qemu have a lot of mechanisms to throttle as well.

Yes, and so? That all exists, if people are satisfied with what exists we can call it a day and not bother adding stuff to spec.

Quoted 128 lines
Quoted 43 lines
Quoted 43 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which looks very hard.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Quoted 16 lines

I don't know how it's done. Do tell.

Quoted 13 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging IOVA.

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

And that's perfect for migration.

Right.

Quoted 41 lines
Quoted 41 lines
Quoted 4 lines

And by the way e.g. vhost has the same
issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>
Quoted 4 lines
Quoted 1 line

All of the above is not what virtio did right now.

Any IOMMU flushes IOTLB on translation changes. If vhost doesn't then it's highly likely to be a bug.

It is exactly what vhost did.

Quoted 27 lines
Quoted 27 lines
Quoted 5 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

Quoted 5 lines
Quoted 1 line

SVA being what here?

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

virtio must always log PA.

How? Without ATS, the device can't see PA since it can only use untranslated requests ...

Please can we speak the spec terms?
It does not matter that there's some IOMMU somewhere and then it wants to call addresses on the physical pci link virtual addresses. device vendors without an iommu only know one kind of address. And so the only place where virtio spec mentions IOVA is in iommu device part. The rest of the spec calls whatever is in the ring "physical address".

Quoted 27 lines
Quoted 22 lines
Quoted 14 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 6 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

Quoted 2 lines

Exactly it's a mess. Instead of making everything 10x more complex, let's just keep talking about PA and leave translation to IOMMU.

For many reasons, the device can't see PA.

Quoted 7 lines

Even with PA, it's still problematic, is it GPA or HPA? GPA may only work if the device is abstracted as two dimension I/O page tables like IOMMU. For HPA, we can't just report it to the userspace which requires a software translation again. What's more, as stated above, there's no way for the device to know if the PA is valid or not (unless there's an ATS), logging an invalid PA is dangerous and may have security implications.

/facepalm

virtio only knows one type of address. it calls it "physical address" for historical reasons. don't program an invalid address into the device otherwise you will break it and get to keep both pieces.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Can't say I understand. All the talk about IOVA is just confusing - what we care about for logging is which page to resend.

See above.

I still see nother relevant above.

Quoted 30 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level easily.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Code walks though.

There's even no code work from Parav to describe how it can work for a hypervisor.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 8 lines
Quoted 2 lines

What do you refer to? People buy hardware and use it for years with no chance to add features.

IOMMU evolves quickly, duplicating its functionality looks like a re-inventing of the wheels.

Again, I think we don't want to suffer from the hard times in bypassing the platform IOMMU again like in the past.

This is just a weird claim. Platforms historically evolved much slower than devices. Which IOMMUs evolve quickly? What is quickly in your world?

Quoted 76 lines
Quoted 67 lines
Quoted 67 lines
Quoted 66 lines
Quoted 58 lines
Quoted 58 lines
Quoted 57 lines
Quoted 9 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 31 lines

So I think this dirty tracking is a good option to have.

Quoted 27 lines
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 6 lines
Quoted 2 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 3 lines
Quoted 3 lines

4) if the platform support is missing, we can use software or
leverage transport for assistance like PRI

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

Quoted 8 lines
Quoted 3 lines

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

My point is PRI causes rx ring underruns and throttling CPU makes it worse not better. And I believe people actually tried, nvidia have a pri implementation in hardware. If they come and say virtio help is needed for performance I tend to believe them.

Quoted 5 lines

I'm not saying I'm not trusting NV. It's not about trust at all, I'm
saying: if they fail with PRI,

1) if there's any fault in virtio that damages the performance of PRI, let's fix it in virtio

PRI is just slow nothing to do with virtio.

Quoted 3 lines

2) if it's not the fault of virtio in the context of PRI, it doesn't necessarily mean logging via virtio is the only way to go, we can seek support from others which fit better

I don't know how is anyone going to do anything useful with feedback like this. monkey see problem monkey fix problem.

Quoted 2 lines

Unfortunately, they didn't explain why they chose to do it in virtio until I pointed out the issues.

More motivation is always nice to have.

Quoted 40 lines
Quoted 40 lines
Quoted 39 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Quoted 3 lines
Quoted 1 line

Dirty tracking in the IOMMU is annoying enough that I am not

What issue did you see? We can report them to platform vendors anyhow.

IIUC there's no log. You need to scan all PTEs to test and clear the dirty bit. This costs CPU time. The issues were discussed when kvm switched to PML - the reason PML is nice is not IMHO as you say that it stops the VM - that's more of a problem for KVM - it is that you don't need to keep rescanning memory.

Quoted 3 lines
Quoted 1 line

sure it's usable. Go ahead but I want to see patches then.

If we agree to log via IOMMU what kind of patches did you expect to see?

A patch to iommufd that lets you find out which memory was modified so you can migrate it.

Quoted 27 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

Quoted 6 lines
Quoted 4 lines

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS).

Maybe. This kind of statement is just vague enough not to be falsifiable.

I don't think so. It could be falsifiable if some vendor comes with real numbers:

Quoted 6 lines

1) demonstrate the possibility of converging a migration when virito is running at wire speed
2) demonstrate logging dirty pages in one VF doesn't damage the performance of other

with reasonable explanations. It's not hard to test the above two simple cases.

what does the above have to do with "unnecessary PCI transactions" and "issues like QOS"?

Quoted 10 lines
Quoted 6 lines
Quoted 4 lines

So I can't
believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

On the other hand they have an extra CPU cost.

This is the way current vhost is working. We know the pros/cons. And there are many ways to limit the bandwidth/QOS of a software based dirty tracking.

So good. Leave it also, it works. You like how it works, whoever is satisfied can just use it. Can we move on?

Quoted 10 lines
Quoted 3 lines

Personally if this is
coming from a hardware vendor, I am inclined to trust them wrt PCI transactions.

The point is not about trust. I think Parav has said in another thread that RX performance is throttled by the dirty tracking.

Quoted 2 lines

But anyway, discussing this at a high level theoretically is pointless -

Quoted 7 lines

As a reviewer, the most important thing for me is to make sure the proposal is theoretically correct before I can go through the details.

Quoted 2 lines

whoever bothers with actual prototyping for performance testing wins,

This part I don't understand.

You just asked for a prototype and performance numbers yourself.

Quoted 4 lines

LingShan has given you the proof that Intel has done it several years ago. And shadow virtqueue is inspired by those works as well. LingShan's proposal is based on those experiences and that's why LingShan's proposal does not come with dirty page tracking.

Fine. So dirty tracking should be optional. Sounds good. And there should be some info showing how dirty tracking if available beings a performance benefit. Sounds even better.

Quoted 4 lines

My understanding is, being an open device standard, the spec needs to seek the best way to go instead of just one of the possible ways to go. We never claim "we are the first so let's go with my way".

Quoted 9 lines
Quoted 2 lines

if no one does I'd expect a back of a napkin estimate to be included.

I'd expect any huge feature like this needs to be prototyped before they can be discussed or it needs to be tagged as RFC.

Thanks

I think this was already done. Parav?

Quoted 86 lines
Quoted 86 lines
Quoted 86 lines
Quoted 9 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

than you should propose that in the dirty page tracking series that you listed above to not do dirty page tracking. Rather depend on PRI, right?

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about why.

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 8 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Quoted 7 lines
Quoted 4 lines

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

Quoted 1 line
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#57 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Jason Wang
Quoted 101 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:02 AM

Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Wed, Nov 8, 2023 at 4:17 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines · Jason Wang
Quoted 1 line

On Tue, Nov 7, 2023 at 3:05 PM Michael S. Tsirkin <[email protected]>

Quoted 12 lines · wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines · Jason Wang
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 7 lines
Quoted 7 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's

no need to duplicate their job.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to

support dirty page tracking.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support. """

Quoted 1 line

Without such platform commitment, virtio also skipping it would

not work.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM

(earlier).

Quoted 4 lines
Quoted 4 lines

To me the important advantage of PML is that it uses limited resources on the host which

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

The device resource reservation starts only when the device migration starts. i.e. with WRITE_RECORDS_START command of patch 7 in the series.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

PML is yet another option that requires small pci writes. In the current proposal, there are no small PCI writes. It is a query interface from the device.

Quoted 6 lines
Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

May not have to be.
Speed of page recording should be fast enough.
It usually improves with subsequent generation.

Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which

looks very hard.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

Quoted 8 lines
Quoted 1 line

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Device page tracking table to be consulted to flush on mapping change.

Quoted 62 lines
Quoted 10 lines

I don't know how it's done. Do tell.

Quoted 7 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 9 lines
Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 2 lines

And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>
Quoted 7 lines

All of the above is not what virtio did right now.

Quoted 2 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

The query interface in this proposal works on the granular boundary to read and clear. This will ensure that mapping is consistent.

Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the
interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

There cannot be logging before the DMA.
Only requirement is before the mapping changes, the dirty page tracking to be synced.

In most common cases where the perf is critical, such mapping wont change so often dynamically anyway.

Quoted 7 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

It is incorrect attribution that platform is duplicated here. It feeds the data to the platform as needed without replicating.

I do agree that there is overlap of IOMMU tracking the dirty and storing it in the per PTE vs device supplying its dirty track via its own interface. Both are consolidated at hypervisor level.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

Quoted 8 lines
Quoted 2 lines

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

PASID to vq assignment won't be duplicated.
It is configured fully by the guest without consulting hypervisor at the device level. Guest IOMMU would consult hypervisor to setup any PASID mapping as part of any mapping method.

Quoted 124 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level

easily.

Quoted 2 lines

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 34 lines

So I think this dirty tracking is a good option to have.

Quoted 30 lines
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 8 lines
Quoted 3 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 4 lines
Quoted 4 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

Quoted 27 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

There is no requirement for mandating PRI either. So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 9 lines
Quoted 3 lines

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Not duplicated. Feeding into the platform.

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

Quoted 7 lines

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

In the current proposal, it does not required PCI transactions, as there is only a hypervisor-initiated query interface. It is a trade off of using svq + pasid vs using something from the device.

Again, both has different use case and value. One uses cpu and one uses device. Depending how much power one wants to spend where..

Quoted 72 lines
Quoted 3 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 1 line

than you should propose that in the dirty page tracking series that

you listed above to not do dirty page tracking. Rather depend on PRI, right?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about

why.

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 8 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Quoted 7 lines
Quoted 4 lines

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

#58 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 106 lines · Parav Pandit
Quoted 106 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:02 AM

Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Wed, Nov 8, 2023 at 4:17 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines · Jason Wang
Quoted 1 line

On Tue, Nov 7, 2023 at 3:05 PM Michael S. Tsirkin <[email protected]>

Quoted 12 lines · wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines · Jason Wang
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 7 lines
Quoted 7 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's

no need to duplicate their job.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to

support dirty page tracking.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support. """

Quoted 1 line

Without such platform commitment, virtio also skipping it would

not work.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM

(earlier).

Quoted 8 lines
Quoted 7 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

The device resource reservation starts only when the device migration starts. i.e. with WRITE_RECORDS_START command of patch 7 in the series.

Right, but this is not the question, see below.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 11 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

PML is yet another option that requires small pci writes. In the current proposal, there are no small PCI writes. It is a query interface from the device.

Well, you've explained in another thread that actually it needs small PCI writes.

E.g during IOTLB invalidation ...

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

May not have to be.
Speed of page recording should be fast enough.
It usually improves with subsequent generation.

If you have something better, let's propose it from the start.

Quoted 138 lines
Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which

looks very hard.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

Quoted 8 lines
Quoted 1 line

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Device page tracking table to be consulted to flush on mapping change.

Quoted 62 lines
Quoted 10 lines

I don't know how it's done. Do tell.

Quoted 7 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 9 lines
Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 2 lines

And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>
Quoted 7 lines

All of the above is not what virtio did right now.

Quoted 2 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

The query interface in this proposal works on the granular boundary to read and clear. This will ensure that mapping is consistent.

Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the
interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 1 line

There cannot be logging before the DMA.

Well, I don't see how this is related to the issue above. Logging after the DMA doesn't mean the device can know what sits behind an IOVA, no?

Quoted 3 lines

Only requirement is before the mapping changes, the dirty page tracking to be synced.

In most common cases where the perf is critical, such mapping wont change so often dynamically anyway.

I've explained the issue in another reply.

Quoted 188 lines
Quoted 8 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

It is incorrect attribution that platform is duplicated here. It feeds the data to the platform as needed without replicating.

I do agree that there is overlap of IOMMU tracking the dirty and storing it in the per PTE vs device supplying its dirty track via its own interface. Both are consolidated at hypervisor level.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

Quoted 8 lines
Quoted 2 lines

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

PASID to vq assignment won't be duplicated.
It is configured fully by the guest without consulting hypervisor at the device level. Guest IOMMU would consult hypervisor to setup any PASID mapping as part of any mapping method.

Quoted 124 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level

easily.

Quoted 2 lines

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 34 lines

So I think this dirty tracking is a good option to have.

Quoted 30 lines
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 8 lines
Quoted 3 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 4 lines
Quoted 4 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

Quoted 27 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

There is no requirement for mandating PRI either. So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 9 lines
Quoted 3 lines

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Not duplicated. Feeding into the platform.

Quoted 16 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

Quoted 8 lines
Quoted 7 lines

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

In the current proposal, it does not required PCI transactions, as there is only a hypervisor-initiated query interface.

Such query requires at least several transactions, no?

Or to make things more clear, could you list the steps how a hypervisor is expected to do the querying?

Quoted 3 lines

It is a trade off of using svq + pasid vs using something from the device.

Again, both has different use case and value. One uses cpu and one uses device.

That's for sure.

Quoted 1 line

Depending how much power one wants to spend where..

But as the author of the proposal, you need to elaborate more on how you expect for a hypervisor instead of letting the reviewer guess.

Thanks

Quoted 66 lines
Quoted 66 lines
Quoted 4 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 1 line

than you should propose that in the dirty page tracking series that

you listed above to not do dirty page tracking. Rather depend on PRI, right?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about

why.

Quoted 25 lines
Quoted 24 lines
Quoted 14 lines

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 7 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Quoted 5 lines
Quoted 1 line

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#59 lore raw

[virtio-comment] RE: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 144 lines

From: Jason Wang <[email protected]>
Sent: Thursday, November 16, 2023 9:49 AM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:02 AM

On Thu, Nov 9, 2023 at 3:59 PM Michael S. Tsirkin <[email protected]>

Quoted 106 lines · wrote:
Quoted 106 lines
Quoted 4 lines · Jason Wang
Quoted 2 lines

On Wed, Nov 8, 2023 at 4:17 PM Michael S. Tsirkin
<[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Jason Wang
Quoted 2 lines

On Tue, Nov 7, 2023 at 3:05 PM Michael S. Tsirkin <[email protected]>

Quoted 12 lines · wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines · Jason Wang
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 7 lines
Quoted 7 lines

3) inventing it in the virtio layer will be
deprecated in the future for sure, as platform
will provide much rich features for logging e.g
it can do it per PASID etc, I don't see any
reason virtio need to compete with the features
that will be provided by the platform

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's

no need to duplicate their job.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu
vendors to

support dirty page tracking.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support. """

Quoted 2 lines

Without such platform commitment, virtio also skipping it would

not work.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM

(earlier).

Quoted 8 lines
Quoted 7 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

The device resource reservation starts only when the device migration starts. i.e. with WRITE_RECORDS_START command of patch 7 in the series.

Right, but this is not the question, see below.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be

addressed.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

PML is yet another option that requires small pci writes. In the current proposal, there are no small PCI writes. It is a query interface from the device.

Well, you've explained in another thread that actually it needs small PCI writes.

No. There may be some misunderstanding.

Quoted 2 lines

E.g during IOTLB invalidation ...

This is not part of the virtio interface.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Problem of course is that it might overflow and cause things like packet drops.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

May not have to be.
Speed of page recording should be fast enough.
It usually improves with subsequent generation.

If you have something better, let's propose it from the start.

It is unproven that current proposal cannot be done by the device. So I prefer to do this incrementally when we get there.

Quoted 141 lines
Quoted 96 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which

looks very hard.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

Quoted 8 lines
Quoted 1 line

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Device page tracking table to be consulted to flush on mapping change.

Quoted 65 lines
Quoted 11 lines

I don't know how it's done. Do tell.

Quoted 8 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 9 lines
Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 2 lines

And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

The query interface in this proposal works on the granular boundary to read

and clear.

Quoted 37 lines

This will ensure that mapping is consistent.

Quoted 35 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 5 lines
Quoted 1 line

There cannot be logging before the DMA.

Well, I don't see how this is related to the issue above. Logging after the DMA doesn't mean the device can know what sits behind an IOVA, no?

Device does not know what sits behind IOVA.
IOVA to GPA mapping is maintained by the iommu.
Hence, device records and hypervor query them.

Quoted 211 lines
Quoted 1 line

Only requirement is before the mapping changes, the dirty page tracking to be

synced.

Quoted 2 lines

In most common cases where the perf is critical, such mapping wont change

so often dynamically anyway.

I've explained the issue in another reply.

Quoted 7 lines
Quoted 7 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as

well.

Quoted 5 lines
Quoted 1 line

It is incorrect attribution that platform is duplicated here. It feeds the data to the platform as needed without replicating.

I do agree that there is overlap of IOMMU tracking the dirty and storing it in

the per PTE vs device supplying its dirty track via its own interface.

Quoted 12 lines

Both are consolidated at hypervisor level.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and

others.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space

feature in core virtio layer:

Quoted 3 lines
Quoted 1 line

PASID to vq assignment won't be duplicated.
It is configured fully by the guest without consulting hypervisor at the device

level.

Quoted 1 line

Guest IOMMU would consult hypervisor to setup any PASID mapping as part

of any mapping method.

Quoted 13 lines
Quoted 13 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

True, but my question is where or not it can be done at the device level

easily.

Quoted 2 lines

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to

do it.

Quoted 37 lines
Quoted 37 lines

And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 18 lines

So I think this dirty tracking is a good option to have.

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 8 lines
Quoted 3 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 4 lines
Quoted 4 lines

4) if the platform support is missing, we can
use software or leverage transport for
assistance like PRI

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can

converge.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Quoted 7 lines
Quoted 2 lines

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is

enabled.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

There is no requirement for mandating PRI either. So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in

virtio.

Quoted 23 lines
Quoted 1 line

Not duplicated. Feeding into the platform.

Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has

Quoted 9 lines

good performance overall.

Quoted 4 lines
Quoted 3 lines

Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

In the current proposal, it does not required PCI transactions, as there is only a

hypervisor-initiated query interface.

Such query requires at least several transactions, no?

It depends on how much memory is unmapped.

For the pinned memory cases with hw iommu's mapping unmapping is not frequent event.

Quoted 3 lines

Or to make things more clear, could you list the steps how a hypervisor is expected to do the querying?

It is listed in other email thread where you described the race condition.

Quoted 11 lines
Quoted 3 lines

It is a trade off of using svq + pasid vs using something from the device.

Again, both has different use case and value. One uses cpu and one uses

device.

That's for sure.

Quoted 1 line

Depending how much power one wants to spend where..

But as the author of the proposal, you need to elaborate more on how you expect for a hypervisor instead of letting the reviewer guess.

Hypervisor would look for ways on how to do functionality and find it in the spec. But I will add a short description on the write record read command.

#60 lore raw

[virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

In reply to Parav Pandit
Quoted 105 lines · Parav Pandit
Quoted 105 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:02 AM

Quoted 3 lines · Michael S. Tsirkin
Quoted 1 line · Jason Wang

On Wed, Nov 8, 2023 at 4:17 PM Michael S. Tsirkin <[email protected]>

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines · Jason Wang
Quoted 1 line

On Tue, Nov 7, 2023 at 3:05 PM Michael S. Tsirkin <[email protected]>

Quoted 12 lines · wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines · Jason Wang
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 7 lines
Quoted 7 lines

3) inventing it in the virtio layer will be deprecated in the future for sure, as platform will provide much rich features for logging e.g it can do it per PASID etc, I don't see any reason virtio need to compete with the features that will be provided by the platform

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's

no need to duplicate their job.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu vendors to

support dirty page tracking.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Let me quote from the above link:

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support. """

Quoted 1 line

Without such platform commitment, virtio also skipping it would

not work.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM

(earlier).

Quoted 8 lines
Quoted 7 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

The device resource reservation starts only when the device migration starts. i.e. with WRITE_RECORDS_START command of patch 7 in the series.

And now your precious VM can't migrate at all because -ENOSPC.

Quoted 126 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be addressed.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

PML is yet another option that requires small pci writes. In the current proposal, there are no small PCI writes. It is a query interface from the device.

Quoted 6 lines
Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

May not have to be.
Speed of page recording should be fast enough.
It usually improves with subsequent generation.

Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which

looks very hard.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

Quoted 8 lines
Quoted 1 line

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Device page tracking table to be consulted to flush on mapping change.

Quoted 64 lines
Quoted 10 lines

I don't know how it's done. Do tell.

Quoted 7 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 9 lines
Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 2 lines

And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

The query interface in this proposal works on the granular boundary to read and clear. This will ensure that mapping is consistent.

By itself it does not, you have to actually keep querying until you flush all dirty info and do it each time there's an invalidation in the IOMMU.

Quoted 204 lines
Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the
interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

There cannot be logging before the DMA.
Only requirement is before the mapping changes, the dirty page tracking to be synced.

In most common cases where the perf is critical, such mapping wont change so often dynamically anyway.

Quoted 7 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as well.

It is incorrect attribution that platform is duplicated here. It feeds the data to the platform as needed without replicating.

I do agree that there is overlap of IOMMU tracking the dirty and storing it in the per PTE vs device supplying its dirty track via its own interface. Both are consolidated at hypervisor level.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and others.

Quoted 8 lines
Quoted 2 lines

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space feature in core virtio layer:

PASID to vq assignment won't be duplicated.
It is configured fully by the guest without consulting hypervisor at the device level. Guest IOMMU would consult hypervisor to setup any PASID mapping as part of any mapping method.

Quoted 128 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

True, but my question is where or not it can be done at the device level

easily.

Quoted 2 lines

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to do it. And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 34 lines

So I think this dirty tracking is a good option to have.

Quoted 30 lines
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 8 lines
Quoted 3 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 4 lines
Quoted 4 lines

4) if the platform support is missing, we can use software or leverage transport for assistance like PRI

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can converge.

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

Quoted 8 lines
Quoted 5 lines

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is enabled.

Quoted 30 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

There is no requirement for mandating PRI either. So it is unusable.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Yea but just vague questions don't help to make sure eiter way.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

Quoted 6 lines
Quoted 5 lines

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in virtio.

Not duplicated. Feeding into the platform.

I mean IOMMU still sets the dirty bit, too. How is that not a duplication?

Quoted 23 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Not true platforms have things like NICs on board and have for many years. It's about performance really.

Quoted 7 lines

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has good performance overall. Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

Quoted 5 lines

In the current proposal, it does not required PCI transactions, as there is only a hypervisor-initiated query interface. It is a trade off of using svq + pasid vs using something from the device.

Again, both has different use case and value. One uses cpu and one uses device. Depending how much power one wants to spend where..

Also how much effort we want to spend on this virtio specific thing. The needs to be a *reason* to do things in virtio as opposed to using platform capabilities, this is exactly the same thing I told Lingshan wrt using SUSPEND for power management as opposed to using PCI PM - relying on platform when we can is right there in the mission statement. For some reason I asssumed you guys have done a PoC and that's the motivation but if it's a "just in case" feature then I'd suggest we focus on merging patches 1-5 first.

Quoted 65 lines
Quoted 65 lines
Quoted 3 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 1 line

than you should propose that in the dirty page tracking series that

you listed above to not do dirty page tracking. Rather depend on PRI, right?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about

why.

Quoted 25 lines
Quoted 24 lines
Quoted 14 lines

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 7 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for years. I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Quoted 5 lines
Quoted 1 line

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#61 lore raw

RE: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 119 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: Friday, November 17, 2023 3:46 PM

Quoted 5 lines · Parav Pandit
Quoted 5 lines

From: Jason Wang <[email protected]>
Sent: Monday, November 13, 2023 9:02 AM

On Thu, Nov 9, 2023 at 3:59 PM Michael S. Tsirkin <[email protected]>

Quoted 106 lines · wrote:
Quoted 106 lines
Quoted 4 lines · Jason Wang
Quoted 2 lines

On Wed, Nov 8, 2023 at 4:17 PM Michael S. Tsirkin
<[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Jason Wang
Quoted 2 lines

On Tue, Nov 7, 2023 at 3:05 PM Michael S. Tsirkin <[email protected]>

Quoted 12 lines · wrote:
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines · Jason Wang
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

Each virtio and non virtio devices who wants to report their dirty page report,

will do their way.

Quoted 7 lines
Quoted 7 lines

3) inventing it in the virtio layer will be
deprecated in the future for sure, as platform
will provide much rich features for logging e.g
it can do it per PASID etc, I don't see any
reason virtio need to compete with the features
that will be provided by the platform

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Can you bring the cpu vendors and committement to virtio tc with timelines

so that virtio TC can omit?

Why do we need to bring CPU vendors in the virtio TC? Virtio needs to be built on top of transport or platform. There's

no need to duplicate their job.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Especially considering that virtio can't do better than them.

I wanted to see a strong commitment for the cpu
vendors to

support dirty page tracking.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The RFC of IOMMUFD support can go back to early 2022. Intel, AMD and ARM are all supporting that now.

Quoted 1 line

And the work seems to have started for some platforms.

Let me quote from the above link:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

"""
Today, AMD Milan (or more recent) supports it while ARM SMMUv3.2 alongside VT-D rev3.x also do support. """

Quoted 2 lines

Without such platform commitment, virtio also skipping it would

not work.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Is the above sufficient? I'm a little bit more familiar with vtd, the hw feature has been there for years.

Repeating myself - I'm not sure that will work well for all workloads.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I think this comment applies to this proposal as well.

Yes - some systems might be better off with platform tracking. And I think supporting shadow vq better would be nice too.

For shadow vq, did you mean the work that is done by Eugenio?

Yes.

That's exactly why vDPA starts with shadow virtqueue. We've evaluated various possible approaches, each of them have their shortcomings and shadow virtqueue is the only one that doesn't require any additional hardware features to work in every platform.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 6 lines

Definitely KVM did
not scan PTEs. It used pagefaults with bit per page and later as VM size grew switched to PLM. This interface is analogous to PLM,

I think you meant PML actually. And it doesn't work like PML. To behave like PML it needs to

1) log buffers were organized as a queue with indices
2) device needs to suspend (as a #vmexit in PML) if it runs out of the buffers
3) device need to send a notification to the driver if it runs out of the buffer

I don't see any of the above in this proposal. If we do that it would be less problematic than what is being proposed here.

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

What is common between this and PML is that you get the addresses directly without scanning megabytes of bitmaps or worse - hundreds of megabytes of page tables.

Yes, it has overhead but this is the method we use for vhost and KVM

(earlier).

Quoted 8 lines
Quoted 7 lines

To me the important advantage of PML is that it uses limited resources on the host which

1) doesn't require resources in the device
2) doesn't scale as the guest memory increases. (but this advantage doesn't exist in neither this nor bitmap)

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

it seems 2 exactly exists here.

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

The device resource reservation starts only when the device migration starts. i.e. with WRITE_RECORDS_START command of patch 7 in the series.

And now your precious VM can't migrate at all because -ENOSPC.

I am not aware of any Linux IOCTL that ensures a guaranteed execution without an error code. :)

As we talked in other email, a VF can be provisioned too as extension and capability can be exposed. This is not going the only error on device migration.

Quoted 137 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be

addressed.

Quoted 124 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

PML is yet another option that requires small pci writes. In the current proposal, there are no small PCI writes. It is a query interface from the device.

Quoted 6 lines
Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

May not have to be.
Speed of page recording should be fast enough.
It usually improves with subsequent generation.

Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which

looks very hard.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

Quoted 8 lines
Quoted 1 line

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Device page tracking table to be consulted to flush on mapping change.

Quoted 70 lines
Quoted 11 lines

I don't know how it's done. Do tell.

Quoted 8 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 9 lines
Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 2 lines

And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Quoted 2 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 7 lines
Quoted 1 line

The query interface in this proposal works on the granular boundary to read

and clear.

Quoted 1 line

This will ensure that mapping is consistent.

By itself it does not, you have to actually keep querying until you flush all dirty info and do it each time there's an invalidation in the IOMMU.

Only during device migration time.
It only applied on those specific cases when unmapping and migration both in progress at same time. But yes, it can slow down unmapping.

Quoted 221 lines
Quoted 36 lines
Quoted 24 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 2 lines

There cannot be logging before the DMA.
Only requirement is before the mapping changes, the dirty page tracking to be

synced.

Quoted 2 lines

In most common cases where the perf is critical, such mapping wont change

so often dynamically anyway.

Quoted 7 lines
Quoted 7 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as

well.

Quoted 5 lines
Quoted 1 line

It is incorrect attribution that platform is duplicated here. It feeds the data to the platform as needed without replicating.

I do agree that there is overlap of IOMMU tracking the dirty and storing it in

the per PTE vs device supplying its dirty track via its own interface.

Quoted 12 lines

Both are consolidated at hypervisor level.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and

others.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space

feature in core virtio layer:

Quoted 3 lines
Quoted 1 line

PASID to vq assignment won't be duplicated.
It is configured fully by the guest without consulting hypervisor at the device

level.

Quoted 1 line

Guest IOMMU would consult hypervisor to setup any PASID mapping as part

of any mapping method.

Quoted 13 lines
Quoted 13 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

True, but my question is where or not it can be done at the device level

easily.

Quoted 2 lines

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to

do it.

Quoted 37 lines
Quoted 37 lines

And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 18 lines

So I think this dirty tracking is a good option to have.

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 8 lines
Quoted 3 lines

If not, we are better off to offer this, and when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 4 lines
Quoted 4 lines

4) if the platform support is missing, we can
use software or leverage transport for
assistance like PRI

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can

converge.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Quoted 7 lines
Quoted 2 lines

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is

enabled.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

There is no requirement for mandating PRI either. So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in

Quoted 6 lines

virtio.

Quoted 2 lines
Quoted 1 line

Not duplicated. Feeding into the platform.

I mean IOMMU still sets the dirty bit, too. How is that not a duplication?

Only if the IOMMU is enabled for it.
For example AMD has DTE HAD bit to enable dirty page tracking in IOMMU.

So if platform does not enable, it can be enabled on the device and vis-versa.

Quoted 33 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 10 lines
Quoted 7 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Not true platforms have things like NICs on board and have for many years. It's about performance really.

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has

good performance overall.

Quoted 4 lines
Quoted 3 lines

Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

In the current proposal, it does not required PCI transactions, as there is only a

hypervisor-initiated query interface.

Quoted 3 lines

It is a trade off of using svq + pasid vs using something from the device.

Again, both has different use case and value. One uses cpu and one uses

device.

Quoted 1 line

Depending how much power one wants to spend where..

Quoted 9 lines

Also how much effort we want to spend on this virtio specific thing. The needs to be a *reason* to do things in virtio as opposed to using platform capabilities, this is exactly the same thing I told Lingshan wrt using SUSPEND for power management as opposed to using PCI PM - relying on platform when we can is right there in the mission statement.
For some reason I asssumed you guys have done a PoC and that's the motivation but if it's a "just in case" feature then I'd suggest we focus on merging patches 1-5 first.

It is not just in case feature.
We learnt that not all cpus have it.

There is ongoing efforts of the poc.
We will have the results in sometime.

We have similar interface on at least two devices already and integrated in the Linux stack, one in upstream, other is in progress. virtio is also in discussion here.

Sure, it is proposed as optional. We can focus on 1-5 first. I will split the series once I have comments.

There is also extension after 1-5 for net device context as well.

Quoted 92 lines
Quoted 50 lines
Quoted 50 lines
Quoted 4 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 2 lines

than you should propose that in the dirty page
tracking series that

you listed above to not do dirty page tracking. Rather depend on PRI, right?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about

why.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for

years.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 8 lines
Quoted 4 lines

I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#62 lore raw

Re: [virtio-comment] Re: [PATCH v3 6/8] admin: Add theory of operation for write recording commands

Quoted 11 lines · Parav Pandit
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Actually not, Parav said the device needs to reserve sufficient resources in another thread.

The device resource reservation starts only when the device migration starts. i.e. with WRITE_RECORDS_START command of patch 7 in the series.

And now your precious VM can't migrate at all because -ENOSPC.

I am not aware of any Linux IOCTL that ensures a guaranteed execution without an error code. :)

As we talked in other email, a VF can be provisioned too as extension and capability can be exposed. This is not going the only error on device migration.

Allocating resources on outgoing migration is a very bad idea. It is common to migrate prcisely because you are out of resources. Incoming is a different story, less of a problem.

Quoted 371 lines
Quoted 137 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The data structure is different but I don't see why it is critical.

I agree that I don't see out of buffers notifications too which implies device has to maintain something like a bitmap internally. Which I guess could be fine but it is not clear to me how large that bitmap has to be. How does the device know? Needs to be

addressed.

Quoted 124 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

This is the question I asked Parav in another thread. Using host memory as a queue with notification (like PML) might be much better.

Well if queue is what you want to do you can just do it internally.

Then it's not the proposal here, Parav has explained it in another reply, and as explained it lacks a lot of other facilities.

PML is yet another option that requires small pci writes. In the current proposal, there are no small PCI writes. It is a query interface from the device.

Quoted 6 lines
Quoted 2 lines

Problem of course is that it might overflow and cause things like packet drops.

Exactly like PML. So sticking to wire speed should not be a general goal in the context of migration. It can be done if the speed of the migration interface is faster than the virtio device that needs to be migrated.

May not have to be.
Speed of page recording should be fast enough.
It usually improves with subsequent generation.

Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Even if we manage to do that, it doesn't mean we won't have issues.

1) For many reasons it can neither see nor log via GPA, so this requires a traversal of the vIOMMU mapping tables by the hypervisor afterwards, it would be expensive and need synchronization with the guest modification of the IO page table which

looks very hard.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

vIOMMU is fast enough to be used on data path but not fast enough for dirty tracking?

We set up SPTEs or using nesting offloading where the PTEs could be iterated by hardware directly which is fast.

Quoted 8 lines
Quoted 1 line

There's a way to have hardware find dirty PTEs for you quickly?

Scanning PTEs on the host is faster and more secure than scanning guests, that's what I want to say:

1) the guest page could be swapped out but not the host one.
2) no guest triggerable behavior

Device page tracking table to be consulted to flush on mapping change.

Quoted 70 lines
Quoted 11 lines

I don't know how it's done. Do tell.

Quoted 8 lines

This is not the case here where software needs to iterate the IO page tables in the guest which could be slow.

Quoted 2 lines

Hard to believe. If true and you want to speed up vIOMMU then you implement an efficient datastructure for that.

Besides the issue of performance, it's also racy, assuming we are logging

IOVA.

Quoted 9 lines
Quoted 7 lines

0) device log IOVA
1) hypervisor fetches IOVA from log buffer
2) guest map IOVA to a new GPA
3) hypervisor traverse guest table to get IOVA to new GPA

Then we lost the old GPA.

Interesting and a good point.

Note that PML logs at GPA as it works at L1 of EPT.

Quoted 2 lines

And by the way e.g. vhost has the same issue. You need to flush dirty tracking info when changing the mappings somehow.

It's not,

1) memory translation is done by vhost
2) vhost knows GPA and it doesn't log via IOVA.

See this for example, and DPDK has similar fixes.

commit cc5e710759470bc7f3c61d11fd54586f15fdbdf4

Author: Jason Wang <[email protected]>
Date:   Wed Jan 16 16:54:42 2019 +0800

vhost: log dirty page correctly

Vhost dirty page logging API is designed to sync through GPA. But we try to log GIOVA when device IOTLB is enabled. This is wrong and may lead to missing data after migration.

To solve this issue, when logging with device IOTLB enabled, we will:

1) reuse the device IOTLB translation result of GIOVA->HVA mapping to get HVA, for writable descriptor, get HVA through iovec. For used ring update, translate its GIOVA to HVA
2) traverse the GPA->HVA mapping to get the possible GPA and log through GPA. Pay attention this reverse mapping is not guaranteed to be unique, so we should log each possible GPA in this case.

This fix the failure of scp to guest during migration. In -next, we will probably support passing GIOVA->GPA instead of GIOVA->HVA.

Fixes 6b1e6cc7855b ("vhost: new device IOTLB API")
Cc Jintack Lim <[email protected]>

All of the above is not what virtio did right now.

Quoted 2 lines

Parav what's the plan for this? Should be addressed in the spec too.

AFAIK, there's no easy/efficient way to do that. I hope I was wrong.

Quoted 7 lines
Quoted 1 line

The query interface in this proposal works on the granular boundary to read

and clear.

Quoted 1 line

This will ensure that mapping is consistent.

By itself it does not, you have to actually keep querying until you flush all dirty info and do it each time there's an invalidation in the IOMMU.

Only during device migration time.
It only applied on those specific cases when unmapping and migration both in progress at same time. But yes, it can slow down unmapping.

Quoted 223 lines
Quoted 36 lines
Quoted 24 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines

2) There are a lot of special or reserved IOVA ranges (for example the interrupt areas in x86) that need special care which is architectural and where it is beyond the scope or knowledge of the virtio device but the platform IOMMU. Things would be more complicated when SVA is enabled.

SVA being what here?

Quoted 7 lines
Quoted 3 lines

For example, IOMMU may treat interrupt ranges differently depending on whether SVA is enabled or not. It's very hard and unnecessary to teach devices about this.

Oh, shared virtual memory. So what you are saying here? virtio does not care, it just uses some addresses and if you want it to it can record writes somewhere.

One example, PCI allows devices to send translated requests, how can a hypervisor know it's a PA or IOVA in this case? We probably need a new bit. But it's not the only thing we need to deal with.

Quoted 12 lines

By definition, interrupt ranges and other reserved ranges should not belong to dirty pages. And the logging should be done before the DMA where there's no way for the device to know whether or not an IOVA is valid or not. It would be more safe to just not report them from the source instead of leaving it to the hypervisor to deal with but this seems impossible at the device level. Otherwise the hypervisor driver needs to communicate with the (v)IOMMU to be reached with the interrupt(MSI) area, RMRR area etc in order to do the correct things or it might have security implications. And those areas don't make sense at L1 when vSVA is enabled. What's more, when vIOMMU could be fully offloaded, there's no easy way to fetch that information.

Quoted 2 lines

There cannot be logging before the DMA.
Only requirement is before the mapping changes, the dirty page tracking to be

synced.

Quoted 2 lines

In most common cases where the perf is critical, such mapping wont change

so often dynamically anyway.

Quoted 7 lines
Quoted 7 lines

Again, it's hard to bypass or even duplicate the functionality of the platform or we need to step into every single detail of a specific transport, architecture or IOMMU to figure out whether or not logging at virtio is correct which is awkward and unrealistic. This proposal suffers from an exact similar issue when inventing things like freeze/stop where I've pointed out other branches of issues as

well.

Quoted 5 lines
Quoted 1 line

It is incorrect attribution that platform is duplicated here. It feeds the data to the platform as needed without replicating.

I do agree that there is overlap of IOMMU tracking the dirty and storing it in

the per PTE vs device supplying its dirty track via its own interface.

Quoted 12 lines

Both are consolidated at hypervisor level.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

And there could be other architecte specific knowledge (e.g PAGE_SIZE) that might be needed. There's no easy way to deal with those cases.

Good point about page size actually - using 4k unconditionally is a waste of resources.

Actually, they are more than just PAGE_SIZE, for example, PASID and

others.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

what does pasid have to do with it? anyway, just give driver control over page size.

For example, two virtqueues have two PASIDs assigned. How can a hypervisor know which specific IOVA belongs to which IOVA? For platform IOMMU, they are handy as it talks to the transport. But I don't think we need to duplicate every transport specific address space

feature in core virtio layer:

Quoted 3 lines
Quoted 1 line

PASID to vq assignment won't be duplicated.
It is configured fully by the guest without consulting hypervisor at the device

level.

Quoted 1 line

Guest IOMMU would consult hypervisor to setup any PASID mapping as part

of any mapping method.

Quoted 13 lines
Quoted 13 lines

1) translated/untranslated request
2) request w/ and w/o PASID

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

We wouldn't need to care about all of them if it is done at platform IOMMU level.

If someone logs at IOMMU level then nothing needs to be done in the spec at all. This is about capability at the device level.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

True, but my question is where or not it can be done at the device level

easily.

Quoted 2 lines

there's no "easily" about live migration ever.

I think I've stated sufficient issues to demonstrate how hard virtio wants to

do it.

Quoted 37 lines
Quoted 37 lines

And I've given the link that it is possible to do that in IOMMU without those issues. So in this context doing it in virtio is much harder.

Quoted 1 line

For example on-device iommus are a thing.

I'm not sure that's the way to go considering the platform IOMMU evolves very quickly.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 7 lines

what Lingshan
proposed is analogous to bit per page - problem unfortunately is you can't easily set a bit by DMA.

I'm not saying bit/bytemap is the best, but it has been used by real hardware. And we have many other options.

Quoted 18 lines

So I think this dirty tracking is a good option to have.

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
Quoted 2 lines

i.e. in first year of 2024?

Why does it matter in 2024?

Because users needs to use it now.

Quoted 8 lines
Quoted 3 lines

If not, we are better off to offer this, and
when/if platform support is, sure,

this feature can be disabled/not used/not enabled.

Quoted 4 lines

4) if the platform support is missing, we can
use software or leverage transport for
assistance like PRI

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

All of these are in theory.
Our experiment shows PRI performance is 21x slower
than page fault rate

done by the cpu.

Quoted 1 line

It simply does not even pass a simple 10Gbps test.

If you stick to the wire speed during migration, it can

converge.

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line

Do you have perf data for this?

No, but it's not hard to imagine the worst case. Wrote a small program that dirty every page by a NIC.

Quoted 1 line

In the internal tests we don’t see this happening.

downtime = dirty_rates * PAGE_SIZE / migration_speed

So if we get very high dirty rates (e.g by a high speed NIC), we can't satisfy the requirement of the downtime. Or if you see the converge, you might get help from the auto converge support by the hypervisors like KVM where it tries to throttle the VCPU then you can't reach the wire speed.

Quoted 7 lines
Quoted 2 lines

Will only work for some device types.

Yes, that's the point. Parav said he doesn't see the issue, it's probably because he is testing a virtio-net and so the vCPU is automatically throttled. It doesn't mean it can work for other virito devices.

Only for TX, and I'm pretty sure they had the foresight to test RX not just TX but let's confirm. Parav did you test both directions?

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

RX speed somehow depends on the speed of refill, so throttling helps more or less.

It doesn't depend on speed of refill you just underrun and drop packets. then your nice 10usec latency becomes more like 10sec.

I miss your point here. If the driver can't achieve wire speed without dirty page tracking, it can neither when dirty page tracking is

enabled.

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

There is no requirement for mandating PRI either.
So it is unusable.

It's not about mandating, it's about doing things in the correct layer. If PRI is slow, PCI can evolve for sure.

You should try.

Not my duty, I just want to make sure things are done in the correct layer, and once it needs to be done in the virtio, there's nothing obviously wrong.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yea but just vague questions don't help to make sure eiter way.

I don't think it's vague, I have explained, if something in the virito slows down the PRI, we can try to fix them.

I don't believe you are going to make PRI fast. No one managed so far.

So it's the fault of PRI not virito, but it doesn't mean we need to do it in virtio.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

I keep saying with this approach we would just say "e1000 emulation is slow and encumbered this is the fault of e1000" and never get virtio at all. Assigning blame only gets you so far.

I think we are discussing different things. My point is virtio needs to leverage the functionality provided by transport or platform (especially considering they evolve faster than virtio). It seems to me it's hard even to duplicate some basic function of platform IOMMU in

virtio.

Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Not duplicated. Feeding into the platform.

I mean IOMMU still sets the dirty bit, too. How is that not a duplication?

Only if the IOMMU is enabled for it.
For example AMD has DTE HAD bit to enable dirty page tracking in IOMMU.

So if platform does not enable, it can be enabled on the device and vis-versa.

So again, if your motivation is on-device IOMMU then say so, and in this case I don't see the point of only adding write tracking without adding the actual device IOMMU interface.
And maybe that is the answer to resource management questions:
there's going to be an IOMMU data structure on the device and it's just an extra bit in the PTE there.
Makes sense but let's see it all together then.
Because separate from on-device IOMMU it looks crazily expensive and just weird.

Quoted 42 lines
Quoted 33 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 10 lines
Quoted 7 lines

Missing functions in
platform or transport is not a good excuse to try to workaround it in the virtio. It's a layer violation and we never had any feature like this in the past.

Yes missing functionality in the platform is exactly why virtio was born in the first place.

Well the platform can't do device specific logic. But that's not the case of dirty page tracking which is device logic agnostic.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Not true platforms have things like NICs on board and have for many years. It's about performance really.

I've stated sufficient issues above. And one more obvious issue for device initiated page logging is that it needs a lot of extra or unnecessary PCI transactions which will throttle the performance of the whole system (and lead to other issues like QOS). So I can't believe it has

good performance overall.

Quoted 4 lines
Quoted 3 lines

Logging via IOMMU or using shadow virtqueue doesn't need any extra PCI transactions at least.

In the current proposal, it does not required PCI transactions, as there is only a

hypervisor-initiated query interface.

Quoted 3 lines

It is a trade off of using svq + pasid vs using something from the device.

Again, both has different use case and value. One uses cpu and one uses

device.

Quoted 1 line

Depending how much power one wants to spend where..

Quoted 9 lines

Also how much effort we want to spend on this virtio specific thing. The needs to be a *reason* to do things in virtio as opposed to using platform capabilities, this is exactly the same thing I told Lingshan wrt using SUSPEND for power management as opposed to using PCI PM - relying on platform when we can is right there in the mission statement.
For some reason I asssumed you guys have done a PoC and that's the motivation but if it's a "just in case" feature then I'd suggest we focus on merging patches 1-5 first.

Quoted 2 lines

It is not just in case feature.
We learnt that not all cpus have it.

Have dirty tracking? Well shadow is portable.

Quoted 4 lines

There is ongoing efforts of the poc.
We will have the results in sometime.

We have similar interface on at least two devices already and integrated in the Linux stack, one in upstream, other is in progress.

Aha. I hear IOMMUFD is working on integrating access to dirty bit. Maybe compare performance to that?
It does not have to be exactly virtio I think for PoC.

Quoted 101 lines

virtio is also in discussion here.

Sure, it is proposed as optional. We can focus on 1-5 first. I will split the series once I have comments.

There is also extension after 1-5 for net device context as well.

Quoted 92 lines
Quoted 50 lines
Quoted 50 lines
Quoted 4 lines

So I'd like Parav to publish some
experiment results and/or some estimates.

That's fine, but the above equation (used by Qemu) is sufficient to demonstrate how hard to stick wire speed in the case.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

In the current state, it is mandating.
And if you think PRI is the only way,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I don't, it's just an example where virtio can leverage from either transport or platform. Or if it's the fault in virtio that slows down the PRI, then it is something we can do.

Quoted 2 lines

than you should propose that in the dirty page
tracking series that

you listed above to not do dirty page tracking. Rather depend on PRI, right?

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Quoted 3 lines

No, the point is to not duplicate works especially considering virtio can't do better than platform or transport.

If someone says they tried and platform's migration support does not work for them and they want to build a solution in virtio then what exactly is the objection?

The discussion is to make sure whether virtio can do this easily and correctly, then we can have a conclusion. I've stated some issues above, and I've asked other questions related to them which are still not answered.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

I think we had a very hard time in bypassing IOMMU in the past that we don't want to repeat.

We've gone through several methods of logging dirty pages in the past (each with pros/cons), but this proposal never explains why it chooses one of them but not others. Spec needs to find the best path instead of just a possible path without any rationale about

why.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Adding more rationale isn't a bad thing.
In particular if platform supplies dirty tracking then how does driver decide which to use platform or device capability? A bit of discussion around this is a good idea.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

virtio is here in the
first place because emulating devices didn't work well.

I don't understand here. We have supported emulated devices for

years.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 8 lines
Quoted 4 lines

I'm pretty sure a lot of issues could be uncovered if this proposal can be prototyped with an emulated device first.

Thanks

virtio was originally PV as opposed to emulation. That there's now hardware virtio and you call software implementation "an emulation" is very meta.

Yes but I don't see how it relates to dirty page tracking. When we find a way it should work for both software and hardware devices.

Thanks

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

It has to work well on a variety of existing platforms. If it does then sure, why would we roll our own.

If virtio can do that in an efficient way without any issues, I agree. But it seems not.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/