device parts handling administration commands
v5 cover device parts handling administration commands
In some systems, there is a need to capture the state of all or part of a device and subsequently restore either the same device or a different one to this captured state. A group owner device can support administration commands to facilitate these get and set operations for the group member devices.
For example, a hypervisor can use the administration commands to capture parts of the device state and save the result as part of a VM snapshot. Later, the hypervisor can retrieve the snapshot and use the administration commands to restore parts of a device to resume VM operation.
As another example, these commands can be used to facilitate VM migration by the
hypervisor: one (source) hypervisor can get parts of a device and send the results to another (destination) hypervisor, which will in turn set (restore) parts of (another) device to resume the VM operation on the destination.
As always, these commands can also be used for debugging the device.
The device comprises many device parts which the driver can get and set. Administration commands are provided to either get and set all the device parts at once, or to get the device parts metadata that indicates which device parts are present, and later to get and set specific device parts.
To get and set the device parts or their metadata, the driver first creates a device parts resource object, indicating whether the object should handle get or set operations but not both simultaneously.
This series introduces administration commands for handling the device parts.
It takes inspiration from the similar idea presented at KVM Forum at [1].
Presently, administration commands are limited to the owner device handling device parts of the member device. However, in the future, this may be extended to a wider range of devices. For example, a member device may be able to handle parts of itself, or an administration device may handle parts of other devices.
Currently, this series does not define specific parts for individual device types. Such device parts will be added subsequent to this work.
This series depends on and uses the capabilities and resources infrastructure introduced several days ago in series [2].
[1] https://static.sched.com/hosted_files/kvmforum2022/3a/KVM22-Migratable-Vhost-vDPA.pdf
[2] https://lore.kernel.org/virtio-comment/[email protected]/T/#t
patch summary:
==============
patch-1 adds device parts theory of operation and overall flow patch-2 extends resource objects command for sriov group type patch-3 defines administration commands related to device parts patch-4 defines common device parts
patch-5 lists requirements for device parts administration commands
changelog:
==========v4->v5:
- removed write recording from the device to future work
- removed device state (aka device context) tracking to future work
- simplified freeze and stopped mode, to a single mode stopped
- reworded from 'device context' to 'device parts'
- simplified the theory of operation description and command descriptions
v3->v4:
- updated commit message to be more precise for mapping
a PCI VF member device to guest virtual machine
- skipped adding msix table and pba table as they are mainly programmed by hypervisor which has the knowledge of vcpus and contains system specific encoded information which may be different on destination hypervisor
- moved vq enabled field to vq config struct so that it can be used for packed vq and split vq uniformly.
- reduce length in the field tlv from 64-bit to 32-bit to be realistic
- added missing num_entries in the feature bits tlv
- fixed copy paste error on structure for VIRTIO_DEV_CTX_VQ_SPLIT_DEV_OWN_DESC field
- corrected discard field type location and enum values
- enhanced queue context
- made device context read command more flexible for device implementations
- improved read command to be finish quickly during freeze mode to have lowest possible downtime
- improved description for query supported fields command and device context to report the length in it
- moved read context command dependency on output field
- added remaining context size bytes in read response so that after the device mode is changed to freeze, the driver can get most accurate information quickly to decide when to stop reading the context from the device
- relaxed device context formation and processing to simplify the device implementationv2->v3:
- updated cover letter to reflect the use for passthrough and for only data path acceleration
- updated cover letter to utilize same infra for nested
- Addressed comments from Michael
- updated read context command to not depend on returned data length for closing the read context stream, instead depend on explicit read response with zero length
- fixed copy paste errors in write context command for fields description
- added device and driver normatives for {_START,_END}_MARKER fields
- wrote member VF device instead of VF devicev1->v2:
- Addressed comments from Michael and Jason
- replaced iova to page/physical address range in write recording commands
- several device specific requirements added to clarify, interaction of device reset, FLR, PCI PM and admin commands
- added device context fields query command to learn compatibility
- split device context field type range into generic and device specific
- added device context extension section to maintain backward and future compatibility
- several rewording in theory of operation
- added requirements to cover config space read/write interaction with device context commands
- added assumption about pci config space and msix table not present in device context, which can be added when hypervisor need themv0->v1:
- enrich device context to cover device configuration layout, feature bits
- fixed alignment of device context fields
- added missing Sign-off for the joint work done with Satananda
- added link to the github issue
Parav Pandit (5):
admin: Add theory of operation for device parts
admin: Extend resource objects for sr-iov group type
admin: Add admin commands for device parts
admin: Define common device parts
admin: Add requirements of device parts commands
admin-cmds-capabilities.tex | 16 ++ admin-cmds-device-parts.tex | 469 ++++++++++++++++++++++++++++++++ admin-cmds-resource-objects.tex | 29 +- admin.tex | 12 +- conformance.tex | 2 + content.tex | 1 + device-parts.tex | 231 ++++++++++++++++ newdevice.tex | 24 ++ 8 files changed, 778 insertions(+), 6 deletions(-) create mode 100644 admin-cmds-device-parts.tex create mode 100644 device-parts.tex
Signature · 1 line
2.34.1
PATCH v5 1/5 admin: Add theory of operation for device parts
A device can get and set the state of the device which is organized as multiple device parts. This can be useful in cases of VM snapshot, VM migration, debug or some other use cases.
Add the theory of operations on how to get/set device parts and how it affects when the device is stopped or resumed.
---
changelog:v4->v5:
- replaced 'device context' with 'device parts'
- updated commit log to be generic device parts related
--- admin-cmds-capabilities.tex | 16 +++++++++++ admin-cmds-device-parts.tex | 55 +++++++++++++++++++++++++++++++++++++ admin.tex | 1 + 3 files changed, 72 insertions(+) create mode 100644 admin-cmds-device-parts.tex
admin-cmds-capabilities.tex +16 -0
@@ -51,6 +51,22 @@ \subsubsection{Device and driver capabilities}\label{sec:Basic Facilities of a V \end{tabularx} \end{table} +Common capabilities are listed: + +\begin{table}[H] +\caption{Common capability ids} +\label{table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities / Common capability ids} +\begin{tabularx}{\textwidth}{ |l|l|X| } +\hline +Id & Name & Description \\ +\hline \hline +0x0000 & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / VIRTIO_DEV_PARTS_CAP]{VIRTIO_DEV_PARTS_CAP} & Device parts capability \\ +\hline +0x0001-0x07ff & - & Generic capability for all device types \\ +\hline +\end{tabularx} +\end{table} + Device type specific capabilities are described separately for each device type under \field{Device and driver capabilities}.
admin-cmds-device-parts.tex new file +55 -0
55 lines not on this page · raw message
admin.tex +1 -0
@@ -328,6 +328,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \input{admin-cmds-legacy-interface.tex} \input{admin-cmds-capabilities.tex} \input{admin-cmds-resource-objects.tex} +\input{admin-cmds-device-parts.tex} \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
Signature · 1 line
2.34.1
PATCH v5 2/5 admin: Extend resource objects for sr-iov group type
In reply to Parav Panditoccasionally the group owner device accesses the device parts of the member device. Extend the usage of resource objects for member device parts access.
--- admin-cmds-resource-objects.tex | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
admin-cmds-resource-objects.tex +11 -5
@@ -69,7 +69,9 @@ \subsubsection{Device resource objects}\label{sec:Basic Facilities of a Virtio D all resource objects created by the driver are destroyed within the device. Following administration commands control device resource objects, -they are only supported for the self group type: +they are supported for the self group type, occasionally some resource +objects can be created for the SR-IOV group type as well. Such sr-iov group +type specific resource objects are listed where such objects is defined. \begin{enumerate} \item VIRTIO_ADMIN_CMD_RESOURCE_OBJ_CREATE
@@ -102,7 +104,8 @@ \subsubsection{Device resource objects}\label{sec:Basic Facilities of a Virtio D for the resource for the specified \field{type}. For the command VIRTIO_ADMIN_CMD_RESOURCE_OBJ_CREATE, \field{opcode} is set to 0xa. -\field{group_member_id} is set to zero. +\field{group_member_id} is set to zero for self-group type and set to +the member device to be accessed for the SR-IOV group type. The \field{command_specific_data} is in the format \field{struct virtio_admin_cmd_resource_obj_create_data}. \field{resource_obj_specific_data} refers to the resource object specific data.
@@ -132,7 +135,8 @@ \subsubsection{Device resource objects}\label{sec:Basic Facilities of a Virtio D For the command VIRTIO_ADMIN_CMD_RESOURCE_OBJ_MODIFY, \field{opcode} is set to 0xb. The \field{command_specific_data} is in the format \field{struct virtio_admin_cmd_resource_modify_data}. -\field{group_member_id} is set to zero. +\field{group_member_id} is set to zero for self-group type and set to +the member device to be accessed for the SR-IOV group type. \field{id} identifies the resource object of type \field{type} whose attributes to modify. This command modifies the attributes supplied in \field{resource_obj_specific_data}.
@@ -154,7 +158,8 @@ \subsubsection{Device resource objects}\label{sec:Basic Facilities of a Virtio D This command queries attributes of the existing resource object. For the command VIRTIO_ADMIN_CMD_RESOURCE_OBJ_QUERY, \field{opcode} is set to 0xc. -\field{group_member_id} is set to zero. +\field{group_member_id} is set to zero for self-group type and set to +the member device to be accessed for the SR-IOV group type. The \field{command_specific_data} is in the format \field{struct virtio_admin_cmd_resource_obj_query_data}. \field{id} identifies the existing resource object of type \field{type} whose
@@ -186,7 +191,8 @@ \subsubsection{Device resource objects}\label{sec:Basic Facilities of a Virtio D For the command VIRTIO_ADMIN_CMD_RESOURCE_OBJ_DESTROY, \field{opcode} is set to 0xd. The \field{command_specific_data} is in the format \field{struct virtio_admin_cmd_resource_obj_cmd_hdr}. -\field{group_member_id} is set to zero. +\field{group_member_id} is set to zero for self-group type and set to +the member device to be accessed for the SR-IOV group type. \field{id} identifies the existing resource object of type \field{type}. This command destroys the specified resource object of \field{type} identified
Signature · 1 line
2.34.1
PATCH v5 3/5 admin: Add admin commands for device parts
In reply to Parav PanditAdd administration commands to handle device parts such as get/set device parts, get metadata of the device parts,
---
changelog:v4->v5:
- using hyperreference to jump to command definition
- rephrase from 'device context' to 'device parts'
- change command headings to match the name from the table
- removed freeze mode
- define marker device parts adjacent to get/set commands
--- admin-cmds-device-parts.tex | 229 ++++++++++++++++++++++++++++++++ admin-cmds-resource-objects.tex | 13 ++ admin.tex | 11 +- 3 files changed, 252 insertions(+), 1 deletion(-)
admin-cmds-device-parts.tex +229 -0
@@ -53,3 +53,232 @@ \subsubsection{Device parts}\label{sec:Basic Facilities of a Virtio Device / Dev resource objects for retrieving the device parts. \field{set_parts_resource_objects_limit} indicates the supported device parts resource objects for restoring the device parts. + +\paragraph{VIRTIO_RESOURCE_OBJ_DEV_PARTS}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / VIRTIO_RESOURCE_OBJ_DEV_PARTS} + +A device parts resource object is used to either get or set the device parts. +Before performing any get or set operation for the device parts, the driver +creates the device parts resource object +VIRTIO_RESOURCE_OBJ_DEV_PARTS using the administration command +\hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects / VIRTIO_ADMIN_CMD_RESOURCE_OBJ_CREATE]{VIRTIO_ADMIN_CMD_RESOURCE_OBJ_CREATE}. +The driver indicates the intended purpose (get or set) at the time of creating the +device parts resource object. +For the device parts resource object, both \field{resource_obj_specific_data} and +\field{resource_obj_specific_result} are in the format +\field{struct virtio_resource_obj_dev_parts}. + +\begin{lstlisting} +struct virtio_resource_obj_dev_parts { + u8 type; +#define VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_GET 0 +#define VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_SET 1 + u8 reserved[7]; +}; +\end{lstlisting} + +When \field{type} is set to VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_GET, +the driver can use the object to capture the device parts and the metadata of +these device parts. When \field{type} is set to +VIRTIO_RESOURCE_OBJ_DEV_PARTS_TYPE_SET, the driver can use the +object to restore the device parts. + +\paragraph{Device parts handling commands}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands} + +The owner driver uses the following resource object handling administration +commands. These commands are only used for the device parts resource +object after the driver creates the VIRTIO_RESOURCE_OBJ_DEV_PARTS object. +These commands are currently only defined for the SR-IOV group type: + +\begin{enumerate} +\item VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET +\item VIRTIO_ADMIN_CMD_DEV_PARTS_GET +\item VIRTIO_ADMIN_CMD_DEV_PARTS_SET +\end{enumerate} + +\subparagraph{VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET} +\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET} + +This command obtains the metadata of the device parts. This metadata includes +the maximum size of the device parts, the count of device parts, and a list of +the device part headers. + +For the command VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET, \field{opcode} is set +to 0xe. The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_dev_parts_metadata_data}. +\field{group_member_id} refers to the member device to be accessed. +The resource object \field{type} in the \field{hdr} is set to +VIRTIO_RESOURCE_OBJ_DEV_PARTS and \field{id} is set to the ID of the +device parts resource object. + +\begin{lstlisting} +struct virtio_admin_cmd_dev_parts_metadata_data { + struct virtio_admin_cmd_resource_obj_cmd_hdr hdr; + u8 type; + u8 reserved[7]; +}; + +#define VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_SIZE 0 +#define VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_COUNT 1 +#define VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_LIST 2 + +struct virtio_admin_cmd_dev_parts_metadata_result { + union { + struct { + le32 size; + le32 reserved; + } parts_size; + struct { + le32 count; + le32 reserved; + } hdr_list_count; + { + le32 count; + le32 reserved; + struct virtio_dev_part_hdr hdrs[]; + } hdr_list; + }; +}; +\end{lstlisting} + +When the command completes successfully, the +\field{command_specific_result} is in the format +\field{struct virtio_admin_cmd_dev_parts_metadata_result}. + +When \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_SIZE, +the device responds with \field{parts_size}. \field{parts_size.size} indicates +the maximum size in bytes for all the device parts. + +When \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_COUNT, the +device responds with \field{hdr_list_count.count}. The +\field{hdr_list_count.count} indicates an count of +\field{struct virtio_dev_part_hdr} metadata entries that the device can +provide when the \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_HDR +in a subsequent VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET command. + +When \field{type} is set to VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_TYPE_HDR, +the device responds with \field{hdr_list}. \field{hdr_list} +indicates the device parts metadata. + +\field{reserved} is reserved and set to 0. + +The command responds with the \field{status} VIRTIO_ADMIN_STATUS_ENOMEM +when the size of \field{command_specific_result} is not sufficient enough +for the response. + +\subparagraph{VIRTIO_ADMIN_CMD_DEV_PARTS_GET} +\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_PARTS_GET} + +This command captures the device parts. For the command +VIRTIO_ADMIN_CMD_DEV_PARTS_GET, \field{opcode} is set to 0xf. +The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_dev_parts_get_data}. +\field{group_member_id} refers to the member device to be accessed. +The resource object \field{type} in the \field{hdr} is set to +VIRTIO_RESOURCE_OBJ_DEV_PARTS and \field{id} is set to the ID of the +device parts resource object. + +\begin{lstlisting} +struct virtio_admin_cmd_dev_parts_get_data { + struct virtio_admin_cmd_resource_obj_cmd_hdr hdr; + u8 type; + u8 reserved[7]; + struct virtio_dev_part_hdr hdr_list[]; +}; + +#define VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_SELECTED 0 +#define VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_ALL 1 + +struct virtio_admin_cmd_dev_parts_get_result { + struct virtio_dev_part parts[]; +}; + +\end{lstlisting} + +When the driver wants to capture specific device parts, \field{type} is set to +VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_SELECTED and \field{hdr_list} is set to the +device parts of interest. + +When the driver wants to retrieve all the device parts, \field{type} is set to +VIRTIO_ADMIN_CMD_DEV_PARTS_GET_TYPE_ALL, and \field{hdr_list} is empty. + +\field{reserved} is reserved and set to 0. + +When the command completes successfully, the \field{command_specific_result} is +in the format \field{struct virtio_admin_cmd_dev_parts_get_result}, containing +either the selected device parts or all the device parts. + +If the requested device part does not exist, the device skips the device part +without any error. + +\subparagraph{VIRTIO_ADMIN_CMD_DEV_PARTS_SET}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_PARTS_SET} + +This command sets one or multiple device parts. For the command +VIRTIO_ADMIN_CMD_DEV_PARTS_SET, \field{opcode} is set to 0x10. +The \field{group_member_id} refers to the member device to be accessed. +The resource object \field{type} in the \field{hdr} is set to +VIRTIO_RESOURCE_OBJ_DEV_PARTS and \field{id} is set to the ID of the +device parts resource object. + +\begin{lstlisting} +struct virtio_admin_cmd_dev_parts_set_data { + struct virtio_admin_cmd_resource_obj_cmd_hdr hdr; + struct virtio_dev_part parts[]; +}; +\end{lstlisting} + +The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_dev_parts_set_data}. + +This command has no command specific result. + +The driver stops the device before setting any device parts. + +When the command completes successfully, the device has updated device +parts to the value supplied in \field{virtio_admin_cmd_dev_parts_set_data}. + +The device parts set by this command take effect when the device is resumed +using the VIRTIO_ADMIN_CMD_DEV_MODE_SET command. + +When the command fails with a status other than VIRTIO_ADMIN_STATUS_OK, the +device does not have any side effects. + +\subparagraph{VIRTIO_ADMIN_CMD_DEV_MODE_SET}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_MODE_SET} + +This command either stops the device from initiating any transport requests or +resumes the device operation. For the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, +\field{opcode} is set to 0x11. \field{group_member_id} indicates the member +device to be accessed. + +The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_dev_mode_set_data}. + +\begin{lstlisting} +struct virtio_admin_cmd_dev_mode_set_data { + u8 flags; +}; + +#define VIRTIO_ADMIN_CMD_DEV_MODE_F_STOPPED 0 +\end{lstlisting} + +This command has no command specific result. + +When the command completes successfully and if the \field{flags} field is set +to VIRTIO_ADMIN_CMD_DEV_MODE_F_STOPPED (bit 0), the device is stopped. +When the device is stopped, the device stops initiating all transport +communications, which includes: + +\begin{enumerate} +\item stopping configuration change notifications +\item stopping all virtqueue notifications +\item stops accessing all virtqueues and the driver buffer memory +\end{enumerate} + +After the device is stopped, the device parts remain unchanged unless +the driver initiates any transport requests. + +When the device is stopped, it writes back any associated descriptors for all +observed buffers to prevent out-of-order processing if the device is resumed. + +When the command completes successfully and if the \field{flags} field +is set to zero, the device resumes its operation. If the command completes +with an error, it does not produce any side effects on the device.
admin-cmds-resource-objects.tex +13 -0
@@ -63,6 +63,19 @@ \subsubsection{Device resource objects}\label{sec:Basic Facilities of a Virtio D \hline \end{tabular} +Following generic resource objects are defined which are described separately. + +\begin{xltabular}{\textwidth}{ |X||X|X| } +\hline +Resource object type & Name & Description \\ +\hline \hline +0x000 & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / VIRTIO_RESOURCE_OBJ_DEV_PARTS]{VIRTIO_RESOURCE_OBJ_DEV_PARTS} & Device parts object \\ +\hline +0x001-0x1ff & - & Generic resource object range reserved \\ +\hline +\hline +\end{xltabular} + When the device resets, it starts with zero resources of each type; the driver can create resources up to the published \field{limit}. The driver can destroy and recreate the resource one or multiple times. Upon device reset,
admin.tex +10 -1
@@ -150,7 +150,16 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \hline 0x000d & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects / VIRTIO_ADMIN_CMD_RESOURCE_OBJ_DESTROY]{VIRTIO_ADMIN_CMD_RESOURCE_OBJ_DESTROY} & Destroy a device resource object \\ \hline -0x000e - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ +0x000e & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET]{VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET} & Get the metadata of the device parts \\ +\hline +0x000f & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_PARTS_GET]{VIRTIO_ADMIN_CMD_DEV_PARTS_GET} & Get the device parts \\ +\hline +0x0010 & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_PARTS_SET]{VIRTIO_ADMIN_CMD_DEV_PARTS_SET} & Set the device parts \\ +\hline +0x0011 & \hyperref[par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts handling commands / VIRTIO_ADMIN_CMD_DEV_MODE_SET]{VIRTIO_ADMIN_CMD_DEV_MODE_SET} & Stop or resume the device \\ +\hline +\hline +0x0013 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\ \hline 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\ \hline
Signature · 1 line
2.34.1
PATCH v5 4/5 admin: Define common device parts
In reply to Parav PanditDefine common device parts that represents the state of the device. The driver can get and set these device parts using administration commands.
---
changelog:v5->v6:
- renamed 'device context' to 'device parts'
- use actual part name for the subsection
- moved device part guidance to newdevice.tex file
- VQ_SPLIT changed to SPLIT_VQ
- dropped PCI prefix from the device parts
- moved common fields to its own subsection
- added hyper reference in the table
- split device status to its own TLV so multiple transport can enjoy it
--- admin-cmds-device-parts.tex | 37 ++++++ content.tex | 1 + device-parts.tex | 231 ++++++++++++++++++++++++++++++++++++ newdevice.tex | 24 ++++ 4 files changed, 293 insertions(+) create mode 100644 device-parts.tex
admin-cmds-device-parts.tex +37 -0
@@ -282,3 +282,40 @@ \subsubsection{Device parts}\label{sec:Basic Facilities of a Virtio Device / Dev When the command completes successfully and if the \field{flags} field is set to zero, the device resumes its operation. If the command completes with an error, it does not produce any side effects on the device. + +\paragraph{Device parts order}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order} + +Device parts are usually captured and restored using get and set administration +commands respectively; when multiple device parts are captured or restored, +they are arranged in the specific order listed: + +Some of the device parts do not need to be written to the device when restored, such +device parts are listed as \field{O}. When a such an optional device part is +exchanged using \field{struct virtio_dev_part}, it is marked as optional by +setting VIRTIO_DEV_PART_F_OPTIONAL(bit 0) in the \field{flags}. + +\begin{table}[H] +\caption{Device parts order} +\label{table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order/ Device parts order} +\begin{xltabular}{\textwidth}{ |l|l|X| } +\hline +Part name & Optional & Mandatory preceding parts \\ +\hline \hline +\hline +VIRTIO_DEV_PART_DEV_FEATURES & O & Nil \\ +\hline +VIRTIO_DEV_PART_DRV_FEATURES & - & Nil \\ +\hline +VIRTIO_DEV_PART_PCI_COMMON_CFG & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES \\ +\hline +VIRTIO_DEV_PART_DEVICE_STATUS & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG \\ +\hline +VIRTIO_DEV_PART_VQ_CFG & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG, + VIRTIO_DEV_PART_DEVICE_STATUS \\ +\hline +VIRTIO_DEV_PART_VQ_NOTIFY_CFG & - & VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_DRV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG, + VIRTIO_DEV_PART_DEVICE_STATUS, VIRTIO_DEV_PART_VQ_CFG \\ +\hline +\hline +\end{xltabular} +\end{table}
content.tex +1 -0
@@ -503,6 +503,7 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. \input{admin.tex} +\input{device-parts.tex} \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
device-parts.tex new file +231 -0
231 lines not on this page · raw message
newdevice.tex +24 -0
@@ -85,3 +85,27 @@ \section{Device Improvements}\label{sec:Creating New Device Types or Extending t can use a single bit, but if one feature makes sense without the others they should not be gratuitously grouped together to conserve feature bits. + +\section{How to define a new device part?}\label{sec:Creating New Device Types or Extending the Device / How to define a new device part?} +Few considerations are necessary when creating new device part or +when extending the device part structure. +If the new field is generic for all the device types or most of the device types, +a new device part should be defined as common device part by claiming a new \field{type} value. +If the new field is unique to a device type, a new device specific device part should +be added. +range. + +\section{When to define a new device part?}\label{sec:Creating New Device Types or Extending the Device / When to define a new device part?} +When the device part for a specific field does not exists, one should +define a new device part. + +\section{How to avoid device part duplication with existing structure?}\label{sec:Creating New Device Types or Extending the Device / How to avoid device part duplication with existing structure?} +Each device should reuse any existing field definition that may exists as part +of device control virtqueue or any other existing structure definition. + +\section{How to extend the existing device part definition?}\label{sec:Creating New Device Types or Extending the Device / How to extend the existing device part definition?} + +When a field is missing in already defined device part, a new field should be added at +the end of the existing device part. New field MUST not be added at beginning or in +the middle of the device part structure. Any field which is already present MUST NOT +be removed.
Signature · 1 line
2.34.1
PATCH v5 5/5 admin: Add requirements of device parts commands
In reply to Parav PanditAdd device and driver side requirements for the device parts related commands.
--- admin-cmds-device-parts.tex | 148 ++++++++++++++++++++++++++++++++++++ conformance.tex | 2 + 2 files changed, 150 insertions(+)
admin-cmds-device-parts.tex +148 -0
@@ -319,3 +319,151 @@ \subsubsection{Device parts}\label{sec:Basic Facilities of a Virtio Device / Dev \hline \end{xltabular} \end{table} + +\devicenormative{\subparagraph}{Device parts}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts} + +A device MUST either support all of, or none of +VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET, +VIRTIO_ADMIN_CMD_DEV_PARTS_GET, VIRTIO_ADMIN_CMD_DEV_PARTS_SET, +VIRTIO_ADMIN_CMD_RESOURCE_OBJ_CREATE, +VIRTIO_ADMIN_CMD_RESOURCE_OBJ_DESTROY, VIRTIO_ADMIN_CMD_RESOURCE_OBJ_MODIFY +VIRTIO_ADMIN_CMD_RESOURCE_OBJ_QUERY, and +VIRTIO_ADMIN_CMD_DEV_MODE_SET commands, where resource commands apply to +the resource object VIRTIO_RESOURCE_OBJ_DEV_PARTS. + +The device MUST support getting the device parts multiple times +with the command VIRTIO_ADMIN_CMD_DEV_PARTS_GET. + +When there are multiple device parts in the command +VIRTIO_ADMIN_CMD_DEV_PARTS_GET, the device MUST respond the device parts in the +same order as listed in the table +\hyperref[table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order/ Device parts order]{device parts order}. + +The device SHOULD respond with an error status for the command +VIRTIO_ADMIN_CMD_DEV_PARTS_SET if the device is not stopped. + +The device MUST support the command VIRTIO_ADMIN_CMD_DEV_PARTS_SET, +allowing the same or different device parts to be set multiple times. + +The device MUST respond with an error for the command +VIRTIO_ADMIN_CMD_DEV_PARTS_SET, if there is a mismatch between the +device part length supplied in the VIRTIO_ADMIN_CMD_DEV_PARTS_SET +and the device part length in the device. + +The device MUST NOT set the device part VIRTIO_DEV_PART_DEV_FEATURES in +the command VIRTIO_ADMIN_CMD_DEV_PARTS_SET; instead, +it must verify that the device features supplied in +VIRTIO_DEV_PART_DEV_FEATURES match those the device has. + +The device may ignore the setting of a device part that has the +VIRTIO_DEV_PART_F_OPTIONAL bit set. + +For the SR-IOV group type, when the device is stopped using the command +VIRTIO_ADMIN_CMD_DEV_MODE_SET, +\begin{itemize} +\item the device MUST not initiate any PCI transaction, +\item the device MUST finish all the outstanding PCI transactions before completing + the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, +\item the device MUST write any associated descriptors to the driver memory for + all the observed buffers, +\item the device MUST accept driver notifications and the device MAY update any + device parts, +\item the device MUST respond with valid values for PCI read requests, +\item the device MUST operate in the same way for the PCI architected interfaces + regardless of the device mode. +\item the device MUST not generate any PCI PME. +\end{itemize} + +When the device is stopped, +\begin{itemize} +\item the device MUST not access any virtqueue memory or any memory referred + by the virtqueue. +\item the device MUST not generate any configuration change notification + or any virtqueue notification. +\end{itemize} + +For the SR-IOV group type, +\begin{itemize} +\item the device MUST respond to the commands +VIRTIO_ADMIN_CMD_DEV_MODE_SET, VIRTIO_ADMIN_CMD_DEV_PARTS_SET +after the member device completes FLR, if the FLR is in progress on the device +when the device receives any of these commands. + +\item the member device MUST respond to the commands +VIRTIO_ADMIN_CMD_DEV_MODE_SET and VIRTIO_ADMIN_CMD_DEV_PARTS_SET +after the device reset completes in the device, if the +device reset is in progress when the device receives any of these commands. + +\item the member device MUST respond to commands +VIRTIO_ADMIN_CMD_DEV_MODE_SET and VIRTIO_ADMIN_CMD_DEV_PARTS_SET +after the device power management state +transition completes on the device, if the power management state transition +is in progress when the device receives any of these commands. +\end{itemize} + +When the \field{flags} is set to VIRTIO_ADMIN_CMD_DEV_MODE_FLAGS_STOPPED +in the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, and if the device is already +stopped before, the device MUST complete the command successfully. + +When the VIRTIO_ADMIN_CMD_DEV_MODE_FLAGS_STOPPED \field{flags} clear, +in the command VIRTIO_ADMIN_CMD_DEV_MODE_SET, and if the device is +not stopped before, the device MUST complete the command successfully. + +For the SR-IOV group type, the device MUST clear all the device parts to +the default value when the member device is reset or undergo an PCI FLR. + +The device MAY NOT respond to the selected device part in \field{hdr_list} +in the command VIRTIO_ADMIN_CMD_DEV_PARTS_GET if the device part is invalid +in the device. + +For the commands VIRTIO_ADMIN_CMD_DEV_PARTS_GET and +VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET, when the device responds with: +\begin{itemize} +\item +VIRTIO_DEV_PART_DRV_FEATURES or VIRTIO_DEV_PART_PCI_COMMON_CFG, it MUST be +preceded by VIRTIO_DEV_PART_DEV_FEATURES. + +\item VIRTIO_DEV_PART_PCI_COMMON_CFG, it MUST be preceded by +VIRTIO_DEV_PART_DEV_FEATURES. + +\item VIRTIO_DEV_PART_PCI_COMMON_CFG, it MUST be preceded by +VIRTIO_DEV_PART_DEV_FEATURES and VIRTIO_DEV_PART_DRV_FEATURES. + +\item VIRTIO_DEV_PART_DEV_CFG, it MUST be preceded by VIRTIO_DEV_PART_DEV_FEATURES. + +\item VIRTIO_DEV_PART_DRV_CFG, it be preceded by VIRTIO_DEV_PART_DEV_FEATURES, +VIRTIO_DEV_PART_DRV_FEATURES and VIRTIO_DEV_PART_DEV_CFG. + +\item VIRTIO_DEV_PART_DEVICE_STAtUS, it is preceded by VIRTIO_DEV_PART_DEV_FEATURES, +VIRTIO_DEV_PART_DRV_FEATURES, and VIRTIO_DEV_PART_DEV_CFG. +\end{itemize} + +When the device receives a VIRTIO_ADMIN_CMD_DEV_PARTS_SET command containing the +parts VIRTIO_DEV_PART_DEV_FEATURES, VIRTIO_DEV_PART_PCI_COMMON_CFG and +VIRTIO_DEV_PART_DEV_CFG, the device SHOULD only verify that the provided configuration is +correct but SHOULD NOT apply it, especially for the fields that are designated +as read-only and invariant. This ensures that the device respects the +immutability of certain configuration aspects while still performing necessary +validation checks. + +\drivernormative{\subparagraph}{Device parts}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts} + +The driver MUST set the mode to VIRTIO_ADMIN_CMD_DEV_MODE_F_STOPPED in +the command VIRTIO_ADMIN_CMD_DEV_MODE_SET before setting parts using the command +VIRTIO_ADMIN_CMD_DEV_PARTS_SET. + +When there are multiple device parts in the command +VIRTIO_ADMIN_CMD_DEV_PARTS_SET, the driver MUST set the device parts in the same +order as listed in the table +\hyperref[table:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts / Device parts order/ Device parts order]{device parts order}. + +For the SR-IOV group type, the driver SHOULD NOT access the device configuration +space described in section +\ref{sec:Basic Facilities of a Virtio Device / Device Configuration Space} +when the device is stopped. + +The driver SHOULD allocate sufficient response buffer to receive all the device +parts metadata in the command VIRTIO_ADMIN_CMD_DEV_PARTS_METADATA_GET. + +The driver SHOULD allocate sufficient response buffer to receive all the device +parts in the command VIRTIO_ADMIN_CMD_DEV_PARTS_GET.
conformance.tex +2 -0
@@ -101,6 +101,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{drivernormative:Reserved Feature Bits} \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities} \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects} +\item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts} \end{itemize} \conformance{\subsection}{PCI Driver Conformance}\label{sec:Conformance / Driver Conformance / PCI Driver Conformance}
@@ -185,6 +186,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{devicenormative:Reserved Feature Bits} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects} +\item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts} \end{itemize} \conformance{\subsection}{PCI Device Conformance}\label{sec:Conformance / Device Conformance / PCI Device Conformance}
Signature · 1 line
2.34.1