Introduce device group and device management
v10 cover Introduce device group and device management
Change log:
since v9:
addressed comments by Parav, Max, Cornelia, David and Zhu Lingshan:
added link to errno header from Linux
rename _MEM to _MEMBER
admin vq num is zero based
clarify who sends commands where
minor english tweaks
clarify command length
specify interaction with sriov capability correct commit log - NumVFs can be 0
i could not decide what should happen when VFs are disabled. for now did not specify.
since v8:
addressed comments by Cornelia - as we agreed on list
since v7:
make high level error codes match linux, with virtio specific codes
in a separate field
renamed _ACCEPT to _USE since that's what it does clarified forward compatibility and non pci transports support multiple admin vqs
conformance statements
lots of changes all over the place to I changed author from Max to myself. Don't need to take credit but also don't want to blame Max for my mistakes.
since v6:
- removed some extentions intended for future use. We'll do them when we get there.
- brought back command list query from v5 in a simplified form - it's here to address the case where a single parent can address multiple groups, such as PF addressing transport vq and sriov vfs.
- attempt to make terminology more formal. In particular a term for whoever controls the group. I am still going back
and forth between "parent" and "owner" - owner might be better after all since it will work if we ever have a self group. For now it's parent.
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
For example, it might make sense to split creating/destroying SIOV devices from the transport passing data from the guest - the driver would then not negotiate VIRTIO_F_SR_IOV (which then means auto-provisioning).
This is out of RFC, since we have two commands which are useful to discover supported group types (ATM can be none or SR-IOV).
Michael S. Tsirkin (10):
virtio: document forward compatibility guarantees
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueues
pci: add admin vq registers to virtio over pci
mmio: document ADMIN_VQ as reserved
ccw: document ADMIN_VQ as reserved
admin: command list discovery
admin: conformance clauses
ccw: document more reserved features
admin.tex | 540 +++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 112 +++++++++- introduction.tex | 3 + 3 files changed, 653 insertions(+), 2 deletions(-) create mode 100644 admin.tex
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 1/10 virtio: document forward compatibility guarantees
Feature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented. Do it now.
--- content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
content.tex +42 -0
@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit. +To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according to +this version of the specification.) Likewise, it is important that +drivers \em{do not} accept feature bits they do not know how to handle +(even though devices are not supposed to offer them in the first place, +according to this version of the specification.) The preferred way for +handling reserved and unexpected features is that the driver ignores +them. + +In particular, this is +especially important for features limited to specific transports, +as enabling these for more transports in future versions of the +specification is highly likely to require changing the behaviour +from drivers and devices. Drivers and devices supporting +multiple transports need to carefully maintain per-transport +lists of allowed features. + \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits} The driver MUST NOT accept a feature which the device did not offer, and MUST NOT accept a feature which requires another feature which was not accepted. +The driver MUST validate the feature bits offered by the device. +The driver MUST ignore and MUST NOT accept any feature bit that is +\begin{itemize} +\item not described in this specification, +\item marked as reserved, +\item not supported for the specific transport, +\item not defined for the device type. +\end{itemize} + The driver SHOULD go into backwards compatibility mode if the device does not offer a feature it understands, otherwise MUST set the FAILED \field{device status} bit and cease initialization. +By contrast, the driver MUST NOT fail solely because a feature +it does not understand has been offered by the device. + \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits} The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK \field{device status} bit when the driver writes it. +The device MUST NOT offer feature bits corresponding to features +it would not support if accepted by the driver (even if the +driver is prohibited from accepting the feature bits by the +specification); for the sake of clarity, this refers to feature +bits not described in this specification, reserved feature bits +and feature bits reserved or not supported for the specific +transport or the specific device type, but this does not preclude +devices written to a future version of this specification from +offering such feature bits should such a specification have a +provision for devices to support the corresponding features. + If a device has successfully negotiated a set of features at least once (by accepting the FEATURES_OK \field{device status} bit during device initialization), then it SHOULD
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 2/10 admin: introduce device group and related concepts
In reply to Michael S. TsirkinEach device group has a type. For now, define one initial group:
SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given PCI SR-IOV physical function (PF). This group may contain zero or more virtio devices according to NumVFs configured.
Each device within a group has a unique identifier. This identifier is the group member identifier.
Note: one can argue both ways whether the new device group handling functionality (this and following patches) is closer
to a new device type or a new transport type.
However, I expect that we will add more features in the near future. To facilitate this as much as possible of the text is located in the new admin chapter.
I did my best to minimize transport-specific text.
--- admin.tex | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 2 ++ 2 files changed, 51 insertions(+) create mode 100644 admin.tex
admin.tex new file +49 -0
@@ -0,0 +1,49 @@ +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups} + +It is occasionally useful to have a device control a group of +other devices. Terminology used in such cases: + +\begin{description} +\item[Device group] + or just group, includes zero or more devices. +\item[Owner device] + or owner, the device controlling the group. +\item[Member device] + a device within a group. The owner device itself is not + a member of the group. +\item[Member identifier] + each member has this identifier, unique within the group + and used to address it through the owner device. +\item[Group type identifier] + specifies what kind of member devices there are in a + group, how is the member identifier is interpreted + and what kind of control the owner has. + A given owner can control a single group of a given type, + thus the type and the owner together identify the group. + \footnote{Even though some group types only support + specific transports, group type identifiers + are global rather than transport-specific - + we don't expect a flood of new group types.} +\end{description} + +The following group types, and their identifiers, are currently specified): +\begin{description} +\item[SR-IOV group type (0x1)] +This device group has a PCI Single Root I/O Virtualization +(SR-IOV) physical function (PF) device as the owner and includes +all its SR-IOV virtual functions (VFs) as members (see +\hyperref[intro:PCIe]{[PCIe]}). + +The PF device itself is not a member of the group. + +The group type identifier for this group is 0x1. + +A member identifier for this group can have a value from 0x1 to 0xFFFF +and equals the SR-IOV VF number of the member device (see +\hyperref[intro:PCIe]{[PCIe]}). + +Both owner and member devices for this group type use the Virtio +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). +\end{description} + +
content.tex +2 -0
@@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 3/10 admin: introduce group administration commands
In reply to Michael S. TsirkinThis introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)
admin.tex +108 -0
@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description} +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device of +a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table: + +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular} + +The \field{group_type} specifies the group type identifier. +The \field{group_member_id} specifies the member identifier within the group. +See section \ref{sec:Introduction / Terminology / Device group} +for the definition of the group type identifier and group member +identifier. + +The following table describes possible \field{status} values; +to simplify common implementations, they are intentionally +matching common \hyperref[intro:errno]{Linux error names and numbers}: + +\begin{tabular}{|l|l|l|} +\hline +Status (decimal) & Name & Description \\ +\hline \hline +00 & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +22 & VIRTIO_ADMIN_STATUS_EINVAL & invalid command \\ +\hline +other & - & group administration command error \\ +\hline +\end{tabular} + +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. + +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, +the following table describes possible \field{status_qialifier} values: +\begin{tabular}{|l|l|l|} +\hline +Status & Name & Description \\ +\hline \hline +0x00 & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND & command error: no additional information \\ +\hline +0x01 & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE & unsupported or invalid \field{opcode} \\ +\hline +0x02 & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD & unsupported or invalid field within \field{command_specific_data} \\ +\hline +0x03 & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP & unsupported or invalid \field{group_type} \\ +\hline +0x04 & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER & unsupported or invalid \field{group_member_id} \\ +\hline +0x05-0xFFFF & - & reserved for future use \\ +\hline +\end{tabular} + +Each command uses a different \field{command_specific_data} and +\field{command_specific_result} structures and the length of +\field{command_specific_data} and \field{command_specific_result} +depends on these structures and is described separately or is +implicit in the structure description.
introduction.tex +3 -0
@@ -68,6 +68,9 @@ \section{Normative References}\label{sec:Normative References} \phantomsection\label{intro:FUSE}\textbf{[FUSE]} & Linux FUSE interface, \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h}\\ + \phantomsection\label{intro:errno}\textbf{[errno]} & + Linux error names and numbers, + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/uapi/asm-generic/errno-base.h}\\ \phantomsection\label{intro:eMMC}\textbf{[eMMC]} & eMMC Electrical Standard (5.1), JESD84-B51, \newline\url{http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf}\\
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 4/10 admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinThe admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
We also support more than one admin virtqueue, for QoS and scalability requirements.
--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)
admin.tex +74 -0
@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commands +placed on different administration virtqueues, because they will +be executed with no order constraints. + +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outside +the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since the +driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
content.tex +5 -2
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 5/10 pci: add admin vq registers to virtio over pci
In reply to Michael S. TsirkinAdd new registers to the PCI common configuration structure.
These registers will be used for querying the indices of the admin virtqueues of the owner device. To configure, reset or enable the admin virtqueues, the driver should follow existing queue configuration/setup sequence.
--- content.tex | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
content.tex +35 -0
@@ -946,6 +946,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport le64 queue_device; /* read-write */ le16 queue_notify_data; /* read-only for driver */ le16 queue_reset; /* read-write */ + + /* About the administration virtqueue. */ + le16 admin_queue_index; /* read-only for driver */ + le16 admin_queue_num; /* read-only for driver */ }; \end{lstlisting}
@@ -1031,6 +1035,20 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport This field exists only if VIRTIO_F_RING_RESET has been negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}). +\item[\field{admin_queue_index}] + The device uses this to report the index of the first administration virtqueue. + This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated. +\item[\field{admin_queue_num}] + The device uses this to report the number of the + supported administration virtqueues. + This is a zero based value. Virtqueues with index + between \field{admin_queue_index} and (\field{admin_queue_index} + + \field{admin_queue_num}) inclusive serve as administration + virtqueues. + Thus the number of administration virtqueues equals + (\field{admin_queue_num} + 1). + This field is valid only if VIRTIO_F_ADMIN_VQ has been + negotiated. \end{description} \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1117,6 +1135,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport were used before the queue reset. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}). +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver +configures any administration virtqueues, the driver MUST +configure the administration virtqueues using the index +in the range \field{admin_queue_index} to +\field{admin_queue_index} + \field{admin_queue_num} inclusive. +The driver MAY configure less administration virtqueues than +supported by the device. + \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
@@ -7684,6 +7710,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more administration virtqueues. + At the moment this feature is only supported for devices using + \ref{sec:Virtio Transport Options / Virtio Over PCI + Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus} + as the transport and is reserved for future use for + devices using other transports (see + \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits} + and + \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for + handling features reserved for future use. \end{description}
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 6/10 mmio: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinAdding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.
--- content.tex | 12 ++++++++++++ 1 file changed, 12 insertions(+)
content.tex +12 -0
@@ -2363,6 +2363,18 @@ \subsection{Legacy interface}\label{sec:Virtio Transport Options / Virtio Over M Notification mechanisms did not change. +\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio Over MMIO / Features reserved for future use} + +At this time, devices and drivers utilizing Virtio Over MMIO +do not support the following features: +\begin{itemize} + +\item VIRTIO_F_ADMIN_VQ + +\end{itemize} + +These features are reserved for future use. + \section{Virtio Over Channel I/O}\label{sec:Virtio Transport Options / Virtio Over Channel I/O} S/390 based virtual machines support neither PCI nor MMIO, so a
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 8/10 admin: command list discovery
In reply to Michael S. TsirkinAdd commands to find out which commands does each group support, as well as enable their use by driver.
This will be especially useful once we have multiple group types.
An alternative is per-type VQs. This is possible but will require more per-transport work. Discovery through the vq helps keep things contained.
--- admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 1 deletion(-)
admin.tex +96 -1
@@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \hline opcode & Name & Command Description \\ \hline \hline -0x0000 - 0x7FFF & - & Group administration commands \\ +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\ +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\ +0x0002 - 0x7FFF & - & Group administration commands \\ \hline 0x8000 - 0xFFFF & - & Reserved \\ \hline
@@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti depends on these structures and is described separately or is implicit in the structure description. +Before sending any administration commands to the device, the driver +needs to communicate to the device which commands it is going to +use. Initially (after reset), only two commands are assumed to be used: +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. + +Before sending any other commands for any member of a specific group to +the device, the driver queries the supported commands via +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via +VIRTIO_ADMIN_CMD_LIST_USE. + +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE +both use the following structure describing the +command opcodes: + +\begin{lstlisting} +struct virtio_admin_cmd_list { + /* Indicates which of the below fields were returned + le64 device_admin_cmds[]; +}; +\end{lstlisting} + +This structure is an array of 64 bit values in little-endian byte +order, in which a bit is set if the specific command opcode +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value +in this array corresponding to opcodes 0 to 63, +\field{device_admin_cmds[1]} is the second 64-bit value +corresponding to opcodes 64 to 127, etc. +For example, the array of size 2 including +the values 0x3 in \field{device_admin_cmds[0]} +and 0x1 in \field{device_admin_cmds[1]} indicates that only +opcodes 0, 1 and 64 are supported. +The length of the array depends on the supported opcodes - it is +large enough to include bits set for all supported opcodes, +that is the length can be calculated by starting with the largest +supported opcode adding one, dividing by 64 and rounding up. +In other words, for +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the +length of \field{command_specific_result} and +\field{command_specific_data} respectively will be +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division +with round up and \field{max_cmd} is the largest available command opcode. + +The array is also allowed to be larger and to additionally +include an arbitrary number of all-zero entries. + +Accordingly, bits 0 and 1 corresponding to opcode 0 +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1 +(VIRTIO_ADMIN_CMD_LIST_USE) are +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY. + +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0. +The \field{group_member_id} is unused. It is set to zero by driver. +This command has no command specific data. +The device, upon success, returns a result in +\field{command_specific_result} in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands supported for the given group. + + +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode} +is set to 0x1. +The \field{group_member_id} is unused. It is set to zero by driver. +The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands used by the driver. +This command has no command specific result. + +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to +query the list of commands valid for this group and before sending +any commands for any member of a group. + +The driver then enables use of some of the opcodes by sending to +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is +both understood and used by the driver. + +If the device supports the command list used by the driver, the +device completes the command with status VIRTIO_ADMIN_STATUS_OK. +If the device does not support the command list, the device +completes the command with status +VIRTIO_ADMIN_STATUS_INVALID_FIELD. + +Note: drivers are assumed not to set bits in device_admin_cmds +if they are not familiar with how the command opcode +is used, since devices could have dependencies between +command opcodes. + +It is assumed that all members in a group support and are used +with the same list of commands. However, for owner devices +supporting multiple group types, the list of supported commands +might differ between different group types. + \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} An administration virtqueue of an owner device is used to submit
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 7/10 ccw: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinAdding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.
Note: there are more features to add to this list. Will be done later with a patch on top.
--- content.tex | 12 ++++++++++++ 1 file changed, 12 insertions(+)
content.tex +12 -0
@@ -2977,6 +2977,18 @@ \subsubsection{Resetting Devices}\label{sec:Virtio Transport Options / Virtio ov MAY also choose to verify reset completion by reading \field{device status} via CCW_CMD_READ_STATUS and checking whether it is 0 afterwards. +\subsection{Features reserved for future use}\label{sec:Virtio Transport Options / Virtio over channel I/O / Features reserved for future use} + +At this time, devices and drivers utilizing Virtio over channel I/O +do not support the following features: +\begin{itemize} + +\item VIRTIO_F_ADMIN_VQ + +\end{itemize} + +These features are reserved for future use. + \chapter{Device Types}\label{sec:Device Types} On top of the queues, config space and feature negotiation facilities
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 9/10 admin: conformance clauses
In reply to Michael S. TsirkinAdd conformance clauses for admin commands and admin virtqueues.
--- admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 215 insertions(+), 1 deletion(-)
admin.tex +215 -1
@@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti supporting multiple group types, the list of supported commands might differ between different group types. +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} + +The device MUST validate \field{opcode}, \field{group_type} and +\field{group_member_id}, and if any of these has an invalid or +unsupported value, set \field{status} to +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier} +accordingly: +\begin{itemize} +\item if \field{group_type} is invalid, \field{status_qualifier} + MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; +\item otherwise, if \field{opcode} is invalid, + \field{status_qualifier} MUST be set to + VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE; +\item otherwise, if \field{group_member_id} is used by the + specific command and is invalid, \field{status_qualifier} MUST be + set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER. +\end{itemize} + +If a command completes successfully, the device MUST set +\field{status} to VIRTIO_ADMIN_STATUS_OK. + +If a command fails, the device MUST set +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK. + +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the +device state MUST NOT change, that is the command MUST NOT have +any side effects on the device, in particular the device MUST not +enter an error state as a result of this command. + +If a command fails, the device state generally SHOULD NOT change, +as far as possible. + +The device MAY enforce additional restrictions and dependencies on +opcodes used by the driver and MAY fail the command +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD +if the list of commands used violate internal device dependencies. + +If the device supports multiple group types, commands for each group +type MUST operate independently of each other, in particular, +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY +for different group types. + +After reset, if the device supports a given group type +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type +the device MUST assume +that the list of legal commands used by the driver consists of +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. + +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully, +the device MUST set the list of legal commands used by the driver +to the one supplied in \field{command_specific_data}. + +The device MUST set the list of legal commands used by the driver +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE. + +The device MUST validate commands against the list used by +the driver and MUST fail any commands not in the list with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +and \field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE. + +The list of supported commands MUST NOT shrink (but MAY expand): +after reporting a given command as supported through +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it +as unsupported. Further, after a given set of commands has been +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a +device or system reset the device SHOULD complete successfully +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails +after a device or system reset, the device MUST not fail it +solely because of the command list used. Failure to do so would +interfere with resuming from suspend and error recovery. + +When processing a command with the SR-IOV group type, +if the device does not have an SR-IOV Extended Capability or +if \field{VF Enable} is clear +then the device MUST fail all commands with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; +otherwise, if \field{group_member_id} is not +between $1$ and \field{NumVFs} inclusive, +the device MUST fail all commands with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER; +\field{NumVFs}, \field{VF Migration Capable} and +\field{VF Enable} refer to registers within the SR-IOV Extended +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. + +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} + +The driver MAY discover whether device supports a specific group type +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching +\field{group_type}. + +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE +and wait for it to be completed with status +VIRTIO_ADMIN_STATUS_OK before issuing any commands +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY +and VIRTIO_ADMIN_CMD_LIST_USE). + +The driver SHOULD NOT set bits in device_admin_cmds +if it is not familiar with how the command opcode +is used, as dependencies between command opcodes might exist. + +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE) +the use of any commands not previously reported as +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY. + +The driver MUST NOT use any commands for a given group type +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct +list of command opcodes and group type. + +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}. + +The driver MUST handle a command error with a reserved \field{status} +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +(except possibly for different error reporting/diagnostic messages). + +The driver MUST handle a command error with a reserved +\field{status_qualifier} value in the same way as +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for +different error reporting/diagnostic messages). + +When sending commands with the SR-IOV group type, +the driver specify a value for \field{group_member_id} +between $1$ and \field{NumVFs} inclusive, +the driver MUST also make sure that as long as any such command +is outstanding, \field{VF Migration Capable} is clear and +\field{VF Enable} is set; +\field{NumVFs}, \field{VF Migration Capable} and +\field{VF Enable} refer to registers within the SR-IOV Extended +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. + \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} An administration virtqueue of an owner device is used to submit
@@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic of the specification are designed, new fields can be added to the tail of a structure, with the driver/device using the full structure without concern for versioning. ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues + +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues} + +The device MUST support device-readable and device-writeable buffers +shorter than described in this specification, by +\begin{enumerate} +\item acting as if any data that would be read outside the +device-readable buffers is set to zero, and +\item discarding data that would be written outside the +specified device-writeable buffers. +\end{enumerate} + +The device MUST support device-readable and device-writeable buffers +longer than described in this specification, by +\begin{enumerate} +\item ignoring any data in device-readable buffers outside +the expected length, and +\item only writing the expected structure to the device-writeable +buffers, ignoring any extra buffers, and reporting the +actual length of data written, in bytes, +as buffer used length. +\end{enumerate} + +The device SHOULD initialize the device-writeable buffer +up to the length of the structure described by this specification or +the length of the buffer supplied by the driver (even if the buffer is +all set to zero), whichever is shorter. + +The device MUST NOT fail a command solely because the buffers +provided are shorter or longer than described in this +specification. + +The device MUST initialize the device-writeable part of +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in +size. + +The device MUST initialize \field{status} in \field{struct +virtio_admin_cmd}. + +The device MUST process commands on a given administration virtqueue +in the order in which they are queued. + +If multiple administration virtqueues have been configured, +device MAY process commands on distinct virtqueues with +no order constraints. + +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues} + +The driver MAY supply device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are longer than described in +this specification. + +The driver SHOULD supply device-readable part of +\field{struct virtio_admin_cmd} that is at least as +large as the structure described by this specification +(even if the structure is all set to zero). + +The driver MUST supply both device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit +in length. + +The device MUST supply both device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are larger than zero in +length. However, \field{command_specific_data} and +\field{command_specific_result} MAY be zero in length, unless +specified otherwise for the command. + +The driver MUST NOT assume that the device will initialize the whole +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead, +the driver MUST act as if the structure +outside the part of the buffer used by the device +is set to zero. + +If multiple administration virtqueues have been configured, +the driver MUST ensure ordering for commands +placed on different administration virtqueues.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
PATCH v10 10/10 ccw: document more reserved features
In reply to Michael S. Tsirkinvq reset and shared memory are unsupported, too.
--- content.tex | 2 ++ 1 file changed, 2 insertions(+)
content.tex +2 -0
@@ -2984,6 +2984,8 @@ \subsection{Features reserved for future use}\label{sec:Virtio Transport Options \begin{itemize} \item VIRTIO_F_ADMIN_VQ +\item VIRTIO_F_RING_RESET +\item Shared memory regions including VIRTIO_PMEM_F_SHMEM_REGION \end{itemize}
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 6 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM
Change log:since v9:
This version is supposed to be v11 from last v10 [1].
https://lists.oasis-open.org/archives/virtio-dev/202302/msg00174.html
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMMy intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Quoted 6 lines
For example, it might make sense to split creating/destroying SIOV devices from the transport passing data from the guest - the driver would then not negotiate
VIRTIO_F_SR_IOV(which then means auto-provisioning).This is out of RFC, since we have two commands which are useful to discover supported group types (ATM can be none or SR-IOV).
I guess you missed expanding the cover letter for our discussion on [1] to list use cases a little more tangible. So let us have them here.
Some I can think of are good starting points are:
1. virtio VF features query and configuration space provisioning
2. virtio VF resource (queues, msix vectors count) provisioning
I plan to post the follow-up patch for #1 as the actual use of AQ.
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00246.html
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] RE: [PATCH v10 01/10] virtio: document forward compatibility guarantees
In reply to Michael S. TsirkinQuoted 11 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMFeature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented.
Do it now.Suggested-by Halil Pasic <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
It may be little painful, but in 10 patch series, it is worth having per patch change log.
Because I do not know my reviewed by tag in [1] was dropped because of a big change, if so what was it or it was simply missed out.
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00237.html
Quoted 35 lines
content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)content.tex +0
-0@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of aVirtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit.
content.tex +19
-0+To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according +to this version of the specification.) Likewise, it is important that +drivers \em{do not} accept feature bits they do not know how to handle +(even though devices are not supposed to offer them in the first place, +according to this version of the specification.) The preferred way for +handling reserved and unexpected features is that the driver ignores +them. + +In particular, this is +especially important for features limited to specific transports, as +enabling these for more transports in future versions of the +specification is highly likely to require changing the behaviour from +drivers and devices. Drivers and devices supporting multiple +transports need to carefully maintain per-transport lists of allowed +features. + \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /Feature Bits} The driver MUST NOT accept a feature which the device did not offer, and MUST NOT accept a feature which requires another feature which was not accepted.
content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
Quoted 8 lines
content.tex +7
-0+The driver MUST validate the feature bits offered by the device. +The driver MUST ignore and MUST NOT accept any feature bit that is +\begin{itemize} \item not described in this specification, \item marked +as reserved, \item not supported for the specific transport, \item not +defined for the device type. +\end{itemize} + The driver SHOULD go into backwards compatibility mode if the device does
The grammar tool is suggesting s/backwards/backward.
content.tex
Quoted 6 lines
not offer a feature it understands, otherwise MUST set the FAILED \field{device status} bit and cease initialization.
content.tex +3
-0+By contrast, the driver MUST NOT fail solely because a feature it does +not understand has been offered by the device. +
Above line can be written without introducing "has been" given we have only two entities (driver, dev) to care about.
By contrast, the driver MUST NOT fail solely because it does not understand the feature offered by the device.
Quoted 3 lines
\devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
Feature Bits} The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of
Extra white space between feature and which, subset and of.
Quoted 3 lines
features the driver accepts, otherwise it MUST fail to set the
FEATURES_OK\field{device status} bit when the driver writes it.
Extra white space between the FEATURE_OK
Quoted 12 lines
content.tex +11
-0+The device MUST NOT offer feature bits corresponding to features it +would not support if accepted by the driver (even if the driver is +prohibited from accepting the feature bits by the specification); for +the sake of clarity, this refers to feature bits not described in this +specification, reserved feature bits and feature bits reserved or not +supported for the specific transport or the specific device type, but +this does not preclude devices written to a future version of this +specification from offering such feature bits should such a +specification have a provision for devices to support the corresponding +features. + If a device has successfully negotiated a set of features at least once (by
Quoted 4 lines
accepting the
FEATURES_OK\field{device status} bit during device initialization), then it SHOULD
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] RE: [PATCH v10 02/10] admin: introduce device group and related concepts
In reply to Michael S. TsirkinQuoted 22 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMEach device group has a type. For now, define one initial group:
SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given PCI SR-IOV physical function (PF). This group may contain zero or more virtio devices according to NumVFs configured.
Each device within a group has a unique identifier. This identifier is the group member identifier.
Note: one can argue both ways whether the new device group handling functionality (this and following patches) is closer to a new device type or a new transport type.
However, I expect that we will add more features in the near future. To facilitate this as much as possible of the text is located in the new admin chapter.
I did my best to minimize transport-specific text.
Quoted 8 lines
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 2 ++ 2 files changed, 51 insertions(+) create mode 100644 admin.tex
You missed addressing the comments of [1].
Can you please fix them?
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00235.html
Quoted 64 lines
admin.tex new file +27
-0@@ -0,0 +1,49 @@ +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / +Device groups} + +It is occasionally useful to have a device control a group of other +devices. Terminology used in such cases: + +\begin{description} +\item[Device group] + or just group, includes zero or more devices. +\item[Owner device] + or owner, the device controlling the group. +\item[Member device] + a device within a group. The owner device itself is not + a member of the group. +\item[Member identifier] + each member has this identifier, unique within the group + and used to address it through the owner device. +\item[Group type identifier] + specifies what kind of member devices there are in a + group, how is the member identifier is interpreted + and what kind of control the owner has. + A given owner can control a single group of a given type, + thus the type and the owner together identify the group. + \footnote{Even though some group types only support + specific transports, group type identifiers + are global rather than transport-specific - + we don't expect a flood of new group types.}\end{description}
admin.tex new file +22
-0+ +The following group types, and their identifiers, are currently specified): +\begin{description} +\item[SR-IOV group type (0x1)] +This device group has a PCI Single Root I/O Virtualization +(SR-IOV) physical function (PF) device as the owner and includes all +its SR-IOV virtual functions (VFs) as members (see +\hyperref[intro:PCIe]{[PCIe]}). + +The PF device itself is not a member of the group. + +The group type identifier for this group is 0x1. + +A member identifier for this group can have a value from 0x1 to 0xFFFF +and equals the SR-IOV VF number of the member device (see +\hyperref[intro:PCIe]{[PCIe]}). + +Both owner and member devices for this group type use the Virtio PCI +transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). +\end{description} + +content.tex +0
-0@@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities ofa Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
Quoted 8 lines
+\input{admin.tex}
+
\chapter{General Initialization And Device Operation}\label{sec:General
Initialization And Device Operation}We start with an overview of device initialization, then expand on the
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] RE: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. Tsirkinadmin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 132
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMadmin.tex +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +
s/status_qialifier/status_qualifier
Missed from v10 of Feb.
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:
s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
And with that, VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER moves under ENODEV error code.
Quoted 14 lines
+\begin{tabular}{|l|l|l|}
+\hline
+Status & Name & Description \\
+\hline \hline
+0x00 &VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND& command
error: no additional information \\
+\hline
+0x01 &VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE& unsupported or
invalid \field{opcode} \\
+\hline
+0x02 &VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD& unsupported or
invalid field within \field{command_specific_data} \\
+\hline
+0x03 &VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP& unsupported or
Quoted 8 lines
invalid \field{group_type} \\
+\hline
+0x04 &VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER& unsupported or
invalid \field{group_member_id} \\
+\hline
+0x05-0xFFFF & - & reserved for future use \\
+\hline
+\end{tabular}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinQuoted 11 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMAdding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.Signed-off-by Michael S. Tsirkin <[email protected]>
Did you miss reviewed-by from [1] or this is an old series reposted?
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html
I will wait for your response before reviewing more, as I am confused now between missing items of v10 of Feb and March.
Can you please have the per-patch change log? I learnt and practice that from netdev community. It has been very useful to have effective reviews (especially when series is long).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] RE: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM+>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
I requested to fix this in v10 of Feb, it was in some other patch. This hunk shifted here.
Review
Quoted 3 lines
(quoted diff) +3
-0+ \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device + exposes one or more administration virtqueues. +
Please remove extra white space between more and administration.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [virtio] [PATCH v10 02/10] admin: introduce device group and related concepts
In reply to Michael S. TsirkinQuoted 1 line · Michael S. Tsirkin
+The following group types, and their identifiers, are currently specified):
spurious ')'
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 18 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 6 lines
admin.tex +2
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description} +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device of
I notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Quoted 45 lines
admin.tex +45
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table:
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 10 lines
admin.tex +10
-0+ +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}
I thought all commands are "group administration commands" but this table makes it look like they are just a subset (0x0000 - 0x7FFF) of group administration commands, which is a paradox.
Quoted 14 lines
admin.tex +14
-0+ +The \field{group_type} specifies the group type identifier. +The \field{group_member_id} specifies the member identifier within the group. +See section \ref{sec:Introduction / Terminology / Device group} +for the definition of the group type identifier and group member +identifier. + +The following table describes possible \field{status} values; +to simplify common implementations, they are intentionally +matching common \hyperref[intro:errno]{Linux error names and numbers}: + +\begin{tabular}{|l|l|l|} +\hline +Status (decimal) & Name & Description \\
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 130
Quoted 10 lines
admin.tex +10
-0+\hline \hline +00 & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +22 & VIRTIO_ADMIN_STATUS_EINVAL & invalid command \\ +\hline +other & - & group administration command error \\ +\hline +\end{tabular} + +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
s/qialifier/qualifier/
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 134
Quoted 4 lines
admin.tex +4
-0+is reserved and set to zero by the device. + +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, +the following table describes possible \field{status_qialifier} values:
s/qialifier/qualifier/
Quoted 13 lines
admin.tex +13
-0+\begin{tabular}{|l|l|l|} +\hline +Status & Name & Description \\ +\hline \hline +0x00 & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND & command error: no additional information \\ +\hline +0x01 & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE & unsupported or invalid \field{opcode} \\ +\hline +0x02 & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD & unsupported or invalid field within \field{command_specific_data} \\ +\hline +0x03 & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP & unsupported or invalid \field{group_type} \\ +\hline +0x04 & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER & unsupported or invalid \field{group_member_id} \\
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 157
Quoted 10 lines
admin.tex +10
-0+\hline +0x05-0xFFFF & - & reserved for future use \\ +\hline +\end{tabular} + +Each command uses a different \field{command_specific_data} and +\field{command_specific_result} structures and the length of +\field{command_specific_data} and \field{command_specific_result} +depends on these structures and is described separately or is +implicit in the structure description.
On more thing:
Does the owner device see commands in order but may complete them in any order?
I think this information might be useful just to make it clear that driver authors shouldn't make assumptions about ordering and completion order, e.g. pipelining a bunch of dependent commands doesn't work because the first command is not necessarily completed before the second command is started.
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Parav PanditQuoted 10 lines · Parav Pandit
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 132
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMadmin.tex +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Only status_qualifier is needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only the status_qualifier really indicates what is going on.
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
Quoted 26 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +8
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 10 lines
admin.tex +10
-0+or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commands +placed on different administration virtqueues, because they will +be executed with no order constraints.
Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
Quoted 46 lines
admin.tex +46
-0+ +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outside
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 10 lines
admin.tex +10
-0+the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since the +driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
A merge conflict line crept into the patch?
Quoted 30 lines
content.tex +5
-2@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
MST
[virtio-dev] Re: [PATCH v10 05/10] pci: add admin vq registers to virtio over pci
In reply to Michael S. TsirkinQuoted 34 lines · Michael S. Tsirkin
Add new registers to the PCI common configuration structure.
These registers will be used for querying the indices of the admin virtqueues of the owner device. To configure, reset or enable the admin virtqueues, the driver should follow existing queue configuration/setup sequence.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)content.tex +6
-0@@ -946,6 +946,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport le64 queue_device; /* read-write */ le16 queue_notify_data; /* read-only for driver */ le16 queue_reset; /* read-write */ + + /* About the administration virtqueue. */ + le16 admin_queue_index; /* read-only for driver */ + le16 admin_queue_num; /* read-only for driver */ }; \end{lstlisting}@@ -1031,6 +1035,20 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport This field exists only if VIRTIO_F_RING_RESET has been negotiated. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}). +\item[\field{admin_queue_index}] + The device uses this to report the index of the first administration virtqueue.
content.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 1049
Quoted 10 lines
content.tex +10
-0+ This field is valid only if VIRTIO_F_ADMIN_VQ has been negotiated. +\item[\field{admin_queue_num}] + The device uses this to report the number of the + supported administration virtqueues. + This is a zero based value. Virtqueues with index + between \field{admin_queue_index} and (\field{admin_queue_index} + + \field{admin_queue_num}) inclusive serve as administration + virtqueues. + Thus the number of administration virtqueues equals + (\field{admin_queue_num} + 1).
Why is this value zero-based? virtio_pci_common_cfg's num_queues field is le16 and not zero-based. It is not possible to have admin_queue_index=0 admin_queue_num=0xffff because num_queues=0x10000 cannot be represented.
I suggest making this field behave the same as virtio_pci_common_cfg's num_queues.
Quoted 6 lines
content.tex +2
-0+ This field is valid only if VIRTIO_F_ADMIN_VQ has been + negotiated. \end{description} \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
content.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 1143
Quoted 9 lines
content.tex +6
-0@@ -1117,6 +1135,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport were used before the queue reset. (see \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}). +If VIRTIO_F_ADMIN_VQ has been negotiated, and if the driver +configures any administration virtqueues, the driver MUST +configure the administration virtqueues using the index +in the range \field{admin_queue_index} to +\field{admin_queue_index} + \field{admin_queue_num} inclusive. +The driver MAY configure less administration virtqueues than
s/less/fewer/
Quoted 24 lines
content.tex +11
-0+supported by the device. + \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability} The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG@@ -7684,6 +7710,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more administration virtqueues. + At the moment this feature is only supported for devices using + \ref{sec:Virtio Transport Options / Virtio Over PCI + Bus}~\nameref{sec:Virtio Transport Options / Virtio Over PCI Bus} + as the transport and is reserved for future use for + devices using other transports (see + \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits} + and + \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for + handling features reserved for future use. \end{description}Signature · 1 line
MST
[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinQuoted 11 lines · Michael S. Tsirkin
Adding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 12 ++++++++++++ 1 file changed, 12 insertions(+)
[virtio-dev] Re: [PATCH v10 07/10] ccw: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinQuoted 7 lines · Michael S. Tsirkin
Adding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.
Quoted 7 lines
Note: there are more features to add to this list. Will be done later with a patch on top.
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 12 ++++++++++++ 1 file changed, 12 insertions(+)
[virtio-dev] Re: [PATCH v10 08/10] admin: command list discovery
In reply to Michael S. TsirkinQuoted 31 lines · Michael S. Tsirkin
Add commands to find out which commands does each group support, as well as enable their use by driver.
This will be especially useful once we have multiple group types.An alternative is per-type VQs. This is possible but will require more per-transport work. Discovery through the vq helps keep things contained.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 1 deletion(-)admin.tex +3
-1@@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \hline opcode & Name & Command Description \\ \hline \hline -0x0000 - 0x7FFF & - & Group administration commands \\ +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\ +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\ +0x0002 - 0x7FFF & - & Group administration commands \\ \hline 0x8000 - 0xFFFF & - & Reserved \\ \hline@@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti depends on these structures and is described separately or is
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 168
Quoted 10 lines
admin.tex +8
-0implicit in the structure description. +Before sending any administration commands to the device, the driver +needs to communicate to the device which commands it is going to +use. Initially (after reset), only two commands are assumed to be used: +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. + +Before sending any other commands for any member of a specific group to +the device, the driver queries the supported commands via +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via
"will use" probably means "is capable of using" rather than "is sure to use"? It might be worth tweaking the language here to make that clear.
Quoted 74 lines
admin.tex +74
-0+VIRTIO_ADMIN_CMD_LIST_USE. + +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE +both use the following structure describing the +command opcodes: + +\begin{lstlisting} +struct virtio_admin_cmd_list { + /* Indicates which of the below fields were returned + le64 device_admin_cmds[]; +}; +\end{lstlisting} + +This structure is an array of 64 bit values in little-endian byte +order, in which a bit is set if the specific command opcode +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value +in this array corresponding to opcodes 0 to 63, +\field{device_admin_cmds[1]} is the second 64-bit value +corresponding to opcodes 64 to 127, etc. +For example, the array of size 2 including +the values 0x3 in \field{device_admin_cmds[0]} +and 0x1 in \field{device_admin_cmds[1]} indicates that only +opcodes 0, 1 and 64 are supported. +The length of the array depends on the supported opcodes - it is +large enough to include bits set for all supported opcodes, +that is the length can be calculated by starting with the largest +supported opcode adding one, dividing by 64 and rounding up. +In other words, for +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the +length of \field{command_specific_result} and +\field{command_specific_data} respectively will be +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division +with round up and \field{max_cmd} is the largest available command opcode. + +The array is also allowed to be larger and to additionally +include an arbitrary number of all-zero entries. + +Accordingly, bits 0 and 1 corresponding to opcode 0 +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1 +(VIRTIO_ADMIN_CMD_LIST_USE) are +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY. + +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0. +The \field{group_member_id} is unused. It is set to zero by driver. +This command has no command specific data. +The device, upon success, returns a result in +\field{command_specific_result} in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands supported for the given group. + + +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode} +is set to 0x1. +The \field{group_member_id} is unused. It is set to zero by driver. +The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands used by the driver. +This command has no command specific result. + +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to +query the list of commands valid for this group and before sending +any commands for any member of a group. + +The driver then enables use of some of the opcodes by sending to +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is +both understood and used by the driver. + +If the device supports the command list used by the driver, the +device completes the command with status VIRTIO_ADMIN_STATUS_OK. +If the device does not support the command list, the device +completes the command with status +VIRTIO_ADMIN_STATUS_INVALID_FIELD.
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 252
Quoted 10 lines
admin.tex +10
-0+ +Note: drivers are assumed not to set bits in device_admin_cmds +if they are not familiar with how the command opcode +is used, since devices could have dependencies between +command opcodes. + +It is assumed that all members in a group support and are used +with the same list of commands. However, for owner devices +supporting multiple group types, the list of supported commands +might differ between different group types.
Is the purpose of VIRTIO_ADMIN_CMD_LIST_USE so that new commands can be added that are mandatory for device operation. Failure to include them in the VIRTIO_ADMIN_CMD_LIST_USE command terminates device initialization with an error?
Are commands evolved by defining flag/version bits in the command specific data struct? For example, if a new field needs to be added to a command and a 0 value carries a meaning (so it cannot simply be added to the end of the existing struct with backwards compatibility).
Stefan
[virtio-dev] Re: [PATCH v10 10/10] ccw: document more reserved features
In reply to Michael S. TsirkinQuoted 6 lines · Michael S. Tsirkin
vq reset and shared memory are unsupported, too.
Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 2 ++ 1 file changed, 2 insertions(+)
Quoted 8 lines · Parav Pandit
Quoted 6 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM
Change log:since v9:
This version is supposed to be v11 from last v10 [1].
https://lists.oasis-open.org/archives/virtio-dev/202302/msg00174.html
Good catch. Sorry.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 01/10] virtio: document forward compatibility guarantees
In reply to Parav PanditQuoted 8 lines · Parav Pandit
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMFeature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented.
Do it now.
Quoted 8 lines
Quoted 3 lines
Suggested-by Halil Pasic <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
It may be little painful, but in 10 patch series, it is worth having per patch change log.
Because I do not know my reviewed by tag in [1] was dropped because of a big change, if so what was it or it was simply missed out.[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00237.html
I just lost it, no change.
Quoted 65 lines
Quoted 35 lines
content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)content.tex +0
-0@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of aVirtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit.
content.tex +19
-0+To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according +to this version of the specification.) Likewise, it is important that +drivers \em{do not} accept feature bits they do not know how to handle +(even though devices are not supposed to offer them in the first place, +according to this version of the specification.) The preferred way for +handling reserved and unexpected features is that the driver ignores +them. + +In particular, this is +especially important for features limited to specific transports, as +enabling these for more transports in future versions of the +specification is highly likely to require changing the behaviour from +drivers and devices. Drivers and devices supporting multiple +transports need to carefully maintain per-transport lists of allowed +features. + \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /Feature Bits} The driver MUST NOT accept a feature which the device did not offer, and MUST NOT accept a feature which requires another feature which was not accepted.
content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
Quoted 8 lines
content.tex +7
-0+The driver MUST validate the feature bits offered by the device. +The driver MUST ignore and MUST NOT accept any feature bit that is +\begin{itemize} \item not described in this specification, \item marked +as reserved, \item not supported for the specific transport, \item not +defined for the device type. +\end{itemize} + The driver SHOULD go into backwards compatibility mode if the device doesThe grammar tool is suggesting s/backwards/backward.
content.tex
Quoted 6 lines
not offer a feature it understands, otherwise MUST set the FAILED \field{device status} bit and cease initialization.
content.tex +3
-0+By contrast, the driver MUST NOT fail solely because a feature it does +not understand has been offered by the device. +Above line can be written without introducing "has been" given we have only two entities (driver, dev) to care about.
By contrast, the driver MUST NOT fail solely because it does not understand the feature offered by the device.
Quoted 3 lines
\devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device /
Feature Bits} The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset ofExtra white space between feature and which, subset and of.
Quoted 3 lines
features the driver accepts, otherwise it MUST fail to set the
FEATURES_OK\field{device status} bit when the driver writes it.Extra white space between the
FEATURE_OK
Quoted 16 lines
Quoted 12 lines
content.tex +11
-0+The device MUST NOT offer feature bits corresponding to features it +would not support if accepted by the driver (even if the driver is +prohibited from accepting the feature bits by the specification); for +the sake of clarity, this refers to feature bits not described in this +specification, reserved feature bits and feature bits reserved or not +supported for the specific transport or the specific device type, but +this does not preclude devices written to a future version of this +specification from offering such feature bits should such a +specification have a provision for devices to support the corresponding +features. + If a device has successfully negotiated a set of features at least once (byQuoted 4 lines
accepting the
FEATURES_OK\field{device status} bit during device initialization), then it SHOULD
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 02/10] admin: introduce device group and related concepts
In reply to Parav PanditQuoted 30 lines · Parav Pandit
Quoted 22 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMEach device group has a type. For now, define one initial group:
SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given PCI SR-IOV physical function (PF). This group may contain zero or more virtio devices according to NumVFs configured.
Each device within a group has a unique identifier. This identifier is the group member identifier.
Note: one can argue both ways whether the new device group handling functionality (this and following patches) is closer to a new device type or a new transport type.
However, I expect that we will add more features in the near future. To facilitate this as much as possible of the text is located in the new admin chapter.
I did my best to minimize transport-specific text.
Quoted 8 lines
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 2 ++ 2 files changed, 51 insertions(+) create mode 100644 admin.tex
Quoted 4 lines
You missed addressing the comments of [1].
Can you please fix them?[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00235.html
Hmm oh ok. At least ): one for sure.
Quoted 64 lines
Quoted 64 lines
admin.tex new file +27
-0@@ -0,0 +1,49 @@ +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / +Device groups} + +It is occasionally useful to have a device control a group of other +devices. Terminology used in such cases: + +\begin{description} +\item[Device group] + or just group, includes zero or more devices. +\item[Owner device] + or owner, the device controlling the group. +\item[Member device] + a device within a group. The owner device itself is not + a member of the group. +\item[Member identifier] + each member has this identifier, unique within the group + and used to address it through the owner device. +\item[Group type identifier] + specifies what kind of member devices there are in a + group, how is the member identifier is interpreted + and what kind of control the owner has. + A given owner can control a single group of a given type, + thus the type and the owner together identify the group. + \footnote{Even though some group types only support + specific transports, group type identifiers + are global rather than transport-specific - + we don't expect a flood of new group types.}\end{description}
admin.tex new file +22
-0+ +The following group types, and their identifiers, are currently specified): +\begin{description} +\item[SR-IOV group type (0x1)] +This device group has a PCI Single Root I/O Virtualization +(SR-IOV) physical function (PF) device as the owner and includes all +its SR-IOV virtual functions (VFs) as members (see +\hyperref[intro:PCIe]{[PCIe]}). + +The PF device itself is not a member of the group. + +The group type identifier for this group is 0x1. + +A member identifier for this group can have a value from 0x1 to 0xFFFF +and equals the SR-IOV VF number of the member device (see +\hyperref[intro:PCIe]{[PCIe]}). + +Both owner and member devices for this group type use the Virtio PCI +transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). +\end{description} + +content.tex +0
-0@@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities ofa Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
Quoted 8 lines
Quoted 8 lines
+\input{admin.tex}
+
\chapter{General Initialization And Device Operation}\label{sec:General
Initialization And Device Operation}We start with an overview of device initialization, then expand on the
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Parav PanditQuoted 10 lines · Parav Pandit
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 132
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMadmin.tex +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
We'll add them when there's a use-case.
Quoted 1 line
And with that,
VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBERmoves under ENODEV error code.
No I think that belongs in in EINVAL - command is malformed. ENODEV is more "The wrong type of device was given to a function that expects a particular sort of device."
Quoted 14 lines
Quoted 14 lines
+\begin{tabular}{|l|l|l|}
+\hline
+Status & Name & Description \\
+\hline \hline
+0x00 &VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND& command
error: no additional information \\
+\hline
+0x01 &VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE& unsupported or
invalid \field{opcode} \\
+\hline
+0x02 &VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD& unsupported or
invalid field within \field{command_specific_data} \\
+\hline
+0x03 &VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP& unsupported or
Quoted 8 lines
Quoted 8 lines
invalid \field{group_type} \\
+\hline
+0x04 &VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER& unsupported or
invalid \field{group_member_id} \\
+\hline
+0x05-0xFFFF & - & reserved for future use \\
+\hline
+\end{tabular}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Parav PanditQuoted 8 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM+>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
I requested to fix this in v10 of Feb, it was in some other patch. This hunk shifted here.
weird I remember fix it.
Quoted 4 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device + exposes one or more administration virtqueues. +Please remove extra white space between more and administration.
I will but it does not matter at all latex ignores it.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Parav PanditQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMAdding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.
Quoted 6 lines
Quoted 1 line
Signed-off-by Michael S. Tsirkin <[email protected]>Did you miss reviewed-by from [1] or this is an old series reposted?
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.htmlI will wait for your response before reviewing more, as I am confused now between missing items of v10 of Feb and March.
There was such a huge flamewar around error codes I probably just missed some useful comments among the noise.
Quoted 2 lines
Can you please have the per-patch change log? I learnt and practice that from netdev community. It has been very useful to have effective reviews (especially when series is long).
Can't promise really I will need to look for tooling for that. Unlikely to happen with this patch.
I'm not asking others to do it either.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Parav PanditQuoted 10 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMAdding relevant registers needs more work and it's not clear what the use-case will be as currently only the PCI transport is supported. But let's keep the door open on this.
We already say it's reserved in a central place, but it does not hurt to remind implementers to mask it.
Quoted 4 lines
Quoted 1 line
Signed-off-by Michael S. Tsirkin <[email protected]>Did you miss reviewed-by from [1] or this is an old series reposted?
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html
yes I lost it.
Quoted 4 lines
I will wait for your response before reviewing more, as I am confused now between missing items of v10 of Feb and March.
Can you please have the per-patch change log? I learnt and practice that from netdev community. It has been very useful to have effective reviews (especially when series is long).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Stefan HajnocziQuoted 22 lines · Stefan Hajnoczi
Quoted 21 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description}
Quoted 7 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 3 lines
admin.tex +2
-0+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device ofI notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are owner, "member device" and "member driver" is always called explicitly.
Quoted 51 lines
Quoted 51 lines
admin.tex +51
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table: + +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\
Quoted 8 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 4 lines
admin.tex +4
-0+\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}I thought all commands are "group administration commands" but this table makes it look like they are just a subset (
0x0000-0x7FFF) of group administration commands, which is a paradox.
Well the rest are reserved, maybe we will have more command types who knows. No?
Quoted 60 lines
Quoted 14 lines
admin.tex +14
-0+ +The \field{group_type} specifies the group type identifier. +The \field{group_member_id} specifies the member identifier within the group. +See section \ref{sec:Introduction / Terminology / Device group} +for the definition of the group type identifier and group member +identifier. + +The following table describes possible \field{status} values; +to simplify common implementations, they are intentionally +matching common \hyperref[intro:errno]{Linux error names and numbers}: + +\begin{tabular}{|l|l|l|} +\hline +Status (decimal) & Name & Description \\admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 130
Quoted 10 lines
admin.tex +10
-0+\hline \hline +00 & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +22 & VIRTIO_ADMIN_STATUS_EINVAL & invalid command \\ +\hline +other & - & group administration command error \\ +\hline +\end{tabular} + +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}s/qialifier/qualifier/
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 134
Quoted 4 lines
admin.tex +4
-0+is reserved and set to zero by the device. + +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, +the following table describes possible \field{status_qialifier} values:s/qialifier/qualifier/
Quoted 13 lines
admin.tex +13
-0+\begin{tabular}{|l|l|l|} +\hline +Status & Name & Description \\ +\hline \hline +0x00 & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND & command error: no additional information \\ +\hline +0x01 & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE & unsupported or invalid \field{opcode} \\ +\hline +0x02 & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD & unsupported or invalid field within \field{command_specific_data} \\ +\hline +0x03 & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP & unsupported or invalid \field{group_type} \\ +\hline +0x04 & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER & unsupported or invalid \field{group_member_id} \\admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 157
Quoted 10 lines
admin.tex +10
-0+\hline +0x05-0xFFFF & - & reserved for future use \\ +\hline +\end{tabular} + +Each command uses a different \field{command_specific_data} and +\field{command_specific_result} structures and the length of +\field{command_specific_data} and \field{command_specific_result} +depends on these structures and is described separately or is +implicit in the structure description.On more thing:
Quoted 8 lines
Does the owner device see commands in order but may complete them in any order?
I think this information might be useful just to make it clear that driver authors shouldn't make assumptions about ordering and completion order, e.g. pipelining a bunch of dependent commands doesn't work because the first command is not necessarily completed before the second command is started.
I think this is discussed when we discuss sending commands through vq. in this section it's just about commands generally whatever the way to send them to device. no?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Stefan HajnocziQuoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Parav Pandit
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 132
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMadmin.tex +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Quoted 4 lines
Only
status_qualifieris needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only thestatus_qualifierreally indicates what is going on.
At a high level at the moment we have only two cases:
- ok
- invalid input supplied by driver
maybe we will have more reasons for a failure - remains to be seen.
Quoted 5 lines
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
Really most drivers just want to propagate errno to userspace. All the detailed reporting is for sure well intentional but in the end it is at best printed into log - end to end people just end up with a switch statement converting these to errno codes.
So we are passing them from device and this way there will be some uniformity.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
Quoted 8 lines
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
yes this commit log is outdated. referred to original proposal by Max which also planned to replace cvq. will update.
Quoted 34 lines
Quoted 34 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +16
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commands
Quoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 2 lines
admin.tex +2
-0+placed on different administration virtqueues, because they will +be executed with no order constraints.Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
we have multiple vqs for that.
Quoted 50 lines
Quoted 50 lines
admin.tex +50
-0+ +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outside +the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since the
Quoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 6 lines
admin.tex +6
-0+driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueuesA merge conflict line crept into the patch?
yes. I fixed it I thought but somehow it's still there :(
Quoted 26 lines
Quoted 26 lines
content.tex +5
-2@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description}
Quoted 4 lines
Quoted 4 lines
content.tex +0
-0\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Parav PanditQuoted 2 lines · Parav Pandit
Did you miss reviewed-by from [1] or this is an old series reposted?
[1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.html
As a general rule, we don't strictly need to track reviewed by since there's a ballot (and presumably people review before voting).
People also tack on Signed-off-by: (and I do it too) but as long as we don't document what it means it's kind of vague, and the process of subscribing to the mailing list is a kind of replacement.
If you think everyone needs to follow practices like netdev does, we really need something written up, and agree on it.
E.g. I work on the linux kernel too, so I can copy practices from there, but even linux isn't uniform.
And I wonder whether it's worth it - it definitely makes contributing to Linux harder, and even within Linux it pushes contributors away. At least for Linux tracking history in a precise way is extremely important since it's helpful with stability. Spec is very different.
Until we have a good contribution documentation I think we should not ask people to follow a pseudo linux work flow with requests like "please split this patchset up" or "track changes across patch versions" simply because there's no good docs to teach people what exactly is the best practice.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 31 lines · Michael S. Tsirkin
Quoted 22 lines · Stefan Hajnoczi
Quoted 21 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description}Quoted 7 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 3 lines
admin.tex +2
-0+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device ofI notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Quoted 3 lines
Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are owner, "member device" and "member driver" is always called explicitly.
Sounds good.
Quoted 60 lines
Quoted 51 lines
Quoted 51 lines
admin.tex +51
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table: + +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\Quoted 8 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 4 lines
admin.tex +4
-0+\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}I thought all commands are "group administration commands" but this table makes it look like they are just a subset (
0x0000-0x7FFF) of group administration commands, which is a paradox.
Quoted 2 lines
Well the rest are reserved, maybe we will have more command types who knows. No?
I see. Does that mean the reserved commands don't need to be in the same format as struct virtio_admin_cmd?
The entire section is called "Group administration commands" but I get the impression it's talking both about admin virtqueue commands in general and specifically about group administration commands.
Is it possible to structure this as follows:
Admin Commands
...common stuff...
Group Administration Commands (0x0000-0x7fff)
...specific to group administration commands...
Reserved (0x8000-0xffff)Quoted 69 lines
Quoted 60 lines
Quoted 14 lines
admin.tex +14
-0+ +The \field{group_type} specifies the group type identifier. +The \field{group_member_id} specifies the member identifier within the group. +See section \ref{sec:Introduction / Terminology / Device group} +for the definition of the group type identifier and group member +identifier. + +The following table describes possible \field{status} values; +to simplify common implementations, they are intentionally +matching common \hyperref[intro:errno]{Linux error names and numbers}: + +\begin{tabular}{|l|l|l|} +\hline +Status (decimal) & Name & Description \\admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 130
Quoted 10 lines
admin.tex +10
-0+\hline \hline +00 & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +22 & VIRTIO_ADMIN_STATUS_EINVAL & invalid command \\ +\hline +other & - & group administration command error \\ +\hline +\end{tabular} + +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}s/qialifier/qualifier/
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 134
Quoted 4 lines
admin.tex +4
-0+is reserved and set to zero by the device. + +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, +the following table describes possible \field{status_qialifier} values:s/qialifier/qualifier/
Quoted 13 lines
admin.tex +13
-0+\begin{tabular}{|l|l|l|} +\hline +Status & Name & Description \\ +\hline \hline +0x00 & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND & command error: no additional information \\ +\hline +0x01 & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE & unsupported or invalid \field{opcode} \\ +\hline +0x02 & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD & unsupported or invalid field within \field{command_specific_data} \\ +\hline +0x03 & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP & unsupported or invalid \field{group_type} \\ +\hline +0x04 & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER & unsupported or invalid \field{group_member_id} \\admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 157
Quoted 10 lines
admin.tex +10
-0+\hline +0x05-0xFFFF & - & reserved for future use \\ +\hline +\end{tabular} + +Each command uses a different \field{command_specific_data} and +\field{command_specific_result} structures and the length of +\field{command_specific_data} and \field{command_specific_result} +depends on these structures and is described separately or is +implicit in the structure description.On more thing:
Quoted 8 lines
Does the owner device see commands in order but may complete them in any order?
I think this information might be useful just to make it clear that driver authors shouldn't make assumptions about ordering and completion order, e.g. pipelining a bunch of dependent commands doesn't work because the first command is not necessarily completed before the second command is started.
Quoted 3 lines
I think this is discussed when we discuss sending commands through vq. in this section it's just about commands generally whatever the way to send them to device. no?
Okay. I'm reading this patch series linearly and the question popped into my mind here. Glad it's already covered elsewhere.
Stefan
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 47 lines · Michael S. Tsirkin
Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Parav Pandit
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 132
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMadmin.tex +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Quoted 4 lines
Only
status_qualifieris needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only thestatus_qualifierreally indicates what is going on.At a high level at the moment we have only two cases:
- ok
- invalid input supplied by drivermaybe we will have more reasons for a failure - remains to be seen.
Quoted 5 lines
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
Quoted 7 lines
Really most drivers just want to propagate errno to userspace. All the detailed reporting is for sure well intentional but in the end it is at best printed into log - end to end people just end up with a switch statement converting these to errno codes.
So we are passing them from device and this way there will be some uniformity.
Please clarify the rationale in the spec. I don't agree with it, as explained in my earlier email, but as long as its documented, people can try to follow it.
Stefan
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Stefan HajnocziQuoted 98 lines · Stefan Hajnoczi
Quoted 31 lines · Michael S. Tsirkin
Quoted 22 lines · Stefan Hajnoczi
Quoted 21 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description}Quoted 7 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 3 lines
admin.tex +2
-0+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device ofI notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Quoted 3 lines
Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are owner, "member device" and "member driver" is always called explicitly.Sounds good.
Quoted 60 lines
Quoted 59 lines
Quoted 45 lines
admin.tex +45
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table:admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 10 lines
admin.tex +10
-0+ +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}I thought all commands are "group administration commands" but this table makes it look like they are just a subset (
0x0000-0x7FFF) of group administration commands, which is a paradox.
Quoted 5 lines
Quoted 2 lines
Well the rest are reserved, maybe we will have more command types who knows. No?
I see. Does that mean the reserved commands don't need to be in the same format as struct
virtio_admin_cmd?
I am not good in predicting future ...
Quoted 11 lines
The entire section is called "Group administration commands" but I get the impression it's talking both about admin virtqueue commands in general and specifically about group administration commands.
Is it possible to structure this as follows:
Admin Commands
...common stuff... Group Administration Commands (0x0000-0x7fff) ...specific to group administration commands... Reserved (0x8000-0xffff)
What if I just write it like this:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
Quoted 4 lines
admin.tex +4
-0+\hline \hline +0x0000 - 0x7FFF & - & commands using struct virtio_admin_cmd \\ +\hline +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
will this address the comment?
Quoted 69 lines
Quoted 69 lines
Quoted 68 lines
Quoted 14 lines
admin.tex +14
-0+ +The \field{group_type} specifies the group type identifier. +The \field{group_member_id} specifies the member identifier within the group. +See section \ref{sec:Introduction / Terminology / Device group} +for the definition of the group type identifier and group member +identifier. + +The following table describes possible \field{status} values; +to simplify common implementations, they are intentionally +matching common \hyperref[intro:errno]{Linux error names and numbers}: + +\begin{tabular}{|l|l|l|} +\hline +Status (decimal) & Name & Description \\admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 130
Quoted 10 lines
admin.tex +10
-0+\hline \hline +00 & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +22 & VIRTIO_ADMIN_STATUS_EINVAL & invalid command \\ +\hline +other & - & group administration command error \\ +\hline +\end{tabular} + +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}s/qialifier/qualifier/
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 134
Quoted 4 lines
admin.tex +4
-0+is reserved and set to zero by the device. + +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, +the following table describes possible \field{status_qialifier} values:s/qialifier/qualifier/
Quoted 13 lines
admin.tex +13
-0+\begin{tabular}{|l|l|l|} +\hline +Status & Name & Description \\ +\hline \hline +0x00 & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND & command error: no additional information \\ +\hline +0x01 & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE & unsupported or invalid \field{opcode} \\ +\hline +0x02 & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD & unsupported or invalid field within \field{command_specific_data} \\ +\hline +0x03 & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP & unsupported or invalid \field{group_type} \\ +\hline +0x04 & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER & unsupported or invalid \field{group_member_id} \\admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 157
Quoted 10 lines
admin.tex +10
-0+\hline +0x05-0xFFFF & - & reserved for future use \\ +\hline +\end{tabular} + +Each command uses a different \field{command_specific_data} and +\field{command_specific_result} structures and the length of +\field{command_specific_data} and \field{command_specific_result} +depends on these structures and is described separately or is +implicit in the structure description.On more thing:
Does the owner device see commands in order but may complete them in any order?
I think this information might be useful just to make it clear that driver authors shouldn't make assumptions about ordering and completion order, e.g. pipelining a bunch of dependent commands doesn't work because the first command is not necessarily completed before the second command is started.
Quoted 8 lines
Quoted 3 lines
I think this is discussed when we discuss sending commands through vq. in this section it's just about commands generally whatever the way to send them to device. no?
Okay. I'm reading this patch series linearly and the question popped into my mind here. Glad it's already covered elsewhere.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Stefan HajnocziQuoted 56 lines · Stefan Hajnoczi
Quoted 47 lines · Michael S. Tsirkin
Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Parav Pandit
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 132
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AMadmin.tex +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Quoted 4 lines
Only
status_qualifieris needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only thestatus_qualifierreally indicates what is going on.At a high level at the moment we have only two cases:
- ok
- invalid input supplied by drivermaybe we will have more reasons for a failure - remains to be seen.
Quoted 5 lines
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
Quoted 7 lines
Really most drivers just want to propagate errno to userspace. All the detailed reporting is for sure well intentional but in the end it is at best printed into log - end to end people just end up with a switch statement converting these to errno codes.
So we are passing them from device and this way there will be some uniformity.
Quoted 5 lines
Please clarify the rationale in the spec. I don't agree with it, as explained in my earlier email, but as long as its documented, people can try to follow it.
Stefan
It's 2:2 for now, you are with Parav, me and Cornelia like it :) Or I will try to document it better.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-dev] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 72 lines · Michael S. Tsirkin
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
Quoted 8 lines
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
yes this commit log is outdated. referred to original proposal by Max which also planned to replace cvq. will update.
Quoted 34 lines
Quoted 34 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +16
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commandsQuoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 2 lines
admin.tex +2
-0+placed on different administration virtqueues, because they will +be executed with no order constraints.Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
Quoted 1 line
we have multiple vqs for that.
This reminds me of the async challenges with QEMU's QMP monitor.
Will it ever be possible to abort an in-flight command? I guess the abort command would need to be submitted on another virtqueue, but how do you identify the in-flight command that you wish to cancel?
Please clarify the blocking semantics in the spec because it wasn't clear to me.
Thanks,
Stefan
[virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 81 lines · Stefan Hajnoczi
Quoted 72 lines · Michael S. Tsirkin
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
Quoted 8 lines
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
yes this commit log is outdated. referred to original proposal by Max which also planned to replace cvq. will update.
Quoted 34 lines
Quoted 34 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +16
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commandsQuoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 2 lines
admin.tex +2
-0+placed on different administration virtqueues, because they will +be executed with no order constraints.Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
Quoted 1 line
we have multiple vqs for that.
This reminds me of the async challenges with QEMU's QMP monitor.
Will it ever be possible to abort an in-flight command? I guess the abort command would need to be submitted on another virtqueue, but how do you identify the in-flight command that you wish to cancel?
Quoted 5 lines
Please clarify the blocking semantics in the spec because it wasn't clear to me.
Thanks,
Stefan
I don't really get what does "blocking" mean. Nothing is required to block I think.
I guess for abort it is still in order but a different vq should not be needed.
Maybe we can use ID of buffer to cancel commands?
1. driver submits command as ID = 1
2. driver submits abort as ID 2
3. device sees the abort and cancels ID 2
4. device uses buffer 1
5. device uses buffer 3 - abort is complete
Device gets commands and processes them in order. It's just like scsi:
multiple queues, if you break it you get to keep both pieces.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 95 lines · Michael S. Tsirkin
Quoted 81 lines · Stefan Hajnoczi
Quoted 72 lines · Michael S. Tsirkin
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
Quoted 8 lines
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
yes this commit log is outdated. referred to original proposal by Max which also planned to replace cvq. will update.
Quoted 34 lines
Quoted 34 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +16
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commandsQuoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 2 lines
admin.tex +2
-0+placed on different administration virtqueues, because they will +be executed with no order constraints.Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
Quoted 1 line
we have multiple vqs for that.
This reminds me of the async challenges with QEMU's QMP monitor.
Will it ever be possible to abort an in-flight command? I guess the abort command would need to be submitted on another virtqueue, but how do you identify the in-flight command that you wish to cancel?
Quoted 5 lines
Please clarify the blocking semantics in the spec because it wasn't clear to me.
Thanks,
StefanI don't really get what does "blocking" mean. Nothing is required to block I think.
I guess for abort it is still in order but a different vq should not be needed.Maybe we can use ID of buffer to cancel commands?
Quoted 8 lines
1. driver submits command as ID = 1
2. driver submits abort as ID 2
3. device sees the abort and cancels ID 2
4. device uses buffer 1
5. device uses buffer 3 - abort is completeDevice gets commands and processes them in order. It's just like scsi:
multiple queues, if you break it you get to keep both pieces.
Now I'm confused because your earlier answer that "we have multiple vqs for that" implied that a command that takes 1 second will stop further processing of that vq (what I called "blocking") and the driver will have to resort to another vq if it wishes to run other commands right away. But your cancel example seems to use a single virtqueue, so that would mean commands don't block after all.
My question rephrased:
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Stefan
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 122 lines · Michael S. Tsirkin
Quoted 98 lines · Stefan Hajnoczi
Quoted 31 lines · Michael S. Tsirkin
Quoted 22 lines · Stefan Hajnoczi
Quoted 21 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description}Quoted 7 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 3 lines
admin.tex +2
-0+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device ofI notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Quoted 3 lines
Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are owner, "member device" and "member driver" is always called explicitly.Sounds good.
Quoted 60 lines
Quoted 59 lines
Quoted 45 lines
admin.tex +45
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table:admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 10 lines
admin.tex +10
-0+ +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}I thought all commands are "group administration commands" but this table makes it look like they are just a subset (
0x0000-0x7FFF) of group administration commands, which is a paradox.Quoted 5 lines
Quoted 2 lines
Well the rest are reserved, maybe we will have more command types who knows. No?
I see. Does that mean the reserved commands don't need to be in the same format as struct
virtio_admin_cmd?I am not good in predicting future ...
Quoted 11 lines
The entire section is called "Group administration commands" but I get the impression it's talking both about admin virtqueue commands in general and specifically about group administration commands.
Is it possible to structure this as follows:
Admin Commands
...common stuff... Group Administration Commands (0x0000-0x7fff) ...specific to group administration commands... Reserved (0x8000-0xffff)What if I just write it like this:
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g
Quoted 4 lines
admin.tex +4
-0+\hline \hline +0x0000 - 0x7FFF & - & commands using struct virtio_admin_cmd \\ +\hline +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\will this address the comment?
Yes, thanks.
Stefan
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 11 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.
NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Stefan
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 11 lines
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 8 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 50 lines · Stefan Hajnoczi
Quoted 35 lines · Jason Wang
Quoted 25 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 6 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Quoted 3 lines
Or device need mandate some kind of QOS here, e.g a request must be complete in some time. Otherwise we don't have sufficient reliability for using it as management task?
Yes, if all commands can be executed in bounded time then a guarantee is possible.
Here is an example where that's hard: imagine a virtio-blk device backed by network storage. When an admin queue command is used to delete a group member, any of the group member's in-flight I/O requests need to be aborted. If the network hangs while the group member is being deleted, then the device can't complete an orderly shutdown of I/O requests in a reasonable time.
Quoted 5 lines
That example shows a basic group admin command that I think Michael is about to propose. We can't avoid this problem by not making it a group admin command - it needs to be a group admin command. So I think it's likely that there will be admin commands that take an unbounded amount of time to complete. One way to achieve what you mentioned is timeouts.
I think that you're getting into device specific implementation details and I'm not sure it's necessary.
I don't think we need to abort admin commands. Admin commands can be flushed/aborted during the device reset phase. Only IO commands should have the possibility to being aborted as you mentioned in NVMe and SCSI (and potentially in virtio-blk).
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 39 lines · Jiri Pirko
Quoted 28 lines · Thu, Mar 02, 2023 at 02:05:06PM CET, [email protected] wrote:
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 7 +++--2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +0
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result}depends on these structures and is described separately or is implicit in the structure description.
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 162
Quoted 5 lines
admin.tex +5
-0+ +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have moreI admit I'm confused. You introduce a concept of admin virtqueue, which sounds quite generic to me, usable in future by much more things than "device groups", correct?
Quoted 7 lines
If yes, here you say "group administration commands" which contradics that idea. On multiple places the text this patchset introduces this very muych tights to groups. Like in struct
virtio_admin_cmdwhich contains fields very specific to groups.If no, isn't it a mistake as the admin queue should be here to handle more than just group commands?
This was my original idea to have various different types of admin operations allowed by an AQ. Not only group management. Somehow this was changed down the road when MST took ownership :)
Maybe it's fine as an initial start but this shouldn't restrict the general idea.
I had in mind things like general health commands, log commands, fw upgrade commands, etc..
Quoted 106 lines
Quoted 61 lines
admin.tex +59
-0+than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commands +placed on different administration virtqueues, because they will +be executed with no order constraints. + +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outsideadmin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 10 lines
admin.tex +10
-0+the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since the +driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueuesConflict resolution leftover.
Quoted 31 lines
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description}\item[0 to 23, and 50 to 127] Feature bits for the specific device type
-\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms-\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description}
\begin{note}
content.tex +3
-0@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
MST
Quoted 5 lines
Quoted 5 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 24 lines · Mon, Mar 06, 2023 at 08:58:12AM CET, [email protected] wrote:
Quoted 22 lines · Michael S. Tsirkin
Quoted 21 lines · Stefan Hajnoczi
Quoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
Quoted 5 lines
Quoted 2 lines
yes this commit log is outdated. referred to original proposal by Max which also planned to replace cvq.
Any advantages of allowing PF to change VF's filters? If you're referring
Which filters are you referring to?
Quoted 4 lines
provisioning, the semantic should be different:
1) using admin virtqueue to provision attributes of mac, #vpqs
2) once the provisioning is done, we should fail another provisioning request
Wait. Why are you assuming provisioning of any kind? In which patch this is covered?
Quoted 140 lines
But 1) should be different from what is used for ctrl vq which is designed for the use of the driver not the management.
Thanks
Quoted 122 lines
will update.
Quoted 33 lines
Quoted 33 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +15
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is theQuoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 3 lines
admin.tex +3
-0+responsibility of the driver to ensure ordering for commands +placed on different administration virtqueues, because they will +be executed with no order constraints.Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
we have multiple vqs for that.
Quoted 49 lines
Quoted 49 lines
admin.tex +49
-0+ +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outside +the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. +Quoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 7 lines
admin.tex +7
-0+This simplifies driver and device implementations since the +driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueuesA merge conflict line crept into the patch?
yes. I fixed it I thought but somehow it's still there :(
Quoted 17 lines
Quoted 17 lines
content.tex +2
-2@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
content.tex +3
-1+ \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} --MST
Quoted 5 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Max GurtovoyQuoted 59 lines · Max Gurtovoy
Quoted 50 lines · Stefan Hajnoczi
Quoted 35 lines · Jason Wang
Quoted 25 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 6 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Quoted 3 lines
Or device need mandate some kind of QOS here, e.g a request must be complete in some time. Otherwise we don't have sufficient reliability for using it as management task?
Yes, if all commands can be executed in bounded time then a guarantee is possible.
Here is an example where that's hard: imagine a virtio-blk device backed by network storage. When an admin queue command is used to delete a group member, any of the group member's in-flight I/O requests need to be aborted. If the network hangs while the group member is being deleted, then the device can't complete an orderly shutdown of I/O requests in a reasonable time.
Quoted 5 lines
That example shows a basic group admin command that I think Michael is about to propose. We can't avoid this problem by not making it a group admin command - it needs to be a group admin command. So I think it's likely that there will be admin commands that take an unbounded amount of time to complete. One way to achieve what you mentioned is timeouts.
Quoted 7 lines
I think that you're getting into device specific implementation details and I'm not sure it's necessary.
I don't think we need to abort admin commands. Admin commands can be flushed/aborted during the device reset phase.
Only IO commands should have the possibility to being aborted as you mentioned in NVMe and SCSI (and potentially in virtio-blk).
It's a general design issue that should be clarified now rather than being left unspecified.
I'm not saying that it must be possible to abort admin commands. There are other options, like requiring the device itself to fail a command after a timeout.
Or we could say that admin commands must complete within bounded time, but I'm not sure that is implementable for some device types like virtio-blk, virtio-scsi, and virtiofs.
Quoted 3 lines
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Forgetting about in-flight requests doesn't necessarily make them go away. It creates a race between forgotten requests and reconnection. In the worst case a forgotten write request takes effect after reconnection, causing data corruption.
Stefan
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 3 lines
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueues
The ordering seems a bit odd to me. I think that at the beginning, the admin virtqueue should be introduced with generic concepts (including the generic command structure), only after that the section introducing device group stuff and related commands.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 02/10] admin: introduce device group and related concepts
In reply to Michael S. TsirkinQuoted 1 line · Michael S. Tsirkin
Each device group has a type. For now, define one initial group:
"one initial type"
Quoted 40 lines
SR-IOV type - PCI SR-IOV virtual functions (VFs) of a given PCI SR-IOV physical function (PF). This group may contain zero or more virtio devices according to NumVFs configured.
Each device within a group has a unique identifier. This identifier is the group member identifier.
Note: one can argue both ways whether the new device group handling functionality (this and following patches) is closer
to a new device type or a new transport type.However, I expect that we will add more features in the near future. To facilitate this as much as possible of the text is located in the new admin chapter.
I did my best to minimize transport-specific text.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 2 ++ 2 files changed, 51 insertions(+) create mode 100644 admin.texadmin.tex new file +9
-0@@ -0,0 +1,49 @@ +\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups} + +It is occasionally useful to have a device control a group of +other devices. Terminology used in such cases: + +\begin{description} +\item[Device group] + or just group, includes zero or more devices. +\item[Owner device]
admin.tex line 19
Quoted 10 lines
admin.tex new file +10
-0+ or owner, the device controlling the group. +\item[Member device] + a device within a group. The owner device itself is not + a member of the group. +\item[Member identifier] + each member has this identifier, unique within the group + and used to address it through the owner device. +\item[Group type identifier] + specifies what kind of member devices there are in a + group, how is the member identifier is interpreted
s/how is/how/
admin.tex line 22
Quoted 3 lines
admin.tex new file +3
-0+ and what kind of control the owner has. + A given owner can control a single group of a given type, + thus the type and the owner together identify the group.
If a given owner can control groups of different types (which is implied by the above?), perhaps worth being explicit that this is the case, or not if that is not the intention.
admin.tex line 29
Quoted 7 lines
admin.tex new file +7
-0+ \footnote{Even though some group types only support + specific transports, group type identifiers + are global rather than transport-specific - + we don't expect a flood of new group types.} +\end{description} + +The following group types, and their identifiers, are currently specified):
Spurious close parenthesis.
Quoted 41 lines
admin.tex new file +20
-0+\begin{description} +\item[SR-IOV group type (0x1)] +This device group has a PCI Single Root I/O Virtualization +(SR-IOV) physical function (PF) device as the owner and includes +all its SR-IOV virtual functions (VFs) as members (see +\hyperref[intro:PCIe]{[PCIe]}). + +The PF device itself is not a member of the group. + +The group type identifier for this group is 0x1. + +A member identifier for this group can have a value from 0x1 to 0xFFFF +and equals the SR-IOV VF number of the member device (see +\hyperref[intro:PCIe]{[PCIe]}). + +Both owner and member devices for this group type use the Virtio +PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). +\end{description} + +content.tex +2
-0@@ -491,6 +491,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on theSignature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 18 lines · Stefan Hajnoczi
Quoted 17 lines · Michael S. Tsirkin
Quoted 15 lines · Stefan Hajnoczi
Quoted 11 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Quoted 7 lines
Quoted 2 lines
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
I'll just copy
The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue. It is the responsibility of the driver to ensure strict request ordering for commands placed on different queues, because they will be consumed with no order constraints.
replacing "request" with "admin".
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 39 lines · Jiri Pirko
Quoted 28 lines · Thu, Mar 02, 2023 at 02:05:06PM CET, [email protected] wrote:
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 7 +++--2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +0
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result}depends on these structures and is described separately or is implicit in the structure description.
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 162
Quoted 5 lines
admin.tex +5
-0+ +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have moreI admit I'm confused. You introduce a concept of admin virtqueue, which sounds quite generic to me, usable in future by much more things than "device groups", correct?
Quoted 7 lines
If yes, here you say "group administration commands" which contradics that idea. On multiple places the text this patchset introduces this very muych tights to groups. Like in struct
virtio_admin_cmdwhich contains fields very specific to groups.If no, isn't it a mistake as the admin queue should be here to handle more than just group commands?
For now, no.
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
This is what groups do.
But if we see more uses we can always add them.
I'd rather avoid being too generic though.
Quoted 105 lines
Quoted 60 lines
admin.tex +59
-0+than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commands +placed on different administration virtqueues, because they will +be executed with no order constraints. + +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outsideadmin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 10 lines
admin.tex +10
-0+the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since the +driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueuesConflict resolution leftover.
Quoted 31 lines
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description}\item[0 to 23, and 50 to 127] Feature bits for the specific device type
-\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms-\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description}
\begin{note}
content.tex +3
-0@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 1 line
MST
Quoted 5 lines
Quoted 5 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Max GurtovoyQuoted 50 lines · Max Gurtovoy
Quoted 39 lines · Jiri Pirko
Quoted 28 lines · Thu, Mar 02, 2023 at 02:05:06PM CET, [email protected] wrote:
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 7 +++--2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +0
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result}depends on these structures and is described separately or is implicit in the structure description.
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 162
Quoted 5 lines
admin.tex +5
-0+ +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have moreI admit I'm confused. You introduce a concept of admin virtqueue, which sounds quite generic to me, usable in future by much more things than "device groups", correct?
Quoted 7 lines
If yes, here you say "group administration commands" which contradics that idea. On multiple places the text this patchset introduces this very muych tights to groups. Like in struct
virtio_admin_cmdwhich contains fields very specific to groups.If no, isn't it a mistake as the admin queue should be here to handle more than just group commands?
Quoted 8 lines
This was my original idea to have various different types of admin operations allowed by an AQ. Not only group management.
Somehow this was changed down the road when MST took ownership :) Maybe it's fine as an initial start but this shouldn't restrict the general idea.I had in mind things like general health commands, log commands, fw upgrade
I'm not sure what if anything unifies these though. I looked at specific proposals on the table and this covered them, and scope was already too broad and was not converging because of this. People are not good at predicting the future, so I confidently predict that we'll make some changes down the road ;)
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to a message not in the archiveQuoted 7 lines · Jiri Pirko
Well, lots of these things (if not majority) are customary, not really written up anywhere in Linux kernel netdev world. It's most of the time just a matter of convenience I would say. For example splitted-up patch changelog. I do it, but most of the people don't (even maintainers). From time to time during review, I request it as I don't want to be lost in the changes. It is nice if the submitter tries to make reviewer's life a bit easier :)
True. With all this lore let's just agree netdev is not the easiest project to get started with, right?
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 29 lines · Michael S. Tsirkin
Quoted 18 lines · Stefan Hajnoczi
Quoted 17 lines · Michael S. Tsirkin
Quoted 15 lines · Stefan Hajnoczi
Quoted 11 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Quoted 7 lines
Quoted 2 lines
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Quoted 8 lines
I'll just copy
The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue. It is the responsibility of the driver to ensure strict request ordering for commands placed on different queues, because they will be consumed with no order constraints.replacing "request" with "admin".
That sounds like it's only about multi-queue because it says "... for commands placed on different queues". What I mentioned about a write followed by a read quest also applies within a single queue.
Can you clarify the semantics in the single queue case?
Stefan
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 37 lines · Stefan Hajnoczi
Quoted 36 lines · Michael S. Tsirkin
Quoted 18 lines · Stefan Hajnoczi
Quoted 17 lines · Michael S. Tsirkin
Quoted 15 lines · Stefan Hajnoczi
Quoted 11 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Quoted 7 lines
Quoted 2 lines
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
I'll just copy
The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue. It is the responsibility of the driver to ensure strict request ordering for commands placed on different queues, because they will be consumed with no order constraints.
Quoted 5 lines
Quoted 1 line
replacing "request" with "admin".
That sounds like it's only about multi-queue because it says "... for commands placed on different queues". What I mentioned about a write followed by a read quest also applies within a single queue.
Well that text is from the virtio scsi spec. turning it around does it need clarification there?
Quoted 3 lines
Can you clarify the semantics in the single queue case?
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees
In reply to a message not in the archiveQuoted 32 lines · David Edmondson
Quoted 15 lines · Michael S. Tsirkin
Feature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented. Do it now.
Suggested-by Halil Pasic <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)content.tex +0
-0content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B · line 121
Quoted 8 lines
content.tex +5
-0@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit. +To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according to +this version of the specification.)I find this (the bit in parenthesis) confusing.
Why are drivers not supposed to accept features that they have been offered, given that they can't know that the device cannot handle the feature that it just offered?
Quoted 7 lines
Is this alluding to the later section:
Quoted 3 lines
feature bits not described in this specification, reserved feature bits and feature bits reserved or not supported for the specific transport or the specific device type
?
exactly. how would you put this better? given an example?
Quoted 64 lines
content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
Quoted 12 lines
Likewise, it is important that
content.tex +11
-0+drivers \em{do not} accept feature bits they do not know how to handle +(even though devices are not supposed to offer them in the first place, +according to this version of the specification.) The preferred way for +handling reserved and unexpected features is that the driver ignores +them. + +In particular, this is +especially important for features limited to specific transports, +as enabling these for more transports in future versions of the +specification is highly likely to require changing the behaviour +from drivers and devices."require changed behaviour from drivers and devices".
Quoted 49 lines
Drivers and devices supporting
content.tex +26
-0+multiple transports need to carefully maintain per-transport +lists of allowed features. + \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits} The driver MUST NOT accept a feature which the device did not offer, and MUST NOT accept a feature which requires another feature which was not accepted. +The driver MUST validate the feature bits offered by the device. +The driver MUST ignore and MUST NOT accept any feature bit that is +\begin{itemize} +\item not described in this specification, +\item marked as reserved, +\item not supported for the specific transport, +\item not defined for the device type. +\end{itemize} + The driver SHOULD go into backwards compatibility mode if the device does not offer a feature it understands, otherwise MUST set the FAILED \field{device status} bit and cease initialization. +By contrast, the driver MUST NOT fail solely because a feature +it does not understand has been offered by the device. + \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits} The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK \field{device status} bit when the driver writes it. +The device MUST NOT offer feature bits corresponding to features +it would not support if accepted by the driver (even if the +driver is prohibited from accepting the feature bits by the +specification); for the sake of clarity, this refers to feature +bits not described in this specification, reserved feature bits +and feature bits reserved or not supported for the specific +transport or the specific device type, but this does not preclude +devices written to a future version of this specification from +offering such feature bits should such a specification have a +provision for devices to support the corresponding features. + If a device has successfully negotiated a set of features at least once (by accepting the FEATURES_OK \field{device status} bit during device initialization), then it SHOULDSignature · 1 line
MST
Quoted 4 lines
Quoted 4 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 8 lines
Quoted 3 lines
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueuesThe ordering seems a bit odd to me. I think that at the beginning, the admin virtqueue should be introduced with generic concepts (including the generic command structure), only after that the section introducing device group stuff and related commands.
That's intentional.
Jason is working on additional ways to pass commands to devices, using memory mapped registers.
Thus admin commands stand by themselves, and the queue is just a way to issue them.
Quoted 7 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 7 lines · Max Gurtovoy
Quoted 5 lines
Quoted 3 lines
This would allow pipelining
dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.we have multiple vqs for that.
why do we need multiple admin vqs ?
You are literally replying to a discussion with a use-case.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Quoted 22 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 5 lines
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Quoted 8 lines
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Sorry to be late to the party, I'm trying to catch up. Is it common to have cover letter for features this brief? I mean, from the cover letter, I'm totally unable to understand what you are introducing here.
Quoted 6 lines
Could you elaborate about what you are aiming to achive with this? Could you shed some usecases perhaps?
I have to be missing something obvious, but I don't get why any notion of SR-IOV could be beneficial for virtio.
Good point, I'll add a bit of motivation.
For SR-IOV, it is not unusual for PFs to excercise control over VFs. There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Quoted 56 lines
Quoted 22 lines
For example, it might make sense to split creating/destroying SIOV devices from the transport passing data from the guest - the driver would then not negotiate
VIRTIO_F_SR_IOV(which then means auto-provisioning).This is out of RFC, since we have two commands which are useful to discover supported group types (ATM can be none or SR-IOV).
Michael S. Tsirkin (10):
virtio: document forward compatibility guarantees
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueues
pci: add admin vq registers to virtio over pci
mmio: documentADMIN_VQas reserved
ccw: documentADMIN_VQas reserved
admin: command list discovery
admin: conformance clauses
ccw: document more reserved featuresQuoted 26 lines
admin.tex | 540 +++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 112 +++++++++- introduction.tex | 3 + 3 files changed, 653 insertions(+), 2 deletions(-)create mode 100644 admin.tex
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 15 lines · Mon, Mar 06, 2023 at 11:48:32PM CET, [email protected] wrote:
Quoted 5 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 8 lines
Quoted 3 lines
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueuesThe ordering seems a bit odd to me. I think that at the beginning, the admin virtqueue should be introduced with generic concepts (including the generic command structure), only after that the section introducing device group stuff and related commands.
Quoted 7 lines
That's intentional.
Jason is working on additional ways to pass commands to devices, using memory mapped registers.
Thus admin commands stand by themselves, and the queue is just a way to issue them.
Okay, in that case, woudnlt it make sense if the generic CMD text (including the generic command structure) would go first, then the "group" related cmds?
Quoted 10 lines
Quoted 10 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 26 lines · Mon, Mar 06, 2023 at 11:54:45PM CET, [email protected] wrote:
Quoted 25 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 5 lines
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Quoted 8 lines
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Sorry to be late to the party, I'm trying to catch up. Is it common to have cover letter for features this brief? I mean, from the cover letter, I'm totally unable to understand what you are introducing here.
Could you elaborate about what you are aiming to achive with this? Could you shed some usecases perhaps?
Quoted 7 lines
Quoted 3 lines
I have to be missing something obvious, but I don't get why any notion of SR-IOV could be beneficial for virtio.
Good point, I'll add a bit of motivation.
For SR-IOV, it is not unusual for PFs to excercise control over VFs.
I understand the concepts of SR-IOV. Yet I fail to see the need of such concept in virtio world. SR-IOV is very specific solution for PCI functions instantiation, and I believe that it is already considered quite limiting in many aspects. Does not make sense to me to introduce it for virtio. But again, I may be missing something crucial, I just would like to see the motivation, needs, usecases for this crystal clear, which is opposite to the current cover letter I'm afraid :/
Quoted 3 lines
There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Yeah, but why? As I asked before, what are the usecases? The fact there is interest in the community does not mean it makes sense to have it :)
Quoted 60 lines
Quoted 60 lines
Quoted 26 lines
For example, it might make sense to split creating/destroying SIOV devices from the transport passing data from the guest - the driver would then not negotiate
VIRTIO_F_SR_IOV(which then means auto-provisioning).This is out of RFC, since we have two commands which are useful to discover supported group types (ATM can be none or SR-IOV).
Michael S. Tsirkin (10):
virtio: document forward compatibility guarantees
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueues
pci: add admin vq registers to virtio over pci
mmio: documentADMIN_VQas reserved
ccw: documentADMIN_VQas reserved
admin: command list discovery
admin: conformance clauses
ccw: document more reserved featuresQuoted 26 lines
admin.tex | 540 +++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 112 +++++++++- introduction.tex | 3 + 3 files changed, 653 insertions(+), 2 deletions(-)create mode 100644 admin.tex
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 45 lines · Mon, Mar 06, 2023 at 07:40:38PM CET, [email protected] wrote:
Quoted 44 lines · Jiri Pirko
Quoted 28 lines · Thu, Mar 02, 2023 at 02:05:06PM CET, [email protected] wrote:
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 7 +++--2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +0
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result}depends on these structures and is described separately or is implicit in the structure description.
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 162
Quoted 5 lines
admin.tex +5
-0+ +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have moreI admit I'm confused. You introduce a concept of admin virtqueue, which sounds quite generic to me, usable in future by much more things than "device groups", correct?
If yes, here you say "group administration commands" which contradics that idea. On multiple places the text this patchset introduces this very muych tights to groups. Like in struct
virtio_admin_cmdwhich contains fields very specific to groups.
Quoted 4 lines
Quoted 2 lines
If no, isn't it a mistake as the admin queue should be here to handle more than just group commands?
For now, no.
Hmm, if not for now, the future exension would not be so simple, I fear.
Quoted 3 lines
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
You mean using the control queue, correct?
From one of the patch description of this patchset I understand that you cannot use control queue for this because control queue is device-specific, yet group control is device-agnostic.
My undestanding therefore was, that the admin queue you are introducing serves as a generic carrier for device-agnostic commands, in parallel for having control queue serving as a generic carrier of device-specific commands. If this is not the case, I think it would be nice to describe the exact monivation and scope of admin queue.
Quoted 3 lines
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
Could you provide examples of such commands please?
Quoted 5 lines
This is what groups do.
But if we see more uses we can always add them.I'd rather avoid being too generic though.
In that case, why not to avoid using generic terms and stay "group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues" "struct virtio_admin_cmd" -> "struct virtio_group_admin_cmd"
Etc. Helps to avoid confusion.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
+than one administration virtqueue.
[...]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinQuoted 11 lines · Mon, Mar 06, 2023 at 07:46:09PM CET, [email protected] wrote:
Quoted 7 lines · Jiri Pirko
Well, lots of these things (if not majority) are customary, not really written up anywhere in Linux kernel netdev world. It's most of the time just a matter of convenience I would say. For example splitted-up patch changelog. I do it, but most of the people don't (even maintainers). From time to time during review, I request it as I don't want to be lost in the changes. It is nice if the submitter tries to make reviewer's life a bit easier :)
True. With all this lore let's just agree netdev is not the easiest project to get started with, right?
No clue, I'm soaked in for too long already :)
Quoted 21 lines
Signature · 19 lines
-- MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to a message not in the archiveQuoted 77 lines · Mon, Mar 06, 2023 at 04:14:09PM CET, [email protected] wrote:
Quoted 72 lines · Michael S. Tsirkin
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
Is this referring to the existing virtio-net, virtio-scsi, etc control virtqueues?
Quoted 8 lines
I see the admin virtqueue as the virtqueue equivalent to transport feature bits. The admin queue does nothing device type-specific (net, scsi, etc) and instead focusses on transport and core virtio tasks.
Keeping the device-specific virtqueue separate from the admin virtqueue is simpler and has fewer potential problems. I don't think creating common infrastructure for device-specific control virtqueues across device types worthwhile or within the scope of this patch series.
yes this commit log is outdated. referred to original proposal by Max which also planned to replace cvq. will update.
Quoted 34 lines
Quoted 34 lines
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +16
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commandsQuoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 175
Quoted 2 lines
admin.tex +2
-0+placed on different administration virtqueues, because they will +be executed with no order constraints.Does "they are processed by the device in the order in which they are queued" mean only 1 admin command can be running at any given time and therefore they will complete in order? This would allow pipelining dependent commands but prevent long-running commands because the virtqueue is blocked while executing a command.
Quoted 1 line
we have multiple vqs for that.
Quoted 7 lines
why do we need multiple admin vqs ?
This is not performance oriented queue.
I don't think we need that.
I introducedF_AQ_IN_ORDERfeature bit to distinguish thein_orderlogic (same as we have for data queues).
Please re-think using this instead of complicating ourselves to multi AQ specification.
Well, out of order processing is parallel to multiple queues isn't it? The only way to achieve that would require to have the same about of queues as possible parallel out of order commands, which does not make sense, does it?
Quoted 100 lines
We have a single ctrl vq so no need to introduce multiple admin vqs...
Quoted 79 lines
Quoted 51 lines
Quoted 51 lines
admin.tex +50
-0+ +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize} + +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects +(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize} + +The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outside +the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since theQuoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 6 lines
admin.tex +6
-0+driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueuesA merge conflict line crept into the patch?
yes. I fixed it I thought but somehow it's still there :(
Quoted 17 lines
Quoted 17 lines
content.tex +2
-2@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.Quoted 10 lines
Quoted 8 lines
Quoted 8 lines
content.tex +3
-1+ \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} --MST
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c rep
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 30 lines
I'll just copy
The driver queues requests to an arbitrary request queue, and they are used by the device on that same queue. It is the responsibility of the driver to ensure strict request ordering for commands placed on different queues, because they will be consumed with no order constraints.replacing "request" with "admin".
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees
In reply to Michael S. TsirkinQuoted 35 lines · Michael S. Tsirkin
Quoted 34 lines · David Edmondson
Quoted 15 lines · Michael S. Tsirkin
Feature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented. Do it now.
Suggested-by Halil Pasic <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)content.tex +0
-0content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B · line 121
Quoted 8 lines
content.tex +5
-0@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit. +To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according to +this version of the specification.)I find this (the bit in parenthesis) confusing.
Why are drivers not supposed to accept features that they have been offered, given that they can't know that the device cannot handle the feature that it just offered?
Is this alluding to the later section:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
feature bits not described in this specification, reserved feature bits and feature bits reserved or not supported for the specific transport or the specific device type
?
exactly. how would you put this better? given an example?
Perhaps it would be enough to say:
Quoted 2 lines
(even though drivers are not supposed to accept unrecognised features in the first place even if offered, according to the specification)
"Unrecognised" is intended as a shorthand for the whole "not described, reserved, ...". Maybe "unrecognised or reserved"?
Quoted 68 lines
Quoted 68 lines
content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
Quoted 12 lines
Likewise, it is important that
content.tex +11
-0+drivers \em{do not} accept feature bits they do not know how to handle +(even though devices are not supposed to offer them in the first place, +according to this version of the specification.) The preferred way for +handling reserved and unexpected features is that the driver ignores +them. + +In particular, this is +especially important for features limited to specific transports, +as enabling these for more transports in future versions of the +specification is highly likely to require changing the behaviour +from drivers and devices."require changed behaviour from drivers and devices".
Quoted 53 lines
Drivers and devices supporting
content.tex +26
-0+multiple transports need to carefully maintain per-transport +lists of allowed features. + \drivernormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits} The driver MUST NOT accept a feature which the device did not offer, and MUST NOT accept a feature which requires another feature which was not accepted. +The driver MUST validate the feature bits offered by the device. +The driver MUST ignore and MUST NOT accept any feature bit that is +\begin{itemize} +\item not described in this specification, +\item marked as reserved, +\item not supported for the specific transport, +\item not defined for the device type. +\end{itemize} + The driver SHOULD go into backwards compatibility mode if the device does not offer a feature it understands, otherwise MUST set the FAILED \field{device status} bit and cease initialization. +By contrast, the driver MUST NOT fail solely because a feature +it does not understand has been offered by the device. + \devicenormative{\subsection}{Feature Bits}{Basic Facilities of a Virtio Device / Feature Bits} The device MUST NOT offer a feature which requires another feature which was not offered. The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK \field{device status} bit when the driver writes it. +The device MUST NOT offer feature bits corresponding to features +it would not support if accepted by the driver (even if the +driver is prohibited from accepting the feature bits by the +specification); for the sake of clarity, this refers to feature +bits not described in this specification, reserved feature bits +and feature bits reserved or not supported for the specific +transport or the specific device type, but this does not preclude +devices written to a future version of this specification from +offering such feature bits should such a specification have a +provision for devices to support the corresponding features. + If a device has successfully negotiated a set of features at least once (by accepting the FEATURES_OK \field{device status} bit during device initialization), then it SHOULDSignature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Signature · 5 lines
He caught a fleeting glimpse of a man, moving uphill pursued by a bus. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [virtio] [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 31 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +9
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description} +\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device of +a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.}
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 68
Quoted 10 lines
admin.tex +10
-0+ +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved
As before, this formatting is potentially confusing. There should be a difference between "value X has meaning Y" and "a range from X to Y".
Quoted 52 lines
admin.tex +52
-0+ */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table: + +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular} + +The \field{group_type} specifies the group type identifier. +The \field{group_member_id} specifies the member identifier within the group. +See section \ref{sec:Introduction / Terminology / Device group} +for the definition of the group type identifier and group member +identifier. + +The following table describes possible \field{status} values; +to simplify common implementations, they are intentionally +matching common \hyperref[intro:errno]{Linux error names and numbers}: + +\begin{tabular}{|l|l|l|} +\hline +Status (decimal) & Name & Description \\
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 130
Quoted 10 lines
admin.tex +10
-0+\hline \hline +00 & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +22 & VIRTIO_ADMIN_STATUS_EINVAL & invalid command \\ +\hline +other & - & group administration command error \\ +\hline +\end{tabular} + +When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier}
"status_qualifier"
Quoted 17 lines
admin.tex +17
-0+is reserved and set to zero by the device. + +When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, +the following table describes possible \field{status_qialifier} values: +\begin{tabular}{|l|l|l|} +\hline +Status & Name & Description \\ +\hline \hline +0x00 & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND & command error: no additional information \\ +\hline +0x01 & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE & unsupported or invalid \field{opcode} \\ +\hline +0x02 & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD & unsupported or invalid field within \field{command_specific_data} \\ +\hline +0x03 & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP & unsupported or invalid \field{group_type} \\ +\hline +0x04 & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER & unsupported or invalid \field{group_member_id} \\
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 157
Quoted 10 lines
admin.tex +10
-0+\hline +0x05-0xFFFF & - & reserved for future use \\ +\hline +\end{tabular} + +Each command uses a different \field{command_specific_data} and +\field{command_specific_result} structures and the length of +\field{command_specific_data} and \field{command_specific_result} +depends on these structures and is described separately or is +implicit in the structure description.
Would it make sense to add some advisory text about alignment of the fields after command_specific_data? It would be annoying if status, status_qualifier, etc. were poorly aligned for a common hardware implementation.
Quoted 22 lines
introduction.tex +3
-0@@ -68,6 +68,9 @@ \section{Normative References}\label{sec:Normative References} \phantomsection\label{intro:FUSE}\textbf{[FUSE]} & Linux FUSE interface, \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h}\\ + \phantomsection\label{intro:errno}\textbf{[errno]} & + Linux error names and numbers, + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/uapi/asm-generic/errno-base.h}\\ \phantomsection\label{intro:eMMC}\textbf{[eMMC]} & eMMC Electrical Standard (5.1), JESD84-B51, \newline\url{http://www.jedec.org/sites/default/files/docs/JESD84-B51.pdf}\\Signature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Signature · 17 lines
There is one bird in my house. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 1 line · Michael S. Tsirkin
The admin virtqueues will be the first interface to issue admin commands.
"first interface used" ?
Quoted 1 line
Currently virtio specification defines control virtqueue to manipulate
"Currently the virtio specification defines the control virtqueue"
Quoted 54 lines
features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in a more generic way, this patch introduces a new admin virtqueue interface.
We also support more than one admin virtqueue, for QoS and scalability requirements.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 7 +++-- 2 files changed, 79 insertions(+), 2 deletions(-)admin.tex +29
-0@@ -155,3 +155,77 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \field{command_specific_data} and \field{command_specific_result} depends on these structures and is described separately or is implicit in the structure description. + +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} + +An administration virtqueue of an owner device is used to submit +group administration commands. An owner device can have more +than one administration virtqueue. + +If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one +or more adminstration virtqueues. The number and locations of the +administration virtqueues are exposed by the owner device in a transport +specific manner. + +The driver submits commands by queueing them to an arbitrary +administration virtqueue, and they are processed by the +device in the order in which they are queued. It is the +responsibility of the driver to ensure ordering for commands +placed on different administration virtqueues, because they will +be executed with no order constraints. + +Administration virtqueues are used as follows: +\begin{itemize} +\item The driver submits the command using the \field{struct virtio_admin_cmd} +structure using a buffer consisting of two parts: a device-readable one followed by a +device-writable one. +\item the device-readable part includes fields from \field{opcode} +through \field{command_specific_data}. +\item the device-writeable buffer includes fields from \field{status} +through \field{command_specific_result} inclusive. +\end{itemize}
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 196
Quoted 10 lines
admin.tex +10
-0+ +For each command, this specification describes a distinct +format structure used for \field{command_specific_data} and +\field{command_specific_result}, the length of these fields +depends on the command. + +However, to ensure forward compatibility +\begin{itemize} +\item drivers are allowed to submit buffers that are longer +than what the device expects
s/what//
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 208
Quoted 12 lines
admin.tex +12
-0+(that is, longer than the length of +\field{opcode} through \field{command_specific_data}). +This allows the driver to maintain +a single format structure even if some structure fields are +unused by the device. +\item drivers are allowed to submit buffers that are shorter +than what the device expects +(that is, shorter than the length of \field{status} through +\field{command_specific_result}). This allows the device to maintain +a single format structure even if some structure fields are +unused by the driver. +\end{itemize}
Given that the previous patch declared this as a single data structure (containing both command and result), with no length indicators for the command specific data or result, how will a device know how much space the driver has allocated for the command_specific_data or command_specific result?
Quoted 12 lines
admin.tex +12
-0+The device compares the length of each part (device-readable and +device-writeable) of the buffer as submitted by driver to what it +expects and then silently truncates the structures to either the +length submitted by the driver, or the length described in this +specification, whichever is shorter. The device silently ignores +any data falling outside the shorter of the two lengths. Any +missing fields are interpreted as set to zero. + +Similarly, the driver compares the used buffer length +of the buffer to what it expects and then silently +truncates the structure to the used buffer length. +The driver silently ignores any data falling outside
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 231
Quoted 10 lines
admin.tex +10
-0+the used buffer length reported by the device. Any missing +fields are interpreted as set to zero. + +This simplifies driver and device implementations since the +driver/device can simply maintain a single large structure (such +as a C structure) for a command and its result. As new versions +of the specification are designed, new fields can be added to the +tail of a structure, with the driver/device using the full +structure without concern for versioning. +>>>>>>> 0edc690... admin: introduce virtio admin virtqueues
Merge detritus.
Quoted 35 lines
content.tex +5
-2@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -7682,6 +7682,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ(41)] This feature indicates that the device exposes one or more + administration virtqueues. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}Signature · 7 lines
MST --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Signature · 17 lines
And removed his hat, in respect of her presence. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery
In reply to Michael S. TsirkinQuoted 48 lines · Michael S. Tsirkin
Add commands to find out which commands does each group support, as well as enable their use by driver.
This will be especially useful once we have multiple group types.An alternative is per-type VQs. This is possible but will require more per-transport work. Discovery through the vq helps keep things contained.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 1 deletion(-)admin.tex +18
-1@@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti \hline opcode & Name & Command Description \\ \hline \hline -0x0000 - 0x7FFF & - & Group administration commands \\ +0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\ +0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\ +0x0002 - 0x7FFF & - & Group administration commands \\ \hline 0x8000 - 0xFFFF & - & Reserved \\ \hline@@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti depends on these structures and is described separately or is implicit in the structure description. +Before sending any administration commands to the device, the driver +needs to communicate to the device which commands it is going to +use. Initially (after reset), only two commands are assumed to be used: +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. + +Before sending any other commands for any member of a specific group to +the device, the driver queries the supported commands via +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via +VIRTIO_ADMIN_CMD_LIST_USE. + +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE +both use the following structure describing the +command opcodes: +
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 185
Quoted 10 lines
admin.tex +10
-0+\begin{lstlisting} +struct virtio_admin_cmd_list { + /* Indicates which of the below fields were returned + le64 device_admin_cmds[]; +}; +\end{lstlisting} + +This structure is an array of 64 bit values in little-endian byte +order, in which a bit is set if the specific command opcode +is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
64-bit.
Quoted 78 lines
admin.tex +68
-0+in this array corresponding to opcodes 0 to 63, +\field{device_admin_cmds[1]} is the second 64-bit value +corresponding to opcodes 64 to 127, etc. +For example, the array of size 2 including +the values 0x3 in \field{device_admin_cmds[0]} +and 0x1 in \field{device_admin_cmds[1]} indicates that only +opcodes 0, 1 and 64 are supported. +The length of the array depends on the supported opcodes - it is +large enough to include bits set for all supported opcodes, +that is the length can be calculated by starting with the largest +supported opcode adding one, dividing by 64 and rounding up. +In other words, for +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the +length of \field{command_specific_result} and +\field{command_specific_data} respectively will be +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division +with round up and \field{max_cmd} is the largest available command opcode. + +The array is also allowed to be larger and to additionally +include an arbitrary number of all-zero entries. + +Accordingly, bits 0 and 1 corresponding to opcode 0 +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1 +(VIRTIO_ADMIN_CMD_LIST_USE) are +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY. + +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0. +The \field{group_member_id} is unused. It is set to zero by driver. +This command has no command specific data. +The device, upon success, returns a result in +\field{command_specific_result} in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands supported for the given group. + + +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode} +is set to 0x1. +The \field{group_member_id} is unused. It is set to zero by driver. +The \field{command_specific_data} is in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands used by the driver. +This command has no command specific result. + +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to +query the list of commands valid for this group and before sending +any commands for any member of a group. + +The driver then enables use of some of the opcodes by sending to +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is +both understood and used by the driver. + +If the device supports the command list used by the driver, the +device completes the command with status VIRTIO_ADMIN_STATUS_OK. +If the device does not support the command list, the device +completes the command with status +VIRTIO_ADMIN_STATUS_INVALID_FIELD. + +Note: drivers are assumed not to set bits in device_admin_cmds +if they are not familiar with how the command opcode +is used, since devices could have dependencies between +command opcodes. + +It is assumed that all members in a group support and are used +with the same list of commands. However, for owner devices +supporting multiple group types, the list of supported commands +might differ between different group types. + \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} An administration virtqueue of an owner device is used to submitSignature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Signature · 17 lines
If I could buy my reasoning, I'd pay to lose. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
In reply to Michael S. TsirkinQuoted 32 lines · Michael S. Tsirkin
Add conformance clauses for admin commands and admin virtqueues.
Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 215 insertions(+), 1 deletion(-)admin.tex +17
-0@@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti supporting multiple group types, the list of supported commands might differ between different group types. +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} + +The device MUST validate \field{opcode}, \field{group_type} and +\field{group_member_id}, and if any of these has an invalid or +unsupported value, set \field{status} to +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier} +accordingly: +\begin{itemize} +\item if \field{group_type} is invalid, \field{status_qualifier} + MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; +\item otherwise, if \field{opcode} is invalid, + \field{status_qualifier} MUST be set to + VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE; +\item otherwise, if \field{group_member_id} is used by the + specific command and is invalid, \field{status_qualifier} MUST be + set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER. +\end{itemize}
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 280
Quoted 10 lines
admin.tex +10
-0+ +If a command completes successfully, the device MUST set +\field{status} to VIRTIO_ADMIN_STATUS_OK. + +If a command fails, the device MUST set +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK. + +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the +device state MUST NOT change, that is the command MUST NOT have +any side effects on the device, in particular the device MUST not
MUST NOT
Quoted 18 lines
admin.tex +18
-0+enter an error state as a result of this command. + +If a command fails, the device state generally SHOULD NOT change, +as far as possible. + +The device MAY enforce additional restrictions and dependencies on +opcodes used by the driver and MAY fail the command +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD +if the list of commands used violate internal device dependencies. + +If the device supports multiple group types, commands for each group +type MUST operate independently of each other, in particular, +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY +for different group types. + +After reset, if the device supports a given group type +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 308
Quoted 10 lines
admin.tex +10
-0+the device MUST assume +that the list of legal commands used by the driver consists of +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. + +After completing VIRTIO_ADMIN_CMD_LIST_USE successfully, +the device MUST set the list of legal commands used by the driver +to the one supplied in \field{command_specific_data}. + +The device MUST set the list of legal commands used by the driver +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.
Are these last two paragraphs not saying the same thing?
Quoted 52 lines
admin.tex +52
-0+The device MUST validate commands against the list used by +the driver and MUST fail any commands not in the list with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +and \field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE. + +The list of supported commands MUST NOT shrink (but MAY expand): +after reporting a given command as supported through +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it +as unsupported. Further, after a given set of commands has been +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a +device or system reset the device SHOULD complete successfully +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails +after a device or system reset, the device MUST not fail it +solely because of the command list used. Failure to do so would +interfere with resuming from suspend and error recovery. + +When processing a command with the SR-IOV group type, +if the device does not have an SR-IOV Extended Capability or +if \field{VF Enable} is clear +then the device MUST fail all commands with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; +otherwise, if \field{group_member_id} is not +between $1$ and \field{NumVFs} inclusive, +the device MUST fail all commands with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER; +\field{NumVFs}, \field{VF Migration Capable} and +\field{VF Enable} refer to registers within the SR-IOV Extended +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. + +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} + +The driver MAY discover whether device supports a specific group type +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching +\field{group_type}. + +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE +and wait for it to be completed with status +VIRTIO_ADMIN_STATUS_OK before issuing any commands +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY +and VIRTIO_ADMIN_CMD_LIST_USE). + +The driver SHOULD NOT set bits in device_admin_cmds +if it is not familiar with how the command opcode +is used, as dependencies between command opcodes might exist. + +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 10 lines
admin.tex +10
-0+the use of any commands not previously reported as +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY. + +The driver MUST NOT use any commands for a given group type +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct +list of command opcodes and group type. + +The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.
This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE are the only commands initially available.
Quoted 24 lines
admin.tex +20
-0+The driver MUST handle a command error with a reserved \field{status} +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +(except possibly for different error reporting/diagnostic messages). + +The driver MUST handle a command error with a reserved +\field{status_qualifier} value in the same way as +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for +different error reporting/diagnostic messages). + +When sending commands with the SR-IOV group type, +the driver specify a value for \field{group_member_id} +between $1$ and \field{NumVFs} inclusive, +the driver MUST also make sure that as long as any such command +is outstanding, \field{VF Migration Capable} is clear and +\field{VF Enable} is set; +\field{NumVFs}, \field{VF Migration Capable} and +\field{VF Enable} refer to registers within the SR-IOV Extended +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. + \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues} An administration virtqueue of an owner device is used to submit
admin.tex \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic · line 326
Quoted 4 lines
admin.tex +0
-1@@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic of the specification are designed, new fields can be added to the tail of a structure, with the driver/device using the full structure without concern for versioning. ->>>>>>> 0edc690... admin: introduce virtio admin virtqueues
Oh, here is where it got removed :-)
Quoted 50 lines
admin.tex +50
-0+ +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues} + +The device MUST support device-readable and device-writeable buffers +shorter than described in this specification, by +\begin{enumerate} +\item acting as if any data that would be read outside the +device-readable buffers is set to zero, and +\item discarding data that would be written outside the +specified device-writeable buffers. +\end{enumerate} + +The device MUST support device-readable and device-writeable buffers +longer than described in this specification, by +\begin{enumerate} +\item ignoring any data in device-readable buffers outside +the expected length, and +\item only writing the expected structure to the device-writeable +buffers, ignoring any extra buffers, and reporting the +actual length of data written, in bytes, +as buffer used length. +\end{enumerate} + +The device SHOULD initialize the device-writeable buffer +up to the length of the structure described by this specification or +the length of the buffer supplied by the driver (even if the buffer is +all set to zero), whichever is shorter. + +The device MUST NOT fail a command solely because the buffers +provided are shorter or longer than described in this +specification. + +The device MUST initialize the device-writeable part of +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in +size. + +The device MUST initialize \field{status} in \field{struct +virtio_admin_cmd}. + +The device MUST process commands on a given administration virtqueue +in the order in which they are queued. + +If multiple administration virtqueues have been configured, +device MAY process commands on distinct virtqueues with +no order constraints. + +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues} + +The driver MAY supply device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are longer than described in
admin.tex \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic · line 524
Quoted 10 lines
admin.tex +10
-0+this specification. + +The driver SHOULD supply device-readable part of +\field{struct virtio_admin_cmd} that is at least as +large as the structure described by this specification +(even if the structure is all set to zero). + +The driver MUST supply both device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit +in length.
This addresses my question on a previous patch.
Quoted 34 lines
admin.tex +15
-0+The device MUST supply both device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are larger than zero in +length. However, \field{command_specific_data} and +\field{command_specific_result} MAY be zero in length, unless +specified otherwise for the command. + +The driver MUST NOT assume that the device will initialize the whole +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead, +the driver MUST act as if the structure +outside the part of the buffer used by the device +is set to zero. + +If multiple administration virtqueues have been configured, +the driver MUST ensure ordering for commands +placed on different administration virtqueues.Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Signature · 17 lines
Music has magic, it's good clear syncopation. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Stefan HajnocziQuoted 129 lines · Fri, Mar 03, 2023 at 09:23:14PM CET, [email protected] wrote:
Quoted 128 lines · Michael S. Tsirkin
Quoted 98 lines · Stefan Hajnoczi
Quoted 31 lines · Michael S. Tsirkin
Quoted 22 lines · Stefan Hajnoczi
Quoted 21 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description}Quoted 7 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 3 lines
admin.tex +2
-0+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device ofI notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Quoted 3 lines
Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are owner, "member device" and "member driver" is always called explicitly.Sounds good.
Quoted 60 lines
Quoted 59 lines
Quoted 45 lines
admin.tex +45
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table:admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 10 lines
admin.tex +10
-0+ +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}I thought all commands are "group administration commands" but this table makes it look like they are just a subset (
0x0000-0x7FFF) of group administration commands, which is a paradox.Quoted 5 lines
Quoted 2 lines
Well the rest are reserved, maybe we will have more command types who knows. No?
I see. Does that mean the reserved commands don't need to be in the same format as struct
virtio_admin_cmd?I am not good in predicting future ...
Quoted 11 lines
The entire section is called "Group administration commands" but I get the impression it's talking both about admin virtqueue commands in general and specifically about group administration commands.
Is it possible to structure this as follows:
Admin Commands
...common stuff... Group Administration Commands (0x0000-0x7fff) ...specific to group administration commands... Reserved (0x8000-0xffff)What if I just write it like this:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
admin.tex +4
-0+\hline \hline +0x0000 - 0x7FFF & - & commands using struct virtio_admin_cmd \\ +\hline +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
Quoted 3 lines
Quoted 1 line
will this address the comment?
Yes, thanks.
What is the reason for having this table? I fail to see it.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c rep
Quoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
Group member creation might have similar issues if it involves acquiring remote resources (e.g. connecting to a Ceph cluster or allocating ports on a distributed network switch). It can be impossible to defer resource acquisition/initialization because because VIRTIO devices must be available as soon as the driver can see them (i.e. how do populate Configuration Space fields if you don't have the details of the resource yet?).
So I have raised two questions:
1. What are the admin queue command completion semantics: in-order or out-of-order command completion?
2. Will there be long-running commands and how will we deal with them when they hang?
Stefan
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 51 lines · Tue, Mar 07, 2023 at 03:39:11PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c repQuoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 8 lines
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
I see. Then I believe that the device should handle the dependencies. Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member B
The device according to internal implementation can either serialize the 2 group member creations or do it in parallel, if it supports it.
Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member A
Here the serialization is necessary and the device is the one to take care of it.
Makes sense?
Quoted 4 lines
Group member creation might have similar issues if it involves acquiring remote resources (e.g. connecting to a Ceph cluster or allocating ports on a distributed network switch). It can be impossible to defer resource
Sidetrack: this is really fuzzy to me, how the new member is going to be plugged into backend (network). Over the time, we learned that the creation of device from the other side (switch side) makes more sense. That is why I asked for motivation to introduce this infra.
Quoted 9 lines
acquisition/initialization because because VIRTIO devices must be available as soon as the driver can see them (i.e. how do populate Configuration Space fields if you don't have the details of the resource yet?).
So I have raised two questions:
1. What are the admin queue command completion semantics: in-order or out-of-order command completion?
I would add "dependencies/serialization" here.
Quoted 3 lines
2. Will there be long-running commands and how will we deal with them when they hang?
Yeah, sounds legit to define it in spec.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 2 lines · Jiri Pirko
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Because the virtqueue is just a virtqueue - a way to pass around commands. Admin commands are used whenever one needs to talk to one device about another one.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 1 line · Jiri Pirko
Hmm, if not for now, the future exension would not be so simple, I fear.
Without knowing what it is I can't say.
Quoted 6 lines
Quoted 4 lines
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
You mean using the control queue, correct?
Depends on the device type. network devices have a control queue, yes.
Quoted 9 lines
Quoted 1 line
From one of the patch description of this patchset I understand that you
cannot use control queue for this because control queue is device-specific, yet group control is device-agnostic.
My undestanding therefore was, that the admin queue you are introducing serves as a generic carrier for device-agnostic commands, in parallel for having control queue serving as a generic carrier of device-specific commands. If this is not the case, I think it would be nice to describe the exact monivation and scope of admin queue.
Nope unfortunately. This queue is just a carrier for admin commands. admin commands are commands that talk to one device about other devices. There's clearly no mechanism in the spec to do that, so we plug this hole.
Quoted 6 lines
Quoted 4 lines
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
Could you provide examples of such commands please?
For example a common feature is to program a vlan and have device put a given VF inside this vlan.
In a virtualization scenario host controls this vlan programming giving the network a measure of protection from VFs. If a VF is passed through to a VM, IOMMU limits VFs to only access guest memory so host has to do this programming through a PF.
Quoted 5 lines
Quoted 5 lines
This is what groups do.
But if we see more uses we can always add them.
Quoted 8 lines
Quoted 1 line
I'd rather avoid being too generic though.
In that case, why not to avoid using generic terms and stay "group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues" "structvirtio_admin_cmd" -> "structvirtio_group_admin_cmd"Etc. Helps to avoid confusion.
Sure, I tried to do that but missed some opportunities. Will address.
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+than one administration virtqueue.
[...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 21 lines · Jiri Pirko
Quoted 20 lines · Mon, Mar 06, 2023 at 11:48:32PM CET, [email protected] wrote:
Quoted 5 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 8 lines
Quoted 3 lines
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueuesThe ordering seems a bit odd to me. I think that at the beginning, the admin virtqueue should be introduced with generic concepts (including the generic command structure), only after that the section introducing device group stuff and related commands.
That's intentional.
Jason is working on additional ways to pass commands to devices, using memory mapped registers.
Quoted 7 lines
Quoted 2 lines
Thus admin commands stand by themselves, and the queue is just a way to issue them.
Okay, in that case, woudnlt it make sense if the generic CMD text (including the generic command structure) would go first, then the "group" related cmds?
there are not other commands though - they are all group related.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 35 lines · Jiri Pirko
Quoted 26 lines · Mon, Mar 06, 2023 at 11:54:45PM CET, [email protected] wrote:
Quoted 25 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 5 lines
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Quoted 8 lines
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Sorry to be late to the party, I'm trying to catch up. Is it common to have cover letter for features this brief? I mean, from the cover letter, I'm totally unable to understand what you are introducing here.
Could you elaborate about what you are aiming to achive with this? Could you shed some usecases perhaps?
Quoted 7 lines
Quoted 3 lines
I have to be missing something obvious, but I don't get why any notion of SR-IOV could be beneficial for virtio.
Good point, I'll add a bit of motivation.
For SR-IOV, it is not unusual for PFs to excercise control over VFs.
Quoted 7 lines
I understand the concepts of SR-IOV. Yet I fail to see the need of such concept in virtio world. SR-IOV is very specific solution for PCI functions instantiation, and I believe that it is already considered quite limiting in many aspects. Does not make sense to me to introduce it for virtio. But again, I may be missing something crucial, I just would like to see the motivation, needs, usecases for this crystal clear, which is opposite to the current cover letter I'm afraid :/
First people are asking for it because it's out there, however limiting it is. In fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is legacy and there's no need to support.
Quoted 8 lines
Quoted 4 lines
There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Yeah, but why? As I asked before, what are the usecases? The fact there is interest in the community does not mean it makes sense to have it :)
If people want to build such hardware it will need some interface. Far better to have it standard.
But generally e.g. intel already said they will reuse this same structure with a different group type for SIOV support. I'll mention this in the cover letter.
Quoted 60 lines
Quoted 60 lines
Quoted 60 lines
Quoted 26 lines
For example, it might make sense to split creating/destroying SIOV devices from the transport passing data from the guest - the driver would then not negotiate
VIRTIO_F_SR_IOV(which then means auto-provisioning).This is out of RFC, since we have two commands which are useful to discover supported group types (ATM can be none or SR-IOV).
Michael S. Tsirkin (10):
virtio: document forward compatibility guarantees
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueues
pci: add admin vq registers to virtio over pci
mmio: documentADMIN_VQas reserved
ccw: documentADMIN_VQas reserved
admin: command list discovery
admin: conformance clauses
ccw: document more reserved featuresQuoted 26 lines
admin.tex | 540 +++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 112 +++++++++- introduction.tex | 3 + 3 files changed, 653 insertions(+), 2 deletions(-)create mode 100644 admin.tex
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] RE: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 6:47 PMQuoted 2 lines
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
We'll add them when there's a use-case.
Errno.h is self-explanatory for the use case.
A device may want the driver to retry the command as it is busy.
Quoted 6 lines
Quoted 2 lines
And with that,
VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBERmoves underENODEV error code.
No I think that belongs in in EINVAL - command is malformed.
Command is not malformed.
The command prepared is correct, the VF number in the well-formed command which has an invalid VF device number.
Quoted 3 lines
ENODEV is more "The wrong type of device was given to a function that expects a particular sort of device."
Yes. Hence the ENODEV when the VF device is accessed is not valid.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] RE: [PATCH v10 03/10] admin: introduce group administration commands
In reply to a message not in the archiveQuoted 78 lines
From: Max Gurtovoy <[email protected]>
Sent: Monday, March 6, 2023 6:05 AMQuoted 56 lines · Michael S. Tsirkin
Quoted 55 lines · Stefan Hajnoczi
Quoted 53 lines · Michael S. Tsirkin
Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Parav Pandit
Review
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM(quoted diff) +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, +\field{status_qialifier} is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following +table describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Quoted 4 lines
Only
status_qualifieris needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only thestatus_qualifierreally indicates what is going on.At a high level at the moment we have only two cases:
- ok
- invalid input supplied by drivermaybe we will have more reasons for a failure - remains to be seen.
Quoted 5 lines
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
Really most drivers just want to propagate errno to userspace. All the detailed reporting is for sure well intentional but in the end it is at best printed into log - end to end people just end up with a switch statement converting these to errno codes.
So we are passing them from device and this way there will be some uniformity.Quoted 8 lines
Quoted 5 lines
Please clarify the rationale in the spec. I don't agree with it, as explained in my earlier email, but as long as its documented, people can try to follow it.
Stefan
It's 2:2 for now, you are with Parav, me and Cornelia like it :) Or I will try to document it better.
I don't understand this
status_qualifieras well and it wasn't included in my original patch set.
I vote for "status" that describe generic status codes and "command_specific_error" that should be inspected by the driver only if "status" is set to "VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR". We discussed this so many times before (and already agreed IIRC) and adding this new qualifiers mechanism sounds not right to me and not intuitive for device and driver developers.
Quoted 8 lines
I suggest:
1.
VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE
2.VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
3.VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP
4.VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER
5.VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).
This is also a good proposal that 3 positives.
1. Tells exactly what was wrong with command or with device
2. generic enough to convert into abstract OS-specific error codes with negligible cost (switch-case)
3. can define command-specific error details
[virtio-comment] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Michael S. TsirkinQuoted 16 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Friday, March 3, 2023 3:34 AMQuoted 3 lines · Parav Pandit
Did you miss reviewed-by from [1] or this is an old series reposted?
[1]
https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.htmlAs a general rule, we don't strictly need to track reviewed by since there's a ballot (and presumably people review before voting).
People also tack on Signed-off-by: (and I do it too) but as long as we don't document what it means it's kind of vague, and the process of subscribing to the mailing list is a kind of replacement.
Quoted 8 lines
If you think everyone needs to follow practices like netdev does, we really need something written up, and agree on it.
E.g. I work on the linux kernel too, so I can copy practices from there, but even linux isn't uniform.
And I wonder whether it's worth it - it definitely makes contributing to Linux harder, and even within Linux it pushes contributors away.
The number of virtio spec contributors are in order of magnitude less than Linux kernel or just the Linux netdev.
Patch split up reduces lot of time author and reviewers.
For example, interrupt moderation at v10 can be very easily split up where prep patch can have RB tag and v11 doesn't need reviewers and author's time. Your work here with 10 patches is yet another good example of it.
I remember Max and I started with 6 patches... and current 10 are more useful way to split them.
Quoted 9 lines
At least for Linux
tracking history in a precise way is extremely important since it's helpful with stability. Spec is very different.Until we have a good contribution documentation I think we should not ask people to follow a pseudo Linux work flow with requests like "please split this patchset up" or "track changes across patch versions"
simply because there's no good docs to teach people what exactly is the best practice.
Yes, I wanted to update the contributing document. It is in my to-do list.
But given the small crowd of contributors right now, almost everyone is familiar with the RB, ack tag. At moment it has two main reasons.
1. Acknowledge the work and efforts that go in the review
2. When in question, reach out later to the spec author or reviewers to know about context/design etc.
3. Same reviewer doesn't need to go through the patch which already has RB tag of him/her.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 77 lines · Jiri Pirko
Quoted 51 lines · Tue, Mar 07, 2023 at 03:39:11PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c repQuoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 8 lines
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
I see. Then I believe that the device should handle the dependencies. Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member BThe device according to internal implementation can either serialize the 2 group member creations or do it in parallel, if it supports it.
Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member A
Quoted 4 lines
Here the serialization is necessary and the device is the one to take care of it.
Makes sense?
Yes, I understand. The spec would need to define ordering rules for specific commands and the device must implement them. It allows the driver to pipeline commands while also allowing out-of-order completion (parallelism) in some cases. The disadvantage of this approach is complexity in the spec and implementations.
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
Quoted 9 lines
Quoted 4 lines
Group member creation might have similar issues if it involves acquiring remote resources (e.g. connecting to a Ceph cluster or allocating ports on a distributed network switch). It can be impossible to defer resource
Sidetrack: this is really fuzzy to me, how the new member is going to be plugged into backend (network). Over the time, we learned that the creation of device from the other side (switch side) makes more sense. That is why I asked for motivation to introduce this infra.
Michael, have you already thought about this?
Quoted 23 lines
Quoted 9 lines
acquisition/initialization because because VIRTIO devices must be available as soon as the driver can see them (i.e. how do populate Configuration Space fields if you don't have the details of the resource yet?).
So I have raised two questions:
1. What are the admin queue command completion semantics: in-order or out-of-order command completion?
I would add "dependencies/serialization" here.
Quoted 3 lines
2. Will there be long-running commands and how will we deal with them when they hang?
Yeah, sounds legit to define it in spec.
Quoted 2 lines
Stefan
[virtio-comment] RE: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 7 lines
From: [email protected] <[email protected]> On Behalf Of Stefan Hajnoczi
I'm not saying that it must be possible to abort admin commands. There are other options, like requiring the device itself to fail a command after a timeout.
An example where EAGAIN, EBUSY can be useful.
RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 10 lines
From: Stefan Hajnoczi <[email protected]>
Sent: Tuesday, March 7, 2023 2:04 PMAn alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
+1.
This has best of both.
1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
2. Ability to execute multiple unrelated commands using a single AQ.
3. stateless device in AQ command execution
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 3 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 7, 2023 12:20 PM
[..]
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
For SR-IOV, it is not unusual for PFs to excercise control over VFs.
I understand the concepts of SR-IOV. Yet I fail to see the need of such concept in virtio world. SR-IOV is very specific solution for PCI functions instantiation, and I believe that it is already considered quite limiting in many aspects. Does not make sense to me to introduce it for virtio. But again, I may be missing something crucial, I just would like to see the motivation, needs, usecases for this crystal clear, which is opposite to the current cover letter I'm afraid :/
Quoted 4 lines
First people are asking for it because it's out there, however limiting it is. In fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is legacy and there's no need to support.
Jiri and I discussed offline today.
SR-IOV is widely used today and in the coming future up to thousands of devices. Many vendors are building it or already built.
SIOV as it stands today is still being crafted at various levels iommu, os, device, pci and more. Not there yet fully.
Jiri and I built the Linux kernel interfaces for SFs which are equivalent to SIOVs that overcome some of the SR-IOV limitations. And Jiri's suggestion was to leverage some of the vision and practice from there.
In other words, can AQ command be useful tomorrow for doing SIOV device add/remove, and provisioning from non-owning PF? The way AQ is crafted today is not there yet, but in the future, it can be extended.
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Yeah, but why? As I asked before, what are the usecases? The fact there is interest in the community does not mean it makes sense to have it :)
Quoted 8 lines
If people want to build such hardware it will need some interface. Far better to have it standard.
But generally e.g. intel already said they will reuse this same structure with a different group type for SIOV support.
I'll mention this in the cover letter.
We have the following already identified use cases of a device agnostic VQ.
1. SR-IOV VF device provisioning at virtio layer (features and config)
2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for example, MSI-X vectors
3. SR-IOV VF Live migration
4. Above #1 to #3 for SIOV devices instead of SRIOV devices
5. cmd q interface for PF/VF/SF/SIOV to transport their own features, config command over queue interface without any parent device
#1 to #3 are well described in past. So, It is worth covering them in the cover letter and things will be clear for new eyes. #4 is broad use case and like #1 to #3 that will eventually happen.
#5 likely needs more discussion, we can differ to a later point because AQ needs to progress in spec 1.3.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to a message not in the archiveQuoted 64 lines · Max Gurtovoy
Quoted 61 lines · Michael S. Tsirkin
Quoted 60 lines · Stefan Hajnoczi
Quoted 47 lines · Michael S. Tsirkin
Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Parav Pandit
Review
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM(quoted diff) +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Quoted 4 lines
Only
status_qualifieris needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only thestatus_qualifierreally indicates what is going on.At a high level at the moment we have only two cases:
- ok
- invalid input supplied by drivermaybe we will have more reasons for a failure - remains to be seen.
Quoted 5 lines
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
Quoted 7 lines
Really most drivers just want to propagate errno to userspace. All the detailed reporting is for sure well intentional but in the end it is at best printed into log - end to end people just end up with a switch statement converting these to errno codes.
So we are passing them from device and this way there will be some uniformity.Please clarify the rationale in the spec. I don't agree with it, as explained in my earlier email, but as long as its documented, people can try to follow it.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Stefan
It's 2:2 for now, you are with Parav, me and Cornelia like it :) Or I will try to document it better.
I don't understand this
status_qualifieras well and it wasn't included in my original patch set.
Sounds like you feel I should drop your S.O.B - is this the complaint? I wanted to give attribution since I started with that but sure, no problem.
Quoted 7 lines
I vote for "status" that describe generic status codes and "
command_specific_error" that should be inspected by the driver only if "status" is set to "VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR". We discussed this so many times before (and already agreed IIRC) and adding this new qualifiers mechanism sounds not right to me and not intuitive for device and driver developers.
Quoted 8 lines
I suggest:
1.
VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE
2.VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
3.VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP
4.VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER
5.VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).
I don't think it's a good idea, we'll have to agree to disagree.
The point is simple. We can have a detailed virtio specific error. Nice for debugging most drivers won't know what to do with it. This is the status_qualifier.
Very detailed but
generally drivers will just have a giant switch statement translating it to a simple error code for userspace.
So to save everyone work, we also added "status"
a generic kind of error class that is easy to pass to userspace with a small switch statement.
COMMAND_SPECIFIC_ERR is just way too much detail - commands generally just should not fail it's a quality of implementation issue.
Quoted 3 lines
Quoted 3 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] RE: [PATCH v10 03/10] admin: introduce group administration commands
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 7, 2023 7:34 PMCOMMAND_SPECIFIC_ERRis just way too much detail - commands generally just should not fail it's a quality of implementation issue.
I disagree partially.
1. Hw device != sw_hypervisor device.
Device may fail or error out that may need want sw driver to retry. Like Stefan's example, it may need to return timeout/retry intermittently. Doesnt means the device is broken at that point.
2. A device implementation may not have imposed a certain locking scheme to synchronize VF enablement with VF provisioning. ENODEV can reflect two commands not synchronized.
So Boolean 0 = success, 22 = error is not the right way to craft the spec.
Many times, those sub-error codes are good indications of what may have gone wrong in the field. Useful for the quality issue you pointed out to debug.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
Quoted 13 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 7, 2023 7:34 PMCOMMAND_SPECIFIC_ERRis just way too much detail - commands generally just should not fail it's a quality of implementation issue.I disagree partially.
1. Hw device !=
sw_hypervisordevice.
Device may fail or error out that may need want sw driver to retry. Like Stefan's example, it may need to return timeout/retry intermittently. Doesnt means the device is broken at that point.
Quoted 6 lines
2. A device implementation may not have imposed a certain locking scheme to synchronize VF enablement with VF provisioning. ENODEV can reflect two commands not synchronized.
So Boolean 0 = success, 22 = error is not the right way to craft the spec.
Many times, those sub-error codes are good indications of what may have gone wrong in the field. Useful for the quality issue you pointed out to debug.
Maybe, but I think we can just leave this stuff for later. Too much hand-waving, when we add commands that actually need this kind of ability that is when we will add the relevant error codes.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] RE: [PATCH v10 03/10] admin: introduce group administration commands
Quoted 23 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 7, 2023 8:06 PMQuoted 6 lines · Parav Pandit
Quoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 7, 2023 7:34 PMCOMMAND_SPECIFIC_ERRis just way too much detail - commands generally just should not fail it's a quality of implementation issue.Quoted 8 lines
I disagree partially.
1. Hw device !=
sw_hypervisordevice.
Device may fail or error out that may need want sw driver to retry. Like Stefan's example, it may need to return timeout/retry intermittently. Doesnt means the device is broken at that point.2. A device implementation may not have imposed a certain locking scheme
to synchronize VF enablement with VF provisioning.
Quoted 2 lines
ENODEV can reflect two commands not synchronized.
Quoted 8 lines
Quoted 2 lines
So Boolean 0 = success, 22 = error is not the right way to craft the spec. Many times, those sub-error codes are good indications of what may have
gone wrong in the field.
Quoted 1 line
Useful for the quality issue you pointed out to debug.
Maybe, but I think we can just leave this stuff for later. Too much hand-waving, when we add commands that actually need this kind of ability that is when we will add the relevant error codes.
Sounds good to me.
I thought the whole reason you wanted to use errno.h was to reuse all applicable error codes from it.
And so doing it in one go appeared natural to me.
As long as adding new error codes doesn't need new feature bits or new/extension of list commands, adding later looks good to me. As long as we agree that adding more error codes to reflect specific errors is fine, (not Boolean 22), seems like a good direction.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 24 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 7, 2023 12:20 PM[..]
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
For SR-IOV, it is not unusual for PFs to excercise control over VFs.
I understand the concepts of SR-IOV. Yet I fail to see the need of such concept in virtio world. SR-IOV is very specific solution for PCI functions instantiation, and I believe that it is already considered quite limiting in many aspects. Does not make sense to me to introduce it for virtio. But again, I may be missing something crucial, I just would like to see the motivation, needs, usecases for this crystal clear, which is opposite to the current cover letter I'm afraid :/
Quoted 4 lines
First people are asking for it because it's out there, however limiting it is. In fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is legacy and there's no need to support.
Jiri and I discussed offline today.
SR-IOV is widely used today and in the coming future up to thousands of devices. Many vendors are building it or already built.
Quoted 8 lines
SIOV as it stands today is still being crafted at various levels iommu, os, device, pci and more. Not there yet fully.
Jiri and I built the Linux kernel interfaces for SFs which are equivalent to SIOVs that overcome some of the SR-IOV limitations. And Jiri's suggestion was to leverage some of the vision and practice from there.
In other words, can AQ command be useful tomorrow for doing SIOV device add/remove, and provisioning from non-owning PF? The way AQ is crafted today is not there yet, but in the future, it can be extended.
There's already a proposal to use the queue as a transport. Any device that can't use a transport specific configuration or provisioning can benefit from that. SIOV is only one of the users for that.
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Yeah, but why? As I asked before, what are the usecases? The fact there is interest in the community does not mean it makes sense to have it :)
Quoted 8 lines
If people want to build such hardware it will need some interface. Far better to have it standard.
But generally e.g. intel already said they will reuse this same structure with a different group type for SIOV support.
I'll mention this in the cover letter.
Quoted 5 lines
We have the following already identified use cases of a device agnostic VQ.
1. SR-IOV VF device provisioning at virtio layer (features and config)
2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for example, MSI-X vectors
The virtio layer device provisioning is transport independent. I don't see anything that is PCI specific there. Even the MSI-X could be reused by other transport.
Quoted 1 line
3. SR-IOV VF Live migration
I think we don't want a partial solution for live migration that only works for PCI VF devices and L1. Instead, there needs to be a general device facility section and the admin virtqueue could be one of the interfaces for the driver to use the facility.
Quoted 1 line
4. Above #1 to #3 for SIOV devices instead of SRIOV devices
If we make the above as a general facility, it would not be hard to re-use them for transport virtqueue or SIOV.
Thanks
Quoted 8 lines
5. cmd q interface for PF/VF/SF/SIOV to transport their own features, config command over queue interface without any parent device
#1 to #3 are well described in past. So, It is worth covering them in the cover letter and things will be clear for new eyes. #4 is broad use case and like #1 to #3 that will eventually happen.
#5 likely needs more discussion, we can differ to a later point because AQ needs to progress in spec 1.3.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Parav PanditQuoted 6 lines · Parav Pandit
Quoted 5 lines
From: Stefan Hajnoczi <[email protected]>
Sent: Tuesday, March 7, 2023 2:04 PM
Quoted 6 lines
Quoted 5 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
+1.
Note that this is the way current ctrl virtqueue works.
Quoted 4 lines
This has best of both.
1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
2. Ability to execute multiple unrelated commands using a single AQ.
3. stateless device in AQ command execution
Does this mean if we want to migrate AQ (not use AQ to migrate), we need to wait for the AQ command to be completed?
Thanks
Quoted 7 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 44 lines · Tue, Mar 07, 2023 at 06:20:03PM CET, [email protected] wrote:
Quoted 35 lines · Jiri Pirko
Quoted 26 lines · Mon, Mar 06, 2023 at 11:54:45PM CET, [email protected] wrote:
Quoted 25 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 5 lines
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Quoted 8 lines
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Sorry to be late to the party, I'm trying to catch up. Is it common to have cover letter for features this brief? I mean, from the cover letter, I'm totally unable to understand what you are introducing here.
Could you elaborate about what you are aiming to achive with this? Could you shed some usecases perhaps?
Quoted 7 lines
Quoted 3 lines
I have to be missing something obvious, but I don't get why any notion of SR-IOV could be beneficial for virtio.
Good point, I'll add a bit of motivation.
For SR-IOV, it is not unusual for PFs to excercise control over VFs.
Quoted 7 lines
I understand the concepts of SR-IOV. Yet I fail to see the need of such concept in virtio world. SR-IOV is very specific solution for PCI functions instantiation, and I believe that it is already considered quite limiting in many aspects. Does not make sense to me to introduce it for virtio. But again, I may be missing something crucial, I just would like to see the motivation, needs, usecases for this crystal clear, which is opposite to the current cover letter I'm afraid :/
Quoted 3 lines
First people are asking for it because it's out there, however limiting it is. In fact Nvidia is - why don't you talk to Parav here and tell him that SR-IOV is legacy and there's no need to support.
Yep, I talked to Parav, makes much more sense now for me as I now understand the motivation. I was not able to reach there just by reading the patchset :/
Quoted 13 lines
Quoted 5 lines
Quoted 7 lines
Quoted 3 lines
There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Yeah, but why? As I asked before, what are the usecases? The fact there is interest in the community does not mean it makes sense to have it :)
Quoted 7 lines
If people want to build such hardware it will need some interface. Far better to have it standard.
But generally e.g. intel already said they will reuse this same structure with a different group type for SIOV support. I'll mention this in the cover letter.
Cool.
Quoted 72 lines
Quoted 61 lines
Quoted 61 lines
Quoted 61 lines
Quoted 27 lines
For example, it might make sense to split creating/destroying SIOV devices from the transport passing data from the guest - the driver would then not negotiate
VIRTIO_F_SR_IOV(which then means auto-provisioning).This is out of RFC, since we have two commands which are useful to discover supported group types (ATM can be none or SR-IOV).
Michael S. Tsirkin (10):
virtio: document forward compatibility guarantees
admin: introduce device group and related concepts
admin: introduce group administration commands
admin: introduce virtio admin virtqueues
pci: add admin vq registers to virtio over pci
mmio: documentADMIN_VQas reserved
ccw: documentADMIN_VQas reserved
admin: command list discovery
admin: conformance clauses
ccw: document more reserved featuresQuoted 26 lines
admin.tex | 540 +++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 112 +++++++++- introduction.tex | 3 + 3 files changed, 653 insertions(+), 2 deletions(-)create mode 100644 admin.tex
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 4 lines · Tue, Mar 07, 2023 at 05:30:18PM CET, [email protected] wrote:
Quoted 1 line · Jiri Pirko
Hmm, if not for now, the future exension would not be so simple, I fear.
Without knowing what it is I can't say.
Yep, so basically you say, for other things if they appear, let's introduce another queue type? If yes, sounds fair to me.
Quoted 20 lines
Quoted 7 lines
Quoted 5 lines
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
You mean using the control queue, correct?
Depends on the device type. network devices have a control queue, yes.
Quoted 9 lines
Quoted 1 line
From one of the patch description of this patchset I understand that you
cannot use control queue for this because control queue is device-specific, yet group control is device-agnostic.
My undestanding therefore was, that the admin queue you are introducing serves as a generic carrier for device-agnostic commands, in parallel for having control queue serving as a generic carrier of device-specific commands. If this is not the case, I think it would be nice to describe the exact monivation and scope of admin queue.
Quoted 4 lines
Nope unfortunately. This queue is just a carrier for admin commands. admin commands are commands that talk to one device about other devices. There's clearly no mechanism in the spec to do that, so we plug this hole.
Okay, in that case "admin" sounds a bit misleading as for me it implicates that this is for "administration" of the device. Yet is is for the administration of other devices (slaves).
Perhaps there could be different term used to clarify? Group leader virtqueue?
Group owner virtqueue?
Group master virtqueue?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
Could you provide examples of such commands please?
For example a common feature is to program a vlan and have device put a given VF inside this vlan.
I don't follow entirely. The way how the VF is connected to network should be ouf of the scope of this interface. The eswitch manager should take care. What you say sounds awfully like the "ip vf" legacy interface, which should not be considered here I believe.
If PF would be the eswitch manager, there are other means to do network programming, using eswitch port representors. But I don't think this is the can of worms we want to open now. I don't think we have a usecase for it currently. Am I wrong Parav?
Quoted 5 lines
In a virtualization scenario host controls this vlan programming giving the network a measure of protection from VFs. If a VF is passed through to a VM, IOMMU limits VFs to only access guest memory so host has to do this programming through a PF.
Understood. This really looks like "ip vf" legacy. I strongly believe it should not be supported.
Any other commands you have in mind?
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
This is what groups do.
But if we see more uses we can always add them.I'd rather avoid being too generic though.
In that case, why not to avoid using generic terms and stay "group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues" "structvirtio_admin_cmd" -> "structvirtio_group_admin_cmd"
Quoted 4 lines
Quoted 1 line
Etc. Helps to avoid confusion.
Sure, I tried to do that but missed some opportunities. Will address.
Cool.
Quoted 21 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
+than one administration virtqueue.
[...]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 7 lines · Tue, Mar 07, 2023 at 05:13:01PM CET, [email protected] wrote:
Quoted 2 lines · Jiri Pirko
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Because the virtqueue is just a virtqueue - a way to pass around commands. Admin commands are used whenever one needs to talk to one device about another one.
Yeah, I got that now. Please check my other reply in this thread suggesting a name change of "admin". Perhaps it would make sense and makes thing a bit clearer for slower people like myself :)
Quoted 20 lines
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 89 lines · Tue, Mar 07, 2023 at 08:03:47PM CET, [email protected] wrote:
Quoted 77 lines · Jiri Pirko
Quoted 51 lines · Tue, Mar 07, 2023 at 03:39:11PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c repQuoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 8 lines
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
I see. Then I believe that the device should handle the dependencies. Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member BThe device according to internal implementation can either serialize the 2 group member creations or do it in parallel, if it supports it.
Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member AQuoted 4 lines
Here the serialization is necessary and the device is the one to take care of it.
Makes sense?
Yes, I understand. The spec would need to define ordering rules for specific commands and the device must implement them. It allows the driver to pipeline commands while also allowing out-of-order completion (parallelism) in some cases. The disadvantage of this approach is complexity in the spec and implementations.
Quoted 6 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
But isn't this duplicating the burden of maintaining dependencies to both driver and device? I mean, device should not depend on driver doing the right thing, that means it has to check the dependencies for every incoming command anyway. The only difference would be to wait instead of returning "-EBUSY" in case the dependency is not satisfied yet.
Device knows exactly what are the dependencies. And I believe, those are device implementation specific. For example, some implementation could support parallel VF config cmd execution, some implementation might need to serialize that. Driver has no clue.
Could you please elaborate a bit more what you mean by "complexity in the spec"?
Thanks!
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Group member creation might have similar issues if it involves acquiring remote resources (e.g. connecting to a Ceph cluster or allocating ports on a distributed network switch). It can be impossible to defer resource
Quoted 30 lines
Quoted 4 lines
Sidetrack: this is really fuzzy to me, how the new member is going to be plugged into backend (network). Over the time, we learned that the creation of device from the other side (switch side) makes more sense. That is why I asked for motivation to introduce this infra.
Michael, have you already thought about this?
Quoted 23 lines
Quoted 9 lines
acquisition/initialization because because VIRTIO devices must be available as soon as the driver can see them (i.e. how do populate Configuration Space fields if you don't have the details of the resource yet?).
So I have raised two questions:
1. What are the admin queue command completion semantics: in-order or out-of-order command completion?
I would add "dependencies/serialization" here.
Quoted 3 lines
2. Will there be long-running commands and how will we deal with them when they hang?
Yeah, sounds legit to define it in spec.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 85 lines · Michael S. Tsirkin
Quoted 64 lines · Max Gurtovoy
Quoted 61 lines · Michael S. Tsirkin
Quoted 60 lines · Stefan Hajnoczi
Quoted 47 lines · Michael S. Tsirkin
Quoted 24 lines · Stefan Hajnoczi
Quoted 10 lines · Parav Pandit
Review
Quoted 7 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 2, 2023 8:05 AM(quoted diff) +3
-0+When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qialifier} +is reserved and set to zero by the device. +s/status_qialifier/status_qualifier
Missed from v10 of Feb.Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+When \field{status} is VIRTIO_ADMIN_STATUS_EINVAL, the following table +describes possible \field{status_qialifier} values:s/status_qialifier/status_qualifier
Can you please add other useful error codes in addition to the EINVAL? Few that we are needed EAGAIN, ENOMEM, EBUSY, ENODEV.
Please define a unique constant for each error condition that can occur instead of sharing catch-all errno constants between multiple error conditions. If a driver wants to squash them together into an errno, that's fine, but I think doing this at the hardware interface level is just propagating the mistakes of errnos.
Quoted 4 lines
Only
status_qualifieris needed and the vague status field can be dropped. It's not clear to me why adding EAGAIN, ENOMEM, EBUSY, and ENODEV is useful. They have no meaning to the driver, only thestatus_qualifierreally indicates what is going on.At a high level at the moment we have only two cases:
- ok
- invalid input supplied by drivermaybe we will have more reasons for a failure - remains to be seen.
Quoted 5 lines
I'm sure you guys have discussed this previously, but please provide rationale in the spec because it looks weird to someone with fresh eyes.
Stefan
Quoted 7 lines
Really most drivers just want to propagate errno to userspace. All the detailed reporting is for sure well intentional but in the end it is at best printed into log - end to end people just end up with a switch statement converting these to errno codes.
So we are passing them from device and this way there will be some uniformity.Please clarify the rationale in the spec. I don't agree with it, as explained in my earlier email, but as long as its documented, people can try to follow it.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Stefan
It's 2:2 for now, you are with Parav, me and Cornelia like it :) Or I will try to document it better.
I don't understand this
status_qualifieras well and it wasn't included in my original patch set.Sounds like you feel I should drop your S.O.B - is this the complaint? I wanted to give attribution since I started with that but sure, no problem.
Quoted 9 lines
I vote for "status" that describe generic status codes and "
command_specific_error" that should be inspected by the driver only if "status" is set to "VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR". We discussed this so many times before (and already agreed IIRC) and adding this new qualifiers mechanism sounds not right to me and not intuitive for device and driver developers.I suggest:
Quoted 8 lines
Quoted 6 lines
1.
VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE
2.VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
3.VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP
4.VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER
5.VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).I don't think it's a good idea, we'll have to agree to disagree.
Ok.
can you explain why isn't this a good idea please ?
remember that this is very common logic in many specifications and many HW device implementation.
Quoted 8 lines
The point is simple. We can have a detailed virtio specific error. Nice for debugging most drivers won't know what to do with it. This is the
status_qualifier.
Very detailed but
generally drivers will just have a giant switch statement translating it to a simple error code for userspace.
how giant is this switch statement ?
NVMe handles fine with this logic. Maybe you can take a look on that to get some more ideas of implementing this kind of things.
Why do we need to add some qualifiers logic into our devices ?
Quoted 3 lines
So to save everyone work, we also added "status"
a generic kind of error class that is easy to pass to userspace with a small switch statement.
if you want something that is easy just pass OK = 0 and not_OK = 1 or != 0. Not need for qualifiers.
In you case the qualifiers is just an extension for the status.
Quoted 3 lines
COMMAND_SPECIFIC_ERRis just way too much detail - commands generally just should not fail it's a quality of implementation issue.
it's the way many specifications work.
And this is the correct way for future extensions IMO. Each new command may have it's own namespace of errors extensions.
why we defer such essential decisions for the future ?
We already introduces 2-3 admin features in the past (MSIX, feature bits config, num_queues config, etc..) - maybe we can learn from it ?
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 68 lines · Stefan Hajnoczi
Quoted 59 lines · Max Gurtovoy
Quoted 50 lines · Stefan Hajnoczi
Quoted 35 lines · Jason Wang
Quoted 25 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 6 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Quoted 3 lines
Or device need mandate some kind of QOS here, e.g a request must be complete in some time. Otherwise we don't have sufficient reliability for using it as management task?
Yes, if all commands can be executed in bounded time then a guarantee is possible.
Here is an example where that's hard: imagine a virtio-blk device backed by network storage. When an admin queue command is used to delete a group member, any of the group member's in-flight I/O requests need to be aborted. If the network hangs while the group member is being deleted, then the device can't complete an orderly shutdown of I/O requests in a reasonable time.
Quoted 5 lines
That example shows a basic group admin command that I think Michael is about to propose. We can't avoid this problem by not making it a group admin command - it needs to be a group admin command. So I think it's likely that there will be admin commands that take an unbounded amount of time to complete. One way to achieve what you mentioned is timeouts.
Quoted 7 lines
I think that you're getting into device specific implementation details and I'm not sure it's necessary.
I don't think we need to abort admin commands. Admin commands can be flushed/aborted during the device reset phase.
Only IO commands should have the possibility to being aborted as you mentioned in NVMe and SCSI (and potentially in virtio-blk).
Quoted 6 lines
It's a general design issue that should be clarified now rather than being left unspecified.
I'm not saying that it must be possible to abort admin commands. There are other options, like requiring the device itself to fail a command after a timeout.
do you have an example of timeout today for control vq ?
Quoted 4 lines
Or we could say that admin commands must complete within bounded time, but I'm not sure that is implementable for some device types like virtio-blk, virtio-scsi, and virtiofs.
No we can't.
Some commands, for example FW upgrade can take 10 minutes and it's perfectly fine. Other commands like setting feature bit will take 1 millisec. Each device implements commands in a different internal logic so we can't expect to complete after X time.
Device can go to so FATAL state in case a command is stuck and causing internal errors in it.
Quoted 9 lines
Quoted 4 lines
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Forgetting about in-flight requests doesn't necessarily make them go away. It creates a race between forgotten requests and reconnection. In the worst case a forgotten write request takes effect after reconnection, causing data corruption.
For making it work without data corruption we need a cooperation of the target side for sure. But this is fine since the target in that case is part of the "virtio-blk backend". One solution is that the target can decide it will flush all the requests to the storage device before accepting new connections.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 5 lines · Jiri Pirko
Quoted 4 lines · Tue, Mar 07, 2023 at 05:13:01PM CET, [email protected] wrote:
Quoted 2 lines · Jiri Pirko
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 7 lines
Quoted 3 lines
Because the virtqueue is just a virtqueue - a way to pass around commands. Admin commands are used whenever one needs to talk to one device about another one.
Yeah, I got that now. Please check my other reply in this thread suggesting a name change of "admin". Perhaps it would make sense and makes thing a bit clearer for slower people like myself :)
Yes I think it's a good idea. Replied there too.
Quoted 12 lines
Quoted 12 lines
Signature · 10 lines
-- MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 8 lines · Jiri Pirko
Quoted 4 lines · Tue, Mar 07, 2023 at 05:30:18PM CET, [email protected] wrote:
Quoted 1 line · Jiri Pirko
Hmm, if not for now, the future exension would not be so simple, I fear.
Without knowing what it is I can't say.
Yep, so basically you say, for other things if they appear, let's introduce another queue type? If yes, sounds fair to me.
Yes. For example I find it likely that live migration/failover support will require a queue where driver pre-adds buffers and then device supplies information as state changes.
Quoted 26 lines
Quoted 21 lines
Quoted 8 lines
Quoted 6 lines
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
You mean using the control queue, correct?
Depends on the device type. network devices have a control queue, yes.
Quoted 9 lines
Quoted 1 line
From one of the patch description of this patchset I understand that you
cannot use control queue for this because control queue is device-specific, yet group control is device-agnostic.
My undestanding therefore was, that the admin queue you are introducing serves as a generic carrier for device-agnostic commands, in parallel for having control queue serving as a generic carrier of device-specific commands. If this is not the case, I think it would be nice to describe the exact monivation and scope of admin queue.
Quoted 4 lines
Nope unfortunately. This queue is just a carrier for admin commands. admin commands are commands that talk to one device about other devices. There's clearly no mechanism in the spec to do that, so we plug this hole.
Quoted 8 lines
Okay, in that case "admin" sounds a bit misleading as for me it implicates that this is for "administration" of the device. Yet is is for the administration of other devices (slaves).
Perhaps there could be different term used to clarify? Group leader virtqueue?
Group owner virtqueue?
Group master virtqueue?
I used group administration virtqueue in a couple of places, just inconsistently. Good enough?
Quoted 52 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
Could you provide examples of such commands please?
For example a common feature is to program a vlan and have device put a given VF inside this vlan.
I don't follow entirely. The way how the VF is connected to network should be ouf of the scope of this interface. The eswitch manager should take care. What you say sounds awfully like the "ip vf" legacy interface, which should not be considered here I believe.
If PF would be the eswitch manager, there are other means to do network programming, using eswitch port representors. But I don't think this is the can of worms we want to open now. I don't think we have a usecase for it currently. Am I wrong Parav?
Quoted 5 lines
In a virtualization scenario host controls this vlan programming giving the network a measure of protection from VFs. If a VF is passed through to a VM, IOMMU limits VFs to only access guest memory so host has to do this programming through a PF.
Understood. This really looks like "ip vf" legacy. I strongly believe it should not be supported.
Any other commands you have in mind?
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
This is what groups do.
But if we see more uses we can always add them.I'd rather avoid being too generic though.
In that case, why not to avoid using generic terms and stay "group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues" "structvirtio_admin_cmd" -> "structvirtio_group_admin_cmd"
Quoted 38 lines
Quoted 4 lines
Quoted 1 line
Etc. Helps to avoid confusion.
Sure, I tried to do that but missed some opportunities. Will address.
Cool.
Quoted 30 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
+than one administration virtqueue.
[...]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery
In reply to a message not in the archiveQuoted 15 lines · Jiri Pirko
Quoted 10 lines · Thu, Mar 02, 2023 at 02:05:22PM CET, [email protected] wrote:
Add commands to find out which commands does each group support, as well as enable their use by driver.
This will be especially useful once we have multiple group types.An alternative is per-type VQs. This is possible but will require more per-transport work. Discovery through the vq helps keep things contained.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>[...]
Quoted 14 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 236
Quoted 9 lines
admin.tex +9
-0+ +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to +query the list of commands valid for this group and before sending +any commands for any member of a group. + +The driver then enables use of some of the opcodes by sending to +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is +both understood and used by the driver.To my untrained ear, this sounds somewhat similar to the feature negotiantion mechanism. Why the fact that device/driver supports some command can't be covered by just another feature? Looks like unnecassary complexicity to negotiate supported commands like this.
Absolutely, it is similar. The issue is that a single device can be an owner for multiple group types.
For example, I think an SRIOV PF itself can have subfunctions with SIOV as well as virtual functions with SRIOV.
And the set of supported commands might differ.
We thus need a command that is kind of like features but per group type.
As a small bonus as we are building this from scratch, I added something which was requested for a long time from features which is blocking access to commands not allowed by features. With standard virtio features we trust the driver not to send commands before enabling a feature and this was considered a maintainance/security problem.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
In reply to David EdmondsonQuoted 38 lines · David Edmondson
Quoted 36 lines · Michael S. Tsirkin
Add conformance clauses for admin commands and admin virtqueues.
Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 215 insertions(+), 1 deletion(-)admin.tex +21
-0@@ -251,6 +251,145 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti supporting multiple group types, the list of supported commands might differ between different group types. +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} + +The device MUST validate \field{opcode}, \field{group_type} and +\field{group_member_id}, and if any of these has an invalid or +unsupported value, set \field{status} to +VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier} +accordingly: +\begin{itemize} +\item if \field{group_type} is invalid, \field{status_qualifier} + MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; +\item otherwise, if \field{opcode} is invalid, + \field{status_qualifier} MUST be set to + VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE; +\item otherwise, if \field{group_member_id} is used by the + specific command and is invalid, \field{status_qualifier} MUST be + set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER. +\end{itemize} + +If a command completes successfully, the device MUST set +\field{status} to VIRTIO_ADMIN_STATUS_OK. +
Quoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 280
Quoted 6 lines
admin.tex +6
-0+If a command fails, the device MUST set +\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK. + +If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the +device state MUST NOT change, that is the command MUST NOT have +any side effects on the device, in particular the device MUST notMUST NOT
/me nods
Quoted 22 lines
Quoted 22 lines
admin.tex +22
-0+enter an error state as a result of this command. + +If a command fails, the device state generally SHOULD NOT change, +as far as possible. + +The device MAY enforce additional restrictions and dependencies on +opcodes used by the driver and MAY fail the command +VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD +if the list of commands used violate internal device dependencies. + +If the device supports multiple group types, commands for each group +type MUST operate independently of each other, in particular, +the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY +for different group types. + +After reset, if the device supports a given group type +and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type +the device MUST assume +that the list of legal commands used by the driver consists of +the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. +
Quoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 308
Quoted 6 lines
admin.tex +6
-0+After completing VIRTIO_ADMIN_CMD_LIST_USE successfully, +the device MUST set the list of legal commands used by the driver +to the one supplied in \field{command_specific_data}. + +The device MUST set the list of legal commands used by the driver +to the one supplied in VIRTIO_ADMIN_CMD_LIST_USE.Are these last two paragraphs not saying the same thing?
They are!
Quoted 59 lines
Quoted 59 lines
admin.tex +59
-0+The device MUST validate commands against the list used by +the driver and MUST fail any commands not in the list with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +and \field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE. + +The list of supported commands MUST NOT shrink (but MAY expand): +after reporting a given command as supported through +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it +as unsupported. Further, after a given set of commands has been +used (via a successful VIRTIO_ADMIN_CMD_LIST_USE), then after a +device or system reset the device SHOULD complete successfully +any following calls to VIRTIO_ADMIN_CMD_LIST_USE with the same +list of commands; if this command VIRTIO_ADMIN_CMD_LIST_USE fails +after a device or system reset, the device MUST not fail it +solely because of the command list used. Failure to do so would +interfere with resuming from suspend and error recovery. + +When processing a command with the SR-IOV group type, +if the device does not have an SR-IOV Extended Capability or +if \field{VF Enable} is clear +then the device MUST fail all commands with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP; +otherwise, if \field{group_member_id} is not +between $1$ and \field{NumVFs} inclusive, +the device MUST fail all commands with +\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER; +\field{NumVFs}, \field{VF Migration Capable} and +\field{VF Enable} refer to registers within the SR-IOV Extended +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. + +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands} + +The driver MAY discover whether device supports a specific group type +by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching +\field{group_type}. + +The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE +and wait for it to be completed with status +VIRTIO_ADMIN_STATUS_OK before issuing any commands +(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY +and VIRTIO_ADMIN_CMD_LIST_USE). + +The driver SHOULD NOT set bits in device_admin_cmds +if it is not familiar with how the command opcode +is used, as dependencies between command opcodes might exist. + +The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE) +the use of any commands not previously reported as +supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY. + +The driver MUST NOT use any commands for a given group type +before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct +list of command opcodes and group type. +
Quoted 8 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 3 lines
admin.tex +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.
Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.
Quoted 23 lines
Quoted 23 lines
admin.tex +20
-0+The driver MUST handle a command error with a reserved \field{status} +value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL +(except possibly for different error reporting/diagnostic messages). + +The driver MUST handle a command error with a reserved +\field{status_qualifier} value in the same way as +\field{status_qualifier} set to +VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for +different error reporting/diagnostic messages). + +When sending commands with the SR-IOV group type, +the driver specify a value for \field{group_member_id} +between $1$ and \field{NumVFs} inclusive, +the driver MUST also make sure that as long as any such command +is outstanding, \field{VF Migration Capable} is clear and +\field{VF Enable} is set; +\field{NumVFs}, \field{VF Migration Capable} and +\field{VF Enable} refer to registers within the SR-IOV Extended +Capability as specified by \hyperref[intro:PCIe]{[PCIe]}. + \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
Quoted 8 lines
admin.tex \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic · line 326
Quoted 6 lines
admin.tex +0
-1An administration virtqueue of an owner device is used to submit@@ -323,4 +462,79 @@ \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic of the specification are designed, new fields can be added to the tail of a structure, with the driver/device using the full structure without concern for versioning. ->>>>>>> 0edc690... admin: introduce virtio admin virtqueuesOh, here is where it got removed :-)
/me blushes
Quoted 89 lines
Quoted 50 lines
admin.tex +50
-0+ +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues} + +The device MUST support device-readable and device-writeable buffers +shorter than described in this specification, by +\begin{enumerate} +\item acting as if any data that would be read outside the +device-readable buffers is set to zero, and +\item discarding data that would be written outside the +specified device-writeable buffers. +\end{enumerate} + +The device MUST support device-readable and device-writeable buffers +longer than described in this specification, by +\begin{enumerate} +\item ignoring any data in device-readable buffers outside +the expected length, and +\item only writing the expected structure to the device-writeable +buffers, ignoring any extra buffers, and reporting the +actual length of data written, in bytes, +as buffer used length. +\end{enumerate} + +The device SHOULD initialize the device-writeable buffer +up to the length of the structure described by this specification or +the length of the buffer supplied by the driver (even if the buffer is +all set to zero), whichever is shorter. + +The device MUST NOT fail a command solely because the buffers +provided are shorter or longer than described in this +specification. + +The device MUST initialize the device-writeable part of +\field{struct virtio_admin_cmd} that is a multiple of 64 bit in +size. + +The device MUST initialize \field{status} in \field{struct +virtio_admin_cmd}. + +The device MUST process commands on a given administration virtqueue +in the order in which they are queued. + +If multiple administration virtqueues have been configured, +device MAY process commands on distinct virtqueues with +no order constraints. + +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues} + +The driver MAY supply device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are longer than described inadmin.tex \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Devic · line 524
Quoted 10 lines
admin.tex +10
-0+this specification. + +The driver SHOULD supply device-readable part of +\field{struct virtio_admin_cmd} that is at least as +large as the structure described by this specification +(even if the structure is all set to zero). + +The driver MUST supply both device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are a multiple of 64 bit +in length.This addresses my question on a previous patch.
Quoted 26 lines
admin.tex +15
-0+The device MUST supply both device-readable or device-writeable parts +of \field{struct virtio_admin_cmd} that are larger than zero in +length. However, \field{command_specific_data} and +\field{command_specific_result} MAY be zero in length, unless +specified otherwise for the command. + +The driver MUST NOT assume that the device will initialize the whole +device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead, +the driver MUST act as if the structure +outside the part of the buffer used by the device +is set to zero. + +If multiple administration virtqueues have been configured, +the driver MUST ensure ordering for commands +placed on different administration virtqueues.Signature · 9 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/Signature · 1 line
Music has magic, it's good clear syncopation.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jason WangQuoted 16 lines · Jason Wang
Quoted 6 lines · Parav Pandit
Quoted 5 lines
From: Stefan Hajnoczi <[email protected]>
Sent: Tuesday, March 7, 2023 2:04 PMQuoted 6 lines
Quoted 5 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
+1.
Note that this is the way current ctrl virtqueue works.
Quoted 7 lines
Quoted 4 lines
This has best of both.
1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
2. Ability to execute multiple unrelated commands using a single AQ.
3. stateless device in AQ command executionDoes this mean if we want to migrate AQ (not use AQ to migrate), we need to wait for the AQ command to be completed?
It depends. Today, if the AQ is emulated by the VMM then it might be possible to migrate while there is a command in-flight. If the AQ is handled by the device then no because VIRTIO currently does not support device state migration.
In the future, device state migration will probably be added to vhost-user and vDPA. In that case a device can migrate in-flight AQ activity - assuming it's possible to pause it, serialize state, deserialize it, and resume it. Otherwise it's still necessary to wait.
To be clear, I'm just taking about AQ commands that are currently in the process of executing. Commands that are in the virtqueue but have not yet been started by the device can be easily migrated using the existing virtqueue migration infrastructure.
Stefan
[virtio-comment] Re: [virtio] RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jason WangQuoted 21 lines · Jason Wang
Quoted 6 lines · Parav Pandit
Quoted 5 lines
From: Stefan Hajnoczi <[email protected]>
Sent: Tuesday, March 7, 2023 2:04 PMQuoted 6 lines
Quoted 5 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
+1.
Note that this is the way current ctrl virtqueue works.
Quoted 4 lines
This has best of both.
1. Command ordering knowledge and hence the decision left to the one which issues them. (driver).
2. Ability to execute multiple unrelated commands using a single AQ.
3. stateless device in AQ command execution
Quoted 4 lines
Does this mean if we want to migrate AQ (not use AQ to migrate), we need to wait for the AQ command to be completed?
Thanks
As with any VQ, if you migrate while an operation is in progress you need to figure out the internal device state.
Quoted 7 lines
Quoted 7 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 7 lines
From: Jason Wang <[email protected]>
Sent: Wednesday, March 8, 2023 12:12 AMQuoted 1 line
In other words, can AQ command be useful tomorrow for doing SIOV device
Quoted 8 lines
add/remove, and provisioning from non-owning PF?
Quoted 1 line
The way AQ is crafted today is not there yet, but in the future, it can be
extended.
There's already a proposal to use the queue as a transport. Any device that can't use a transport specific configuration or provisioning can benefit from that. SIOV is only one of the users for that.
o.k. Better to include that link in the cover letter.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 10 lines
Quoted 5 lines
There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Yeah, but why? As I asked before, what are the usecases? The fact there is interest in the community does not mean it makes sense to have it :)
If people want to build such hardware it will need some interface. Far better to have it standard.
But generally e.g. intel already said they will reuse this same structure with a different group type for SIOV support. I'll mention this in the cover letter.
Quoted 8 lines
Quoted 6 lines
We have the following already identified use cases of a device agnostic VQ.
1. SR-IOV VF device provisioning at virtio layer (features and config)
2. SR IOV (SR-PCIM interface of the PCI spec) for VF provisioning, for example, MSI-X vectorsThe virtio layer device provisioning is transport independent.
Yes. I listed known use cases above. If there are more that can use, sure.
Quoted 4 lines
I don't see
anything that is PCI specific there. Even the MSI-X could be reused by other transport.
Maybe yes.
I just don’t know which other transport of virtio has defined MSI-X. If there is, sure there are more users of AQ.
Quoted 7 lines
Quoted 1 line
3. SR-IOV VF Live migration
I think we don't want a partial solution for live migration that only works for PCI VF devices and L1. Instead, there needs to be a general device facility section and the admin virtqueue could be one of the interfaces for the driver to use the facility.
May be there are more use cases of AQ than what I listed.
Quoted 5 lines
Quoted 1 line
4. Above #1 to #3 for SIOV devices instead of SRIOV devices
If we make the above as a general facility, it would not be hard to re-use them for transport virtqueue or SIOV.
Ok. thanks.
[virtio-comment] Re: [virtio-dev] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Jiri PirkoQuoted 130 lines · Jiri Pirko
Quoted 129 lines · Fri, Mar 03, 2023 at 09:23:14PM CET, [email protected] wrote:
Quoted 128 lines · Michael S. Tsirkin
Quoted 98 lines · Stefan Hajnoczi
Quoted 31 lines · Michael S. Tsirkin
Quoted 22 lines · Stefan Hajnoczi
Quoted 21 lines · Michael S. Tsirkin
This introduces a general structure for group administration commands, used to control device groups through their owner.
Following patches will introduce specific commands and an interface for submitting these commands to the owner.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- admin.tex | 108 +++++++++++++++++++++++++++++++++++++++++++++++ introduction.tex | 3 ++ 2 files changed, 111 insertions(+)admin.tex +0
-0@@ -46,4 +46,112 @@ \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}). \end{description}Quoted 7 lines
admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 51
Quoted 3 lines
admin.tex +2
-0+\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands} +The driver sends group administration commands to the owner device ofI notice that the terminology is simply "the driver". "Owner driver" and "group member driver" might be clearer because there will be two (possibly different) drivers involved.
Quoted 3 lines
Hmm I don't really want to repeat owner everywhere.
I will clarify that in this section simple "driver" and "device" are owner, "member device" and "member driver" is always called explicitly.Sounds good.
Quoted 60 lines
Quoted 59 lines
Quoted 45 lines
admin.tex +45
-0+a group to control member devices of the group. +This mechanism can +be used, for example, to configure a member device before it is +initialized by its driver. +\footnote{The term "administration" is intended to be interpreted +widely to include any kind of control. See specific commands +for detail.} + +All the group administration commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + le16 opcode; + /* + * 1 - SR-IOV + * 2 - 65535 reserved + */ + le16 group_type; + /* unused, reserved for future extensions */ + u8 reserved1[12]; + le64 group_member_id; + u8 command_specific_data[]; + + /* Device-writable part */ + le16 status; + le16 status_qualifier; + /* unused, reserved for future extensions */ + u8 reserved2[4]; + u8 command_specific_result[]; +}; +\end{lstlisting} + +For all commands, \field{opcode}, \field{group_type} and if +necessary \field{group_member_id} and \field{command_specific_data} are +set by the driver, and the owner device sets \field{status} and if +needed \field{status_qualifier} and +\field{command_specific_result}. + +Generally, any unused device-readable fields are set to zero by the driver +and ignored by the device. Any unused device-writeable fields are set to zero +by the device and ignored by the driver. + +\field{opcode} specifies the command. The valid +values for \field{opcode} can be found in the following table:admin.tex \section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device g · line 106
Quoted 10 lines
admin.tex +10
-0+ +\begin{tabular}{|l|l|} +\hline +opcode & Name & Command Description \\ +\hline \hline +0x0000 - 0x7FFF & - & Group administration commands \\ +\hline +0x8000 - 0xFFFF & - & Reserved \\ +\hline +\end{tabular}I thought all commands are "group administration commands" but this table makes it look like they are just a subset (
0x0000-0x7FFF) of group administration commands, which is a paradox.Quoted 5 lines
Quoted 2 lines
Well the rest are reserved, maybe we will have more command types who knows. No?
I see. Does that mean the reserved commands don't need to be in the same format as struct
virtio_admin_cmd?I am not good in predicting future ...
Quoted 11 lines
The entire section is called "Group administration commands" but I get the impression it's talking both about admin virtqueue commands in general and specifically about group administration commands.
Is it possible to structure this as follows:
Admin Commands
...common stuff... Group Administration Commands (0x0000-0x7fff) ...specific to group administration commands... Reserved (0x8000-0xffff)What if I just write it like this:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
admin.tex +4
-0+\hline \hline +0x0000 - 0x7FFF & - & commands using struct virtio_admin_cmd \\ +\hline +0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
will this address the comment?
Yes, thanks.
What is the reason for having this table? I fail to see it.
I think it's more a place-holder table in which follow-up patches add entries. I'll check and see if we can get rid of it.
Quoted 4 lines
Quoted 3 lines
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Max GurtovoyQuoted 7 lines · Max Gurtovoy
Quoted 4 lines
Quoted 2 lines
5.
VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).I don't think it's a good idea, we'll have to agree to disagree.
Ok.
can you explain why isn't this a good idea please ?
Pointless complexity for spec, devices and drivers. In the end drivers don't really do anything with all this detailed info.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Michael S. TsirkinQuoted 5 lines · Wed, Mar 08, 2023 at 12:50:48PM CET, [email protected] wrote:
Quoted 4 lines · Jiri Pirko
Quoted 3 lines · Tue, Mar 07, 2023 at 05:30:18PM CET, [email protected] wrote:
Quoted 1 line · Jiri Pirko
Hmm, if not for now, the future exension would not be so simple, I fear.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Without knowing what it is I can't say.
Yep, so basically you say, for other things if they appear, let's introduce another queue type? If yes, sounds fair to me.
Yes. For example I find it likely that live migration/failover support will require a queue where driver pre-adds buffers and then device supplies information as state changes.
I see. So there would be a queue called for example "child state virtqueue" or something like that for the sole purpose of getting the state of VF? Hmm, wouldn't it make more sense to have this done as a part of "group administrarion queue"? I mean, there is already notion of addresing child/VF here. So from my perspective, it is just another "group administration" command.
We would have to prealloc buffers and use it for this purpose as well. I mean, the cmd payload for other group commands could be put in this buffers as well, making all group command infrastructure consistent.
Something like:
opcode Xgroup_member_id 1
cmd_payload-> buffer A
opcode Ygroup_member_id 1
cmd_payload-> buffer B
Basically this would replace command_specific_data/command_specific_result
Quoted 31 lines
Quoted 31 lines
Quoted 22 lines
Quoted 9 lines
Quoted 7 lines
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
You mean using the control queue, correct?
Depends on the device type. network devices have a control queue, yes.
Quoted 9 lines
Quoted 1 line
From one of the patch description of this patchset I understand that you
cannot use control queue for this because control queue is device-specific, yet group control is device-agnostic.
My undestanding therefore was, that the admin queue you are introducing serves as a generic carrier for device-agnostic commands, in parallel for having control queue serving as a generic carrier of device-specific commands. If this is not the case, I think it would be nice to describe the exact monivation and scope of admin queue.
Quoted 4 lines
Nope unfortunately. This queue is just a carrier for admin commands. admin commands are commands that talk to one device about other devices. There's clearly no mechanism in the spec to do that, so we plug this hole.
Okay, in that case "admin" sounds a bit misleading as for me it implicates that this is for "administration" of the device. Yet is is for the administration of other devices (slaves).
Quoted 7 lines
Quoted 4 lines
Perhaps there could be different term used to clarify? Group leader virtqueue?
Group owner virtqueue?
Group master virtqueue?I used group administration virtqueue in a couple of places, just inconsistently. Good enough?
Yep, sounds good. Thanks!
Quoted 83 lines
Quoted 83 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
Could you provide examples of such commands please?
For example a common feature is to program a vlan and have device put a given VF inside this vlan.
I don't follow entirely. The way how the VF is connected to network should be ouf of the scope of this interface. The eswitch manager should take care. What you say sounds awfully like the "ip vf" legacy interface, which should not be considered here I believe.
If PF would be the eswitch manager, there are other means to do network programming, using eswitch port representors. But I don't think this is the can of worms we want to open now. I don't think we have a usecase for it currently. Am I wrong Parav?
Quoted 5 lines
In a virtualization scenario host controls this vlan programming giving the network a measure of protection from VFs. If a VF is passed through to a VM, IOMMU limits VFs to only access guest memory so host has to do this programming through a PF.
Understood. This really looks like "ip vf" legacy. I strongly believe it should not be supported.
Any other commands you have in mind?
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
This is what groups do.
But if we see more uses we can always add them.I'd rather avoid being too generic though.
In that case, why not to avoid using generic terms and stay "group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues" "structvirtio_admin_cmd" -> "structvirtio_group_admin_cmd"Quoted 4 lines
Quoted 1 line
Etc. Helps to avoid confusion.
Sure, I tried to do that but missed some opportunities. Will address.
Cool.
Quoted 21 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
+than one administration virtqueue.
[...]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery
In reply to Michael S. TsirkinQuoted 20 lines · Thu, Mar 02, 2023 at 02:05:22PM CET, [email protected] wrote:
Add commands to find out which commands does each group support, as well as enable their use by driver.
This will be especially useful once we have multiple group types.An alternative is per-type VQs. This is possible but will require more per-transport work. Discovery through the vq helps keep things contained.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>---
admin.tex | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-1 file changed, 96 insertions(+), 1 deletion(-)admin.tex +0
-0@@ -99,7 +99,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti\hline
Quoted 8 lines
opcode & Name & Command Description \\
\hline \hline
-0x0000 -0x7FFF& - & Group administration commands \\
+0x0000 &VIRTIO_ADMIN_CMD_LIST_QUERY& Provides to driver list of commands supported for this group type \\
+0x0001 &VIRTIO_ADMIN_CMD_LIST_USE& Provides to device list of commands used for this group type \\
+0x0002 -0x7FFF& - & Group administration commands \\
\hline0x8000-0xFFFF& - & Reserved \\
This table would not be needed at all, as all the commands here are "group administration commands". Could you remove it?
Quoted 19 lines
\hline
admin.tex +0
-0@@ -156,6 +158,99 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virtidepends on these structures and is described separately or is implicit in the structure description.
admin.tex +14
-0+Before sending any administration commands to the device, the driver +needs to communicate to the device which commands it is going to +use. Initially (after reset), only two commands are assumed to be used: +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE. + +Before sending any other commands for any member of a specific group to +the device, the driver queries the supported commands via +VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it will use via +VIRTIO_ADMIN_CMD_LIST_USE. + +Commands VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE +both use the following structure describing the +command opcodes:
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 184
Quoted 10 lines
admin.tex +10
-0+ +\begin{lstlisting} +struct virtio_admin_cmd_list { + /* Indicates which of the below fields were returned + le64 device_admin_cmds[]; +}; +\end{lstlisting} + +This structure is an array of 64 bit values in little-endian byte +order, in which a bit is set if the specific command opcode
Ah wait, you are having this as bitfield. How many commands do you expect to have? I mean, could this be rather just plain array of opcodes.
If you want to save some space, the opcode certainly does not need to be 64 bit. I think that 16 bits is more than enough for this :)
Then to make this consistent with struct virtio_admin_cmd, where you use term "opcode" and with the rest of the text where you use "CMD" and "cmd", perhaps "cmd_opcode" in struct virtio_admin_cmd and "cmd_opcodes[]" here would work better?
Quoted 1 line
+is supported. Thus, \field{device_admin_cmds[0]} refers to the first 32-bit value
You say "32", it is "64".
Quoted 23 lines
admin.tex +23
-0+in this array corresponding to opcodes 0 to 63, +\field{device_admin_cmds[1]} is the second 64-bit value +corresponding to opcodes 64 to 127, etc. +For example, the array of size 2 including +the values 0x3 in \field{device_admin_cmds[0]} +and 0x1 in \field{device_admin_cmds[1]} indicates that only +opcodes 0, 1 and 64 are supported. +The length of the array depends on the supported opcodes - it is +large enough to include bits set for all supported opcodes, +that is the length can be calculated by starting with the largest +supported opcode adding one, dividing by 64 and rounding up. +In other words, for +VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the +length of \field{command_specific_result} and +\field{command_specific_data} respectively will be +$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division +with round up and \field{max_cmd} is the largest available command opcode. + +The array is also allowed to be larger and to additionally +include an arbitrary number of all-zero entries. + +Accordingly, bits 0 and 1 corresponding to opcode 0 +(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 218
Quoted 10 lines
admin.tex +10
-0+(VIRTIO_ADMIN_CMD_LIST_USE) are +always set in \field{device_admin_cmds[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY. + +For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0. +The \field{group_member_id} is unused. It is set to zero by driver. +This command has no command specific data. +The device, upon success, returns a result in +\field{command_specific_result} in the format +\field{struct virtio_admin_cmd_list} describing the +list of administration commands supported for the given group.
Hmm, IIUC, the list is per group type (not group). If yes, could you please consistently use "group type"?
Also, I think you are missing requirement for driver to specify \field{group_type}.
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 224
Quoted 6 lines
admin.tex +6
-0+ + +For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode} +is set to 0x1. +The \field{group_member_id} is unused. It is set to zero by driver. +The \field{command_specific_data} is in the format
Missing \field{group_type} here as well.
Quoted 5 lines
+\field{struct
virtio_admin_cmd_list} describing theadmin.tex +4
-0+list of administration commands used by the driver. +This command has no command specific result. + +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 239
Quoted 10 lines
admin.tex +10
-0+query the list of commands valid for this group and before sending +any commands for any member of a group. + +The driver then enables use of some of the opcodes by sending to +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is +both understood and used by the driver. + +If the device supports the command list used by the driver, the +device completes the command with status VIRTIO_ADMIN_STATUS_OK.
How can the device not support the command list passed by driver? In the para above you say it is a subset of query list. Wouldn't it be better to say that driver would have to pass cmd op which was not in the query list? IDK, maybe better to emphasize this.
Quoted 16 lines
admin.tex +14
-0+If the device does not support the command list, the device +completes the command with status +VIRTIO_ADMIN_STATUS_INVALID_FIELD. + +Note: drivers are assumed not to set bits in device_admin_cmds +if they are not familiar with how the command opcode +is used, since devices could have dependencies between +command opcodes. + +It is assumed that all members in a group support and are used +with the same list of commands. However, for owner devices +supporting multiple group types, the list of supported commands +might differ between different group types. +\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
Quoted 9 lines
An administration virtqueue of an owner device is used to submit
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio-dev] [PATCH v10 08/10] admin: command list discovery
In reply to Michael S. TsirkinQuoted 31 lines · Wed, Mar 08, 2023 at 12:54:52PM CET, [email protected] wrote:
Quoted 15 lines · Jiri Pirko
Quoted 10 lines · Thu, Mar 02, 2023 at 02:05:22PM CET, [email protected] wrote:
Add commands to find out which commands does each group support, as well as enable their use by driver.
This will be especially useful once we have multiple group types.An alternative is per-type VQs. This is possible but will require more per-transport work. Discovery through the vq helps keep things contained.
Signed-off-by Max Gurtovoy <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>[...]
Quoted 14 lines
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 236
Quoted 9 lines
admin.tex +9
-0+ +The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to +query the list of commands valid for this group and before sending +any commands for any member of a group. + +The driver then enables use of some of the opcodes by sending to +the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset +of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is +both understood and used by the driver.To my untrained ear, this sounds somewhat similar to the feature negotiantion mechanism. Why the fact that device/driver supports some command can't be covered by just another feature? Looks like unnecassary complexicity to negotiate supported commands like this.
Quoted 7 lines
Absolutely, it is similar. The issue is that a single device can be an owner for multiple group types.
For example, I think an SRIOV PF itself can have subfunctions with SIOV as well as virtual functions with SRIOV.
And the set of supported commands might differ.We thus need a command that is kind of like features but per group type.
I see. But that does not mean it can't be done using existing features:
_F_SRIOV_CMD1_F_SRIOV_CMD2
.._F_SIOV_CMD1_F_SIOV_CMD2
...
Quoted 7 lines
As a small bonus as we are building this from scratch, I added something which was requested for a long time from features which is blocking access to commands not allowed by features. With standard virtio features we trust the driver not to send commands before enabling a feature and this was considered a maintainance/security problem.
I understand. Is it unfixable for features not to break buggy drivers?
Quoted 5 lines
Signature · 2 lines
-- MST
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 97 lines · Jiri Pirko
Quoted 89 lines · Tue, Mar 07, 2023 at 08:03:47PM CET, [email protected] wrote:
Quoted 77 lines · Jiri Pirko
Quoted 51 lines · Tue, Mar 07, 2023 at 03:39:11PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c repQuoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 8 lines
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
I see. Then I believe that the device should handle the dependencies. Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member BThe device according to internal implementation can either serialize the 2 group member creations or do it in parallel, if it supports it.
Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member AQuoted 4 lines
Here the serialization is necessary and the device is the one to take care of it.
Makes sense?
Yes, I understand. The spec would need to define ordering rules for specific commands and the device must implement them. It allows the driver to pipeline commands while also allowing out-of-order completion (parallelism) in some cases. The disadvantage of this approach is complexity in the spec and implementations.
Quoted 6 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
Quoted 5 lines
But isn't this duplicating the burden of maintaining dependencies to both driver and device? I mean, device should not depend on driver doing the right thing, that means it has to check the dependencies for every incoming command anyway. The only difference would be to wait instead of returning "
-EBUSY" in case the dependency is not satisfied yet.
The device does not need to reject commands that have dependencies with -EBUSY. The result of commands with dependencies is either A -> B or B -> A.
For example:
1. Create Group Member A
2. Delete Group Member A
Command 2 might succeed or it might fail with -ENOENT because Group Member A doesn't exist yet.
Quoted 4 lines
Device knows exactly what are the dependencies. And I believe, those are device implementation specific. For example, some implementation could support parallel VF config cmd execution, some implementation might need to serialize that. Driver has no clue.
Yes, that's up to the device. Out-of-order completion is a superset of in-order completion. So the device is allowed to run commands in series when it wants. A driver designed for out-of-order completion will work fine either way.
Quoted 2 lines
Could you please elaborate a bit more what you mean by "complexity in the spec"?
When adding commands to the spec, the dependency relationships with other commands need to be thought about and documented.
Device implementers need to get those relationships right. That means they need to remember that command B waits for command A.
Driver implementers have to understand that command B waits for command A but not command C according to the spec.
That seems complex to me.
Stefan
[virtio-comment] Re: [virtio] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Quoted 108 lines · Wed, Mar 08, 2023 at 01:44:18PM CET, [email protected] wrote:
Quoted 97 lines · Jiri Pirko
Quoted 89 lines · Tue, Mar 07, 2023 at 08:03:47PM CET, [email protected] wrote:
Quoted 77 lines · Jiri Pirko
Quoted 51 lines · Tue, Mar 07, 2023 at 03:39:11PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c repQuoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 8 lines
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
I see. Then I believe that the device should handle the dependencies. Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member BThe device according to internal implementation can either serialize the 2 group member creations or do it in parallel, if it supports it.
Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member AQuoted 4 lines
Here the serialization is necessary and the device is the one to take care of it.
Makes sense?
Yes, I understand. The spec would need to define ordering rules for specific commands and the device must implement them. It allows the driver to pipeline commands while also allowing out-of-order completion (parallelism) in some cases. The disadvantage of this approach is complexity in the spec and implementations.
Quoted 6 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
Quoted 5 lines
But isn't this duplicating the burden of maintaining dependencies to both driver and device? I mean, device should not depend on driver doing the right thing, that means it has to check the dependencies for every incoming command anyway. The only difference would be to wait instead of returning "
-EBUSY" in case the dependency is not satisfied yet.The device does not need to reject commands that have dependencies with
-EBUSY. The result of commands with dependencies is either A -> B or B -> A.
Quoted 6 lines
For example:
1. Create Group Member A
2. Delete Group Member ACommand 2 might succeed or it might fail with
-ENOENTbecause Group Member A doesn't exist yet.
Yeah, correct.
Quoted 10 lines
Quoted 5 lines
Device knows exactly what are the dependencies. And I believe, those are device implementation specific. For example, some implementation could support parallel VF config cmd execution, some implementation might need to serialize that. Driver has no clue.
Yes, that's up to the device. Out-of-order completion is a superset of in-order completion. So the device is allowed to run commands in series when it wants. A driver designed for out-of-order completion will work fine either way.
Agreed.
Quoted 9 lines
Quoted 3 lines
Could you please elaborate a bit more what you mean by "complexity in the spec"?
When adding commands to the spec, the dependency relationships with other commands need to be thought about and documented.
Device implementers need to get those relationships right. That means they need to remember that command B waits for command A.
That's my point, this is device implementation specific. Spec should not define anything like this, that might be limiting in some of the cases.
Quoted 5 lines
Driver implementers have to understand that command B waits for command A but not command C according to the spec.
That seems complex to me.
Sure, I agree this does not belong to the spec, it can't, really. For the record, I never suggested it.
I think we are in agreement.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
In reply to Michael S. TsirkinQuoted 11 lines · Michael S. Tsirkin
Quoted 9 lines · David Edmondson
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 3 lines · Michael S. Tsirkin
admin.tex +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.
Quoted 7 lines
Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.
This puzzles me.
I can see the point of blocking LIST_USE, but not of blocking LIST_QUERY. What's the purpose of this?
Signature · 17 lines
Time is waiting to explain, why refuse? This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
Quoted 16 lines · Wed, Mar 08, 2023 at 01:59:10PM CET, [email protected] wrote:
Quoted 14 lines · Michael S. Tsirkin
Quoted 9 lines · David Edmondson
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 3 lines · Michael S. Tsirkin
admin.tex +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
Quoted 8 lines
Quoted 4 lines
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.This puzzles me.
I can see the point of blocking
LIST_USE, but not of blocking
What is that point, I'm curious.
Quoted 1 line
LIST_QUERY. What's the purpose of this?
I think that this might be just for the sake of treating all commands equally.
Quoted 8 lines
Signature · 6 lines
Time is waiting to explain, why refuse? --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Michael S. TsirkinQuoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Max Gurtovoy
Quoted 3 lines
Quoted 2 lines
5.
VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
I don't think it's a good idea, we'll have to agree to disagree.
Ok.
can you explain why isn't this a good idea please ?Pointless complexity for spec, devices and drivers. In the end drivers don't really do anything with all this detailed info.
Pointless according to your own opinion. As I mentioned, other specifications and devices work this way. I didn't invent it.
All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
In reply to Jiri PirkoQuoted 26 lines · Jiri Pirko
Quoted 16 lines · Wed, Mar 08, 2023 at 01:59:10PM CET, [email protected] wrote:
Quoted 14 lines · Michael S. Tsirkin
Quoted 9 lines · David Edmondson
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 3 lines · Michael S. Tsirkin
admin.tex +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
Quoted 8 lines
Quoted 4 lines
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.This puzzles me.
I can see the point of blocking
LIST_USE, but not of blocking
Quoted 8 lines
What is that point, I'm curious.
Quoted 1 line
LIST_QUERY. What's the purpose of this?I think that this might be just for the sake of treating all commands equally.
Right. There was this vague idea that blocking all commands is good to have. E.g. if you are making some drastic changes to the device?
Quoted 8 lines
Quoted 8 lines
Signature · 6 lines
Time is waiting to explain, why refuse? --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
In reply to Jiri PirkoQuoted 22 lines · Jiri Pirko
Quoted 21 lines · Wed, Mar 08, 2023 at 01:59:10PM CET, [email protected] wrote:
Quoted 18 lines · Michael S. Tsirkin
Quoted 9 lines · David Edmondson
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 3 lines · Michael S. Tsirkin
admin.tex +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.
Quoted 5 lines
Quoted 3 lines
This puzzles me.
I can see the point of blocking
LIST_USE, but not of blockingWhat is that point, I'm curious.
I might restrict the commands permitted by a device before handing it off to some subordinate less trusted code.
Quoted 15 lines
Quoted 2 lines
LIST_QUERY. What's the purpose of this?I think that this might be just for the sake of treating all commands equally.
Quoted 8 lines
Signature · 6 lines
Time is waiting to explain, why refuse? --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
Signature · 17 lines
I used to get mad at my school, the teachers who taught me weren't cool. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
Quoted 24 lines · Wed, Mar 08, 2023 at 02:44:08PM CET, [email protected] wrote:
Quoted 22 lines · Jiri Pirko
Quoted 21 lines · Wed, Mar 08, 2023 at 01:59:10PM CET, [email protected] wrote:
Quoted 18 lines · Michael S. Tsirkin
Quoted 9 lines · David Edmondson
admin.tex \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti · line 371
Quoted 3 lines · Michael S. Tsirkin
admin.tex +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
This puzzles me.
I can see the point of blocking
LIST_USE, but not of blockingWhat is that point, I'm curious.
I might restrict the commands permitted by a device before handing it off to some subordinate less trusted code.
Hmm, that is theoretical, or is there a usecase for that?
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
LIST_QUERY. What's the purpose of this?I think that this might be just for the sake of treating all commands equally.
Quoted 10 lines
Quoted 8 lines
Quoted 8 lines
Signature · 6 lines
Time is waiting to explain, why refuse? --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.phpSignature · 1 line
I used to get mad at my school, the teachers who taught me weren't cool.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [virtio] Re: [virtio-comment] [PATCH v10 09/10] admin: conformance clauses
Quoted 27 lines · Jiri Pirko
Quoted 26 lines · Wed, Mar 08, 2023 at 02:44:08PM CET, [email protected] wrote:
Quoted 24 lines · Jiri Pirko
Quoted 23 lines · Wed, Mar 08, 2023 at 01:59:10PM CET, [email protected] wrote:
Quoted 11 lines · Michael S. Tsirkin
Quoted 9 lines · David Edmondson
Review
Quoted 3 lines · Michael S. Tsirkin
(quoted diff) +3
-0+The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and +VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE +with respective bits cleared in \field{command_specific_data}.This runs contrary to the assertion "The list of supported commands MUST NOT shrink", given that a driver is told to assume that
VIRTIO_ADMIN_CMD_LIST_QUERYandVIRTIO_ADMIN_CMD_LIST_USEare the only commands initially available.Quoted 7 lines
Commands are still available just disabled, the meaning of "MUST NOT shrink" is clarified by the following:
> > +after reporting a given command as supported through > > +VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT later report it > > +as unsupported.
I will stick an "i.e." there to make it hopefully clearer.This puzzles me.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
I can see the point of blocking
LIST_USE, but not of blockingWhat is that point, I'm curious.
I might restrict the commands permitted by a device before handing it off to some subordinate less trusted code.
Hmm, that is theoretical, or is there a usecase for that?
Maybe there will be a future version replacing VIRTIO_ADMIN_CMD_LIST_QUERY with a different command. Then we start with a device supporting both VIRTIO_ADMIN_CMD_LIST_QUERY and the new VIRTIO_ADMIN_FOO, and we block VIRTIO_ADMIN_CMD_LIST_QUERY for testing, to make sure driver is able to handle future devices without VIRTIO_ADMIN_CMD_LIST_QUERY.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
LIST_QUERY. What's the purpose of this?I think that this might be just for the sake of treating all commands equally.
Quoted 3 lines
Signature · 1 line
Time is waiting to explain, why refuse?
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.phpSignature · 1 line
I used to get mad at my school, the teachers who taught me weren't cool.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Max GurtovoyQuoted 71 lines · Max Gurtovoy
Quoted 68 lines · Stefan Hajnoczi
Quoted 66 lines · Max Gurtovoy
Quoted 50 lines · Stefan Hajnoczi
Quoted 35 lines · Jason Wang
Quoted 25 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 6 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Quoted 3 lines
Or device need mandate some kind of QOS here, e.g a request must be complete in some time. Otherwise we don't have sufficient reliability for using it as management task?
Yes, if all commands can be executed in bounded time then a guarantee is possible.
Here is an example where that's hard: imagine a virtio-blk device backed by network storage. When an admin queue command is used to delete a group member, any of the group member's in-flight I/O requests need to be aborted. If the network hangs while the group member is being deleted, then the device can't complete an orderly shutdown of I/O requests in a reasonable time.
Quoted 5 lines
That example shows a basic group admin command that I think Michael is about to propose. We can't avoid this problem by not making it a group admin command - it needs to be a group admin command. So I think it's likely that there will be admin commands that take an unbounded amount of time to complete. One way to achieve what you mentioned is timeouts.
I think that you're getting into device specific implementation details and I'm not sure it's necessary.
I don't think we need to abort admin commands. Admin commands can be flushed/aborted during the device reset phase.
Only IO commands should have the possibility to being aborted as you mentioned in NVMe and SCSI (and potentially in virtio-blk).
Quoted 8 lines
Quoted 6 lines
It's a general design issue that should be clarified now rather than being left unspecified.
I'm not saying that it must be possible to abort admin commands. There are other options, like requiring the device itself to fail a command after a timeout.
do you have an example of timeout today for control vq ?
Do you mean the virtio-net control virtqueue? I don't think it has any commands with an unbounded execution time.
Quoted 10 lines
Quoted 4 lines
Or we could say that admin commands must complete within bounded time, but I'm not sure that is implementable for some device types like virtio-blk, virtio-scsi, and virtiofs.
No we can't.
Some commands, for example FW upgrade can take 10 minutes and it's perfectly fine. Other commands like setting feature bit will take 1 millisec. Each device implements commands in a different internal logic so we can't expect to complete after X time.
When I say bounded time, I mean that it finishes in a finite amount of time. I'm not saying there is a specific time X that all device implementations must satisfy. Unbounded means it might never finish.
Quoted 13 lines
Device can go to so FATAL state in case a command is stuck and causing internal errors in it.
Quoted 9 lines
Quoted 4 lines
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Forgetting about in-flight requests doesn't necessarily make them go away. It creates a race between forgotten requests and reconnection. In the worst case a forgotten write request takes effect after reconnection, causing data corruption.
Quoted 5 lines
For making it work without data corruption we need a cooperation of the target side for sure. But this is fine since the target in that case is part of the "virtio-blk backend".
One solution is that the target can decide it will flush all the requests to the storage device before accepting new connections.
This solution shifts the unbounded time from disconnection to connection. The Group Member Delete command will complete quickly but a subsequent Group Member Create command for the same underlying storage device would need to wait until the requests are done.
Therefore I think the admin queue must be designed under the assumption that some commands take a very long time.
Stefan
[virtio-comment] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees
In reply to David EdmondsonQuoted 45 lines · David Edmondson
Quoted 35 lines · Michael S. Tsirkin
Quoted 34 lines · David Edmondson
Quoted 15 lines · Michael S. Tsirkin
Feature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented. Do it now.
Suggested-by Halil Pasic <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)content.tex +0
-0content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B · line 121
Quoted 8 lines
content.tex +5
-0@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit. +To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according to +this version of the specification.)I find this (the bit in parenthesis) confusing.
Why are drivers not supposed to accept features that they have been offered, given that they can't know that the device cannot handle the feature that it just offered?
Is this alluding to the later section:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
feature bits not described in this specification, reserved feature bits and feature bits reserved or not supported for the specific transport or the specific device type
?
exactly. how would you put this better? given an example?
Quoted 7 lines
Perhaps it would be enough to say:
Quoted 2 lines
(even though drivers are not supposed to accept unrecognised features in the first place even if offered, according to the specification)
"Unrecognised" is intended as a shorthand for the whole "not described, reserved, ...". Maybe "unrecognised or reserved"?
Hm, what about
"even though drivers are not supposed to accept any unspecified, reserved, or unsupported features even if offered..."
?
I'm not sure how we can make this both short and descriptive enough...
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Max GurtovoyQuoted 10 lines · Max Gurtovoy
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Max Gurtovoy
Quoted 4 lines
Quoted 2 lines
5.
VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).I don't think it's a good idea, we'll have to agree to disagree.
Ok.
can you explain why isn't this a good idea please ?
Quoted 8 lines
Quoted 3 lines
Pointless complexity for spec, devices and drivers. In the end drivers don't really do anything with all this detailed info.
Pointless according to your own opinion. As I mentioned, other specifications and devices work this way. I didn't invent it.
All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?
FWIW, I've spent enough time translating detailed-but-not-really-that-useful error status codes to -EINVAL for s390-specific things...
The only thing that might make sense to transmit to a consumer would be the difference between "this thing won't work" and "this thing might work if you retry later" (if the consumer is even in a place to retry.)
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
Quoted 21 lines · Cornelia Huck
Quoted 10 lines · Max Gurtovoy
Quoted 9 lines · Michael S. Tsirkin
Quoted 7 lines · Max Gurtovoy
Quoted 4 lines
Quoted 2 lines
5.
VIRTIO_ADMIN_STATUS_COMMAND_SPECIFIC_ERR(for more info read thecommand_specific_errorfield).I don't think it's a good idea, we'll have to agree to disagree.
Ok.
can you explain why isn't this a good idea please ?Quoted 8 lines
Quoted 3 lines
Pointless complexity for spec, devices and drivers. In the end drivers don't really do anything with all this detailed info.
Pointless according to your own opinion. As I mentioned, other specifications and devices work this way. I didn't invent it.
All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?
Quoted 7 lines
FWIW, I've spent enough time translating detailed-but-not-really-that-useful error status codes to
-EINVALfor s390-specific things...The only thing that might make sense to transmit to a consumer would be the difference between "this thing won't work" and "this thing might work if you retry later" (if the consumer is even in a place to retry.)
One place where that extra detail might be helpful is device debugging. But there an error code out of list is too restrictive anyway, you want "Turbo Encabulator 42 logarithmic casing failure". I am still looking for something that drivers are not likely to abuse. Maybe an ASCII string?
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [virtio] [PATCH v10 01/10] virtio: document forward compatibility guarantees
In reply to Cornelia HuckQuoted 55 lines · Cornelia Huck
Quoted 45 lines · David Edmondson
Quoted 35 lines · Michael S. Tsirkin
Quoted 34 lines · David Edmondson
Quoted 15 lines · Michael S. Tsirkin
Feature negotiation forms the basis of forward compatibility guarantees of virtio but has never been properly documented. Do it now.
Suggested-by Halil Pasic <[email protected]>Signed-off-by Michael S. Tsirkin <[email protected]>--- content.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)content.tex +0
-0content.tex \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B · line 121
Quoted 8 lines
content.tex +5
-0@@ -114,21 +114,63 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B In particular, new fields in the device configuration space are indicated by offering a new feature bit. +To keep the feature negotiation mechanism extensible, it is important +that devices \em{do not} offer any feature bits that they would not be +able to handle if the driver accepted them (even though drivers are not +supposed to accept them in the first place even if offered, according to +this version of the specification.)I find this (the bit in parenthesis) confusing.
Why are drivers not supposed to accept features that they have been offered, given that they can't know that the device cannot handle the feature that it just offered?
Is this alluding to the later section:
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
feature bits not described in this specification, reserved feature bits and feature bits reserved or not supported for the specific transport or the specific device type
?
exactly. how would you put this better? given an example?
Quoted 7 lines
Perhaps it would be enough to say:
Quoted 2 lines
(even though drivers are not supposed to accept unrecognised features in the first place even if offered, according to the specification)
"Unrecognised" is intended as a shorthand for the whole "not described, reserved, ...". Maybe "unrecognised or reserved"?
Quoted 6 lines
Hm, what about
"even though drivers are not supposed to accept any unspecified, reserved, or unsupported features even if offered..."
?
ok
Quoted 1 line
I'm not sure how we can make this both short and descriptive enough...
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[virtio-comment] Re: [PATCH v10 03/10] admin: introduce group administration commands
In reply to Max GurtovoyQuoted 1 line · Max Gurtovoy
All drivers in SW world ? All drivers in Linux ? or Virtio drivers ?
Most of the above.
Signature · 18 lines
MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 75 lines · Stefan Hajnoczi
Quoted 74 lines · Max Gurtovoy
Quoted 71 lines · Stefan Hajnoczi
Quoted 59 lines · Max Gurtovoy
Quoted 50 lines · Stefan Hajnoczi
Quoted 35 lines · Jason Wang
Quoted 25 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 6 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Quoted 3 lines
Or device need mandate some kind of QOS here, e.g a request must be complete in some time. Otherwise we don't have sufficient reliability for using it as management task?
Yes, if all commands can be executed in bounded time then a guarantee is possible.
Here is an example where that's hard: imagine a virtio-blk device backed by network storage. When an admin queue command is used to delete a group member, any of the group member's in-flight I/O requests need to be aborted. If the network hangs while the group member is being deleted, then the device can't complete an orderly shutdown of I/O requests in a reasonable time.
Quoted 5 lines
That example shows a basic group admin command that I think Michael is about to propose. We can't avoid this problem by not making it a group admin command - it needs to be a group admin command. So I think it's likely that there will be admin commands that take an unbounded amount of time to complete. One way to achieve what you mentioned is timeouts.
Quoted 7 lines
I think that you're getting into device specific implementation details and I'm not sure it's necessary.
I don't think we need to abort admin commands. Admin commands can be flushed/aborted during the device reset phase.
Only IO commands should have the possibility to being aborted as you mentioned in NVMe and SCSI (and potentially in virtio-blk).It's a general design issue that should be clarified now rather than being left unspecified.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
I'm not saying that it must be possible to abort admin commands. There are other options, like requiring the device itself to fail a command after a timeout.
do you have an example of timeout today for control vq ?
Do you mean the virtio-net control virtqueue? I don't think it has any commands with an unbounded execution time.
Correct. So why introducing it now ?
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Or we could say that admin commands must complete within bounded time, but I'm not sure that is implementable for some device types like virtio-blk, virtio-scsi, and virtiofs.
No we can't.
Some commands, for example FW upgrade can take 10 minutes and it's perfectly fine. Other commands like setting feature bit will take 1 millisec. Each device implements commands in a different internal logic so we can't expect to complete after X time.
Quoted 3 lines
When I say bounded time, I mean that it finishes in a finite amount of time. I'm not saying there is a specific time X that all device implementations must satisfy. Unbounded means it might never finish.
There might be a chance that any command for any virtio device type will never finish. Nothing new here in the adminq.
what one can do is to set a timeout for himself and if this timeout expire - check the device status. If it needs_reset - do a reset. if status is ok, then wait some more time. After X retries, unmap buffers or reset the adminq.
Quoted 20 lines
Quoted 14 lines
Device can go to so FATAL state in case a command is stuck and causing internal errors in it.
Quoted 9 lines
Quoted 4 lines
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Forgetting about in-flight requests doesn't necessarily make them go away. It creates a race between forgotten requests and reconnection. In the worst case a forgotten write request takes effect after reconnection, causing data corruption.
Quoted 5 lines
For making it work without data corruption we need a cooperation of the target side for sure. But this is fine since the target in that case is part of the "virtio-blk backend".
One solution is that the target can decide it will flush all the requests to the storage device before accepting new connections.
Quoted 7 lines
This solution shifts the unbounded time from disconnection to connection. The Group Member Delete command will complete quickly but a subsequent Group Member Create command for the same underlying storage device would need to wait until the requests are done.
Therefore I think the admin queue must be designed under the assumption that some commands take a very long time.
For sure an admin command may take long time. FW upgrade can take 10 minutes for example. But each device is free to implement internal logic as he choose.
Same for live migration, when we stop/quiesce a device we must make sure it doesn't master any DMA operations. Thus, in some implementations we need to wait for all inflights to end fast. In others, we can invalidate the access to host/guest memory and wait for completions until the freeze state.
Bottom line, this is device implementation specific consideration.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 7 lines
From: Stefan Hajnoczi <[email protected]>
Sent: Wednesday, March 8, 2023 9:13 AMTherefore I think the admin queue must be designed under the assumption that some commands take a very long time.
+1
Re: [virtio-comment] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Parav PanditQuoted 21 lines · Parav Pandit
Quoted 21 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Friday, March 3, 2023 3:34 AMQuoted 3 lines · Parav Pandit
Did you miss reviewed-by from [1] or this is an old series reposted?
[1]
https://lists.oasis-open.org/archives/virtio-dev/202302/msg00242.htmlAs a general rule, we don't strictly need to track reviewed by since there's a ballot (and presumably people review before voting).
People also tack on Signed-off-by: (and I do it too) but as long as we don't document what it means it's kind of vague, and the process of subscribing to the mailing list is a kind of replacement.
If you think everyone needs to follow practices like netdev does, we really need something written up, and agree on it.
E.g. I work on the linux kernel too, so I can copy practices from there, but even linux isn't uniform.
Quoted 7 lines
Quoted 2 lines
And I wonder whether it's worth it - it definitely makes contributing to Linux harder, and even within Linux it pushes contributors away.
The number of virtio spec contributors are in order of magnitude less than Linux kernel or just the Linux netdev.
Patch split up reduces lot of time author and reviewers.
For example, interrupt moderation at v10 can be very easily split up where prep patch can have RB tag and v11 doesn't need reviewers and author's time. Your work here with 10 patches is yet another good example of it.
I remember Max and I started with 6 patches... and current 10 are more useful way to split them.
I'd argue that splitting changes in a way that makes it easy for reviewers is a good thing for any project (although practices on many forges seem to go into another direction...)
Quoted 11 lines
Quoted 10 lines
At least for Linux
tracking history in a precise way is extremely important since it's helpful with stability. Spec is very different.Until we have a good contribution documentation I think we should not ask people to follow a pseudo Linux work flow with requests like "please split this patchset up" or "track changes across patch versions"
simply because there's no good docs to teach people what exactly is the best practice.
Quoted 5 lines
Yes, I wanted to update the contributing document. It is in my to-do list.
But given the small crowd of contributors right now, almost everyone is familiar with the RB, ack tag. At moment it has two main reasons.1. Acknowledge the work and efforts that go in the review
I agree, and I try to include tags when applying.
Quoted 2 lines
2. When in question, reach out later to the spec author or reviewers to know about context/design etc.
3. Same reviewer doesn't need to go through the patch which already has RB tag of him/her.
I tend to use R-b in that way as well, but it relies on the patch author not doing substantial changes between versions...
I guess the usage of R-b et al in the virtio spec stems from the fact that many contributors are also Linux and/or QEMU contributors -- not sure if it makes sense to enshrine their usage, but
4. We can get an R-b from a non-TC member who is an expert on the topic (and follows standard Linux/QEMU/... practices.)
In the end, how the TC members are voting is the only thing that matters for inclusion.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Parav PanditQuoted 10 lines · Parav Pandit
Quoted 2 lines
And I wonder whether it's worth it - it definitely makes contributing to Linux harder, and even within Linux it pushes contributors away.
The number of virtio spec contributors are in order of magnitude less than Linux kernel or just the Linux netdev. Patch split up reduces lot of time author and reviewers. For example, interrupt moderation at v10 can be very easily split up where prep patch can have RB tag and v11 doesn't need reviewers and author's time. Your work here with 10 patches is yet another good example of it. I remember Max and I started with 6 patches... and current 10 are more useful way to split them.
It's a big project though. All I ask is that we try to be gentler than netdev to new contributors. If there's a 10 line patch it is not necessary to be pedantic and ask for it to be split up to a 3 patch series.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 30 lines · Michael S. Tsirkin
Quoted 22 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 5 lines
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Quoted 8 lines
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Sorry to be late to the party, I'm trying to catch up. Is it common to have cover letter for features this brief? I mean, from the cover letter, I'm totally unable to understand what you are introducing here.
Quoted 6 lines
Could you elaborate about what you are aiming to achive with this? Could you shed some usecases perhaps?
I have to be missing something obvious, but I don't get why any notion of SR-IOV could be beneficial for virtio.
Quoted 6 lines
Good point, I'll add a bit of motivation.
For SR-IOV, it is not unusual for PFs to excercise control over VFs. There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Unfortunately, information in the cover letter does not make it into git -- should things like that go into the github issue (and into the ballot?) It's useful both for reviewing (cover letter) and understanding the rationale later (github/ballot).
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [virtio-comment] RE: [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved
In reply to Cornelia HuckQuoted 11 lines
From: Cornelia Huck <[email protected]>
Sent: Wednesday, March 8, 2023 11:24 AMQuoted 3 lines
Quoted 2 lines
And I wonder whether it's worth it - it definitely makes contributing to Linux harder, and even within Linux it pushes contributors away.
The number of virtio spec contributors are in order of magnitude less than
Linux kernel or just the Linux netdev.
Quoted 2 lines
Patch split up reduces lot of time author and reviewers.
For example, interrupt moderation at v10 can be very easily split up where
Quoted 9 lines
prep patch can have RB tag and v11 doesn't need reviewers and author's time.
Quoted 2 lines
Your work here with 10 patches is yet another good example of it. I remember Max and I started with 6 patches... and current 10 are more
useful way to split them.
I'd argue that splitting changes in a way that makes it easy for reviewers is a good thing for any project (although practices on many forges seem to go into another direction...)
In my experience in dpdk, Linux netdev, rdma, nvme, tells me that patch splitting is common practice.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
At least for Linux
tracking history in a precise way is extremely important since it's helpful with stability. Spec is very different.Until we have a good contribution documentation I think we should not ask people to follow a pseudo Linux work flow with requests like "please split this patchset up" or "track changes across patch versions" simply because there's no good docs to teach people what exactly is the best practice.
Quoted 9 lines
Quoted 2 lines
Yes, I wanted to update the contributing document. It is in my to-do list. But given the small crowd of contributors right now, almost everyone is
familiar with the RB, ack tag.
Quoted 3 lines
At moment it has two main reasons.
1. Acknowledge the work and efforts that go in the review
I agree, and I try to include tags when applying.
Sure. It is not about you.
It is about the patch author when he/she sends v9 to v10, adding R-b tag in v10 (for the already reviewed work of v9) helps to ack and speed up the v10 review.
Quoted 8 lines
Quoted 1 line
2. When in question, reach out later to the spec author or reviewers to know
about context/design etc.
Quoted 1 line
3. Same reviewer doesn't need to go through the patch which already has RB
tag of him/her.
I tend to use R-b in that way as well, but it relies on the patch author not doing substantial changes between versions...
Quoted 7 lines
I guess the usage of R-b et al in the virtio spec stems from the fact that many contributors are also Linux and/or QEMU contributors -- not sure if it makes sense to enshrine their usage, but
4. We can get an R-b from a non-TC member who is an expert on the topic (and follows standard Linux/QEMU/... practices.)
The intent here was between vN to v(N+1).
Virtio spec maintainers are likely adding R-b when they merge anyway. No issue there.
Quoted 2 lines
In the end, how the TC members are voting is the only thing that matters for inclusion.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Max GurtovoyQuoted 82 lines · Max Gurtovoy
Quoted 79 lines · Stefan Hajnoczi
Quoted 78 lines · Max Gurtovoy
Quoted 74 lines · Stefan Hajnoczi
Quoted 59 lines · Max Gurtovoy
Quoted 50 lines · Stefan Hajnoczi
Quoted 35 lines · Jason Wang
Quoted 25 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device allowed to process the next command from the virtqueue during this time, possibly completing it before the first command?
This requires additional clarification in the spec because "they are processed by the device in the order in which they are queued" does not explain whether commands block the virtqueue (in order completion) or not (out of order completion).
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
They rely on multi-queue for abort operations:
In virtio-scsi the abort requests (
VIRTIO_SCSI_T_TMF_ABORT_TASK) are sent on the control virtqueue. The the request identifier namespace is shared across all virtqueues so it's possible to abort a request that was submitted to any command virtqueue.NVMe also follows the same design where abort commands are sent on the Admin Submission Queue instead of an I/O Submission Queue. It's possible to identify NVMe requests by <Submission Queue ID, Command Identifier>.
Quoted 6 lines
virtio-blk doesn't support aborting requests.
I think the logic behind this design is that if a queue gets stuck processing long-running requests, then the device should not be forced to perform lookahead in the queue to find abort commands. A separate control/admin queue is used for the abort requests.
Quoted 3 lines
Or device need mandate some kind of QOS here, e.g a request must be complete in some time. Otherwise we don't have sufficient reliability for using it as management task?
Yes, if all commands can be executed in bounded time then a guarantee is possible.
Here is an example where that's hard: imagine a virtio-blk device backed by network storage. When an admin queue command is used to delete a group member, any of the group member's in-flight I/O requests need to be aborted. If the network hangs while the group member is being deleted, then the device can't complete an orderly shutdown of I/O requests in a reasonable time.
Quoted 5 lines
That example shows a basic group admin command that I think Michael is about to propose. We can't avoid this problem by not making it a group admin command - it needs to be a group admin command. So I think it's likely that there will be admin commands that take an unbounded amount of time to complete. One way to achieve what you mentioned is timeouts.
Quoted 7 lines
I think that you're getting into device specific implementation details and I'm not sure it's necessary.
I don't think we need to abort admin commands. Admin commands can be flushed/aborted during the device reset phase.
Only IO commands should have the possibility to being aborted as you mentioned in NVMe and SCSI (and potentially in virtio-blk).It's a general design issue that should be clarified now rather than being left unspecified.
I'm not saying that it must be possible to abort admin commands. There are other options, like requiring the device itself to fail a command after a timeout.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
do you have an example of timeout today for control vq ?
Do you mean the virtio-net control virtqueue? I don't think it has any commands with an unbounded execution time.
Correct. So why introducing it now ?
The examples I've given are the create and delete group member operations. I think those operations will take unbounded time in some device implementations.
Quoted 17 lines
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
Or we could say that admin commands must complete within bounded time, but I'm not sure that is implementable for some device types like virtio-blk, virtio-scsi, and virtiofs.
Quoted 5 lines
No we can't.
Some commands, for example FW upgrade can take 10 minutes and it's perfectly fine. Other commands like setting feature bit will take 1 millisec. Each device implements commands in a different internal logic so we can't expect to complete after X time.Quoted 3 lines
When I say bounded time, I mean that it finishes in a finite amount of time. I'm not saying there is a specific time X that all device implementations must satisfy. Unbounded means it might never finish.
Quoted 7 lines
There might be a chance that any command for any virtio device type will never finish. Nothing new here in the adminq.
what one can do is to set a timeout for himself and if this timeout expire - check the device status. If it
needs_reset- do a reset. if status is ok, then wait some more time.
After X retries, unmap buffers or reset the adminq.
Michael: What effect does resetting the group owner device have on group member devices?
I'm concerned that this approach disrupts all group member devices. For example, you try to add a new device but the command hangs. In order to recover you now have to reset the group owner device and this breaks all the group member devices.
Quoted 32 lines
Quoted 20 lines
Quoted 14 lines
Device can go to so FATAL state in case a command is stuck and causing internal errors in it.
Quoted 9 lines
Quoted 4 lines
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Forgetting about in-flight requests doesn't necessarily make them go away. It creates a race between forgotten requests and reconnection. In the worst case a forgotten write request takes effect after reconnection, causing data corruption.
Quoted 5 lines
For making it work without data corruption we need a cooperation of the target side for sure. But this is fine since the target in that case is part of the "virtio-blk backend".
One solution is that the target can decide it will flush all the requests to the storage device before accepting new connections.Quoted 7 lines
This solution shifts the unbounded time from disconnection to connection. The Group Member Delete command will complete quickly but a subsequent Group Member Create command for the same underlying storage device would need to wait until the requests are done.
Therefore I think the admin queue must be designed under the assumption that some commands take a very long time.
For sure an admin command may take long time. FW upgrade can take 10 minutes for example.
But each device is free to implement internal logic as he choose.
Quoted 6 lines
Same for live migration, when we stop/quiesce a device we must make sure it doesn't master any DMA operations. Thus, in some implementations we need to wait for all inflights to end fast. In others, we can invalidate the access to host/guest memory and wait for completions until the freeze state.
Bottom line, this is device implementation specific consideration.
What I'm asking is that the spec clarifies the command completion order semantics (in-order or out-of-order), whether there is a mechanism to abort commands, etc.
Device implementers can then take advantage of those aspects to implement devices that don't hang (e.g. health monitoring becomes unavailable when there is a long running command).
If the spec doesn't cover this, then device implementers will not be able to work around it when implementing standard commands like create/delete group member.
Does that make sense?
Stefan
[virtio-comment] Re: [virtio] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 149 lines · Jiri Pirko
Quoted 108 lines · Wed, Mar 08, 2023 at 01:44:18PM CET, [email protected] wrote:
Quoted 97 lines · Jiri Pirko
Quoted 89 lines · Tue, Mar 07, 2023 at 08:03:47PM CET, [email protected] wrote:
Quoted 77 lines · Jiri Pirko
Quoted 51 lines · Tue, Mar 07, 2023 at 03:39:11PM CET, [email protected] wrote:
Quoted 41 lines · Jiri Pirko
Quoted 22 lines · Mon, Mar 06, 2023 at 07:37:31PM CET, [email protected] wrote:
Quoted 21 lines · Stefan Hajnoczi
Quoted 19 lines · Michael S. Tsirkin
Quoted 11 lines · Stefan Hajnoczi
Quoted 10 lines · Michael S. Tsirkin
Quoted 8 lines · Stefan Hajnoczi
What happens if a command takes 1 second to complete, is the device
allowed to process the next command from the virtqueue during this time,
possibly completing it before the first command?This requires additional clarification in the spec because "they are
processed by the device in the order in which they are queued" does not
explain whether commands block the virtqueue (in order completion) or
not (out of order completion).Quoted 4 lines
Quoted 1 line
Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
virtio-scsi, virtio-blk, and NVMe requests may complete out of order. Several may be processed by the device at the same time.
Let's say I submit a write followed by read - is read guaranteed to return an up to date info?
Quoted 6 lines
Quoted 4 lines
In general, no. The driver must wait for the write completion before submitting the read if it wants consistency.
Stefan
I see. I think it's a good design to follow then.
Hmm, is it suitable to have this approach for configuration interface? Storage device is a different beast, having parallel reads and writes makes complete sense for performance.
->read a req
->read b req
->read c req
<-read a rep
<-read b rep
<-read c repQuoted 8 lines
There is no dependency, even between writes.
But in case of configuration, does not make any sense to me. Why is it needed? To pass the burden of consistency of configuration to driver sounds odd at least.
I sense there is no concete idea about what the "admin virtqueue" should serve for exactly.
Quoted 8 lines
It's useful for long-running commands because they prevent other commands from executing.
An example I've given is that deleting a group member might require waiting for the group member's I/O activity to finish. If that I/O activity cannot be cancelled instantaneously, then it could take an unbounded amount of time to delete the group member. The device would be unable to process futher admin commands.
I see. Then I believe that the device should handle the dependencies. Example 1:
-> REQ cmd to create group member A
-> REQ cmd to create group member B
<- REP cmd to create group member A
<- REP cmd to create group member BThe device according to internal implementation can either serialize the 2 group member creations or do it in parallel, if it supports it.
Example 2:
-> REQ cmd to create group member A
-> REQ cmd config group member A
<- REP cmd to create group member A
<- REP cmd config group member AQuoted 4 lines
Here the serialization is necessary and the device is the one to take care of it.
Makes sense?
Yes, I understand. The spec would need to define ordering rules for specific commands and the device must implement them. It allows the driver to pipeline commands while also allowing out-of-order completion (parallelism) in some cases. The disadvantage of this approach is complexity in the spec and implementations.
Quoted 6 lines
An alternative is unconditional out-of-order completion, where there are no per-command ordering rules. The driver must wait for a command to complete if it relies on the results of that command for its next command. I like this approach because it's less complex in the spec and for device implementers, while the burden on the driver implementer is still reasonable.
Quoted 5 lines
But isn't this duplicating the burden of maintaining dependencies to both driver and device? I mean, device should not depend on driver doing the right thing, that means it has to check the dependencies for every incoming command anyway. The only difference would be to wait instead of returning "
-EBUSY" in case the dependency is not satisfied yet.The device does not need to reject commands that have dependencies with
-EBUSY. The result of commands with dependencies is either A -> B or B -> A.Quoted 6 lines
For example:
1. Create Group Member A
2. Delete Group Member ACommand 2 might succeed or it might fail with
-ENOENTbecause Group Member A doesn't exist yet.Yeah, correct.
Quoted 10 lines
Quoted 5 lines
Device knows exactly what are the dependencies. And I believe, those are device implementation specific. For example, some implementation could support parallel VF config cmd execution, some implementation might need to serialize that. Driver has no clue.
Yes, that's up to the device. Out-of-order completion is a superset of in-order completion. So the device is allowed to run commands in series when it wants. A driver designed for out-of-order completion will work fine either way.
Agreed.
Quoted 9 lines
Quoted 3 lines
Could you please elaborate a bit more what you mean by "complexity in the spec"?
When adding commands to the spec, the dependency relationships with other commands need to be thought about and documented.
Device implementers need to get those relationships right. That means they need to remember that command B waits for command A.
That's my point, this is device implementation specific. Spec should not define anything like this, that might be limiting in some of the cases.
Quoted 4 lines
Driver implementers have to understand that command B waits for command A but not command C according to the spec.
Quoted 6 lines
Quoted 1 line
That seems complex to me.
Sure, I agree this does not belong to the spec, it can't, really. For the record, I never suggested it.
I think we are in agreement.
Okay. In that case I think I misunderstood what you meant. Sorry!
Stefan
Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Stefan HajnocziQuoted 17 lines · Stefan Hajnoczi
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Or we could say that admin commands must complete within bounded time, but I'm not sure that is implementable for some device types like virtio-blk, virtio-scsi, and virtiofs.
No we can't.
Some commands, for example FW upgrade can take 10 minutes and it's perfectly fine. Other commands like setting feature bit will take 1 millisec. Each device implements commands in a different internal logic so we can't expect to complete after X time.Quoted 3 lines
When I say bounded time, I mean that it finishes in a finite amount of time. I'm not saying there is a specific time X that all device implementations must satisfy. Unbounded means it might never finish.
There might be a chance that any command for any virtio device type will never finish. Nothing new here in the adminq.
Quoted 7 lines
Quoted 4 lines
what one can do is to set a timeout for himself and if this timeout expire - check the device status. If it
needs_reset- do a reset. if status is ok, then wait some more time.
After X retries, unmap buffers or reset the adminq.Michael: What effect does resetting the group owner device have on group member devices?
virtio level reset? It's a good question. I'd expect them all to be reset no?
Quoted 4 lines
I'm concerned that this approach disrupts all group member devices. For example, you try to add a new device but the command hangs. In order to recover you now have to reset the group owner device and this breaks all the group member devices.
I agree. How about a VQ level reset though? Seems like exactly what's needed here?
Quoted 47 lines
Quoted 32 lines
Quoted 20 lines
Quoted 14 lines
Device can go to so FATAL state in case a command is stuck and causing internal errors in it.
Quoted 9 lines
Quoted 4 lines
For your example, stopping a member is possible even it there are some errors in the network. You can for example destroy all the connections to the remote target and complete all the BIOS with some error.
Forgetting about in-flight requests doesn't necessarily make them go away. It creates a race between forgotten requests and reconnection. In the worst case a forgotten write request takes effect after reconnection, causing data corruption.
Quoted 5 lines
For making it work without data corruption we need a cooperation of the target side for sure. But this is fine since the target in that case is part of the "virtio-blk backend".
One solution is that the target can decide it will flush all the requests to the storage device before accepting new connections.Quoted 7 lines
This solution shifts the unbounded time from disconnection to connection. The Group Member Delete command will complete quickly but a subsequent Group Member Create command for the same underlying storage device would need to wait until the requests are done.
Therefore I think the admin queue must be designed under the assumption that some commands take a very long time.
For sure an admin command may take long time. FW upgrade can take 10 minutes for example.
But each device is free to implement internal logic as he choose.Quoted 6 lines
Same for live migration, when we stop/quiesce a device we must make sure it doesn't master any DMA operations. Thus, in some implementations we need to wait for all inflights to end fast. In others, we can invalidate the access to host/guest memory and wait for completions until the freeze state.
Bottom line, this is device implementation specific consideration.
What I'm asking is that the spec clarifies the command completion order semantics (in-order or out-of-order), whether there is a mechanism to abort commands, etc.
Device implementers can then take advantage of those aspects to implement devices that don't hang (e.g. health monitoring becomes unavailable when there is a long running command).
Quoted 7 lines
If the spec doesn't cover this, then device implementers will not be able to work around it when implementing standard commands like create/delete group member.
Does that make sense?
Stefan
It does, to me.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 39 lines · Cornelia Huck
Quoted 30 lines · Michael S. Tsirkin
Quoted 22 lines · Jiri Pirko
Thu, Mar 02, 2023 at 02:04:48PM CET, [email protected] wrote:
[...]
Quoted 5 lines
TODO (maybe?) - probably ok to defer until this part is upstream:
Add "all members" member id.
Quoted 8 lines
Add commands for MSI, feature discovery.
Add commands for transport vq.
My intent is to try and support both SR-IOV and SIOV usecases with the same structure and maybe even the same VQ.
Sorry to be late to the party, I'm trying to catch up. Is it common to have cover letter for features this brief? I mean, from the cover letter, I'm totally unable to understand what you are introducing here.
Quoted 6 lines
Could you elaborate about what you are aiming to achive with this? Could you shed some usecases perhaps?
I have to be missing something obvious, but I don't get why any notion of SR-IOV could be beneficial for virtio.
Quoted 6 lines
Good point, I'll add a bit of motivation.
For SR-IOV, it is not unusual for PFs to excercise control over VFs. There is interest in the community to include an interface to allow this in the virtio spec, when the PF is a virtio device. This is what this patch does.
Quoted 4 lines
Unfortunately, information in the cover letter does not make it into git -- should things like that go into the github issue (and into the ballot?) It's useful both for reviewing (cover letter) and understanding the rationale later (github/ballot).
Well the github issue has a link to the archives, but sure it's possible to copy that.
Signature · 6 lines
MST --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-comment] Re: [virtio] [PATCH v10 04/10] admin: introduce virtio admin virtqueues
In reply to Jiri PirkoQuoted 15 lines · Jiri Pirko
Quoted 5 lines · Wed, Mar 08, 2023 at 12:50:48PM CET, [email protected] wrote:
Quoted 4 lines · Jiri Pirko
Quoted 3 lines · Tue, Mar 07, 2023 at 05:30:18PM CET, [email protected] wrote:
Quoted 1 line · Jiri Pirko
Hmm, if not for now, the future exension would not be so simple, I fear.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Without knowing what it is I can't say.
Yep, so basically you say, for other things if they appear, let's introduce another queue type? If yes, sounds fair to me.
Yes. For example I find it likely that live migration/failover support will require a queue where driver pre-adds buffers and then device supplies information as state changes.
Quoted 6 lines
I see. So there would be a queue called for example "child state virtqueue" or something like that for the sole purpose of getting the state of VF? Hmm, wouldn't it make more sense to have this done as a part of "group administrarion queue"? I mean, there is already notion of addresing child/VF here. So from my perspective, it is just another "group administration" command.
For sure VF Live Migration, MSIX config of VF, VF feature bits config and others should be admin commands on admin vq. I don't see any reason introducing another type of admin-like vq.
Also we don't need to have multiple admin vqs. This AQ is not aimed for performance.
Quoted 55 lines
We would have to prealloc buffers and use it for this purpose as well. I mean, the cmd payload for other group commands could be put in this buffers as well, making all group command infrastructure consistent.
Something like:
opcode X
group_member_id1
cmd_payload-> buffer Aopcode Y
group_member_id1
cmd_payload-> buffer BBasically this would replace command_specific_data/command_specific_result
Quoted 36 lines
Quoted 35 lines
Quoted 22 lines
Quoted 9 lines
Quoted 7 lines
Passing commands to devices themselves is already covered in spec reasonably well though not in a generic way.
You mean using the control queue, correct?
Depends on the device type. network devices have a control queue, yes.
Quoted 9 lines
Quoted 1 line
From one of the patch description of this patchset I understand that you
cannot use control queue for this because control queue is device-specific, yet group control is device-agnostic.
My undestanding therefore was, that the admin queue you are introducing serves as a generic carrier for device-agnostic commands, in parallel for having control queue serving as a generic carrier of device-specific commands. If this is not the case, I think it would be nice to describe the exact monivation and scope of admin queue.
Quoted 4 lines
Nope unfortunately. This queue is just a carrier for admin commands. admin commands are commands that talk to one device about other devices. There's clearly no mechanism in the spec to do that, so we plug this hole.
Okay, in that case "admin" sounds a bit misleading as for me it implicates that this is for "administration" of the device. Yet is is for the administration of other devices (slaves).
Perhaps there could be different term used to clarify? Group leader virtqueue?
Group owner virtqueue?
Group master virtqueue?
Quoted 99 lines
Quoted 2 lines
I used group administration virtqueue in a couple of places, just inconsistently. Good enough?
Yep, sounds good. Thanks!
Quoted 93 lines
Quoted 92 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What we lack is passing commands about one device to another device. E.g. control VFs through PFs.
Could you provide examples of such commands please?
For example a common feature is to program a vlan and have device put a given VF inside this vlan.
I don't follow entirely. The way how the VF is connected to network should be ouf of the scope of this interface. The eswitch manager should take care. What you say sounds awfully like the "ip vf" legacy interface, which should not be considered here I believe.
If PF would be the eswitch manager, there are other means to do network programming, using eswitch port representors. But I don't think this is the can of worms we want to open now. I don't think we have a usecase for it currently. Am I wrong Parav?
Quoted 5 lines
In a virtualization scenario host controls this vlan programming giving the network a measure of protection from VFs. If a VF is passed through to a VM, IOMMU limits VFs to only access guest memory so host has to do this programming through a PF.
Understood. This really looks like "ip vf" legacy. I strongly believe it should not be supported.
Any other commands you have in mind?
Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
This is what groups do.
But if we see more uses we can always add them.I'd rather avoid being too generic though.
In that case, why not to avoid using generic terms and stay "group-centric"? What I mean is:
"Administration Virtqueues" -> "Group Administration Virtqueues" "structvirtio_admin_cmd" -> "structvirtio_group_admin_cmd"Quoted 4 lines
Quoted 1 line
Etc. Helps to avoid confusion.
Sure, I tried to do that but missed some opportunities. Will address.
Cool.
Quoted 30 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
+than one administration virtqueue.
[...]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/