Add virtio Admin Virtqueue specification
RFC PATCH v2 1/2 Add virtio Admin Virtqueue specification
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific class
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size, 512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+) create mode 100644 admin-virtq.tex
admin-virtq.tex new file +241 -0
@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class. The following table describes the commands +supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} + +The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.
content.tex +4 -0
@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}
Signature · 1 line
2.21.0
RFC PATCH v2 2/2 virtio-blk: add support for VIRTIO_F_ADMIN_VQ
Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation. Also restrict the negotiation of VIRTIO_F_ADMIN_VQ to be possible only if VIRTIO_BLK_F_MQ is negotiated.
--- content.tex | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
content.tex +16 -0
@@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by \field{num_queues}. + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the + request queues are shifted as the following: +\begin{description} +\item[1] requestq1 +\item[\ldots] +\item[N] requestqN +\end{description} + + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be + negotiated. \subsection{Feature bits}\label{sec:Device Types / Block Device / Feature bits} \begin{description}
@@ -4574,6 +4584,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Block Device / Devic The driver MUST NOT read \field{writeback} before setting the FEATURES_OK \field{device status} bit. +The driver MUST NOT negotiate VIRTIO_F_ADMIN_VQ, unless VIRTIO_BLK_F_MQ +is negotiated. + \devicenormative{\subsubsection}{Device Initialization}{Device Types / Block Device / Device Initialization} Devices SHOULD always offer VIRTIO_BLK_F_FLUSH, and MUST offer it
@@ -4585,6 +4598,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Block Device / Devic The device MUST initialize padding bytes \field{unused0} and \field{unused1} to 0. +The device MUST NOT offer VIRTIO_F_ADMIN_VQ, unless it offer +VIRTIO_BLK_F_MQ. + \subsubsection{Legacy Interface: Device Initialization}\label{sec:Device Types / Block Device / Device Initialization / Legacy Interface: Device Initialization} Because legacy devices do not have FEATURES_OK, transitional devices
Signature · 1 line
2.21.0
Re: [RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ
Quoted 14 lines · Max Gurtovoy
Set the relevant index in case of
VIRTIO_F_ADMIN_VQnegotiation. Also restrict the negotiation ofVIRTIO_F_ADMIN_VQto be possible only ifVIRTIO_BLK_F_MQis negotiated.Signed-off-by Max Gurtovoy <[email protected]>--- content.tex | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)content.tex +0
-0
content.tex \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} · line 4410
Quoted 5 lines
content.tex +2
-0@@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by \field{num_queues}. + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the + request queues are shifted as the following:
"as follows:"
content.tex \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} · line 4418
Quoted 8 lines
content.tex +8
-0+\begin{description} +\item[1] requestq1 +\item[\ldots] +\item[N] requestqN +\end{description} + + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be + negotiated.
Why?
Re: [virtio-comment] Re: [RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ
Quoted 30 lines · Stefan Hajnoczi
Quoted 14 lines · Max Gurtovoy
Set the relevant index in case of
VIRTIO_F_ADMIN_VQnegotiation. Also restrict the negotiation ofVIRTIO_F_ADMIN_VQto be possible only ifVIRTIO_BLK_F_MQis negotiated.Signed-off-by Max Gurtovoy <[email protected]>--- content.tex | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)content.tex +0
-0content.tex \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} · line 4410
Quoted 5 lines
content.tex +2
-0@@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by \field{num_queues}. + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the + request queues are shifted as the following:"as follows:"
content.tex \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} · line 4418
Quoted 8 lines
content.tex +8
-0+\begin{description} +\item[1] requestq1 +\item[\ldots] +\item[N] requestqN +\end{description} + + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be + negotiated.
Quoted 1 line
Why?
To distinguish between adminq and requestq.
Re: [virtio-comment] Re: [RFC PATCH v2 2/2] virtio-blk: add support for VIRTIO_F_ADMIN_VQ
Quoted 31 lines · Max Gurtovoy
Quoted 29 lines · Stefan Hajnoczi
Quoted 14 lines · Max Gurtovoy
Set the relevant index in case of
VIRTIO_F_ADMIN_VQnegotiation. Also restrict the negotiation ofVIRTIO_F_ADMIN_VQto be possible only ifVIRTIO_BLK_F_MQis negotiated.Signed-off-by Max Gurtovoy <[email protected]>--- content.tex | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)content.tex +0
-0content.tex \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} · line 4409
Quoted 4 lines
content.tex +2
-0@@ -4406,6 +4406,16 @@ \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by \field{num_queues}. + If VIRTIO_F_ADMIN_VQ is negotiated, queue0 is an admin virtqueue and the + request queues are shifted as the following:"as follows:"
Quoted 8 lines
content.tex +8
-0+\begin{description} +\item[1] requestq1 +\item[\ldots] +\item[N] requestqN +\end{description} + + if VIRTIO_BLK_F_MQ is not negotiated, VIRTIO_F_ADMIN_VQ MUST NOT be + negotiated.
Quoted 3 lines
Quoted 1 line
Why?
To distinguish between adminq and requestq.
I don't see a conflict between VIRTIO_BLK_F_MQ and VIRTIO_F_ADMIN_VQ. VIRTIO_BLK_F_MQ just indicates to the driver that it can fetch the num_queues device configuration field for the number of request queues.
In the case where VIRTIO_F_ADMIN_VQ is negotiated but VIRTIO_BLK_F_MQ is not negotiated the device's virtqueue layout is still:
1. Admin queue
2. requestq1
...
N+1. requestqN
The driver will probably only initialize the admin queue and requestq1.
Stefan
Quoted 103 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size, 512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+) create mode 100644 admin-virtq.texadmin-virtq.tex new file +61
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128
admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST
"advertises the VIRTIO_F_ADMIN_VQ feature"
Quoted 42 lines
admin-virtq.tex new file +42
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} +
admin-virtq.tex line 123
Quoted 10 lines
admin-virtq.tex new file +10
-0+This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\
These fields are not defined. I wonder why they are necessary - the driver should already have this information.
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
Quoted 143 lines
admin-virtq.tex new file +118
-0+\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class. The following table describes the commands +supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} + +The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 78 lines · Stefan Hajnoczi
Quoted 77 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size, 512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+) create mode 100644 admin-virtq.texadmin-virtq.tex new file +35
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 +
Quoted 8 lines
admin-virtq.tex line 43
Quoted 8 lines
admin-virtq.tex new file +8
-0+/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting}
This is very complex, and it feels like we're overengineering this.
Quoted 78 lines
Quoted 18 lines
admin-virtq.tex new file +18
-0+ +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST"advertises the
VIRTIO_F_ADMIN_VQfeature"Quoted 47 lines
admin-virtq.tex new file +47
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline
Quoted 8 lines
admin-virtq.tex line 123
Quoted 5 lines
admin-virtq.tex new file +5
-0+Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\These fields are not defined. I wonder why they are necessary - the driver should already have this information.
Agreed.
Quoted 5 lines
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
A concrete example would be good, but I think we can come up with a bare-bones spec to start with.
- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not device specificic and device specific); I don't think we need separate classes for transport or vendor specific
- make the format for the request simple (command + length + payload?)
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 81 lines · Cornelia Huck
Quoted 79 lines · Stefan Hajnoczi
Quoted 78 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +36
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */
Quoted 8 lines
Quoted 7 lines
admin-virtq.tex line 43
Quoted 7 lines
admin-virtq.tex new file +7
-0+#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting}This is very complex, and it feels like we're overengineering this.
Do you mean the status type and the status ?
Quoted 85 lines
Quoted 77 lines
Quoted 19 lines
admin-virtq.tex new file +18
-0+ +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST"advertises the
VIRTIO_F_ADMIN_VQfeature"Quoted 46 lines
admin-virtq.tex new file +46
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|}Quoted 8 lines
admin-virtq.tex line 123
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\These fields are not defined. I wonder why they are necessary - the driver should already have this information.
Quoted 1 line
Agreed.
These are initial fields.
We can add also model, serial_number and more in the future.
Quoted 6 lines
Quoted 6 lines
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
admin virtq is not only for migration.
You'll be able to configure virtio device properties using user space tools like: virtio-cli.
For example: format a block device, manage virtual function resources using its PF, query for error logs, device health and more.
In the SW world maybe all the above were redundant, but now that you have more and more HW virtio devices the protocol should be more flexible and adjust.
Few weeks ago I've sent a concrete commands for live migration but then I was told that new infrastructure (admin virtq) should be developed and this is what I did in this RFC.
if you combine the 2 RFCs you can imagine what is needed here for adding Live migration support.
But I want to add it step by step.
We need to agree on the infrastructure.
Quoted 8 lines
A concrete example would be good, but I think we can come up with a bare-bones spec to start with.
- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not device specificic and device specific); I don't think we need separate classes for transport or vendor specific
We need it for live migration probably. It will be a transport class.
Vendor specific is also important to allow vendors develop their special souse.
Quoted 1 line
- make the format for the request simple (command + length + payload?)
I used almost the same format as virtio net ctrl queue.
Quoted 4 lines
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
Quoted 193 lines · Max Gurtovoy
Quoted 81 lines · Cornelia Huck
Quoted 79 lines · Stefan Hajnoczi
Quoted 78 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +36
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */Quoted 8 lines
Quoted 7 lines
admin-virtq.tex line 43
Quoted 7 lines
admin-virtq.tex new file +7
-0+#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting}This is very complex, and it feels like we're overengineering this.
Do you mean the status type and the status ?
Quoted 85 lines
Quoted 77 lines
Quoted 19 lines
admin-virtq.tex new file +18
-0+ +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST"advertises the
VIRTIO_F_ADMIN_VQfeature"Quoted 46 lines
admin-virtq.tex new file +46
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|}Quoted 8 lines
admin-virtq.tex line 123
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\These fields are not defined. I wonder why they are necessary - the driver should already have this information.
Quoted 1 line
Agreed.
These are initial fields.
We can add also model,
serial_numberand more in the future.Quoted 6 lines
Quoted 6 lines
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
Quoted 7 lines
admin virtq is not only for migration.
You'll be able to configure virtio device properties using user space tools
like: virtio-cli.For example: format a block device, manage virtual function resources using its PF, query for error logs, device health and more.
That sounds good.
Quoted 3 lines
In the SW world maybe all the above were redundant, but now that you have more and more HW virtio devices the protocol should be more flexible and adjust.
HW is not special in this regard, I think this will be useful for software too. In-band admin commands are necessary for nested virtualization, for example. They also provide a standard admin interface for out-of-process devices (vhost-user, etc).
Quoted 20 lines
Few weeks ago I've sent a concrete commands for live migration but then I was told that new infrastructure (admin virtq) should be developed and this is what I did in this RFC.
if you combine the 2 RFCs you can imagine what is needed here for adding Live migration support.
But I want to add it step by step.
We need to agree on the infrastructure.
Quoted 8 lines
A concrete example would be good, but I think we can come up with a bare-bones spec to start with.
- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not device specificic and device specific); I don't think we need separate classes for transport or vendor specific
Quoted 8 lines
We need it for live migration probably. It will be a transport class.
Vendor specific is also important to allow vendors develop their special souse.
Quoted 1 line
- make the format for the request simple (command + length + payload?)
I used almost the same format as virtio net ctrl queue.
The virtio_net_ctrl packet format looks good to me, it's close to what Cornelia's command + length + payload suggestion:
struct virtio_net_ctrl {
u8 class;
u8 command;
u8 command-specific-data[];
u8 ack;
};
/* ack values */
#define VIRTIO_NET_OK 0
#define VIRTIO_NET_ERR 1I'm not sure how vendor commands will be allocated though. Will each vendor get a unique class id to prevent collisions? If we want to support cross-implementation migration then it may be necessary to allow vendor command availability to change while the device is running.
I prefer the simpler struct virtio_net_ctrl format to the more complicated one proposed in this patch series.
Quoted 9 lines
Quoted 5 lines
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
struct virtio_admin_report_cmds {
/* Bitmap of available admin commands [Device->Driver]
* bool command_present =
* command_bits[class * 32 + command / 8] & (command % 8);
*/
u8 command_bits[8192];};
Stefan
Quoted 226 lines · Stefan Hajnoczi
Quoted 187 lines · Max Gurtovoy
Quoted 81 lines · Cornelia Huck
Quoted 79 lines · Stefan Hajnoczi
Quoted 78 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +36
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */Quoted 8 lines
Quoted 7 lines
admin-virtq.tex line 43
Quoted 7 lines
admin-virtq.tex new file +7
-0+#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting}This is very complex, and it feels like we're overengineering this.
Do you mean the status type and the status ?
Quoted 84 lines
Quoted 76 lines
Quoted 18 lines
admin-virtq.tex new file +18
-0+ +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST"advertises the
VIRTIO_F_ADMIN_VQfeature"Quoted 46 lines
admin-virtq.tex new file +46
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|}Quoted 8 lines
admin-virtq.tex line 123
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\These fields are not defined. I wonder why they are necessary - the driver should already have this information.
Quoted 1 line
Agreed.
These are initial fields.
We can add also model,
serial_numberand more in the future.Quoted 5 lines
Quoted 5 lines
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
Quoted 7 lines
admin virtq is not only for migration.
You'll be able to configure virtio device properties using user space tools
like: virtio-cli.For example: format a block device, manage virtual function resources using its PF, query for error logs, device health and more.
That sounds good.
Quoted 3 lines
In the SW world maybe all the above were redundant, but now that you have more and more HW virtio devices the protocol should be more flexible and adjust.
HW is not special in this regard, I think this will be useful for software too. In-band admin commands are necessary for nested virtualization, for example. They also provide a standard admin interface for out-of-process devices (vhost-user, etc).
Quoted 21 lines
Few weeks ago I've sent a concrete commands for live migration but then I was told that new infrastructure (admin virtq) should be developed and this is what I did in this RFC.
if you combine the 2 RFCs you can imagine what is needed here for adding Live migration support.
But I want to add it step by step.
We need to agree on the infrastructure.
Quoted 8 lines
A concrete example would be good, but I think we can come up with a bare-bones spec to start with.
- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not device specificic and device specific); I don't think we need separate classes for transport or vendor specificWe need it for live migration probably. It will be a transport class.
Quoted 7 lines
Quoted 5 lines
Vendor specific is also important to allow vendors develop their special souse.
Quoted 1 line
- make the format for the request simple (command + length + payload?)
I used almost the same format as virtio net ctrl queue.
The
virtio_net_ctrlpacket format looks good to me, it's close to what Cornelia's command + length + payload suggestion:
I guess I didn't understand Cornelia suggestion.
Quoted 12 lines
struct virtio_net_ctrl {
u8 class;
u8 command;
u8 command-specific-data[];
u8 ack;
};
/* ack values */
#define VIRTIO_NET_OK 0
#define VIRTIO_NET_ERR 1Quoted 4 lines
I'm not sure how vendor commands will be allocated though. Will each vendor get a unique class id to prevent collisions? If we want to support cross-implementation migration then it may be necessary to allow vendor command availability to change while the device is running.
vendor specific commands can collide.
Vendor A can implement class 192 to do X and Vendor B can implement class 192 to do Y.
what do you mean "support cross-implementation migration" ?
Quoted 2 lines
I prefer the simpler struct
virtio_net_ctrlformat to the more complicated one proposed in this patch series.
This is the same besides adding status type
u8 status_type : 4;
u8 reserved : 4;
I split "u8 command-specific-data[];"
to
"u8 command-specific-data[];
u8 command-specific-result[];"
to emphasize that there is some data that can be written by the device and some data written by the driver in the same command. And this is also the case in virtio-net-ctrl, right ?
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
Quoted 7 lines
struct virtio_admin_report_cmds {
/* Bitmap of available admin commands [Device->Driver]
* bool command_present =
* command_bits[class * 32 + command / 8] & (command % 8);
*/
u8 command_bits[8192];
};Yes, I divided it to multiple commands per class to cover the case we will need more than 1 bit to describe a command.
But I guess we can add it later on.
I think the above should be:
bool command_present = command_bits[class * 32 + command / 8] & (1 << (command % 8));
isn't it ?
Also what do you think about renaming <class, command> to <opcode, opmod> ?
Quoted 2 lines
Stefan
Quoted 255 lines · Max Gurtovoy
Quoted 226 lines · Stefan Hajnoczi
Quoted 187 lines · Max Gurtovoy
Quoted 81 lines · Cornelia Huck
Quoted 79 lines · Stefan Hajnoczi
Quoted 78 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +36
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */Quoted 8 lines
Quoted 7 lines
admin-virtq.tex line 43
Quoted 7 lines
admin-virtq.tex new file +7
-0+#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting}This is very complex, and it feels like we're overengineering this.
Do you mean the status type and the status ?
Quoted 84 lines
Quoted 76 lines
Quoted 18 lines
admin-virtq.tex new file +18
-0+ +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST"advertises the
VIRTIO_F_ADMIN_VQfeature"Quoted 46 lines
admin-virtq.tex new file +46
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|}Quoted 8 lines
admin-virtq.tex line 123
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\These fields are not defined. I wonder why they are necessary - the driver should already have this information.
Quoted 1 line
Agreed.
These are initial fields.
We can add also model,
serial_numberand more in the future.Quoted 5 lines
Quoted 5 lines
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
Quoted 7 lines
admin virtq is not only for migration.
You'll be able to configure virtio device properties using user space tools
like: virtio-cli.For example: format a block device, manage virtual function resources using its PF, query for error logs, device health and more.
That sounds good.
Quoted 3 lines
In the SW world maybe all the above were redundant, but now that you have more and more HW virtio devices the protocol should be more flexible and adjust.
HW is not special in this regard, I think this will be useful for software too. In-band admin commands are necessary for nested virtualization, for example. They also provide a standard admin interface for out-of-process devices (vhost-user, etc).
Quoted 21 lines
Few weeks ago I've sent a concrete commands for live migration but then I was told that new infrastructure (admin virtq) should be developed and this is what I did in this RFC.
if you combine the 2 RFCs you can imagine what is needed here for adding Live migration support.
But I want to add it step by step.
We need to agree on the infrastructure.
Quoted 8 lines
A concrete example would be good, but I think we can come up with a bare-bones spec to start with.
- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not device specificic and device specific); I don't think we need separate classes for transport or vendor specificWe need it for live migration probably. It will be a transport class.
Quoted 7 lines
Quoted 5 lines
Vendor specific is also important to allow vendors develop their special souse.
Quoted 1 line
- make the format for the request simple (command + length + payload?)
I used almost the same format as virtio net ctrl queue.
The
virtio_net_ctrlpacket format looks good to me, it's close to what Cornelia's command + length + payload suggestion:I guess I didn't understand Cornelia suggestion.
Quoted 11 lines
struct virtio_net_ctrl { u8 class; u8 command; u8 command-specific-data[]; u8 ack; }; /* ack values */ #define VIRTIO_NET_OK 0 #define VIRTIO_NET_ERR 1Quoted 4 lines
I'm not sure how vendor commands will be allocated though. Will each vendor get a unique class id to prevent collisions? If we want to support cross-implementation migration then it may be necessary to allow vendor command availability to change while the device is running.
Quoted 6 lines
vendor specific commands can collide.
Vendor A can implement class 192 to do X and Vendor B can implement class 192 to do Y.
what do you mean "support cross-implementation migration" ?
Migrating from vhost_net to vDPA virtio-net, for example. Or migrating between two different vDPA virtio-net implementations.
If vendor commands are all in a single namespace then the guest cannot use them without the risk of the command accidentally executing on the migration destination (where it has a different effect because the vendor has changed!).
Quoted 7 lines
Quoted 2 lines
I prefer the simpler struct
virtio_net_ctrlformat to the more complicated one proposed in this patch series.This is the same besides adding status type
u8
status_type: 4;
u8 reserved : 4;
I'm not sure why it's needed.
Quoted 7 lines
I split "u8 command-specific-data[];"
to
"u8 command-specific-data[];
u8 command-specific-result[];"to emphasize that there is some data that can be written by the device and some data written by the driver in the same command. And this is also the case in virtio-net-ctrl, right ?
The split makes sense to me.
Quoted 21 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
Quoted 7 lines
struct virtio_admin_report_cmds { /* Bitmap of available admin commands [Device->Driver] * bool command_present = * command_bits[class * 32 + command / 8] & (command % 8); */ u8 command_bits[8192]; };Yes, I divided it to multiple commands per class to cover the case we will need more than 1 bit to describe a command.
Quoted 7 lines
But I guess we can add it later on.
I think the above should be:
bool
command_present= command_bits[class * 32 + command / 8] & (1 << (command % 8));isn't it ?
You're right. I forgot to shift the bit :D.
Quoted 1 line
Also what do you think about renaming <class, command> to <opcode, opmod> ?
I need to understand how opcode and opmod values are used. I'm not sure.
Stefan
Quoted 266 lines · Stefan Hajnoczi
Quoted 251 lines · Max Gurtovoy
Quoted 226 lines · Stefan Hajnoczi
Quoted 187 lines · Max Gurtovoy
Quoted 81 lines · Cornelia Huck
Quoted 79 lines · Stefan Hajnoczi
Quoted 78 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +36
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */Quoted 8 lines
Quoted 7 lines
admin-virtq.tex line 43
Quoted 7 lines
admin-virtq.tex new file +7
-0+#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting}This is very complex, and it feels like we're overengineering this.
Do you mean the status type and the status ?
Quoted 84 lines
Quoted 76 lines
Quoted 18 lines
admin-virtq.tex new file +18
-0+ +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST"advertises the
VIRTIO_F_ADMIN_VQfeature"Quoted 46 lines
admin-virtq.tex new file +46
-0+support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular} + +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|}Quoted 8 lines
admin-virtq.tex line 123
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\These fields are not defined. I wonder why they are necessary - the driver should already have this information.
Quoted 1 line
Agreed.
These are initial fields.
We can add also model,
serial_numberand more in the future.Quoted 5 lines
Quoted 5 lines
In general, I'm a little concerned that this whole infrastructure will increase the complexity of VIRTIO significantly with little benefit. I do think an admin virtqueue makes sense, e.g. for migration, but would prefer it if we focus on actual commands first instead of infrastructure. That way it will be clear what infrastructure is needed.
Quoted 7 lines
admin virtq is not only for migration.
You'll be able to configure virtio device properties using user space tools
like: virtio-cli.For example: format a block device, manage virtual function resources using its PF, query for error logs, device health and more.
That sounds good.
Quoted 3 lines
In the SW world maybe all the above were redundant, but now that you have more and more HW virtio devices the protocol should be more flexible and adjust.
HW is not special in this regard, I think this will be useful for software too. In-band admin commands are necessary for nested virtualization, for example. They also provide a standard admin interface for out-of-process devices (vhost-user, etc).
Quoted 21 lines
Few weeks ago I've sent a concrete commands for live migration but then I was told that new infrastructure (admin virtq) should be developed and this is what I did in this RFC.
if you combine the 2 RFCs you can imagine what is needed here for adding Live migration support.
But I want to add it step by step.
We need to agree on the infrastructure.
Quoted 8 lines
A concrete example would be good, but I think we can come up with a bare-bones spec to start with.
- feature bit for the admin vq, as defined here
- location of the admin vq is device specific
- I think we can get away with two classes, as for feature bits (not device specificic and device specific); I don't think we need separate classes for transport or vendor specificWe need it for live migration probably. It will be a transport class.
Quoted 7 lines
Quoted 5 lines
Vendor specific is also important to allow vendors develop their special souse.
Quoted 1 line
- make the format for the request simple (command + length + payload?)
I used almost the same format as virtio net ctrl queue.
The
virtio_net_ctrlpacket format looks good to me, it's close to what Cornelia's command + length + payload suggestion:I guess I didn't understand Cornelia suggestion.
Quoted 10 lines
struct virtio_net_ctrl { u8 class; u8 command; u8 command-specific-data[]; u8 ack; }; /* ack values */ #define VIRTIO_NET_OK 0 #define VIRTIO_NET_ERR 1Quoted 4 lines
I'm not sure how vendor commands will be allocated though. Will each vendor get a unique class id to prevent collisions? If we want to support cross-implementation migration then it may be necessary to allow vendor command availability to change while the device is running.
Quoted 6 lines
vendor specific commands can collide.
Vendor A can implement class 192 to do X and Vendor B can implement class 192 to do Y.
what do you mean "support cross-implementation migration" ?
Migrating from
vhost_netto vDPA virtio-net, for example. Or migrating between two different vDPA virtio-net implementations.If vendor commands are all in a single namespace then the guest cannot use them without the risk of the command accidentally executing on the migration destination (where it has a different effect because the vendor has changed!).
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
I prefer the simpler struct
virtio_net_ctrlformat to the more complicated one proposed in this patch series.This is the same besides adding status type
u8
status_type: 4;
u8 reserved : 4;I'm not sure why it's needed.
If we can live with 256 status code, I guess we can drop it and divide it to groups:
/* status values that are transport, device and vendor independent */
#define VIRTIO_ADMIN_STATUS_GENERIC_START 0
#define VIRTIO_ADMIN_STATUS_GENERIC_END 63
/* status values that are transport specific */
#define VIRTIO_ADMIN_STATUS_TRANSPORT_START 64
#define VIRTIO_ADMIN_STATUS_TRANSPORT_END 127
/* status values that are device specific */
#define VIRTIO_ADMIN_STATUS_DEVICE_START 128
#define VIRTIO_ADMIN_STATUS_DEVICE_END 191
/* status values that are vendor specific */
#define VIRTIO_ADMIN_STATUS_VENDOR_START 192
#define VIRTIO_ADMIN_STATUS_VENDOR_END 255
Quoted 34 lines
Quoted 8 lines
I split "u8 command-specific-data[];"
to
"u8 command-specific-data[];
u8 command-specific-result[];"to emphasize that there is some data that can be written by the device and some data written by the driver in the same command. And this is also the case in virtio-net-ctrl, right ?
The split makes sense to me.
Quoted 24 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
Quoted 7 lines
struct virtio_admin_report_cmds { /* Bitmap of available admin commands [Device->Driver] * bool command_present = * command_bits[class * 32 + command / 8] & (command % 8); */ u8 command_bits[8192]; };Yes, I divided it to multiple commands per class to cover the case we will need more than 1 bit to describe a command.
But I guess we can add it later on.
I think the above should be:
Quoted 7 lines
Quoted 3 lines
bool
command_present= command_bits[class * 32 + command / 8] & (1 << (command % 8));isn't it ?
You're right. I forgot to shift the bit :D.
Quoted 1 line
Also what do you think about renaming <class, command> to <opcode, opmod> ?
I need to understand how opcode and opmod values are used. I'm not sure.
Same as class and command just with different naming.
Quoted 2 lines
Stefan
Quoted 33 lines · Max Gurtovoy
Quoted 32 lines · Stefan Hajnoczi
Quoted 31 lines · Max Gurtovoy
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Max Gurtovoy
Quoted 5 lines · Cornelia Huck
Quoted 1 line · Stefan Hajnoczi
On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
Quoted 7 lines
struct virtio_admin_report_cmds { /* Bitmap of available admin commands [Device->Driver] * bool command_present = * command_bits[class * 32 + command / 8] & (command % 8); */ u8 command_bits[8192]; };Yes, I divided it to multiple commands per class to cover the case we will need more than 1 bit to describe a command.
But I guess we can add it later on.
I think the above should be:
bool
command_present= command_bits[class * 32 + command / 8] & (1 << (command % 8));
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
isn't it ?
You're right. I forgot to shift the bit :D.
Quoted 1 line
Also what do you think about renaming <class, command> to <opcode, opmod> ?
I need to understand how opcode and opmod values are used. I'm not sure.
Same as class and command just with different naming.
Is opcode the same as command? What does opmod do?
Stefan
Quoted 34 lines · Stefan Hajnoczi
Quoted 33 lines · Max Gurtovoy
Quoted 32 lines · Stefan Hajnoczi
Quoted 31 lines · Max Gurtovoy
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Max Gurtovoy
Quoted 5 lines · Cornelia Huck
Quoted 1 line · Stefan Hajnoczi
On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
Quoted 7 lines
struct virtio_admin_report_cmds { /* Bitmap of available admin commands [Device->Driver] * bool command_present = * command_bits[class * 32 + command / 8] & (command % 8); */ u8 command_bits[8192]; };Yes, I divided it to multiple commands per class to cover the case we will need more than 1 bit to describe a command.
But I guess we can add it later on.
I think the above should be:
bool
command_present= command_bits[class * 32 + command / 8] & (1 << (command % 8));
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
isn't it ?
You're right. I forgot to shift the bit :D.
Quoted 1 line
Also what do you think about renaming <class, command> to <opcode, opmod> ?
I need to understand how opcode and opmod values are used. I'm not sure.
Same as class and command just with different naming.
Is opcode the same as command? What does opmod do?
no. opcode == class and opmod == command.
Quoted 2 lines
Stefan
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 37 lines · Max Gurtovoy
Quoted 35 lines · Stefan Hajnoczi
Quoted 34 lines · Max Gurtovoy
Quoted 33 lines · Stefan Hajnoczi
Quoted 32 lines · Max Gurtovoy
Quoted 14 lines · Stefan Hajnoczi
Quoted 9 lines · Max Gurtovoy
Quoted 5 lines · Cornelia Huck
Quoted 1 line · Stefan Hajnoczi
On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
How many different (groups of) commands can we reasonably expect? Do we need a generic discovery command, or can we get away with a feature bit covering each new group of commands?
I can't predict the future but IMO we need a discovery command.
We have many devices and more can be added in the future.
A <u8 class, u8 command> space is 65536 bits or 8KB. I think admin commands would not be included in VIRTIO Feature Bits but instead reported via a separate admin command that returns up to 8KB of data:
Quoted 7 lines
struct virtio_admin_report_cmds { /* Bitmap of available admin commands [Device->Driver] * bool command_present = * command_bits[class * 32 + command / 8] & (command % 8); */ u8 command_bits[8192]; };Yes, I divided it to multiple commands per class to cover the case we will need more than 1 bit to describe a command.
But I guess we can add it later on.
I think the above should be:
bool
command_present= command_bits[class * 32 + command / 8] & (1 << (command % 8));isn't it ?
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
You're right. I forgot to shift the bit :D.
Quoted 1 line
Also what do you think about renaming <class, command> to <opcode, opmod> ?
I need to understand how opcode and opmod values are used. I'm not sure.
Same as class and command just with different naming.
Is opcode the same as command? What does opmod do?
no. opcode == class and opmod == command.
I find those terms confusing. "class" and "command" was clearer to me, at least.
Stefan
Quoted 2 lines · Max Gurtovoy
Vendor specific is also important to allow vendors develop their special souse.
Most of the value of virtio is that it's generic.
So we added VIRTIO_PCI_CAP_VENDOR_CFG which is very easy for the hypervisor to intercept. Maybe that is not enough but that needs justification, I am not sure it's a good idea to add multiple ways devices can differ without a lot of thought.
Signature · 1 line
MST
Quoted 9 lines · Michael S. Tsirkin
Quoted 2 lines · Max Gurtovoy
Vendor specific is also important to allow vendors develop their special souse.
Most of the value of virtio is that it's generic.
So we addedVIRTIO_PCI_CAP_VENDOR_CFGwhich is very easy for the hypervisor to intercept. Maybe that is not enough but that needs justification, I am not sure it's a good idea to add multiple ways devices can differ without a lot of thought.
Agreed, we should be careful about adding devices that differ in a lot of little things.
My thought is that the default should be to concentrate on common features that can be specified, just to keep things movable between different implementations. If a device is really very different from the generic one, it should probably become a different device type. But my preference would be that everyone tries to make their device stick to a common specification.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 23 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size, 512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.
Quoted 7 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>
Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.
Quoted 112 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+) create mode 100644 admin-virtq.texadmin-virtq.tex new file +100
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|}
admin-virtq.tex line 110
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}
So there are several problems with this approach. One is that there is no two way negotiation.
No way for device to know what will driver use in the end. This breaks things like e.g. accelerating some configurations but not others.
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.
Quoted 122 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +
admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.
Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have VIRTIO_PCI_CAP_VENDOR_CFG and if that is not sufficient I would like to know why before we add more vendor specific stuff.
Quoted 24 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Quoted 5 lines
Quoted 1 line
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.
This will be defined per command.
For example, funcion_id will be given as command data.
Quoted 117 lines
Quoted 117 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +104
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\
Quoted 8 lines
admin-virtq.tex line 110
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach.
One is that there is no two way negotiation.
you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
I don't understand what it breaks exactly.
Quoted 4 lines
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
we'll develop virtio-cli tool.
I don't know what should be changed in the device by the hypervisor.
Quoted 4 lines
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.
device will support admin q with mandatory commands.
Old driver shouldn't care and it won't negotiate this feature.
New drivers will adjust and will be extended.
Quoted 7 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.
This is initial commit to add infrastructure. I can't add 255 commands at once.
We need to add it incrementally.
Quoted 140 lines
Quoted 125 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.
Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Quoted 26 lines
Quoted 26 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
admin-virtq.tex line 241
Quoted 6 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
admin-virtq.tex new file +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
Quoted 8 lines
Quoted 4 lines
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 21 lines · Cornelia Huck
Quoted 19 lines · Max Gurtovoy
Quoted 18 lines · Michael S. Tsirkin
admin-virtq.tex line 241
Quoted 6 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
admin-virtq.tex new file +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.
Quoted 7 lines
Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
You can implement a compliant device and a compliant. why do you think you can't ?
Some features are vendor/sub-vendor specific.
And as mentioned, you already added it to the spec so I guess it was for a reason.
If someone would like to add its special souse to virtio device and we have a flexible admin queue to manage it with a spec compliant manner, why not ?
The feature might not get support by the working group so do you really want to limit vendor from implementing features that are not agreed here on the mailing list ?
Adding a vendor specific command set is trivial.
Quoted 6 lines
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
We have an interface. Its the admin queue.
Virtio-cli will be a user space tool to send commands via this admin queue and configure the device.
The driver will just open a char dev to supply a channel to this admin queue from user space.
Examples:
1. format a virtio-blk device to add data integrity checks (T-10)
2. set 5 msix to VF_1, 12 msix to VF_2 and 2 msix to VF_3 before enabling SRIOV
3. Get telemetry log
4. Get error log
5. Get vendor specific HW health report
6. Get generic virtio device health report
7. set 5 queues to VF_1, 12 queues to VF_2 and 2 queues to VF_3 before enabling SRIOV
This knowledge is needed for the sys-admin and not the driver. The driver will provide the channel to enable this configuration by an orchestrator/admin.
Quoted 1 line
Quoted 31 lines · Max Gurtovoy
Quoted 21 lines · Cornelia Huck
Quoted 19 lines · Max Gurtovoy
Quoted 18 lines · Michael S. Tsirkin
admin-virtq.tex line 241
Quoted 6 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
admin-virtq.tex new file +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 7 lines
Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
Quoted 7 lines
You can implement a compliant device and a compliant. why do you think you can't ?
Some features are vendor/sub-vendor specific.
And as mentioned, you already added it to the spec so I guess it was for a reason.
Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities we'll add in the future.
And such a capability can prove useful for things like identifying the device to enable workarounds, etc.
That is a far cry from a fat interface which generic drivers will need to support and who's sole purpose is vendor specific extensions.
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
Quoted 8 lines
If someone would like to add its special souse to virtio device and we have a flexible admin queue to manage it with a spec compliant manner, why not ?
The feature might not get support by the working group so do you really want to limit vendor from implementing features that are not agreed here on the mailing list ?
Adding a vendor specific command set is trivial.
Supporting all this mess leter isn't. So we better make sure just how is all this vendor stuff going to be limited.
Quoted 41 lines
Quoted 6 lines
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
We have an interface. Its the admin queue.
Virtio-cli will be a user space tool to send commands via this admin queue and configure the device.
The driver will just open a char dev to supply a channel to this admin queue from user space.
Examples:
1. format a virtio-blk device to add data integrity checks (T-10)
2. set 5 msix to
VF_1, 12 msix toVF_2and 2 msix toVF_3before enabling SRIOV3. Get telemetry log
4. Get error log
5. Get vendor specific HW health report
6. Get generic virtio device health report
7. set 5 queues to
VF_1, 12 queues toVF_2and 2 queues toVF_3before enabling SRIOVThis knowledge is needed for the sys-admin and not the driver. The driver will provide the channel to enable this configuration by an orchestrator/admin.
Quoted 1 line
Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Max Gurtovoy
Quoted 21 lines · Cornelia Huck
Quoted 19 lines · Max Gurtovoy
Quoted 18 lines · Michael S. Tsirkin
admin-virtq.tex line 241
Quoted 6 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
admin-virtq.tex new file +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 7 lines
Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
You can implement a compliant device and a compliant. why do you think you can't ?
Quoted 7 lines
Quoted 4 lines
Some features are vendor/sub-vendor specific.
And as mentioned, you already added it to the spec so I guess it was for a reason.
Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities we'll add in the future.
it shouldn't bother you as a driver maintainer. I don't think vendor specific code should go to Linux kernel.
No conflict will happen.
There is no harm in having a virtio-cli tool that will pass-through commands from command line to the device using the admin queue.
All the HW devices I know have sw tool that can access it and virtio shouldn't be different.
Also, I can develop my own vendor specific user space driver for virtio-blk for example (lets say in SPDK framework). And this driver will be aware of this vendor specific capabilities.
Quoted 5 lines
And such a capability can prove useful for things like identifying the device to enable workarounds, etc.
That is a far cry from a fat interface which generic drivers will need to support and who's sole purpose is vendor specific extensions.
No vendor specific commands will be added to generic driver. I never said this.
I'll enable the option to create devices that support vendor specific commands.
Quoted 4 lines
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
I don't get your point.
If someone would like to do crazy stuff, you can't stop him.
Quoted 5 lines
Quoted 5 lines
If someone would like to add its special souse to virtio device and we have a flexible admin queue to manage it with a spec compliant manner, why not ?
Quoted 7 lines
Quoted 5 lines
The feature might not get support by the working group so do you really want to limit vendor from implementing features that are not agreed here on the mailing list ?
Adding a vendor specific command set is trivial.
Supporting all this mess leter isn't. So we better make sure just how is all this vendor stuff going to be limited.
You define a range of commands for vendors and that's it.
I don't see a scenario to add some vendor code to virtio generic drivers.
Quoted 39 lines
Quoted 39 lines
Quoted 6 lines
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
We have an interface. Its the admin queue.
Virtio-cli will be a user space tool to send commands via this admin queue and configure the device.
The driver will just open a char dev to supply a channel to this admin queue from user space.
Examples:
1. format a virtio-blk device to add data integrity checks (T-10)
2. set 5 msix to
VF_1, 12 msix toVF_2and 2 msix toVF_3before enabling SRIOV3. Get telemetry log
4. Get error log
5. Get vendor specific HW health report
6. Get generic virtio device health report
7. set 5 queues to
VF_1, 12 queues toVF_2and 2 queues toVF_3before enabling SRIOVThis knowledge is needed for the sys-admin and not the driver. The driver will provide the channel to enable this configuration by an orchestrator/admin.
Quoted 6 lines · Max Gurtovoy
Quoted 4 lines
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
I don't get your point.
Talking about virtio admin interface (you are using a VQ now), it seems to include or be planned to include several unrelated features, which I'm not sure is appropriate and it sure makes review harder. I would advise focusing on hypervisor management features such as live migration etc, or some other subset. Also, there's overlap with existing functionality such as feature negotiation.
Talking about vendor specific commands, I don't feel you demonstrated a need for these beyond what already exists in the spec. In particular it is unclear why there a need to layer vendor specific commands on top of virtio. Using the existing capability would work at the PCI level which seems more appropriate as vendor things are likely to be low level.
It might be a good idea to address these concerns in your next revision, narrowing the scope as much as possible and explaining the motivation for the new interfaces.
Thanks!
Signature · 1 line
MST
Quoted 11 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
Quoted 3 lines
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
I don't get your point.
Talking about virtio admin interface (you are using a VQ now), it seems to include or be planned to include several unrelated features, which I'm not sure is appropriate and it sure makes review harder. I would advise focusing on hypervisor management features such as live migration etc, or some other subset. Also, there's overlap with existing functionality such as feature negotiation.
Only one feature is negotiated in the RFC and it's the admin virtq support.
You asked me to give examples for future features and I did. These features are not in the RFC.
So I don't understand what is hard to review. I've answered all the points you raised.
If I'll add Live Migration to the RFC we'll discuss it for months and then we'll discuss the admin queue interface again.
I don't want that. We already agreed to upstream first the Admin q interface.
Please don't change the order and make me run in cycles.
Quoted 6 lines
Talking about vendor specific commands, I don't feel you demonstrated a need for these beyond what already exists in the spec. In particular it is unclear why there a need to layer vendor specific commands on top of virtio. Using the existing capability would work at the PCI level which seems more appropriate as vendor things are likely to be low level.
I did explain it.
Vendors might have their own special souse that might seems weird for the TC or maybe some internal management stuff that are not related to virtio, such as HW counters for example.
You're trying to address all the commands in one shut and then you ask the opposite to narrow it to minimal.
I did already all the needed work: sent an RFC for LM and send an RFC for new admin q with only 2 commands for us to have easy submission.
This RFC gives you all the flexibility to add Jason's stop/start queue commands in the future, Live migration RFC that I sent and all the features you'll ever think to add to virtio instead of creating a dedicated interface each time and run long review cycles.
Quoted 4 lines
It might be a good idea to address these concerns in your next revision, narrowing the scope as much as possible and explaining the motivation for the new interfaces.
narrow to what ? we have only 2 classes here, come on.
Quoted 3 lines
Thanks!
Quoted 29 lines · Max Gurtovoy
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Max Gurtovoy
Quoted 25 lines · Cornelia Huck
Quoted 23 lines · Max Gurtovoy
Quoted 13 lines · Michael S. Tsirkin
Review
Quoted 7 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic
Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}(quoted diff) +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
You can implement a compliant device and a compliant. why do you think you can't ?
Some features are vendor/sub-vendor specific.
Please don't do this. This breaks the efforts for making virtio as a standard and generic device.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
And as mentioned, you already added it to the spec so I guess it was for a reason.
Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities we'll add in the future.
it shouldn't bother you as a driver maintainer. I don't think vendor specific code should go to Linux kernel.
Quoted 7 lines
No conflict will happen.
There is no harm in having a virtio-cli tool that will pass-through commands from command line to the device using the admin queue.
All the HW devices I know have sw tool that can access it and virtio shouldn't be different.
Also, I can develop my own vendor specific user space driver for virtio-blk for example (lets say in SPDK framework). And this driver will be aware of this vendor specific capabilities.
That's even worse since the driver can only work for the vendor specific device which is in fact not a standard device.
For the vendor specific capability, the spec has already said that it will be only used for debugging/reporting purpose:
"""
The optional Vendor data capability allows the device to present vendor-specific data to the driver, without
conflicts, for debugging and/or reporting purposes, and without conflicting with standard functionality.
"""
It looks to me what you're trying to invent may violate the spec.
Thanks
Quoted 81 lines
Quoted 7 lines
And such a capability can prove useful for things like identifying the device to enable workarounds, etc.
That is a far cry from a fat interface which generic drivers will need to support and who's sole purpose is vendor specific extensions.
No vendor specific commands will be added to generic driver. I never said this.
I'll enable the option to create devices that support vendor specific commands.
Quoted 4 lines
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
I don't get your point.
If someone would like to do crazy stuff, you can't stop him.
Quoted 5 lines
Quoted 5 lines
If someone would like to add its special souse to virtio device and we have a flexible admin queue to manage it with a spec compliant manner, why not ?
Quoted 7 lines
Quoted 5 lines
The feature might not get support by the working group so do you really want to limit vendor from implementing features that are not agreed here on the mailing list ?
Adding a vendor specific command set is trivial.
Supporting all this mess leter isn't. So we better make sure just how is all this vendor stuff going to be limited.
You define a range of commands for vendors and that's it.
I don't see a scenario to add some vendor code to virtio generic drivers.
Quoted 39 lines
Quoted 39 lines
Quoted 6 lines
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
We have an interface. Its the admin queue.
Virtio-cli will be a user space tool to send commands via this admin queue and configure the device.
The driver will just open a char dev to supply a channel to this admin queue from user space.
Examples:
1. format a virtio-blk device to add data integrity checks (T-10)
2. set 5 msix to
VF_1, 12 msix toVF_2and 2 msix toVF_3before enabling SRIOV3. Get telemetry log
4. Get error log
5. Get vendor specific HW health report
6. Get generic virtio device health report
7. set 5 queues to
VF_1, 12 queues toVF_2and 2 queues toVF_3before enabling SRIOVThis knowledge is needed for the sys-admin and not the driver. The driver will provide the channel to enable this configuration by an orchestrator/admin.
Quoted 36 lines · Jason Wang
Quoted 34 lines · Max Gurtovoy
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Max Gurtovoy
Quoted 30 lines · Cornelia Huck
Quoted 17 lines · Max Gurtovoy
Quoted 16 lines · Michael S. Tsirkin
Review
Quoted 8 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic
Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}(quoted diff) +2
-0+ +The Vendor specific command set is a group of classes andcommands
(quoted diff) +3
-0+within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
Quoted 7 lines
Quoted 4 lines
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
You can implement a compliant device and a compliant. why do you think you can't ?
Some features are vendor/sub-vendor specific.
Please don't do this. This breaks the efforts for making virtio as a standard and generic device.
The correct way is to generalize those features and introduce them in the spec instead.
Thanks
Quoted 119 lines
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
And as mentioned, you already added it to the spec so I guess it was for a reason.
Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities we'll add in the future.
it shouldn't bother you as a driver maintainer. I don't think vendor specific code should go to Linux kernel.
Quoted 7 lines
No conflict will happen.
There is no harm in having a virtio-cli tool that will pass-through commands from command line to the device using the admin queue.
All the HW devices I know have sw tool that can access it and virtio shouldn't be different.
Also, I can develop my own vendor specific user space driver for virtio-blk for example (lets say in SPDK framework). And this driver will be aware of this vendor specific capabilities.
That's even worse since the driver can only work for the vendor specific device which is in fact not a standard device.
For the vendor specific capability, the spec has already said that it will be only used for debugging/reporting purpose:
"""
The optional Vendor data capability allows the device to present vendor-specific data to the driver, without
conflicts, for debugging and/or reporting purposes, and without conflicting with standard functionality."""
It looks to me what you're trying to invent may violate the spec.
Thanks
Quoted 81 lines
Quoted 7 lines
And such a capability can prove useful for things like identifying the device to enable workarounds, etc.
That is a far cry from a fat interface which generic drivers will need to support and who's sole purpose is vendor specific extensions.
No vendor specific commands will be added to generic driver. I never said this.
I'll enable the option to create devices that support vendor specific commands.
Quoted 4 lines
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
I don't get your point.
If someone would like to do crazy stuff, you can't stop him.
Quoted 5 lines
Quoted 5 lines
If someone would like to add its special souse to virtio device and we have a flexible admin queue to manage it with a spec compliant manner, why not ?
Quoted 7 lines
Quoted 5 lines
The feature might not get support by the working group so do you really want to limit vendor from implementing features that are not agreed here on the mailing list ?
Adding a vendor specific command set is trivial.
Supporting all this mess leter isn't. So we better make sure just how is all this vendor stuff going to be limited.
You define a range of commands for vendors and that's it.
I don't see a scenario to add some vendor code to virtio generic drivers.
Quoted 39 lines
Quoted 39 lines
Quoted 6 lines
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
We have an interface. Its the admin queue.
Virtio-cli will be a user space tool to send commands via this admin queue and configure the device.
The driver will just open a char dev to supply a channel to this admin queue from user space.
Examples:
1. format a virtio-blk device to add data integrity checks (T-10)
2. set 5 msix to
VF_1, 12 msix toVF_2and 2 msix toVF_3before enabling SRIOV3. Get telemetry log
4. Get error log
5. Get vendor specific HW health report
6. Get generic virtio device health report
7. set 5 queues to
VF_1, 12 queues toVF_2and 2 queues toVF_3before enabling SRIOVThis knowledge is needed for the sys-admin and not the driver. The driver will provide the channel to enable this configuration by an orchestrator/admin.
Quoted 36 lines · Jason Wang
Quoted 34 lines · Max Gurtovoy
Quoted 32 lines · Michael S. Tsirkin
Quoted 31 lines · Max Gurtovoy
Quoted 30 lines · Cornelia Huck
Quoted 17 lines · Max Gurtovoy
Quoted 16 lines · Michael S. Tsirkin
Review
Quoted 8 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic
Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}(quoted diff) +2
-0+ +The Vendor specific command set is a group of classes andcommands
(quoted diff) +3
-0+within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
Quoted 7 lines
Quoted 4 lines
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
You can implement a compliant device and a compliant. why do you think you can't ?
Some features are vendor/sub-vendor specific.
Please don't do this. This breaks the efforts for making virtio as a standard and generic device.
no it doesn't.
virtio already has vendor specific section.
Quoted 12 lines
Quoted 12 lines
Quoted 7 lines
Quoted 4 lines
And as mentioned, you already added it to the spec so I guess it was for a reason.
Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities we'll add in the future.
it shouldn't bother you as a driver maintainer. I don't think vendor specific code should go to Linux kernel.
No conflict will happen.
Quoted 8 lines
Quoted 5 lines
There is no harm in having a virtio-cli tool that will pass-through commands from command line to the device using the admin queue.
All the HW devices I know have sw tool that can access it and virtio shouldn't be different.
Also, I can develop my own vendor specific user space driver for virtio-blk for example (lets say in SPDK framework). And this driver will be aware of this vendor specific capabilities.
That's even worse since the driver can only work for the vendor specific device which is in fact not a standard device.
this is not true.
Quoted 5 lines
For the vendor specific capability, the spec has already said that it will be only used for debugging/reporting purpose:
"""
Quoted 9 lines
The optional Vendor data capability allows the device to present vendor-specific data to the driver, without
conflicts, for debugging and/or reporting purposes, and without conflicting with standard functionality."""
It looks to me what you're trying to invent may violate the spec.
this is also not true.
Quoted 85 lines
Thanks
Quoted 81 lines
Quoted 7 lines
And such a capability can prove useful for things like identifying the device to enable workarounds, etc.
That is a far cry from a fat interface which generic drivers will need to support and who's sole purpose is vendor specific extensions.
No vendor specific commands will be added to generic driver. I never said this.
I'll enable the option to create devices that support vendor specific commands.
Quoted 4 lines
And yes, it also allows you to unload the generic driver and load a vendor driver. Which can then go wild if it wants to - nothing we can or want to do about it.
I don't get your point.
If someone would like to do crazy stuff, you can't stop him.
Quoted 5 lines
Quoted 5 lines
If someone would like to add its special souse to virtio device and we have a flexible admin queue to manage it with a spec compliant manner, why not ?
Quoted 7 lines
Quoted 5 lines
The feature might not get support by the working group so do you really want to limit vendor from implementing features that are not agreed here on the mailing list ?
Adding a vendor specific command set is trivial.
Supporting all this mess leter isn't. So we better make sure just how is all this vendor stuff going to be limited.
You define a range of commands for vendors and that's it.
I don't see a scenario to add some vendor code to virtio generic drivers.
Quoted 39 lines
Quoted 39 lines
Quoted 6 lines
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Things like that are part of the driver as in the spec sense. The spec does not care how you actually split the implementation, or what controls you are giving to whom. We need a defined interface.
We have an interface. Its the admin queue.
Virtio-cli will be a user space tool to send commands via this admin queue and configure the device.
The driver will just open a char dev to supply a channel to this admin queue from user space.
Examples:
1. format a virtio-blk device to add data integrity checks (T-10)
2. set 5 msix to
VF_1, 12 msix toVF_2and 2 msix toVF_3before enabling SRIOV3. Get telemetry log
4. Get error log
5. Get vendor specific HW health report
6. Get generic virtio device health report
7. set 5 queues to
VF_1, 12 queues toVF_2and 2 queues toVF_3before enabling SRIOVThis knowledge is needed for the sys-admin and not the driver. The driver will provide the channel to enable this configuration by an orchestrator/admin.
Quoted 36 lines · Max Gurtovoy
Quoted 35 lines · Jason Wang
Quoted 33 lines · Max Gurtovoy
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Max Gurtovoy
Quoted 29 lines · Cornelia Huck
Quoted 27 lines · Max Gurtovoy
Quoted 15 lines · Michael S. Tsirkin
Review
Quoted 9 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
(quoted diff) +2
-0+ +The Vendor specific command set is a group of classes andcommands
(quoted diff) +3
-0+within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already haveVIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.You can implement a compliant device and a compliant. why do you think you
can't ?
If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
Quoted 10 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Some features are vendor/sub-vendor specific.
Please don't do this. This breaks the efforts for making virtio as a standard and generic device.
no it doesn't.
virtio already has vendor specific section.
But only as a pci-specific capability, with a very narrow scope and constraints; that is a very far cry from what you're proposing!
Quoted 28 lines
Quoted 28 lines
Quoted 26 lines
Quoted 11 lines
Quoted 7 lines
And as mentioned, you already added it to the spec so I guess it was for a
reason.Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities
we'll add in the future.it shouldn't bother you as a driver maintainer. I don't think vendor specific code should go to Linux kernel.
No conflict will happen.
There is no harm in having a virtio-cli tool that will pass-through commands from command line to the device using the admin queue.
All the HW devices I know have sw tool that can access it and virtio shouldn't be different.
Also, I can develop my own vendor specific user space driver for virtio-blk for example (lets say in SPDK framework). And this driver will be aware of this vendor specific capabilities.
Quoted 4 lines
Quoted 2 lines
That's even worse since the driver can only work for the vendor specific device which is in fact not a standard device.
this is not true.
What is not true? Adding some magic vendor-specific stuff that is needed for the device to function does make it de facto non-standard.
Quoted 13 lines
Quoted 13 lines
For the vendor specific capability, the spec has already said that it will be only used for debugging/reporting purpose:
"""
The optional Vendor data capability allows the device to present vendor-specific data to the driver, without
conflicts, for debugging and/or reporting purposes, and without conflicting with standard functionality.
Quoted 5 lines
Quoted 4 lines
"""
It looks to me what you're trying to invent may violate the spec.
this is also not true.
What is not true about that? That capability has even more restrictions than what is cited here.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 41 lines · Cornelia Huck
Quoted 39 lines · Max Gurtovoy
Quoted 38 lines · Jason Wang
Quoted 37 lines · Max Gurtovoy
Quoted 36 lines · Michael S. Tsirkin
Quoted 35 lines · Max Gurtovoy
Quoted 34 lines · Cornelia Huck
Quoted 24 lines · Max Gurtovoy
Quoted 15 lines · Michael S. Tsirkin
Review
Quoted 9 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
(quoted diff) +2
-0+ +The Vendor specific command set is a group of classes andcommands
(quoted diff) +3
-0+within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already haveVIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
You can implement a compliant device and a compliant. why do you think you
can't ?If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
again, Linux driver will be generic.
It will include a char dev to user space virtio-cli that will enable sending admin commands to do administration.
Nobody asked to add vendor specific code to the driver.
The spec allows vendor specific stuff today that are not usable and I'm trying to create a usable API without adding one line of vendor code to driver.
You just need to open a window for few commands like others do and like you already did in the spec.
NVIDIA HW internals are different from VENDOR_A and VENDOR_B.
I want to debug my HW or enable some accelerations in it.
This is the way to do it.
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Some features are vendor/sub-vendor specific.
Please don't do this. This breaks the efforts for making virtio as a standard and generic device.
no it doesn't.
virtio already has vendor specific section.
But only as a pci-specific capability, with a very narrow scope and constraints; that is a very far cry from what you're proposing!
why creating some pci vendor specific channel is ok but creating few admin command isn't ?
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 9 lines
Quoted 5 lines
And as mentioned, you already added it to the spec so I guess it was for a
reason.Well we basically just reserved a capability ID so if people add their vendor specific ones they don't conflict with each other or capabilities
we'll add in the future.it shouldn't bother you as a driver maintainer. I don't think vendor specific code should go to Linux kernel.
No conflict will happen.
There is no harm in having a virtio-cli tool that will pass-through commands from command line to the device using the admin queue.
All the HW devices I know have sw tool that can access it and virtio shouldn't be different.
Also, I can develop my own vendor specific user space driver for virtio-blk for example (lets say in SPDK framework). And this driver will be aware of this vendor specific capabilities.
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
That's even worse since the driver can only work for the vendor specific device which is in fact not a standard device.
this is not true.
What is not true? Adding some magic vendor-specific stuff that is needed for the device to function does make it de facto non-standard.
This is not true because I can write a user space driver any way I want.
This discussion is repeating in cycles.
You already added vendor specific channel to the spec so why you ask me the need for it ?
lets progress to other sections.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
For the vendor specific capability, the spec has already said that it will be only used for debugging/reporting purpose:
"""
The optional Vendor data capability allows the device to present vendor-specific data to the driver, without
conflicts, for debugging and/or reporting purposes, and without conflicting with standard functionality.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
"""
It looks to me what you're trying to invent may violate the spec.
this is also not true.
What is not true about that? That capability has even more restrictions than what is cited here.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 1 line · Max Gurtovoy
The spec allows vendor specific stuff today that are not usable
Maybe start there and explain what are the issues you are trying to address. I think there's a wide consensus in the TC that significant features should be part of the spec not a vendor specific thing. Existing vendor specific capability is thus geared towards enabling minor extensions such as bug work arounds.
Quoted 2 lines
trying to create a usable API without adding one line of vendor code to driver.
Driver in the virtio terminology is taken widely to mean all software interacting with the device through the virtio interface. Your virtio-cli tool would be part of the driver.
Signature · 1 line
MST
Quoted 52 lines · Max Gurtovoy
Quoted 43 lines · Cornelia Huck
Quoted 41 lines · Max Gurtovoy
Quoted 40 lines · Jason Wang
Quoted 39 lines · Max Gurtovoy
Quoted 38 lines · Michael S. Tsirkin
Quoted 37 lines · Max Gurtovoy
Quoted 36 lines · Cornelia Huck
Quoted 26 lines · Max Gurtovoy
Quoted 17 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
(quoted diff) +2
-0+ +The Vendor specific command set is a group of classes andcommands
+within each of these classes which are vendor specific. Refer to
+each vendor specification for more information on the supported
+commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already haveVIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
You can implement a compliant device and a compliant. why do you think you
can't ?If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
Quoted 5 lines
again, Linux driver will be generic.
It will include a char dev to user space virtio-cli that will enable sending admin commands to do administration.
Nobody asked to add vendor specific code to the driver.
It's not about the general driver but about the general device. We want to present a general standard device which means we can migrate virtio devices among different vendors.
If each vendor choose to implement their own vendor specific features via vendor specific commands that are not ruled by the spec, there will be little value of the spec.
Quoted 8 lines
The spec allows vendor specific stuff today that are not usable and I'm trying to create a usable API without adding one line of vendor code to driver.
You just need to open a window for few commands like others do and like you already did in the spec.
NVIDIA HW internals are different from
VENDOR_AandVENDOR_B.I want to debug my HW or enable some accelerations in it.
A big question is that, on what layer do you want to provide those functions?
1) virtio level
2) vendor level
For 1), it needs to be done in the spec instead of introducing backdoor for vendor specific command
For 2), virtio keeps simple as is, functionality is provided in a vendor specific device interface. In this case, you can't use virtio-pci directly, then you need present virtio on the upper layer (with some software meditation)
Quoted 2 lines
This is the way to do it.
Again, for debug or acceleration, if you want to do it at the virtio level, you need generalize those features and define them in the spec.
Maybe we can start the work on generalizing the debug facility first. (adding Laurent who is working on this).
Tahnks
Quoted 54 lines · Jason Wang
Quoted 52 lines · Max Gurtovoy
Quoted 49 lines · Cornelia Huck
Quoted 36 lines · Max Gurtovoy
Quoted 35 lines · Jason Wang
Quoted 34 lines · Max Gurtovoy
Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Max Gurtovoy
Quoted 31 lines · Cornelia Huck
Quoted 29 lines · Max Gurtovoy
Quoted 17 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
(quoted diff) +2
-0+ +The Vendor specific command set is a group of classes andcommands
+within each of these classes which are vendor specific. Refer to
+each vendor specification for more information on the supported
+commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already haveVIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.You can implement a compliant device and a compliant. why do you think you
can't ?If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
Quoted 8 lines
Quoted 5 lines
again, Linux driver will be generic.
It will include a char dev to user space virtio-cli that will enable sending admin commands to do administration.
Nobody asked to add vendor specific code to the driver.
It's not about the general driver but about the general device. We want to present a general standard device which means we can migrate virtio devices among different vendors.
I don't know if it's reasonable.
You're looking on a device as a SW component since it was like this from day 1. But it's not. It's HW, and HW has its own internal state and implementation that is known only to the vendor.
Quoted 3 lines
If each vendor choose to implement their own vendor specific features via vendor specific commands that are not ruled by the spec, there will be little value of the spec.
Not at all.
I'm very surprised that you are negative to this RFC instead of working on defining admin commands to stop/start queues and other commands that can be helpful for vDPA migration.
Quoted 12 lines
Quoted 10 lines
The spec allows vendor specific stuff today that are not usable and I'm trying to create a usable API without adding one line of vendor code to driver.
You just need to open a window for few commands like others do and like you already did in the spec.
NVIDIA HW internals are different from
VENDOR_AandVENDOR_B.I want to debug my HW or enable some accelerations in it.
Quoted 6 lines
A big question is that, on what layer do you want to provide those functions?
1) virtio level
2) vendor levelFor 1), it needs to be done in the spec instead of introducing backdoor for vendor specific command
I'm repeating myself. You already have this backdoor for workarounding bugs.
I don't ask for workarounds. I want to create a clean spec definition for vendor to debug/control their special souse.
Quoted 2 lines
For 2), virtio keeps simple as is, functionality is provided in a vendor specific device interface. In this case, you can't use virtio-pci directly, then you need present virtio on the upper layer (with some software meditation)
you mean VDPA ?
Quoted 9 lines
Quoted 4 lines
This is the way to do it.
Again, for debug or acceleration, if you want to do it at the virtio level, you need generalize those features and define them in the spec.
Maybe we can start the work on generalizing the debug facility first. (adding Laurent who is working on this).
Virtio debug stuff can be generalized.
Vendor debug stuff should use vendor specific commands. A HW device has more than virtio state. It might have FW that controls it. It might be an SoC with a lot of components that are used to accelerate this device.
It's not always a QEMU code that implements virtio device.
Quoted 3 lines
Tahnks
Quoted 60 lines · Max Gurtovoy
Quoted 58 lines · Jason Wang
Quoted 56 lines · Max Gurtovoy
Quoted 43 lines · Cornelia Huck
Quoted 41 lines · Max Gurtovoy
Quoted 40 lines · Jason Wang
Quoted 39 lines · Max Gurtovoy
Quoted 38 lines · Michael S. Tsirkin
Quoted 37 lines · Max Gurtovoy
Quoted 36 lines · Cornelia Huck
Quoted 26 lines · Max Gurtovoy
Quoted 17 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
+\subsubsection{Vendor specific command set}\label{sec:Basic
Facilities of a Virtio Device / Admin Virtqueues / Admin
command set / Vendor specific command set}
+
+The Vendor specific command set is a group of classes and
commands
+within each of these classes which are vendor specific.
Refer to
+each vendor specification for more information on the
supported
+commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like
this directly in the spec. We already haveVIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
You can implement a compliant device and a compliant. why do you think you
can't ?If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
again, Linux driver will be generic.
It will include a char dev to user space virtio-cli that will enable sending admin commands to do administration.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Nobody asked to add vendor specific code to the driver.
It's not about the general driver but about the general device. We want to present a general standard device which means we can migrate virtio devices among different vendors.
I don't know if it's reasonable.
You're looking on a device as a SW component since it was like this from day 1. But it's not. It's HW, and HW has its own internal state and implementation that is known only to the vendor.
Spec doesn't forbid device internal state. But it cares the state that is visible to the driver. If we had vendor specific state, it would be very hard to claim to be a standard virtio device.
Quoted 8 lines
Quoted 4 lines
If each vendor choose to implement their own vendor specific features via vendor specific commands that are not ruled by the spec, there will be little value of the spec.
Not at all.
I'm very surprised that you are negative to this RFC instead of working on defining admin commands to stop/start queues and other commands that can be helpful for vDPA migration.
I think there are some misunderstanding here. I'm not negative to admin virtqueue, actually such concept has been discussed by the end of last year. If you search the archive, I've posted RFC for net specific admin virtqueue sometime early this year.
But what I want to be negative is the vendor specific commands/features through the admin virtqueue. The admin virtqueue should be used for general facility like (device stop and migration as you said above).
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
The spec allows vendor specific stuff today that are not usable and I'm trying to create a usable API without adding one line of vendor code to driver.
You just need to open a window for few commands like others do and like you already did in the spec.
NVIDIA HW internals are different from
VENDOR_AandVENDOR_B.I want to debug my HW or enable some accelerations in it.
Quoted 8 lines
Quoted 6 lines
A big question is that, on what layer do you want to provide those functions?
1) virtio level
2) vendor levelFor 1), it needs to be done in the spec instead of introducing backdoor for vendor specific command
I'm repeating myself. You already have this backdoor for workarounding bugs.
To say the truth, I don't like that. And it was probably too late to fix. And I am not aware of any vendor that implements the VENDOR_CFG capability.
But we can do better now by not introducing any new vendor specific stuffs.
Quoted 2 lines
I don't ask for workarounds. I want to create a clean spec definition for vendor to debug/control their special souse.
That's fine, but if it can't be generalized, it's better to be done at the layer under virtio.
Quoted 5 lines
Quoted 3 lines
For 2), virtio keeps simple as is, functionality is provided in a vendor specific device interface. In this case, you can't use virtio-pci directly, then you need present virtio on the upper layer (with some software meditation)
you mean VDPA ?
vDPA is one approach but not all.
Another example is the IFCVF, it has a dedicated bar for live migration which is not a part of the virtio-pci capability. (I'm not saying it's a good design, but it's indeed another approach).
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
This is the way to do it.
Quoted 7 lines
Quoted 3 lines
Again, for debug or acceleration, if you want to do it at the virtio level, you need generalize those features and define them in the spec.
Maybe we can start the work on generalizing the debug facility first. (adding Laurent who is working on this).
Virtio debug stuff can be generalized.
Vendor debug stuff should use vendor specific commands. A HW device has more than virtio state. It might have FW that controls it. It might be an SoC with a lot of components that are used to accelerate this device.
If the states/statistics/counters are highly vendor specific, let's do that under virtio(pci). Such method has been used by other vendors.
Quoted 2 lines
It's not always a QEMU code that implements virtio device.
It's unrelated to whether or not it's a software device. It's about the effort of the spec to present a standard device.
What I don't understand is that, the mlx5-vdpa approach has been proved to succeed in all points you raised above (I meant the vendor specific part). Why need to bother them in the virtio layer if the feature can't be generalized?
Thanks
Quoted 5 lines
Quoted 4 lines
Tahnks
Quoted 115 lines · Jason Wang
Quoted 60 lines · Max Gurtovoy
Quoted 58 lines · Jason Wang
Quoted 56 lines · Max Gurtovoy
Quoted 43 lines · Cornelia Huck
Quoted 41 lines · Max Gurtovoy
Quoted 40 lines · Jason Wang
Quoted 39 lines · Max Gurtovoy
Quoted 38 lines · Michael S. Tsirkin
Quoted 37 lines · Max Gurtovoy
Quoted 36 lines · Cornelia Huck
Quoted 26 lines · Max Gurtovoy
Quoted 25 lines · Michael S. Tsirkin
On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
+\subsubsection{Vendor specific command set}\label{sec:Basic
Facilities of a Virtio Device / Admin Virtqueues / Admin
command set / Vendor specific command set}
+
+The Vendor specific command set is a group of classes and
commands
+within each of these classes which are vendor specific.
Refer to
+each vendor specification for more information on the
supported
+commands.
Here's another example.
It's important that there is a way to make a device completely
generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific
stuff like
this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why
before we add more vendor specific stuff.Quoted 3 lines
We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
You can implement a compliant device and a compliant. why do you think you
can't ?If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
again, Linux driver will be generic.
It will include a char dev to user space virtio-cli that will enable sending admin commands to do administration.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Nobody asked to add vendor specific code to the driver.
It's not about the general driver but about the general device. We want to present a general standard device which means we can migrate virtio devices among different vendors.
I don't know if it's reasonable.
You're looking on a device as a SW component since it was like this from day 1. But it's not. It's HW, and HW has its own internal state and implementation that is known only to the vendor.
Spec doesn't forbid device internal state. But it cares the state that is visible to the driver. If we had vendor specific state, it would be very hard to claim to be a standard virtio device.
Quoted 8 lines
Quoted 4 lines
If each vendor choose to implement their own vendor specific features via vendor specific commands that are not ruled by the spec, there will be little value of the spec.
Not at all.
I'm very surprised that you are negative to this RFC instead of working on defining admin commands to stop/start queues and other commands that can be helpful for vDPA migration.
I think there are some misunderstanding here. I'm not negative to admin virtqueue, actually such concept has been discussed by the end of last year. If you search the archive, I've posted RFC for net specific admin virtqueue sometime early this year.
But what I want to be negative is the vendor specific commands/features through the admin virtqueue. The admin virtqueue should be used for general facility like (device stop and migration as you said above).
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
The spec allows vendor specific stuff today that are not usable and I'm trying to create a usable API without adding one line of vendor code to driver.
You just need to open a window for few commands like others do and like you already did in the spec.
NVIDIA HW internals are different from
VENDOR_AandVENDOR_B.I want to debug my HW or enable some accelerations in it.
Quoted 8 lines
Quoted 6 lines
A big question is that, on what layer do you want to provide those functions?
1) virtio level
2) vendor levelFor 1), it needs to be done in the spec instead of introducing backdoor for vendor specific command
I'm repeating myself. You already have this backdoor for workarounding bugs.
To say the truth, I don't like that. And it was probably too late to fix. And I am not aware of any vendor that implements the
VENDOR_CFGcapability.
Quoted 9 lines
But we can do better now by not introducing any new vendor specific stuffs.
Quoted 2 lines
I don't ask for workarounds. I want to create a clean spec definition for vendor to debug/control their special souse.
That's fine, but if it can't be generalized, it's better to be done at the layer under virtio.
what layer are you talking about ?
There is the virtio layer and the transport layer. That's it.
The transport layer has a small window for configuration and handshake.
The virtio layer has ability to transfer data to/from driver from/to device.
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
For 2), virtio keeps simple as is, functionality is provided in a vendor specific device interface. In this case, you can't use virtio-pci directly, then you need present virtio on the upper layer (with some software meditation)
you mean VDPA ?
vDPA is one approach but not all.
Another example is the IFCVF, it has a dedicated bar for live migration which is not a part of the virtio-pci capability. (I'm not saying it's a good design, but it's indeed another approach).
Sorry, I don't this virtio spec should care about vDPA and IFCVF. This should be the other way around.
And you try to enforce the opposite.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
This is the way to do it.
Again, for debug or acceleration, if you want to do it at the virtio level, you need generalize those features and define them in the spec.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Maybe we can start the work on generalizing the debug facility first. (adding Laurent who is working on this).
Virtio debug stuff can be generalized.
Vendor debug stuff should use vendor specific commands. A HW device has more than virtio state. It might have FW that controls it. It might be an SoC with a lot of components that are used to accelerate this device.
If the states/statistics/counters are highly vendor specific, let's do that under virtio(pci). Such method has been used by other vendors.
No other layers.
Lets stop mentioning this.
Quoted 9 lines
Quoted 4 lines
It's not always a QEMU code that implements virtio device.
It's unrelated to whether or not it's a software device. It's about the effort of the spec to present a standard device.
What I don't understand is that, the mlx5-vdpa approach has been proved to succeed in all points you raised above (I meant the vendor specific part). Why need to bother them in the virtio layer if the feature can't be generalized?
vDPA code shouldn't change at all.
Let's keep vDPA out of this discussion.
I can invent another my_virtio_data_path_acceleration and when I'll do that I don't expect the TC to change the spec to fit it.
Quoted 10 lines
Thanks
Quoted 5 lines
Quoted 4 lines
Tahnks
Quoted 125 lines · Max Gurtovoy
Quoted 123 lines · Jason Wang
Quoted 61 lines · Max Gurtovoy
Quoted 59 lines · Jason Wang
Quoted 57 lines · Max Gurtovoy
Quoted 44 lines · Cornelia Huck
Quoted 42 lines · Max Gurtovoy
Quoted 41 lines · Jason Wang
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Michael S. Tsirkin
Quoted 38 lines · Max Gurtovoy
Quoted 37 lines · Cornelia Huck
Quoted 30 lines · Max Gurtovoy
On 7/28/2021 3:48 PM, Michael S. Tsirkin wrote:
On Mon, Jul 26, 2021 at 07:52:53PM +0300, Max Gurtovoy wrote:
+\subsubsection{Vendor specific command
set}\label{sec:Basic
Facilities of a Virtio Device / Admin Virtqueues / Admin
command set / Vendor specific command set}
+
+The Vendor specific command set is a group of classes and
commands
+within each of these classes which are vendor specific.
Refer to
+each vendor specification for more information on the
supported
+commands.
Here's another example.
It's important that there is a way to make a device completely
generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific
stuff like
this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFG
and if that is not sufficient I would like to know why
before we add more vendor specific stuff.
We are adding an option to add vendor specific commands. We're not
adding it to the spec since each vendor will have its own manual for
that.IMHO, that way madness lies. I want to be able to look at the spec and
be able to implement a compliant device or a compliant driver. If a vendor has some special feature they want to support, put it in the spec, so that it is possible to actually implement it.Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
You can implement a compliant device and a compliant. why do you think you
can't ?If I want to implement a device/driver, I need a standard to refer to. How can something vendor-specific be standard? If it is useful, add it to the virtio spec.
again, Linux driver will be generic.
It will include a char dev to user space virtio-cli that will enable sending admin commands to do administration.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Nobody asked to add vendor specific code to the driver.
It's not about the general driver but about the general device. We want to present a general standard device which means we can migrate virtio devices among different vendors.
I don't know if it's reasonable.
You're looking on a device as a SW component since it was like this from day 1. But it's not. It's HW, and HW has its own internal state and implementation that is known only to the vendor.
Spec doesn't forbid device internal state. But it cares the state that is visible to the driver. If we had vendor specific state, it would be very hard to claim to be a standard virtio device.
Quoted 8 lines
Quoted 4 lines
If each vendor choose to implement their own vendor specific features via vendor specific commands that are not ruled by the spec, there will be little value of the spec.
Not at all.
I'm very surprised that you are negative to this RFC instead of working on defining admin commands to stop/start queues and other commands that can be helpful for vDPA migration.
I think there are some misunderstanding here. I'm not negative to admin virtqueue, actually such concept has been discussed by the end of last year. If you search the archive, I've posted RFC for net specific admin virtqueue sometime early this year.
But what I want to be negative is the vendor specific commands/features through the admin virtqueue. The admin virtqueue should be used for general facility like (device stop and migration as you said above).
Quoted 13 lines
Quoted 13 lines
Quoted 11 lines
The spec allows vendor specific stuff today that are not usable and I'm trying to create a usable API without adding one line of vendor code to driver.
You just need to open a window for few commands like others do and like you already did in the spec.
NVIDIA HW internals are different from
VENDOR_AandVENDOR_B.I want to debug my HW or enable some accelerations in it.
Quoted 8 lines
Quoted 6 lines
A big question is that, on what layer do you want to provide those functions?
1) virtio level
2) vendor levelFor 1), it needs to be done in the spec instead of introducing backdoor for vendor specific command
I'm repeating myself. You already have this backdoor for workarounding bugs.
To say the truth, I don't like that. And it was probably too late to fix. And I am not aware of any vendor that implements the
VENDOR_CFGcapability.But we can do better now by not introducing any new vendor specific stuffs.
Quoted 2 lines
I don't ask for workarounds. I want to create a clean spec definition for vendor to debug/control their special souse.
Quoted 7 lines
Quoted 2 lines
That's fine, but if it can't be generalized, it's better to be done at the layer under virtio.
what layer are you talking about ?
There is the virtio layer and the transport layer. That's it.
The transport layer has a small window for configuration and handshake.
Nope. The spec only cares about virtio related features. It doesn't prevent you from adding vendor specific features in the transport layer.
Quoted 10 lines
The virtio layer has ability to transfer data to/from driver from/to device.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
For 2), virtio keeps simple as is, functionality is provided in a vendor specific device interface. In this case, you can't use virtio-pci directly, then you need present virtio on the upper layer (with some software meditation)
you mean VDPA ?
Quoted 7 lines
Quoted 3 lines
vDPA is one approach but not all.
Another example is the IFCVF, it has a dedicated bar for live migration which is not a part of the virtio-pci capability. (I'm not saying it's a good design, but it's indeed another approach).
Sorry, I don't this virtio spec should care about vDPA and IFCVF. This should be the other way around.
And you try to enforce the opposite.
No. It's not related to vDPA but just an example of adding vendor specific features.
If you look at the IFCVF driver codes, it's easy to see that the device could be probed by the standard virtio-pci driver.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 8 lines
This is the way to do it.
Again, for debug or acceleration, if you want to do it at the virtio level, you need generalize those features and define them in the spec.
Maybe we can start the work on generalizing the debug facility first. (adding Laurent who is working on this).
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Virtio debug stuff can be generalized.
Vendor debug stuff should use vendor specific commands. A HW device has more than virtio state. It might have FW that controls it. It might be an SoC with a lot of components that are used to accelerate this device.
If the states/statistics/counters are highly vendor specific, let's do that under virtio(pci). Such method has been used by other vendors.
No other layers.
Virtio can't work by itself.
Quoted 2 lines
Lets stop mentioning this.
Let's keep the discussion constructive. I'm not obliged to design hardware for you, but obliged to figure out the design that violates the goal of the virtio spec. What I mentioned has been used by other vendor and it works well without bothering the virtio spec at all.
I think I've stated many times that adding vendor specific features in the spec is wrong, we want to have general features within a standard device. And you don't even explain why it must be done at virtio spec, (actually, not even defined in the spec but just leave a door for vendor to implement anything). Technically, you can have non virtio PF with virtio VF.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
It's not always a QEMU code that implements virtio device.
It's unrelated to whether or not it's a software device. It's about the effort of the spec to present a standard device.
Quoted 7 lines
Quoted 1 line
What I don't understand is that, the mlx5-vdpa approach has been proved to succeed in all points you raised above (I meant the vendor specific part). Why need to bother them in the virtio layer if the feature can't be generalized?
vDPA code shouldn't change at all.
Let's keep vDPA out of this discussion.
I can invent another
my_virtio_data_path_accelerationand when I'll do that I don't expect the TC to change the spec to fit it.
I think we need some clarification on the concept "vDPA". To me it means virtio + vendor specific features (out of the spec).
Thanks
Quoted 28 lines
Quoted 11 lines
Thanks
Quoted 5 lines
Quoted 4 lines
Tahnks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 33 lines · Max Gurtovoy
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>
Quoted 8 lines
Quoted 4 lines
Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.This will be defined per command.
For example,
funcion_idwill be given as command data.
Why? Sounds like a generic thing to me.
Quoted 136 lines
Quoted 118 lines
Quoted 118 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +104
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\Quoted 8 lines
admin-virtq.tex line 110
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach.
One is that there is no two way negotiation.you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
Quoted 6 lines
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
I don't understand what it breaks exactly.
Long practice taught us that it is good for device to know what is driver going to use.
For example, some features can be implemented in hardware and some in hypervisor software. If driver is going to use software features then you need to switch to a slower software implementation. Doing that dynamically at time of use is often much harder that up-front at negotiation time.
Quoted 7 lines
Quoted 5 lines
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
we'll develop virtio-cli tool.
will that be part of the spec somehow then?
I'm not sure it can be ...
Quoted 1 line
I don't know what should be changed in the device by the hypervisor.
Look at vdpa as an example, guest device can be different from host device.
Quoted 11 lines
Quoted 5 lines
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.device will support admin q with mandatory commands.
Old driver shouldn't care and it won't negotiate this feature.
New drivers will adjust and will be extended.
Does not sound like I got the issue across.
Operating VQs just to know whether to bind to device is problematic since there's an assumption that once DRIVER_OK is set driver is happy. Before DRIVER_OK device will not consume buffers.
Quoted 9 lines
Quoted 8 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.
Quoted 4 lines
This is initial commit to add infrastructure. I can't add 255 commands at once.
We need to add it incrementally.
How about a list of future extensions?
Quoted 142 lines
Quoted 142 lines
Quoted 125 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
Quoted 7 lines
Quoted 4 lines
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
You didn't actually answer the question though.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Opaque strings passed all the way from guest userspace to host device? Not sure why is that a good idea.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Max Gurtovoy
Quoted 31 lines · Michael S. Tsirkin
Quoted 30 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.This will be defined per command.
For example,
funcion_idwill be given as command data.Why? Sounds like a generic thing to me.
Generic to a command that handles virtualization management.
It has nothing to do with a discovery command for the device itself.
I guess we can add few bytes to be mandatory after the <u8 class, u8 command> pair.
Something like <u8 class, u8 command, u16 mandatory_command_data >.
But I think it's not a must and it's already covered by: "u8 command-specific-data[];"
Quoted 137 lines
Quoted 137 lines
Quoted 118 lines
Quoted 118 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +104
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\Quoted 8 lines
admin-virtq.tex line 110
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach.
One is that there is no two way negotiation.you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
Quoted 8 lines
Quoted 1 line
I don't understand what it breaks exactly.
Long practice taught us that it is good for device to know what is driver going to use.
For example, some features can be implemented in hardware and some in hypervisor software. If driver is going to use software features then you need to switch to a slower software implementation. Doing that dynamically at time of use is often much harder that up-front at negotiation time.
I don't think we should write specifications that should consider hypervisor SW.
You might use virtio device without hypervisor at all.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
we'll develop virtio-cli tool.
will that be part of the spec somehow then?
I'm not sure it can be ...
of course not.
This is just a channel for the admin to send admin command to the virtio device.
Quoted 4 lines
Quoted 2 lines
I don't know what should be changed in the device by the hypervisor.
Look at vdpa as an example, guest device can be different from host device.
VDPA is an acceleration. I can write different acceleration on to of virtio device.
Is shouldn't be considered in the specification.
Nevertheless, there is nothing in this RFC that VDPA can't handle.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.device will support admin q with mandatory commands.
Old driver shouldn't care and it won't negotiate this feature.
Quoted 7 lines
Quoted 1 line
New drivers will adjust and will be extended.
Does not sound like I got the issue across.
Operating VQs just to know whether to bind to device is problematic since there's an assumption that once
DRIVER_OKis set driver is happy. BeforeDRIVER_OKdevice will not consume buffers.
Nothing is changed in DRIVER_OK negotiation rules.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.
Quoted 5 lines
Quoted 4 lines
This is initial commit to add infrastructure. I can't add 255 commands at once.
We need to add it incrementally.
How about a list of future extensions?
Added a partial list in the commit message and in the answer to Cornelia.
Quoted 149 lines
Quoted 141 lines
Quoted 141 lines
Quoted 126 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
Quoted 7 lines
You didn't actually answer the question though.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Opaque strings passed all the way from guest userspace to host device? Not sure why is that a good idea.
Where did I mentioned a guest ?
Virtio-cli will control a device on the same host that it runs.
If it's a bare metal host so it will manage the virtio attached device.
If it's a guest it will manage the device attached to the guest.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 16 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific class
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.
I still don't really see what do all these things have in common? Why are they grouped behind the same feature bit? Share same VQ?
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
This will be defined per command.
For example,
funcion_idwill be given as command data.Why? Sounds like a generic thing to me.
Generic to a command that handles virtualization management.
It could be that mixing up virtualization management and arbitrary other management commands in the same interface is a mistake. Or maybe not - do we want host to have ability to run health tests for a VF without loading VF driver? Get error logs?
In fact besides migration and virtualization management the rest of examples that you give all seem to be more or less device specific, with the possible exception of 3. Encrypt/Decrypt descriptors. what does this one imply, exactly?
Quoted 9 lines
It has nothing to do with a discovery command for the device itself.
I guess we can add few bytes to be mandatory after the <u8 class, u8 command> pair.
Something like <u8 class, u8 command, u16
mandatory_command_data>.But I think it's not a must and it's already covered by: "u8 command-specific-data[];"
I would say we should include the device ID as well as a definition what the ID is at least for SRIOV. This isn't a command specific data, any virt management command will need this.
Quoted 147 lines
Quoted 138 lines
Quoted 138 lines
Quoted 119 lines
Quoted 119 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +104
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\Quoted 8 lines
admin-virtq.tex line 110
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach.
One is that there is no two way negotiation.you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
Quoted 8 lines
Quoted 1 line
I don't understand what it breaks exactly.
Long practice taught us that it is good for device to know what is driver going to use.
For example, some features can be implemented in hardware and some in hypervisor software. If driver is going to use software features then you need to switch to a slower software implementation. Doing that dynamically at time of use is often much harder that up-front at negotiation time.
Quoted 2 lines
I don't think we should write specifications that should consider hypervisor SW.
Well considering hypervisors is clearly one of the purposes of virtio TC. Look it up in the charter, section 2. Statement of Purpose.
Quoted 1 line
You might use virtio device without hypervisor at all.
Yes and supporting that is also clearly an objective:
With the 1.1, 1.2 and future revisions of the Specification, we aim to evolve the VIRTIO standard further, addressing such new requirements while both continuing to honor the goals of the 1.0 Specification and including new objectives.
Quoted 36 lines
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
we'll develop virtio-cli tool.
will that be part of the spec somehow then?
I'm not sure it can be ...of course not.
This is just a channel for the admin to send admin command to the virtio device.
Quoted 4 lines
Quoted 2 lines
I don't know what should be changed in the device by the hypervisor.
Look at vdpa as an example, guest device can be different from host device.
VDPA is an acceleration. I can write different acceleration on to of virtio device.
Is shouldn't be considered in the specification.
Nevertheless, there is nothing in this RFC that VDPA can't handle.
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.device will support admin q with mandatory commands.
Old driver shouldn't care and it won't negotiate this feature.
New drivers will adjust and will be extended.
Quoted 8 lines
Quoted 6 lines
Does not sound like I got the issue across.
Operating VQs just to know whether to bind to device is problematic since there's an assumption that once
DRIVER_OKis set driver is happy. BeforeDRIVER_OKdevice will not consume buffers.Nothing is changed in
DRIVER_OKnegotiation rules.
Right and that means you can't really use the admin vq to validate device during driver probe.
Quoted 174 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.Quoted 5 lines
Quoted 4 lines
This is initial commit to add infrastructure. I can't add 255 commands at once.
We need to add it incrementally.
How about a list of future extensions?
Added a partial list in the commit message and in the answer to Cornelia.
Quoted 149 lines
Quoted 141 lines
Quoted 141 lines
Quoted 126 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
Quoted 7 lines
You didn't actually answer the question though.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Opaque strings passed all the way from guest userspace to host device? Not sure why is that a good idea.
Quoted 7 lines
Where did I mentioned a guest ?
Virtio-cli will control a device on the same host that it runs.
If it's a bare metal host so it will manage the virtio attached device.
If it's a guest it will manage the device attached to the guest.
Opaqueness of all this is what worries at least me and Cornelia.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 27 lines · Michael S. Tsirkin
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.
Quoted 1 line
I still don't really see what do all these things have in common?
I don't think you need to look on this in that direction.
This is a queue for administration.
Cornelia and Stefan already agreed on this approach.
Please lets progress and not go back to the beginning.
Quoted 1 line
Why are they grouped behind the same feature bit? Share same VQ?
They are grouped behind an admin q.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
This will be defined per command.
For example,
funcion_idwill be given as command data.Why? Sounds like a generic thing to me.
Generic to a command that handles virtualization management.
It could be that mixing up virtualization management and arbitrary other management commands in the same interface is a mistake.
It's not a mistake.
This is the right design.
Creating a new interface for each feature is madness.
Quoted 2 lines
Or maybe not - do we want host to have ability to run health tests for a VF without loading VF driver? Get error logs?
This can be an optional feature.
We need to define it in the future. We can't define the entire command set now.
We need to define the infrastructure.
Quoted 5 lines
In fact besides migration and virtualization management the rest of examples that you give all seem to be more or less device specific, with the possible exception of 3. Encrypt/Decrypt descriptors. what does this one imply, exactly?
For storage devices there is an option to have a self encrypted drive.
Maybe we can develop encryption/decryption also for net devices.
This will be developed in the future.
But the infrastructure will allow it. This is the beauty if it, you create infrastructure today and add optional commands tomorrow.
Nobody can think of thousands of features and commands today, and we also can't push thousands of pages to the spec.
Lets push 2-3 mandatory commands with the infrastructure and build new features incrementally.
Quoted 8 lines
Quoted 8 lines
It has nothing to do with a discovery command for the device itself.
I guess we can add few bytes to be mandatory after the <u8 class, u8 command> pair.
Quoted 7 lines
Quoted 4 lines
Something like <u8 class, u8 command, u16
mandatory_command_data>.But I think it's not a must and it's already covered by: "u8 command-specific-data[];"
I would say we should include the device ID as well as a definition what the ID is at least for SRIOV. This isn't a command specific data, any virt management command will need this.
of course it will be defined for virt management. But is only for PCI devices.
Not needed for MMIO devices, right ?
Quoted 152 lines
Quoted 146 lines
Quoted 138 lines
Quoted 138 lines
Quoted 119 lines
Quoted 119 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +104
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\Quoted 8 lines
admin-virtq.tex line 110
Quoted 6 lines
admin-virtq.tex new file +6
-0+\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach.
One is that there is no two way negotiation.you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
Quoted 8 lines
Quoted 1 line
I don't understand what it breaks exactly.
Long practice taught us that it is good for device to know what is driver going to use.
For example, some features can be implemented in hardware and some in hypervisor software. If driver is going to use software features then you need to switch to a slower software implementation. Doing that dynamically at time of use is often much harder that up-front at negotiation time.Quoted 2 lines
I don't think we should write specifications that should consider hypervisor SW.
Well considering hypervisors is clearly one of the purposes of virtio TC. Look it up in the charter, section 2. Statement of Purpose.
Quoted 9 lines
Quoted 1 line
You might use virtio device without hypervisor at all.
Yes and supporting that is also clearly an objective:
With the 1.1, 1.2 and future revisions of the Specification, we aim to evolve the VIRTIO standard further, addressing such new requirements while both continuing to honor the goals of the 1.0 Specification and including new objectives.
There is nothing in admin queue that doesn't honor old spec.
Old driver will not be aware of it.
Quoted 33 lines
Quoted 33 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
we'll develop virtio-cli tool.
will that be part of the spec somehow then?
I'm not sure it can be ...of course not.
This is just a channel for the admin to send admin command to the virtio device.
Quoted 3 lines
Quoted 1 line
I don't know what should be changed in the device by the hypervisor.
Look at vdpa as an example, guest device can be different from host device.
VDPA is an acceleration. I can write different acceleration on to of virtio device.
Is shouldn't be considered in the specification.
Nevertheless, there is nothing in this RFC that VDPA can't handle.
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.device will support admin q with mandatory commands.
Old driver shouldn't care and it won't negotiate this feature.
New drivers will adjust and will be extended.
Does not sound like I got the issue across.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Operating VQs just to know whether to bind to device is problematic since there's an assumption that once
DRIVER_OKis set driver is happy. BeforeDRIVER_OKdevice will not consume buffers.Nothing is changed in
DRIVER_OKnegotiation rules.Right and that means you can't really use the admin vq to validate device during driver probe.
You said that before DRIVER_OK is set the device will not consume buffers, so what is the question/problem here ?
Do you think we need to change it ? I don't think it's needed.
Quoted 170 lines
Quoted 170 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.Quoted 5 lines
Quoted 4 lines
This is initial commit to add infrastructure. I can't add 255 commands at once.
We need to add it incrementally.
How about a list of future extensions?
Added a partial list in the commit message and in the answer to Cornelia.
Quoted 154 lines
Quoted 139 lines
Quoted 139 lines
Quoted 124 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
You didn't actually answer the question though.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Opaque strings passed all the way from guest userspace to host device? Not sure why is that a good idea.
Quoted 8 lines
Quoted 7 lines
Where did I mentioned a guest ?
Virtio-cli will control a device on the same host that it runs.
If it's a bare metal host so it will manage the virtio attached device.
If it's a guest it will manage the device attached to the guest.
Opaqueness of all this is what worries at least me and Cornelia.
guest is not aware of host devices.
Sending raw command from Linux cmdline to a device is not something I invented.
If a user is aware of its HW he can use the virtio-cli tool to configure its unique features.
And if the user is not so smart, we can help him with adding vendor classes to virtio-cli management tool.
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 29 lines · Max Gurtovoy
Quoted 27 lines · Michael S. Tsirkin
Quoted 26 lines · Max Gurtovoy
Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Max Gurtovoy
Quoted 23 lines · Michael S. Tsirkin
Quoted 22 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.
Quoted 7 lines
Quoted 1 line
I still don't really see what do all these things have in common?
I don't think you need to look on this in that direction.
This is a queue for administration.
Cornelia and Stefan already agreed on this approach.
We're discussing these patches and I'm trying to understand what you're proposing. It's a bit early to say I've agreed to anything.
Stefan
Quoted 32 lines · Stefan Hajnoczi
Quoted 31 lines · Max Gurtovoy
Quoted 28 lines · Michael S. Tsirkin
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.I still don't really see what do all these things have in common?
Quoted 8 lines
Quoted 5 lines
I don't think you need to look on this in that direction.
This is a queue for administration.
Cornelia and Stefan already agreed on this approach.
We're discussing these patches and I'm trying to understand what you're proposing. It's a bit early to say I've agreed to anything.
For the record, same for me.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 32 lines · Cornelia Huck
Quoted 30 lines · Stefan Hajnoczi
Quoted 29 lines · Max Gurtovoy
Quoted 28 lines · Michael S. Tsirkin
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.I still don't really see what do all these things have in common?
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
I don't think you need to look on this in that direction.
This is a queue for administration.
Cornelia and Stefan already agreed on this approach.
We're discussing these patches and I'm trying to understand what you're proposing. It's a bit early to say I've agreed to anything.
For the record, same for me.
didn't you agree on admin q approach ?
I'm not talking about the command set.
Quoted 1 line
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 38 lines · Max Gurtovoy
Quoted 36 lines · Cornelia Huck
Quoted 34 lines · Stefan Hajnoczi
Quoted 33 lines · Max Gurtovoy
Quoted 27 lines · Michael S. Tsirkin
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.Quoted 1 line
I still don't really see what do all these things have in common?
I don't think you need to look on this in that direction.
This is a queue for administration.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
Cornelia and Stefan already agreed on this approach.
We're discussing these patches and I'm trying to understand what you're proposing. It's a bit early to say I've agreed to anything.
For the record, same for me.
didn't you agree on admin q approach ?
I'm not talking about the command set.
I like the idea of a common admin virtqueue but it's still necessary to answer questions that have been raised. If someone identifies a significant issue or suggests an alternative then that needs to be dealt with on its technical merit.
Stefan
Quoted 34 lines · Max Gurtovoy
Quoted 27 lines · Michael S. Tsirkin
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.Quoted 1 line
I still don't really see what do all these things have in common?
I don't think you need to look on this in that direction.
This is a queue for administration.
Quoted 7 lines
Cornelia and Stefan already agreed on this approach.
Please lets progress and not go back to the beginning.
Quoted 1 line
Why are they grouped behind the same feature bit? Share same VQ?
They are grouped behind an admin q.
It's fine for a variety of things to use the admin q. But I think each feature should come with its own feature bit that depends on the admin vq.
What actually makes sense to use the admin vq is also worth further discussion.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.This will be defined per command.
For example,
funcion_idwill be given as command data.Why? Sounds like a generic thing to me.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Generic to a command that handles virtualization management.
It could be that mixing up virtualization management and arbitrary other management commands in the same interface is a mistake.
It's not a mistake.
This is the right design.
Well, we're clearly not all in agreement what the "right design" is. Figuring that out is the whole point of this discussion.
Quoted 26 lines
Creating a new interface for each feature is madness.
Quoted 2 lines
Or maybe not - do we want host to have ability to run health tests for a VF without loading VF driver? Get error logs?
This can be an optional feature.
We need to define it in the future. We can't define the entire command set now.
We need to define the infrastructure.
Quoted 5 lines
In fact besides migration and virtualization management the rest of examples that you give all seem to be more or less device specific, with the possible exception of 3. Encrypt/Decrypt descriptors. what does this one imply, exactly?
For storage devices there is an option to have a self encrypted drive.
Maybe we can develop encryption/decryption also for net devices.
This will be developed in the future.
Quoted 8 lines
But the infrastructure will allow it. This is the beauty if it, you create infrastructure today and add optional commands tomorrow.
Nobody can think of thousands of features and commands today, and we also can't push thousands of pages to the spec.
Lets push 2-3 mandatory commands with the infrastructure and build new features incrementally.
Why "mandatory commands"? Just make them covered by a feature bit.
Quoted 36 lines
Quoted 27 lines
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 2 lines
So there are several problems with this approach.
One is that there is no two way negotiation.you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
Quoted 8 lines
Quoted 1 line
I don't understand what it breaks exactly.
Long practice taught us that it is good for device to know what is driver going to use.
For example, some features can be implemented in hardware and some in hypervisor software. If driver is going to use software features then you need to switch to a slower software implementation. Doing that dynamically at time of use is often much harder that up-front at negotiation time.Quoted 2 lines
I don't think we should write specifications that should consider hypervisor SW.
Well considering hypervisors is clearly one of the purposes of virtio TC. Look it up in the charter, section 2. Statement of Purpose.
Quoted 9 lines
Quoted 1 line
You might use virtio device without hypervisor at all.
Yes and supporting that is also clearly an objective:
With the 1.1, 1.2 and future revisions of the Specification, we aim to evolve the VIRTIO standard further, addressing such new requirements while both continuing to honor the goals of the 1.0 Specification and including new objectives.
Quoted 3 lines
There is nothing in admin queue that doesn't honor old spec.
Old driver will not be aware of it.
I don't see how this helps; negotiating the admin vq only tells the device that the driver wants to use the admin vq, but not what it actually wants to use. This is a departure from the feature negotiation method used up to now.
Quoted 37 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
Review
Quoted 6 lines
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
(quoted diff) +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
You didn't actually answer the question though.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Opaque strings passed all the way from guest userspace to host device? Not sure why is that a good idea.
Quoted 8 lines
Quoted 7 lines
Where did I mentioned a guest ?
Virtio-cli will control a device on the same host that it runs.
If it's a bare metal host so it will manage the virtio attached device.
If it's a guest it will manage the device attached to the guest.
Opaqueness of all this is what worries at least me and Cornelia.
guest is not aware of host devices.
Quoted 8 lines
Sending raw command from Linux cmdline to a device is not something I invented.
If a user is aware of its HW he can use the virtio-cli tool to configure its unique features.
And if the user is not so smart, we can help him with adding vendor classes to virtio-cli management tool.
There's something very wrong in relying on an external tool to configure a supposedly standardized device. This spec is supposed to be platform-agnostic. Everything must be implementable by a random OS or HW maker, and for that, it needs to be properly specified.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 37 lines · Cornelia Huck
Quoted 35 lines · Max Gurtovoy
Quoted 27 lines · Michael S. Tsirkin
Quoted 15 lines · Max Gurtovoy
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Max Gurtovoy
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classQuoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
The above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.Quoted 1 line
I still don't really see what do all these things have in common?
I don't think you need to look on this in that direction.
This is a queue for administration.
Cornelia and Stefan already agreed on this approach.
Quoted 7 lines
Quoted 4 lines
Please lets progress and not go back to the beginning.
Quoted 1 line
Why are they grouped behind the same feature bit? Share same VQ?
They are grouped behind an admin q.
It's fine for a variety of things to use the admin q. But I think each feature should come with its own feature bit that depends on the admin vq.
There is a specific command in the admin command set that query all these feature bits.
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.
please think of 5-10 years from today.
Quoted 20 lines
What actually makes sense to use the admin vq is also worth further discussion.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.This will be defined per command.
For example,
funcion_idwill be given as command data.Why? Sounds like a generic thing to me.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Generic to a command that handles virtualization management.
It could be that mixing up virtualization management and arbitrary other management commands in the same interface is a mistake.
It's not a mistake.
This is the right design.
Well, we're clearly not all in agreement what the "right design" is. Figuring that out is the whole point of this discussion.
I suggested a solid infrastructure for adding any feature easily in the future without having half a year waste on debates.
I didn't see any other suggestion so far.
Only some half baked phrases and answers.
Quoted 26 lines
Quoted 26 lines
Creating a new interface for each feature is madness.
Quoted 2 lines
Or maybe not - do we want host to have ability to run health tests for a VF without loading VF driver? Get error logs?
This can be an optional feature.
We need to define it in the future. We can't define the entire command set now.
We need to define the infrastructure.
Quoted 4 lines
In fact besides migration and virtualization management the rest of examples that you give all seem to be more or less device specific, with the possible exception of 3. Encrypt/Decrypt descriptors. what does this one imply, exactly?
For storage devices there is an option to have a self encrypted drive.
Maybe we can develop encryption/decryption also for net devices.
This will be developed in the future.
But the infrastructure will allow it. This is the beauty if it, you create infrastructure today and add optional commands tomorrow.
Quoted 6 lines
Quoted 5 lines
Nobody can think of thousands of features and commands today, and we also can't push thousands of pages to the spec.
Lets push 2-3 mandatory commands with the infrastructure and build new features incrementally.
Why "mandatory commands"? Just make them covered by a feature bit.
mandatory for anyone that supports admin q.
They are covered by admin q bit.
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
So there are several problems with this approach.
One is that there is no two way negotiation.you negotiate the adminq feature.
then you can send admin commands and discover the supported commands.
Quoted 1 line
No way for device to know what will driver use in the end.
device will be designed to support mandatory admin commands and some optional.
It doesn't need to care whether the driver will use it or not.
Quoted 2 lines
This breaks things like e.g. accelerating some configurations but not others.
Quoted 8 lines
Quoted 1 line
I don't understand what it breaks exactly.
Long practice taught us that it is good for device to know what is driver going to use.
For example, some features can be implemented in hardware and some in hypervisor software. If driver is going to use software features then you need to switch to a slower software implementation. Doing that dynamically at time of use is often much harder that up-front at negotiation time.Quoted 2 lines
I don't think we should write specifications that should consider hypervisor SW.
Well considering hypervisors is clearly one of the purposes of virtio TC. Look it up in the charter, section 2. Statement of Purpose.
Quoted 1 line
You might use virtio device without hypervisor at all.
Yes and supporting that is also clearly an objective:
With the 1.1, 1.2 and future revisions of the Specification, we aim to evolve the VIRTIO standard further, addressing such new requirements while both continuing to honor the goals of the 1.0 Specification and including new objectives.
Quoted 7 lines
Quoted 3 lines
There is nothing in admin queue that doesn't honor old spec.
Old driver will not be aware of it.
I don't see how this helps; negotiating the admin vq only tells the device that the driver wants to use the admin vq, but not what it actually wants to use. This is a departure from the feature negotiation method used up to now.
The driver can use all the admin q commands the device is capable to support.
The HW tells the cap of admin queue and this is negotiated.
Then all the supported commands should be usable and supported by the device.
Quoted 43 lines
Quoted 43 lines
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Review
Quoted 7 lines
+\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
(quoted diff) +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.Quoted 8 lines
Quoted 6 lines
That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.We are adding an option to add vendor specific commands. We're not adding it to the spec since each vendor will have its own manual for that.
You didn't actually answer the question though.
Quoted 3 lines
For example, we can use virtio-cli to pass command from command line to device in pass-through manner without changing driver.
Opaque strings passed all the way from guest userspace to host device? Not sure why is that a good idea.
Quoted 8 lines
Quoted 7 lines
Where did I mentioned a guest ?
Virtio-cli will control a device on the same host that it runs.
If it's a bare metal host so it will manage the virtio attached device.
If it's a guest it will manage the device attached to the guest.
Opaqueness of all this is what worries at least me and Cornelia.
guest is not aware of host devices.
Sending raw command from Linux cmdline to a device is not something I invented.
If a user is aware of its HW he can use the virtio-cli tool to configure its unique features.
Quoted 6 lines
Quoted 2 lines
And if the user is not so smart, we can help him with adding vendor classes to virtio-cli management tool.
There's something very wrong in relying on an external tool to configure a supposedly standardized device. This spec is supposed to be platform-agnostic. Everything must be implementable by a random OS or HW maker, and for that, it needs to be properly specified.
Where do I rely on it ? I said it's an option.
How would you like to format a virtio block device without tools ? the driver can't do this job out of the blue.
It is platform agnostic.
And it is a proper spec.
Opening a vendor channel is already there.
Nothing new here.
Quoted 7 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
Quoted 24 lines
Quoted 24 lines
What actually makes sense to use the admin vq is also worth further discussion.
Quoted 19 lines
Quoted 19 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
This will be defined per command.
For example,
funcion_idwill be given as command data.Why? Sounds like a generic thing to me.
Generic to a command that handles virtualization management.
It could be that mixing up virtualization management and arbitrary other management commands in the same interface is a mistake.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
It's not a mistake.
This is the right design.
Well, we're clearly not all in agreement what the "right design" is. Figuring that out is the whole point of this discussion.
I suggested a solid infrastructure for adding any feature easily in the future without having half a year waste on debates.
Well we don't really normally waste time on debating adding feature bits. We do that a lot without a lot of fuss.
Signature · 1 line
MST
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.
Quoted 8 lines
Quoted 2 lines
please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
I recall that we had a discussion about that years ago when we introduced VERSION_1; we explicitly agreed that we can extend features beyond 64 bit once we need it. (A quick search did not turn up that mail exchange, though.)
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.Quoted 7 lines
Quoted 2 lines
please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
Quoted 5 lines
I recall that we had a discussion about that years ago when we introduced
VERSION_1; we explicitly agreed that we can extend features beyond 64 bit once we need it. (A quick search did not turn up that mail exchange, though.)
E.g PCI transport has feature_select.
le32 device_feature_select; /* read-write */ le32 device_feature; /* read-only for driver */ le32 driver_feature_select; /* read-write */ le32 driver_feature; /* read-write */
Technically it can support 32*32 different features.
Thanks
Quoted 22 lines · Jason Wang
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.Quoted 7 lines
Quoted 2 lines
please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
Quoted 5 lines
I recall that we had a discussion about that years ago when we introduced
VERSION_1; we explicitly agreed that we can extend features beyond 64 bit once we need it. (A quick search did not turn up that mail exchange, though.)
Quoted 8 lines
E.g PCI transport has
feature_select.le32 device_feature_select; /* read-write */ le32 device_feature; /* read-only for driver */ le32 driver_feature_select; /* read-write */ le32 driver_feature; /* read-write */
Technically it can support 32*32 different features.
MMIO uses a similar scheme; CCW uses
struct virtio_feature_desc {
le32 features;
u8 index;
};so there's plenty of room to spare.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 31 lines · Cornelia Huck
Quoted 21 lines · Jason Wang
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.Quoted 7 lines
Quoted 2 lines
please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
Quoted 5 lines
I recall that we had a discussion about that years ago when we introduced
VERSION_1; we explicitly agreed that we can extend features beyond 64 bit once we need it. (A quick search did not turn up that mail exchange, though.)Quoted 8 lines
E.g PCI transport has
feature_select.le32 device_feature_select; /* read-write */ le32 device_feature; /* read-only for driver */ le32 driver_feature_select; /* read-write */ le32 driver_feature; /* read-write */
Technically it can support 32*32 different features.
Quoted 8 lines
MMIO uses a similar scheme; CCW uses
struct virtio_feature_desc { le32 features; u8 index; };so there's plenty of room to spare.
Feature negotiation can be done in admin q level. You don't want this branches in the spec per each transport or device type.
We have an admin command to get all device admin q features and we can create a command to set all the driver supported.
Again, I don't see the point of doing that. If a device is supporting command_A, the driver should be able to use it. Telling the device "hi, I'm going to use command_A in the future" is redundant.
The device knows already how to handle it.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 39 lines · Max Gurtovoy
Quoted 38 lines · Cornelia Huck
Quoted 21 lines · Jason Wang
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.Quoted 7 lines
Quoted 2 lines
please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
Quoted 5 lines
I recall that we had a discussion about that years ago when we introduced
VERSION_1; we explicitly agreed that we can extend features beyond 64 bit once we need it. (A quick search did not turn up that mail exchange, though.)Quoted 8 lines
E.g PCI transport has
feature_select.le32 device_feature_select; /* read-write */ le32 device_feature; /* read-only for driver */ le32 driver_feature_select; /* read-write */ le32 driver_feature; /* read-write */
Technically it can support 32*32 different features.
MMIO uses a similar scheme; CCW uses
struct virtio_feature_desc { le32 features; u8 index; };
Quoted 4 lines
Quoted 1 line
so there's plenty of room to spare.
Feature negotiation can be done in admin q level. You don't want this branches in the spec per each transport or device type.
This has nothing to do with transports. We only wanted to demonstrate that all of the transports can be extended, so feature bits are not a scarce resource.
Quoted 9 lines
We have an admin command to get all device admin q features and we can create a command to set all the driver supported.
Again, I don't see the point of doing that. If a device is supporting
command_A, the driver should be able to use it. Telling the device "hi, I'm going to usecommand_Ain the future" is redundant.The device knows already how to handle it.
If the driver negotiates a certain feature, it can mean that the device can make certain optimizations.
That's not a particularily exotic pattern.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 55 lines · Cornelia Huck
Quoted 34 lines · Max Gurtovoy
Quoted 33 lines · Cornelia Huck
Quoted 21 lines · Jason Wang
Quoted 15 lines · Cornelia Huck
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Max Gurtovoy
There is no much bits left in the generic feature field for all the features we would like to add.
I mentioned only 5-6 in the above example and it will bring us to bit 46 already.Quoted 7 lines
Quoted 2 lines
please think of 5-10 years from today.
IIUC nothing prevents adding more once we exhaust 64 bits. IMHO it's actually pretty important to make sure the feature negotiation works well and covers relevant usecases. If we have limitations preventing that I'd like to at least try to fix that not replacing feature negotiation with something else.
Quoted 5 lines
I recall that we had a discussion about that years ago when we introduced
VERSION_1; we explicitly agreed that we can extend features beyond 64 bit once we need it. (A quick search did not turn up that mail exchange, though.)Quoted 8 lines
E.g PCI transport has
feature_select.le32 device_feature_select; /* read-write */ le32 device_feature; /* read-only for driver */ le32 driver_feature_select; /* read-write */ le32 driver_feature; /* read-write */
Technically it can support 32*32 different features.
MMIO uses a similar scheme; CCW uses
Quoted 8 lines
Quoted 6 lines
struct virtio_feature_desc { le32 features; u8 index; };so there's plenty of room to spare.
Feature negotiation can be done in admin q level. You don't want this branches in the spec per each transport or device type.
This has nothing to do with transports. We only wanted to demonstrate that all of the transports can be extended, so feature bits are not a scarce resource.
Quoted 7 lines
We have an admin command to get all device admin q features and we can create a command to set all the driver supported.
Again, I don't see the point of doing that. If a device is supporting
command_A, the driver should be able to use it. Telling the device "hi, I'm going to usecommand_Ain the future" is redundant.
Quoted 5 lines
Quoted 1 line
The device knows already how to handle it.
If the driver negotiates a certain feature, it can mean that the device can make certain optimizations.
That's not a particularily exotic pattern.
It's also doesn't allow forward compatibility.
Do you really want to add SW for each new HW cap ?
We're having difficulties because of this non fwd-compatible handshake.
Quoted 165 lines · Michael S. Tsirkin
Quoted 23 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size,512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.Quoted 7 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.Quoted 112 lines
--- admin-virtq.tex | 241 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 4 + 2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +100
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueues are used to send administrative commands to manipulate +various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and \field{command-specific-data} are +set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the following form: + +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set} + +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set} + +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class} + +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|}admin-virtq.tex line 110
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach. One is that there is no two way negotiation.
No way for device to know what will driver use in the end. This breaks things like e.g. accelerating some configurations but not others.
Quoted 7 lines
Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.
Yes, so I think it makes sense to make the admin virtqueue as a transport and then build other stuffs on top.
Quoted 7 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.
It should work (especially for PCI).
Thanks
Quoted 172 lines
Quoted 125 lines
admin-virtq.tex new file +60
-0+ +\subparagraph{Device identity command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command} + +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class} + +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +61
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in the following structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sections will +follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Transport command set} + +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Device command set} +admin-virtq.tex line 241
Quoted 10 lines
admin-virtq.tex new file +10
-0+The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set} + +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 24 lines
content.tex +4
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} data to be provided in driver notification and the delivery method is transport specific. For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that + the device supports administration virtqueue negotiation. \end{description}Signature · 1 line
2.21.0
Quoted 179 lines · Jason Wang
Quoted 177 lines · Michael S. Tsirkin
Quoted 23 lines · Max Gurtovoy
Admin virtqueues will be used to send administrative commands to manipulate various features of the device which would not easily map into the configuration space.
The same Admin command format will be used for all virtio devices. The Admin command set will include 4 types of command classes:
1. The generic common class
2. The transport specific class
3. The device specific class
4. The vendor specific classThe above mechanism will enable adding various features to the virtio specification, e.g.:
1. Format virtio-blk devices in various configurations (512B block size, 512B + 8B T10-DIF, 4K block size, 4k + 8B T10-DIF, etc..).
2. Live migration management.
3. Encrypt/Decrypt descriptors.
4. Virtualization management.
5. Get device error logs.
6. Implement advanced vendor/device/transport specific features.
7. Run device health test.
8. More.Quoted 7 lines
As virtio evolves beyond the para-virt/sw-emulated world, it's mandatory for the specification to become flexible and allow a wider feature set. The corrent ctrl virtq that is defined for some of the virtio devices is device specific and wasn't designed to be a generic virtq for admininistration.
Signed-off-by Max Gurtovoy <[email protected]>Lots of things on this list seem to make sense when done from PF and affecting a VF.
I think from this POV the generic structure should include a way to address one device from another.Quoted 120 lines
---
admin-virtq.tex | 241
++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 4 +2 files changed, 245 insertions(+)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,241 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +2
-0+ +Admin virtqueues are used to send administrative commands tomanipulate
admin-virtq.tex new file +42
-0+various features of the device which would not easily map into the +configuration space. + +Use of Admin virtqueues is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u8 class; + u8 command; + u8 command-specific-data[]; + + /* Device-writable part */ + u8 command-specific-result[]; + u8 status_type : 4; + u8 reserved : 4; + u8 status; +}; + +/* Status type values */ +#define VIRTIO_ADMIN_STATUS_TYPE_GENERIC 0 +#define VIRTIO_ADMIN_STATUS_TYPE_CLASS_SPECIFIC 1 +#define VIRTIO_ADMIN_STATUS_TYPE_COMMAND_SPECIFIC 2 +#define VIRTIO_ADMIN_STATUS_TYPE_TRANSPORT_SPECIFIC 3 +#define VIRTIO_ADMIN_STATUS_TYPE_DEVICE_SPECIFIC 4 +#define VIRTIO_ADMIN_STATUS_TYPE_VENDOR_SPECIFIC 5 + +/* Generic status values */ +#define VIRTIO_ADMIN_STATUS_GENERIC_OK 0 +#define VIRTIO_ADMIN_STATUS_GENERIC_ERR 1 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_CLASS 2 +#define VIRTIO_ADMIN_STATUS_GENERIC_INVALID_COMMAND 3 +#define VIRTIO_ADMIN_STATUS_GENERIC_DATA_TRANSFER_ERR 4 +#define VIRTIO_ADMIN_STATUS_GENERIC_DEVICE_INTERNAL_ERR 5 +\end{lstlisting} + +The \field{class}, \field{command} and\field{command-specific-data} are
admin-virtq.tex new file +4
-0+set by the driver, and the device sets the \field{status_type}, the +\field{status} and the \field{command-specific-result}, if needed. + +The virtio Admin command class codes are divided in the followingform:
admin-virtq.tex new file +20
-0+ +\begin{lstlisting} +/* class values that are transport, device and vendor independent */ +#define VIRTIO_ADMIN_COMMON_CLASS_START 0 +#define VIRTIO_ADMIN_COMMON_CLASS_END 63 + +/* class values that are transport specific */ +#define VIRTIO_ADMIN_TRANSPORT_CLASS_START 64 +#define VIRTIO_ADMIN_TRANSPORT_CLASS_END 127 + +/* class values that are device specific */ +#define VIRTIO_ADMIN_DEVICE_CLASS_START 128 +#define VIRTIO_ADMIN_DEVICE_CLASS_END 191 + +/* class values that are vendor specific */ +#define VIRTIO_ADMIN_VENDOR_CLASS_START 192 +#define VIRTIO_ADMIN_VENDOR_CLASS_END 255 +\end{lstlisting} + +\subsection{Admin command set}\label{sec:Basic Facilities of aVirtio Device / Admin Virtqueues / Admin command set}
admin-virtq.tex new file +6
-0+ +Each virtio device that advertise VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also +one or more optional admin commands. + +\subsubsection{Common command set}\label{sec:Basic Facilities of aVirtio Device / Admin Virtqueues / Admin command set / Common command set}
admin-virtq.tex new file +19
-0+ +The Common command set is a group of classes and commands within each +of these classes which are transport, device and vendor independent. +A mandatory class is a class that has at least one mandatory command. +The Common command set is summarized in following table: + +\begin{tabular}{|l|l|l|} +\hline +Class & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_CLASS_COMMANDS & M \\ +\hline +2-63 & reserved & - \\ +\hline +\end{tabular} + +\paragraph{Discover device class}\label{sec:Basic Facilities of aVirtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class}
admin-virtq.tex new file +6
-0+ +This class (opcode: 0) of commands is used to query generic device +information. The following table describes the commands supported for +this class: + +\begin{tabular}{|l|l|l|}admin-virtq.tex line 110
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_DISCOVER_DEVICE_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_DISCOVER_DEVICE_SUPPORTED_CLASSES & M \\ +\hline +2-255 & reserved & - \\ +\hline +\end{tabular}So there are several problems with this approach. One is that there is no two way negotiation.
No way for device to know what will driver use in the end. This breaks things like e.g. accelerating some configurations but not others.Another is that everything is going through the admin vq. Hard for hypervisor to intervene and change just some aspects of the device.
Quoted 6 lines
Quoted 3 lines
This is also a problem for validation, tricky
to find out during probe what does device support and whether driver can work with it.Yes, so I think it makes sense to make the admin virtqueue as a transport and then build other stuffs on top.
what transport exactly ?
it should be a virtq for devices to open.
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 5 lines
Can't we map this stuff into memory instead?
Maybe have an admin config structure
that is separate from regular config?
Only issue we can't then make it too big.
But so far I don't see a lot of info used, most is reserved.It should work (especially for PCI).
you're trying to invent some mechanism that is not flexible and non trivial.
You have already a virtq - so lets use it.
Quoted 184 lines
Thanks
Quoted 179 lines
Quoted 139 lines
+
+\subparagraph{Device identity command}\label{sec:Basic Facilities
of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device identity command}admin-virtq.tex new file +17
-0+ +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO DEVICE ID & M \\ +\hline +05:04 & VIRTIO TRANSPORT ID & M \\ +\hline +4095:06 & reserved & - \\ + \hline +\end{tabular} + +\subparagraph{Device supported classes command}\label{sec:BasicFacilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class / Device supported classes command}
admin-virtq.tex new file +25
-0+ +This mandatory command should return device identity in the following +structure: + +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_CLASS_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_CLASS_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_CLASS_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_CLASS_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a device supports a certain class code. The +following 3072B are reserved. each of the non-reserved 4B sectionswill
admin-virtq.tex new file +13
-0+follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\paragraph{Discover device class commands class}\label{sec:BasicFacilities of a Virtio Device / Admin Virtqueues / Admin command set / Common command set / Discover device class commands class}
admin-virtq.tex new file +3
-0+ +This class (opcode: 1) of commands is used to query supported commands +for each supported device class.The following table describes the commands
admin-virtq.tex new file +19
-0+supported for this class: + +\begin{tabular}{|l|l|l|} +\hline +Command & Description & M/O \\ +\hline \hline +0 & VIRTIO_ADMIN_CLASS_0_COMMANDS_IDENTITY & M \\ +\hline +1 & VIRTIO_ADMIN_CLASS_1_COMMANDS_IDENTITY & M \\ +\hline +2 & VIRTIO_ADMIN_CLASS_2_COMMANDS_IDENTITY & M \\ +\hline +... & ... & M \\ +\hline +255 & VIRTIO_ADMIN_CLASS_255_COMMANDS_IDENTITY & M \\ +\hline +\end{tabular} + +Each command in this class, will return class identity in thefollowing structure:
admin-virtq.tex new file +22
-0+ +\begin{tabular}{|l|l|l|} +\hline +Bytes & Description & M/O \\ +\hline \hline +03:00 & VIRTIO_ADMIN_COMMAND_0_PROPERTIES & M \\ +\hline +07:04 & VIRTIO_ADMIN_COMMAND_1_PROPERTIES & M \\ + \hline +11:08 & VIRTIO_ADMIN_COMMAND_2_PROPERTIES & M \\ +\hline +... & ... & M \\ +\hline +1023:1020 & VIRTIO_ADMIN_COMMAND_255_PROPERTIES & M \\ +\hline +4095:1024 & reserved & - \\ +\hline +\end{tabular} + +The above structure is divided to 256 sections of 4B each, that will +describe whether a class supports a certain command code. The +following 3072B are reserved. each of the non-reserved 4B sectionswill
admin-virtq.tex new file +13
-0+follow the following data structure: + +\begin{tabular}{|l|l|l|} +\hline +Bits & Description & M/O \\ +\hline \hline +0 & SUPPORTED (1: yes, 0:no) & M \\ +\hline +31:01 & reserved & - \\ + \hline +\end{tabular} + +\subsubsection{Transport command set}\label{sec:Basic Facilities ofa Virtio Device / Admin Virtqueues / Admin command set / Transport command set}
admin-virtq.tex new file +6
-0+ +The Transport command set is a group of classes and commands within +each of these classes which are transport specific. Refer to each +transport section for more information on the supported commands. + +\subsubsection{Device command set}\label{sec:Basic Facilities of aVirtio Device / Admin Virtqueues / Admin command set / Device command set}
admin-virtq.tex new file +6
-0+ +The Device command set is a group of classes and commands within +each of these classes which are device specific. Refer to each +device section for more information on the supported commands. + +\subsubsection{Vendor specific command set}\label{sec:BasicFacilities of a Virtio Device / Admin Virtqueues / Admin command set / Vendor specific command set}
admin-virtq.tex line 241
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +The Vendor specific command set is a group of classes and commands +within each of these classes which are vendor specific. Refer to +each vendor specification for more information on the supported +commands.Here's another example.
It's important that there is a way to make a device completely generic without vendor specific expensions.
Would be hard to do here.That's a generic comment.
but specifically I am very reluctant to add vendor specific stuff like this directly in the spec. We already have
VIRTIO_PCI_CAP_VENDOR_CFGand if that is not sufficient I would like to know why before we add more vendor specific stuff.Quoted 22 lines
content.tex +0
-0@@ -407,6 +407,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4
UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
+\input{admin-virtq.tex}
+
\chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on thecontent.tex +0
-0@@ -6671,6 +6673,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}data to be provided in driver notification and the delivery method is
transport specific.
For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}. + \item[VIRTIO_F_ADMIN_VQ(40)] This feature indicates that
+ the device supports administration virtqueue negotiation.
\end{description}
--
2.21.0
Quoted 2 lines · Jason Wang
Yes, so I think it makes sense to make the admin virtqueue as a transport and then build other stuffs on top.
As an alternative to pci/mmio etc? We'd need to better define what we are trying to achieve here ...
Signature · 1 line
MST
Quoted 6 lines · Michael S. Tsirkin
Quoted 2 lines · Jason Wang
Yes, so I think it makes sense to make the admin virtqueue as a transport and then build other stuffs on top.
As an alternative to pci/mmio etc? We'd need to better define what we are trying to achieve here ...
I've posted an RFC for this.
The motivation is to slice the device at the virtio level (transport independent).
But it could be extended for other usage.
Thanks