VIRTIO: Provision maximum MSI-X vectors for a VF
v3 cover VIRTIO: Provision maximum MSI-X vectors for a VF
Hi,
In a PCI SR-IOV configuration, MSI-X vectors of the device is precious device resource. Hence making efficient use of it based on the use case that aligns to the VM configuration is desired for best system performance.
For example, today's static assignment of the amount of MSI-X vectors doesn't allow sophisticated utilization of resources.
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. Each VM might have a different purpose and different amount of resources accordingly (e.g. number of CPUs). A common driver usage of device's MSI-X vectors is proportional to the number of CPUs in the VM. Since the system administrator might know the amount of CPUs in the requested VM, he can also configure the VF's MSI-X vectors amount proportional to the number of CPUs in the VM. In this way, the utilization of the physical hardware will be improved.
Today we have some operating systems that support provisioning MSI-X vectors for PCI VFs.
Update the specification to have a method to change the number of MSI-X vectors supported by a VF using the PF admin virtqueue interface. For that, create a generic infrastructure for managing PCI resources of the managed VF by its parent PF.
The admin virtqueue interface will be extended in the future with more and more features that some of them already in discussions. Some of these features don't fit to MMIO/config_space characteristics, therefore a queue is selected to address admin commands.
Motivation for choosing admin queue:
1. It is anticipated that admin queue will be used for managing and configuring many different type of resources. For example,
a. PCI PF configuring PCI VF attributes in [1].
b. virtio device creating/destroying/configuring subfunctions discussed in [2] c. composing device config space of VF or SF such as mac address, number of VQs, virtio features
Mapping all of them as configuration registers to MMIO will require large MMIO space, if done for each VF/SF. Such MMIO implementation in physical devices such as PCI PF and VF requires on-chip resources to complete within MMIO access latencies. Such resources are very expensive.
2. Such limitation can be overcome by having smaller MMIO register set to build a command request response interface. However, such MMIO based command interface will be limited to serve single outstanding command execution. Such limitation can resulting in high device creation and composing time which can affect VM startup time. Often device can queue and service multiple commands in parallel, such command interface cannot use parallelism offered by the device.
3. When a command wants to DMA data from one or more physical addresses, for example in the future a live migration command may need to fetch device state consist of config space, tens of VQs state, VLAN and MAC table, per VQ partial outstanding block IO list database and more. Packing one or more DMA addresses over new command interface will be burden some and continue to suffer single outstanding command execution latencies. Such limitation is not good for time sensitive live migration use cases.
4. A virtio queue overcomes all the above limitations. It also supports DMA and multiple outstanding descriptors. Similar mechanism exist today for device specific configuration - the control VQ.
[1] This proposal
[2] https://lists.oasis-open.org/archives/virtio-comment/202108/msg00025.html
Patch (1/4) introduce the admin virtqueue concept and feature bit. Patch (2/4) introduce the miscellaneous configuration structure for PCI. Patch (3/4) introduce the device management concept.
Patch (4/4) introduce MSI-X control support.
Open issues:
1. Should we have command to reset all VFs vectors to zero before enabling SR-IOV? Yes/No. If yes, is this optimization command must in current series? Or it can be added later?
2. CCW and MMIO specification for admin_queue_index register
---
changes from V2:
1. Addressed below comments from Jason:
- split admin virtqueue section from admin command set section
- added device management section in "Basic Facilities of a Virtio Device"
- mention that management interface may be admin virtqueue (but not a must)
- add MSI-X configuration sequence
2. Addressed below comments from MST:
- added conformance sections
- use decimal instead of hex for bit numbering
- remove the OOO definitions for AQ
- explain more about default configuration of MSI-X for VFs
3. Addressed below comments from Cornelia:
- reword AQ index definitions
- introduce misc config for PCI transport
4. added vfs_assigned_msix_count attr to VIRTIO_ADMIN_PCI_SRIOV_ATTRS cmd
5. removed the net/blk AQ index definitions (not needed in the new model)
6. rebased on top of master branch commit 88895f56e642aca ("Reserve more feature bits for device type usage")
changes from V1:
1. Addressed below comments from MST:
- updated cover letter for admin queue motivation
- removed VIRTIO_F_ADMIN_PCI_VIRT_MANAGER
- simplified admin queue interface by removing VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER feature bits
- added a subsection for VF MSI-X control capability in PCI section
- re-designed interrupt vector management admin commands
- added a mandatory admin command to expose admin capabilities
- improved commit messages
- described error code for unsupported command
- described error code for errors on invalid VF
- described system software requirements for supporting MSI-X configuration
2. Addressed comments from Parav
- extended command opcode to 16-bit
- improve commit messages
3. Added more command specific error codes
---
Max Gurtovoy (4):
Add virtio Admin virtqueue
Add miscellaneous configuration structure for PCI Add device management facility
Add support for MSI-X vectors configuration for PCI VFs
admin.tex | 252 ++++++++++++++++++++++++++++++++++++++++++++++++ conformance.tex | 4 + content.tex | 119 ++++++++++++++++++++++- 3 files changed, 373 insertions(+), 2 deletions(-) create mode 100644 admin.tex
Signature · 1 line
2.21.0
PATCH v3 1/4 Add virtio Admin virtqueue
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue interface. Admin virtqueue is proposed as one of the interfaces to issue admin commands that have the same command format for all type of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
--- admin.tex | 94 +++++++++++++++++++++++++++++++++++++++++++++++++ conformance.tex | 1 + content.tex | 8 +++-- 3 files changed, 101 insertions(+), 2 deletions(-) create mode 100644 admin.tex
admin.tex new file +94 -0
@@ -0,0 +1,94 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueue is one of the management interface that used to send administrative +commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFs +of a parent PCI PF device are grouped together. These devices can be optionally +managed by its parent PCI PF using its admin virtqueue.). + +An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is +negotiated. The index of the admin virtqueue exposed by the device in a +transport specific manner. + +When VIRTIO_F_ADMIN_VQ is negotiated with the device, driver will send all admin commands +through the admin virtqueue. + +\section{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admin command set} + +The Admin command set defines the commands that can be issued to a well defined management +interface, such as the admin virtqueue. All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported or invalid opcode \\ +\hline +\end{tabular} + +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driver only if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error} +holds the command specific error. If \field{status} is not set to VIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined. + +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular} + +\devicenormative{\subsection}{Admin command set}{Basic Facilities of a Virtio Device / Admin command set} +A device that advertises VIRTIO_F_ADMIN_VQ capability MUST support all the mandatory admin commands. + +A device that advertises VIRTIO_F_ADMIN_VQ capability MAY support one or more optional admin commands. + + +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN CAPS IDENTIFY command} + +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about the supported +admin capabilities by the device. This information is of form: +\begin{lstlisting} +struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0 - Bit 7 are reserved + * caps[1] - Bit 0 - Bit 7 are reserved + * caps[2] - Bit 0 - Bit 7 are reserved + * .... + * caps[8191] - Bit 0 - Bit 7 are reserved + */ + u8 caps[8192]; +}; +\end{lstlisting}
conformance.tex +1 -0
@@ -341,6 +341,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions} \item \ref{devicenormative:Reserved Feature Bits} +\item \ref{devicenormative:Basic Facilities of a Virtio Device / Admin command set} \end{itemize} \conformance{\subsection}{PCI Device Conformance}\label{sec:Conformance / Device Conformance / PCI Device Conformance}
content.tex +6 -2
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23, and 50 to 127] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 to 49, and 128 and above] Feature bits reserved for future extensions. +\item[42 to 49, and 128 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
@@ -6847,6 +6849,8 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that an administration virtqueue is supported. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.21.0
[virtio-dev] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
admin.tex
Quoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue interface. Admin virtqueue is proposed as one of the interfaces to issue admin commands that have the same command format for all type of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin.tex | 94 +++++++++++++++++++++++++++++++++++++++++++++++++ conformance.tex | 1 + content.tex | 8 +++-- 3 files changed, 101 insertions(+), 2 deletions(-) create mode 100644 admin.texadmin.tex new file +0
-0@@ -0,0 +1,94 @@
[some wording only, have not yet thought about the rest]
admin.tex line 3
Quoted 3 lines
admin.tex new file +3
-0+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueue is one of the management interface that used to send administrative
"An admin virtqueue is a management interface of a device that can be used..."
admin.tex line 5
Quoted 2 lines
admin.tex new file +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFs
Maybe add
"Which devices are actually considered a group is transport specific."
?
admin.tex line 7
Quoted 2 lines
admin.tex new file +2
-0+of a parent PCI PF device are grouped together. These devices can be optionally +managed by its parent PCI PF using its admin virtqueue.).
I would move the content in the brackets to a separate paragraph. Maybe
"An example of a group is PCI virtual functions (VFs) being grouped together with their parent PCI physical function (PF). These VFs can be optionally managed by their parent PF using its admin virtqueue."
admin.tex line 10
Quoted 3 lines
admin.tex new file +3
-0+ +An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is +negotiated. The index of the admin virtqueue exposed by the device in a
s/exposed/is exposed/
admin.tex line 14
Quoted 4 lines
admin.tex new file +4
-0+transport specific manner. + +When VIRTIO_F_ADMIN_VQ is negotiated with the device, driver will send all admin commands +through the admin virtqueue.
That sounds a bit like the driver might use an alternative interface for the admin commands as well? What about
"If VIRTIO_F_ADMIN_VQ has been negotiated, the driver used the admin virtqueue to send admin commands."
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 41 lines · Cornelia Huck
admin.tex
Quoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue interface. Admin virtqueue is proposed as one of the interfaces to issue admin commands that have the same command format for all type of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin.tex | 94 +++++++++++++++++++++++++++++++++++++++++++++++++ conformance.tex | 1 + content.tex | 8 +++-- 3 files changed, 101 insertions(+), 2 deletions(-)create mode 100644 admin.tex
admin.tex new file +0
-0@@ -0,0 +1,94 @@[some wording only, have not yet thought about the rest]
Quoted 4 lines
admin.tex line 3
Quoted 3 lines
admin.tex new file +3
-0+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueue is one of the management interface that used to send administrative"An admin virtqueue is a management interface of a device that can be used..."
OK thanks.
Quoted 8 lines
admin.tex line 5
Quoted 3 lines
admin.tex new file +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFsMaybe add
"Which devices are actually considered a group is transport specific."
?
Not sure we want to restrict ourselves for that.
Quoted 8 lines
admin.tex line 7
Quoted 3 lines
admin.tex new file +2
-0+of a parent PCI PF device are grouped together. These devices can be optionally +managed by its parent PCI PF using its admin virtqueue.).I would move the content in the brackets to a separate paragraph. Maybe
"An example of a group is PCI virtual functions (VFs) being grouped together with their parent PCI physical function (PF). These VFs can be optionally managed by their parent PF using its admin virtqueue."
Sounds good.
Quoted 5 lines
admin.tex line 10
Quoted 4 lines
admin.tex new file +3
-0+ +An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is +negotiated. The index of the admin virtqueue exposed by the device in as/exposed/is exposed/
thanks.
Quoted 7 lines
admin.tex line 14
Quoted 5 lines
admin.tex new file +4
-0+transport specific manner. + +When VIRTIO_F_ADMIN_VQ is negotiated with the device, driver will send all admin commands +through the admin virtqueue.That sounds a bit like the driver might use an alternative interface for the admin commands as well? What about
Yes if there will be an alternative for AQ and this feature bit will not be negotiated so the driver will use a different channel.
This was explicitly discussed in previous versions.
What is the issue with this assumption ?
Quoted 3 lines
"If
VIRTIO_F_ADMIN_VQhas been negotiated, the driver used the admin virtqueue to send admin commands."
maybe:
"If VIRTIO_F_ADMIN_VQ has been negotiated, the driver will use the admin virtqueue to send all admin commands."
Quoted 1 line
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 55 lines · Max Gurtovoy
Quoted 41 lines · Cornelia Huck
admin.tex
Quoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue interface. Admin virtqueue is proposed as one of the interfaces to issue admin commands that have the same command format for all type of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin.tex | 94 +++++++++++++++++++++++++++++++++++++++++++++++++ conformance.tex | 1 + content.tex | 8 +++-- 3 files changed, 101 insertions(+), 2 deletions(-)create mode 100644 admin.tex
admin.tex new file +0
-0@@ -0,0 +1,94 @@[some wording only, have not yet thought about the rest]
Quoted 4 lines
admin.tex line 3
Quoted 3 lines
admin.tex new file +3
-0+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueue is one of the management interface that used to send administrative"An admin virtqueue is a management interface of a device that can be used..."
OK thanks.
Quoted 5 lines
admin.tex line 5
Quoted 3 lines
admin.tex new file +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFsMaybe add
Quoted 5 lines
Quoted 3 lines
"Which devices are actually considered a group is transport specific."
?
Not sure we want to restrict ourselves for that.
do restrict this please, if we want to extend the scope we can always do that down the road.
Quoted 26 lines
Quoted 8 lines
admin.tex line 7
Quoted 3 lines
admin.tex new file +2
-0+of a parent PCI PF device are grouped together. These devices can be optionally +managed by its parent PCI PF using its admin virtqueue.).I would move the content in the brackets to a separate paragraph. Maybe
"An example of a group is PCI virtual functions (VFs) being grouped together with their parent PCI physical function (PF). These VFs can be optionally managed by their parent PF using its admin virtqueue."
Sounds good.
Quoted 5 lines
admin.tex line 10
Quoted 4 lines
admin.tex new file +3
-0+ +An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is +negotiated. The index of the admin virtqueue exposed by the device in as/exposed/is exposed/
thanks.
Quoted 7 lines
admin.tex line 14
Quoted 5 lines
admin.tex new file +4
-0+transport specific manner. + +When VIRTIO_F_ADMIN_VQ is negotiated with the device, driver will send all admin commands +through the admin virtqueue.That sounds a bit like the driver might use an alternative interface for the admin commands as well? What about
Quoted 6 lines
Yes if there will be an alternative for AQ and this feature bit will not be negotiated so the driver will use a different channel.
This was explicitly discussed in previous versions.
What is the issue with this assumption ?
it's not an issue down the road but we want to be clear that right now that is the only way, to make sure reader does not waste time looking for more ways in the spec. maybe just say so.
Quoted 12 lines
Quoted 3 lines
"If
VIRTIO_F_ADMIN_VQhas been negotiated, the driver used the admin virtqueue to send admin commands."maybe:
"If
VIRTIO_F_ADMIN_VQhas been negotiated, the driver will use the admin virtqueue to send
all admin commands."Quoted 1 line
[virtio-dev] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Max Gurtovoy
Quoted 6 lines · Cornelia Huck
admin.tex line 5
Quoted 2 lines · Max Gurtovoy
admin.tex new file +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFsMaybe add
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
"Which devices are actually considered a group is transport specific."
?
Not sure we want to restrict ourselves for that.
do restrict this please, if we want to extend the scope we can always do that down the road.
I'm also not sure how grouping can _not_ be transport specific... the PF/VF example is obviously a pci thing; for ccw, in a non-virtio context, there's sometimes the concept of some subchannels/devices being grouped together with no clear hierarchy, and for mmio, I don't really have an idea how "grouping" might work there.
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
admin.tex line 14
Quoted 2 lines
admin.tex new file +2
-0+When VIRTIO_F_ADMIN_VQ is negotiated with the device, driver will send all admin commands +through the admin virtqueue.That sounds a bit like the driver might use an alternative interface for the admin commands as well? What about
Yes if there will be an alternative for AQ and this feature bit will not be negotiated so the driver will use a different channel.
Quoted 8 lines
Quoted 3 lines
This was explicitly discussed in previous versions.
What is the issue with this assumption ?
it's not an issue down the road but we want to be clear that right now that is the only way, to make sure reader does not waste time looking for more ways in the spec. maybe just say so.
Yes, we should be explicit that admin commands are independent of the conduit they are using, and that currently the only conduit is the admin vq. Someone reading the spec does not know about previous discussions on the mailing list.
Maybe reorder this? First have a section where the admin commands are defined, and then have a section that lists the different channels admin commands can use, where the admin vq is the only one currently supported?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 12 lines · Cornelia Huck
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Max Gurtovoy
Quoted 8 lines · Cornelia Huck
admin.tex line 5
Quoted 2 lines · Max Gurtovoy
admin.tex new file +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFsMaybe add
"Which devices are actually considered a group is transport specific."
?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Not sure we want to restrict ourselves for that.
do restrict this please, if we want to extend the scope we can always do that down the road.
I'm also not sure how grouping can _not_ be transport specific... the PF/VF example is obviously a pci thing; for ccw, in a non-virtio context, there's sometimes the concept of some subchannels/devices being grouped together with no clear hierarchy, and for mmio, I don't really have an idea how "grouping" might work there.
Yes today it's transport specific.
But if one day there will be a definition for virtio fabric (over TCP/RDMA) it might not be true.
Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
admin.tex line 14
Quoted 2 lines
admin.tex new file +2
-0+When VIRTIO_F_ADMIN_VQ is negotiated with the device, driver will send all admin commands +through the admin virtqueue.That sounds a bit like the driver might use an alternative interface for the admin commands as well? What about
Yes if there will be an alternative for AQ and this feature bit will not be negotiated so the driver will use a different channel.
Quoted 7 lines
Quoted 3 lines
This was explicitly discussed in previous versions.
What is the issue with this assumption ?
it's not an issue down the road but we want to be clear that right now that is the only way, to make sure reader does not waste time looking for more ways in the spec. maybe just say so.
Yes, we should be explicit that admin commands are independent of the conduit they are using, and that currently the only conduit is the admin vq. Someone reading the spec does not know about previous discussions on the mailing list.
Quoted 4 lines
Maybe reorder this? First have a section where the admin commands are defined, and then have a section that lists the different channels admin commands can use, where the admin vq is the only one currently supported?
I'll reorder it for V4.
Quoted 1 line
[virtio-comment] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 22 lines · Max Gurtovoy
Quoted 12 lines · Cornelia Huck
Quoted 10 lines · Michael S. Tsirkin
Quoted 9 lines · Max Gurtovoy
Quoted 8 lines · Cornelia Huck
admin.tex line 5
Quoted 2 lines · Max Gurtovoy
admin.tex new file +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFsMaybe add
"Which devices are actually considered a group is transport specific."
?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Not sure we want to restrict ourselves for that.
do restrict this please, if we want to extend the scope we can always do that down the road.
I'm also not sure how grouping can _not_ be transport specific... the PF/VF example is obviously a pci thing; for ccw, in a non-virtio context, there's sometimes the concept of some subchannels/devices being grouped together with no clear hierarchy, and for mmio, I don't really have an idea how "grouping" might work there.
Quoted 4 lines
Yes today it's transport specific.
But if one day there will be a definition for virtio fabric (over TCP/RDMA) it might not be true.
I don't think that is contadictory; we can simply extend the meaning of what "grouping" means when needed.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 24 lines · Cornelia Huck
Quoted 22 lines · Max Gurtovoy
Quoted 20 lines · Cornelia Huck
Quoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Max Gurtovoy
Quoted 5 lines · Cornelia Huck
Review
Quoted 2 lines · Max Gurtovoy
(quoted diff) +2
-0+commands to manipulate various features of the device and/or to manipulate +various features, if possible, of another device within the same group (e.g. PCI VFsMaybe add
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
"Which devices are actually considered a group is transport specific."
?
Not sure we want to restrict ourselves for that.
do restrict this please, if we want to extend the scope we can always do that down the road.
I'm also not sure how grouping can _not_ be transport specific... the PF/VF example is obviously a pci thing; for ccw, in a non-virtio context, there's sometimes the concept of some subchannels/devices being grouped together with no clear hierarchy, and for mmio, I don't really have an idea how "grouping" might work there.
Quoted 7 lines
Quoted 4 lines
Yes today it's transport specific.
But if one day there will be a definition for virtio fabric (over TCP/RDMA) it might not be true.
I don't think that is contadictory; we can simply extend the meaning of what "grouping" means when needed.
Yes but as long it is transport specific I don't see why not call this out.
Signature · 1 line
MST
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Max Gurtovoyadmin.tex line 32
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
admin.tex new file +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}
ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?
The point of all this is to allow making sense of commands and e.g. virtualizing them for nested virt without necessarily knowing all of the detail about the specific command.
Signature · 1 line
MST
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 23 lines · Michael S. Tsirkin
admin.tex line 32
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
admin.tex new file +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and
some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.
Quoted 4 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?
For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld group_id for commands that are targeted to the whole group.
Some commands are referring to a specific device in the group so only a vdev_id is needed.
Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for vdev_id nor group_id.
So I'm sure sure we can improve common virtio_admin_cmd structure to have these attributes since they are not mandatory because of the reasons I've mentioned.
Quoted 4 lines
The point of all this is to allow making sense of commands and e.g. virtualizing them for nested virt without necessarily knowing all of the detail about the specific command.
I don't understand this, sorry.
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 42 lines · Max Gurtovoy
Quoted 23 lines · Michael S. Tsirkin
admin.tex line 32
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
admin.tex new file +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and
some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.Quoted 4 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld
group_idfor commands that are targeted to the whole group.Some commands are referring to a specific device in the group so only a
vdev_idis needed.
Quoted 7 lines
Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for
vdev_idnorgroup_id.So I'm sure sure we can improve common
virtio_admin_cmdstructure to have these attributes since they are not mandatory because of the reasons I've mentioned.
I'm not sure I understand 100%, but try to address in the next revision and we'll discuss.
Quoted 5 lines
Quoted 4 lines
The point of all this is to allow making sense of commands and e.g. virtualizing them for nested virt without necessarily knowing all of the detail about the specific command.
I don't understand this, sorry.
Basically try to move stuff into generic format so it's possible to understand things without knowing detail of the command.
Signature · 1 line
MST
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 45 lines · Michael S. Tsirkin
Quoted 44 lines · Max Gurtovoy
Quoted 23 lines · Michael S. Tsirkin
admin.tex line 32
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
admin.tex new file +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and
some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.Quoted 4 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld
group_idfor commands that are targeted to the whole group.Some commands are referring to a specific device in the group so only a
vdev_idis needed.Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for
vdev_idnorgroup_id.
Quoted 5 lines
Quoted 3 lines
So I'm sure sure we can improve common
virtio_admin_cmdstructure to have these attributes since they are not mandatory because of the reasons I've mentioned.I'm not sure I understand 100%, but try to address in the next revision and we'll discuss.
I meant to say that I'm *not* sure we can improve the common structure...
It was a typo.
And I don't understand why this info can't be in the command_specific_data because of all the reasons I mentioned above.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
The point of all this is to allow making sense of commands and e.g. virtualizing them for nested virt without necessarily knowing all of the detail about the specific command.
I don't understand this, sorry.
Basically try to move stuff into generic format so it's possible to understand things without knowing detail of the command.
But we don't develop a networking protocol here. The management device is not sending packets towards its managed devices, right ?
This is an interface for a specific device that can manage others but also manage itself.
We didn't introduce a notion of broadcasting admin commands for other devices.
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 53 lines · Max Gurtovoy
Quoted 45 lines · Michael S. Tsirkin
Quoted 44 lines · Max Gurtovoy
Quoted 23 lines · Michael S. Tsirkin
admin.tex line 32
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
admin.tex new file +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and
some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.Quoted 4 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld
group_idfor commands that are targeted to the whole group.Some commands are referring to a specific device in the group so only a
vdev_idis needed.Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for
vdev_idnorgroup_id.Quoted 5 lines
Quoted 3 lines
So I'm sure sure we can improve common
virtio_admin_cmdstructure to have these attributes since they are not mandatory because of the reasons I've mentioned.I'm not sure I understand 100%, but try to address in the next revision and we'll discuss.
Quoted 6 lines
I meant to say that I'm *not* sure we can improve the common structure...
It was a typo.
And I don't understand why this info can't be in the
command_specific_databecause of all the reasons I mentioned above.
It can, but as declared admin commands are there to handle groups of VFs, so let's standardize how they refer to groups.
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
The point of all this is to allow making sense of commands and e.g. virtualizing them for nested virt without necessarily knowing all of the detail about the specific command.
I don't understand this, sorry.
Basically try to move stuff into generic format so it's possible to understand things without knowing detail of the command.
Quoted 8 lines
But we don't develop a networking protocol here. The management device is not sending packets towards its managed devices, right ?
This is an interface for a specific device that can manage others but also manage itself.
We didn't introduce a notion of broadcasting admin commands for other devices.
It's all entities communicating by message passing whether you call these "commands", "packets" or whatever.
Signature · 1 line
MST
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 52 lines · Michael S. Tsirkin
Quoted 51 lines · Max Gurtovoy
Quoted 50 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 23 lines · Michael S. Tsirkin
admin.tex line 32
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
admin.tex new file +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and
some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.Quoted 4 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld
group_idfor commands that are targeted to the whole group.Some commands are referring to a specific device in the group so only a
vdev_idis needed.Quoted 7 lines
Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for
vdev_idnorgroup_id.So I'm sure sure we can improve common
virtio_admin_cmdstructure to have these attributes since they are not mandatory because of the reasons I've mentioned.I'm not sure I understand 100%, but try to address in the next revision and we'll discuss.
Quoted 8 lines
Quoted 6 lines
I meant to say that I'm *not* sure we can improve the common structure...
It was a typo.
And I don't understand why this info can't be in the
command_specific_databecause of all the reasons I mentioned above.It can, but as declared admin commands are there to handle groups of VFs, so let's standardize how they refer to groups.
"Admin virtqueue is one of the management interface that used to send administrative commands to manipulate various features of the *device* and/or to manipulate various features, if possible, of *another device* within the same group"
It's not only for groups.
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
The point of all this is to allow making sense of commands and e.g. virtualizing them for nested virt without necessarily knowing all of the detail about the specific command.
I don't understand this, sorry.
Basically try to move stuff into generic format so it's possible to understand things without knowing detail of the command.
But we don't develop a networking protocol here. The management device is not sending packets towards its managed devices, right ?
Quoted 7 lines
Quoted 5 lines
This is an interface for a specific device that can manage others but also manage itself.
We didn't introduce a notion of broadcasting admin commands for other devices.
It's all entities communicating by message passing whether you call these "commands", "packets" or whatever.
We're extending a spec that introduce a driver-device communication via queue.
Your suggestion is an internal fabric with packets, headers, broadcasting commands (what about error flows ? is this fabric reliable ? etc etc..).
I feel like this can be a nice future extension and shouldn't block current basic (and already big enough) proposal.
Quoted 1 line
[virtio-dev] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 59 lines · Max Gurtovoy
Quoted 58 lines · Michael S. Tsirkin
Quoted 57 lines · Max Gurtovoy
Quoted 45 lines · Michael S. Tsirkin
Quoted 44 lines · Max Gurtovoy
Quoted 23 lines · Michael S. Tsirkin
Review
Quoted 12 lines · Max Gurtovoy
+\begin{lstlisting}
(quoted diff) +11
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting}ok this abstraction is an improvement, thanks!
What I'd like to see is moving a bit more format to this generic structure.
From what I could gather, some commands affect a group as a whole, and
some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.Quoted 4 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld
group_idfor commands that are targeted to the whole group.Some commands are referring to a specific device in the group so only a
vdev_idis needed.Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for
vdev_idnorgroup_id.Quoted 5 lines
Quoted 3 lines
So I'm sure sure we can improve common
virtio_admin_cmdstructure to have these attributes since they are not mandatory because of the reasons I've mentioned.I'm not sure I understand 100%, but try to address in the next revision and we'll discuss.
I meant to say that I'm *not* sure we can improve the common structure...
It was a typo.
And I don't understand why this info can't be in the
command_specific_databecause of all the reasons I mentioned above.
Quoted 8 lines
Quoted 2 lines
It can, but as declared admin commands are there to handle groups of VFs, so let's standardize how they refer to groups.
"Admin virtqueue is one of the management interface that used to send administrative commands to manipulate various features of the *device* and/or to manipulate various features, if possible, of *another device* within the same group"
It's not only for groups.
What I don't understand: Why can't we simply add target information to the common definition? If a certain command doesn't need that target information, it is simply ignored.
The benefit of reserving a field for target information and specifying how groups and devices are supposed to be addressed is that not every command will have to roll their own.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 70 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 6:39 PMQuoted 18 lines · Max Gurtovoy
Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Max Gurtovoy
Quoted 15 lines · Michael S. Tsirkin
Quoted 14 lines · Max Gurtovoy
Quoted 13 lines · Michael S. Tsirkin
Review
Quoted 10 lines · Max Gurtovoy
+\begin{lstlisting}
(quoted diff) +9
-0+struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; }; \end{lstlisting}ok this abstraction is an improvement, thanks!
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
What I'd like to see is moving a bit more format to this generic
structure.
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 27 lines
Quoted 27 lines
Quoted 11 lines
From what I could gather, some commands affect a group as a
whole, and some commands just a single member of the group. We could have a "destination" field for that, and a special "all of the group" destination for commands affecting the whole group.Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?For now we have only a PCI group that composed of VFs and the PF.
What you suggest, IMO is a definition of a generic virtio group/subsystem that I've mentioned in the discussion of V1.
Once we have virtio group - it should have a group id and them the admin command can have a field calld
group_idfor commands that are targeted to the whole group.Some commands are referring to a specific device in the group so only a
vdev_idis needed.Some commands are even targeted to the same device to query some info (we have examples in this series for that), so in this case there is no need for
vdev_idnorgroup_id.Quoted 5 lines
Quoted 3 lines
So I'm sure sure we can improve common
virtio_admin_cmdstructure to have these attributes since they are not mandatory because of the reasons I've mentioned.I'm not sure I understand 100%, but try to address in the next revision and we'll discuss.
I meant to say that I'm *not* sure we can improve the common structure...
It was a typo.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
And I don't understand why this info can't be in the
command_specific_databecause of all the reasons I mentioned above.It can, but as declared admin commands are there to handle groups of VFs, so let's standardize how they refer to groups.
"Admin virtqueue is one of the management interface that used to send administrative commands to manipulate various features of the *device* and/or to manipulate various features, if possible, of *another device* within
Quoted 6 lines
the same group"
Quoted 2 lines
It's not only for groups.
What I don't understand: Why can't we simply add target information to the common definition?
A command in the admin queue is always targeted to the device on which AQ resides. There is no need to add any explicit target field. AQ is just like any other queue.
A command in the AQ is modifying the attributes of a VF.
If you call this VF as _target_ than related AQ commands already contain the VF id.
Quoted 2 lines
If a certain command doesn't need that target
information, it is simply ignored.
There is no point in adding any arbitrary and unknown length of reserved bytes which are supposed to be ignored.
Quoted 4 lines
The benefit of reserving a field for target information and specifying how groups and devices are supposed to be addressed is that not every command will have to roll their own.
There isn't a good proposal available about how explicit grouping to be done for which some arbitrary group id or target id is requested. So I think we should not introduce some unknown fields that driver and devices doesn't know how to use it.
Better to define such new fields when an actual explicit grouping scheme is defined.
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Max GurtovoyQuoted 5 lines · Max Gurtovoy
"Admin virtqueue is one of the management interface that used to send administrative commands to manipulate various features of the *device* and/or to manipulate various features, if possible, of *another device* within the same group"
It's not only for groups.
First this sentence is unclear what does "to manipulate" refer to? What is done to manipulate these things?
I think the vq is to send commands. The commands are to manipulate etc. So pls cut this up and use shorter sentences please.
And I think the vq is useful mostly for groups. But yes, thinkably if admin commands are submitted via e.g. MMIO then they can refer to the same device where they are sent, not to a group. And this is exactly why Cornelia's suggesting a way to differentiate. We should not need a completely separate command just because it's going to the same device and not to another one.
Signature · 1 line
MST
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Michael S. TsirkinQuoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMNext, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level.
How does one differentiate between the two?
Maybe reserve a field for "destination type"?
We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows what that would be. We just cannot reserve some arbitrary bytes for unknown.
You suggested in v2 to reserve 4 bytes for sf_id, and I explained you that 4 bytes may not be enough.
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely different spec.
Whether PF will manage SFs of the VFs or it will be done nested manner by VF etc is a completely different discussion than what is being proposed here. Whether PF will manage the SF is yet another big question. We work with users and they dislike this.
To address it, some OS has a different management interface (not visible to PF) for SF life cycle even though SFs are anchored on a PF.
So SF/iov extension discussion has long way to go for community to first understand the use cases before crafting some extension.
So lets not complicate and mix things here for a blurry definition of scalable iov/sf extension.
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 14 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMNext, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level.
How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows what that would be. We just cannot reserve some arbitrary bytes for unknown.
You suggested in v2 to reserve 4 bytes forsf_id, and I explained you that 4 bytes may not be enough.
Quoted 8 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely different spec.
Whether PF will manage SFs of the VFs or it will be done nested manner by VF etc is a completely different discussion than what is being proposed here. Whether PF will manage the SF is yet another big question. We work with users and they dislike this.
To address it, some OS has a different management interface (not visible to PF) for SF life cycle even though SFs are anchored on a PF.So SF/iov extension discussion has long way to go for community to first understand the use cases before crafting some extension.
So lets not complicate and mix things here for a blurry definition of scalable iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
Signature · 1 line
MST
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 30 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 11 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMNext, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level.
How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 2 lines
We just cannot reserve some arbitrary bytes for unknown.
You suggested in v2 to reserve 4 bytes forsf_id, and I explained you that 4bytes may not be enough.
Quoted 2 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 1 line
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 1 line
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 1 line
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 1 line
Quoted 8 lines
Quoted 1 line
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reserving bytes for different device id type, interrupt type and more. We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iov and its configuration.
[virtio-comment] RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 33 lines · Parav Pandit
Quoted 33 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 11 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMNext, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level.
How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 2 lines
We just cannot reserve some arbitrary bytes for unknown.
You suggested in v2 to reserve 4 bytes forsf_id, and I explained you that 4bytes may not be enough.
Quoted 2 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 1 line
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 1 line
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 1 line
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 2 lines
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
Quoted 8 lines
Quoted 4 lines
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reserving bytes for different device id type, interrupt type and more. We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iov and its configuration.
I have the feeling that we might be overcomplicating this. We have some groups of targets (a device, a group, that more complicated SF thingy), and we want to distinguish between them. That's easy enough to cover via some kind of enum-equivalent (0 == same dev, 1 == target a dev id, 2 == target a group id, 3 == multi-layer target) and some spec how 1 and 2 should look like (as I'd expect them to be common for many different things). The SF thingy can be covered by 3, and we'll probably want to make that one command-specific, as the whole setup does not have enough things we can standardize on.
Does that make sense? This standardizes the simpler setups, and gives enough flexibility for the more complicated ones. If we have another simpler setup in the future, it can become type 4, 5, etc.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 56 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 6:50 PMQuoted 39 lines · Parav Pandit
Quoted 39 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMQuoted 8 lines
Quoted 5 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 3 lines
We just cannot reserve some arbitrary bytes for unknown. You suggested in v2 to reserve 4 bytes for
sf_id, and I explained you that 4bytes may not be enough.
Quoted 3 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 2 lines
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 2 lines
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 2 lines
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 3 lines
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reservingbytes for different device id type, interrupt type and more.
Quoted 2 lines
We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iov
Quoted 8 lines
and its configuration.
I have the feeling that we might be overcomplicating this. We have some groups of targets (a device, a group, that more complicated SF thingy), and we want to distinguish between them. That's easy enough to cover via some kind of enum-equivalent (0 == same dev, 1 == target a dev id, 2 == target a group id, 3 == multi-layer target) and some spec how 1 and 2 should look like (as I'd expect them to be common for many different things).
Do we have a concrete example of a command that can be targeted for same device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
Quoted 3 lines
The SF thingy can be covered
by 3, and we'll probably want to make that one command-specific, as the whole setup does not have enough things we can standardize on.
This comment has underlying assumption that there are nested layers. And that assumption may not be true at all.
At least for iov extension of intel and SF of Mellanox (already open source for 1+ year) doesn't have the nested layers as today as far as I understand.
Quoted 4 lines
Does that make sense? This standardizes the simpler setups, and gives enough flexibility for the more complicated ones. If we have another simpler setup in the future, it can become type 4, 5, etc.
I feel that without a use case we are over complicating the commands by introducing group/target id etc.
So better to first come up with a valid use case and a device that supports it, which needs a group.
Otherwise a target id can be a long string of PCI device = 0000:03:00.0 or a BDF or a VF number or a VF of a different PCI PF or a SF number 32-bit or SF UUID or a VF on a remote DPU system or PCI device on transparent bridge or something else.
Without knowing the grouping and a nonexistence device we shouldn't complicate the commands.
There are enough opcodes (64K) that can define new structure for more complex devices.
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 58 lines · Parav Pandit
Quoted 58 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 6:50 PMQuoted 39 lines · Parav Pandit
Quoted 39 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMQuoted 8 lines
Quoted 5 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 3 lines
We just cannot reserve some arbitrary bytes for unknown. You suggested in v2 to reserve 4 bytes for
sf_id, and I explained you that 4bytes may not be enough.
Quoted 3 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 2 lines
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 2 lines
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 2 lines
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 3 lines
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reservingbytes for different device id type, interrupt type and more.
Quoted 2 lines
We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iovand its configuration.
Quoted 7 lines
Quoted 6 lines
I have the feeling that we might be overcomplicating this. We have some groups of targets (a device, a group, that more complicated SF thingy), and we want to distinguish between them. That's easy enough to cover via some kind of enum-equivalent (0 == same dev, 1 == target a dev id, 2 == target a group id, 3 == multi-layer target) and some spec how 1 and 2 should look like (as I'd expect them to be common for many different things).
Do we have a concrete example of a command that can be targeted for same device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
Quoted 6 lines
Quoted 3 lines
The SF thingy can be covered
by 3, and we'll probably want to make that one command-specific, as the whole setup does not have enough things we can standardize on.This comment has underlying assumption that there are nested layers. And that assumption may not be true at all.
At least for iov extension of intel and SF of Mellanox (already open source for 1+ year) doesn't have the nested layers as today as far as I understand.Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Does that make sense? This standardizes the simpler setups, and gives enough flexibility for the more complicated ones. If we have another simpler setup in the future, it can become type 4, 5, etc.
I feel that without a use case we are over complicating the commands by introducing group/target id etc.
So better to first come up with a valid use case and a device that supports it, which needs a group.
Otherwise a target id can be a long string of PCI device = 0000:03:00.0 or a BDF or a VF number or a VF of a different PCI PF or a SF number 32-bit or SF UUID or a VF on a remote DPU system or PCI device on transparent bridge or something else.
Well PASID is IIRC just 20 bit on express. I find it unlikely that we'll need more than 64 bit. Yes, it's hard to predict the future but just doing 16 bit here seems frankly like a premature optimization. UUID for a transient thing such as SF just seems unnecessary. 32 or 64 bit seem both acceptable.
Quoted 3 lines
Without knowing the grouping and a nonexistence device we shouldn't complicate the commands.
There are enough opcodes (64K) that can define new structure for more complex devices.
I think you are asking for a bit much frankly, it's up to you to build the interface. Just like with code, if the design does not feel robust the bar is much higher even if one can not prove there's a locking problem. Same here, this interface design does not yet feel very robust yet - so either we build it in a way that seems robust based basically on our gut feeling, or actually spend time predicting and addressing future use-cases to prove they can be addressed. I dare say we've developed some intuition about what makes an extensible interface and where things are likely to go here at the TC, so I wouldn't discard all feedback as unnecessary complication even if it does not always come with concrete use-case examples.
Signature · 1 line
MST
[virtio-comment] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Parav Pandit
Quoted 64 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 6:50 PMQuoted 39 lines · Parav Pandit
Quoted 39 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMQuoted 8 lines
Quoted 5 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 3 lines
We just cannot reserve some arbitrary bytes for unknown. You suggested in v2 to reserve 4 bytes for
sf_id, and I explained you that 4bytes may not be enough.
Quoted 3 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 2 lines
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 2 lines
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 2 lines
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 3 lines
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reservingbytes for different device id type, interrupt type and more.
Quoted 2 lines
We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iovand its configuration.
I have the feeling that we might be overcomplicating this. We have some groups of targets (a device, a group, that more complicated SF thingy), and we want to distinguish between them. That's easy enough to cover via some kind of enum-equivalent (0 == same dev, 1 == target a dev id, 2 == target a group id, 3 == multi-layer target) and some spec how 1 and 2 should look like (as I'd expect them to be common for many different things).
Quoted 4 lines
Quoted 1 line
Do we have a concrete example of a command that can be targeted for same device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
Yes, that would be an example.
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
The SF thingy can be covered
by 3, and we'll probably want to make that one command-specific, as the whole setup does not have enough things we can standardize on.This comment has underlying assumption that there are nested layers. And that assumption may not be true at all.
At least for iov extension of intel and SF of Mellanox (already open source for 1+ year) doesn't have the nested layers as today as far as I understand.Quoted 4 lines
Does that make sense? This standardizes the simpler setups, and gives enough flexibility for the more complicated ones. If we have another simpler setup in the future, it can become type 4, 5, etc.
I feel that without a use case we are over complicating the commands by introducing group/target id etc.
Quoted 8 lines
Quoted 2 lines
So better to first come up with a valid use case and a device that supports it, which needs a group.
Otherwise a target id can be a long string of PCI device = 0000:03:00.0 or a BDF or a VF number or a VF of a different PCI PF or a SF number 32-bit or SF UUID or a VF on a remote DPU system or PCI device on transparent bridge or something else.Well PASID is IIRC just 20 bit on express. I find it unlikely that we'll need more than 64 bit. Yes, it's hard to predict the future but just doing 16 bit here seems frankly like a premature optimization. UUID for a transient thing such as SF just seems unnecessary. 32 or 64 bit seem both acceptable.
However, if we want to be able to accommodate targets we have no idea about yet (and that may have nothing to do with PCI at all), we should maybe standardize on something that can fit a uuid -- if all else fails, you can identify anything with a uuid.
64 bits might be enough in practice, 32 bit seems a bit short.
Quoted 5 lines
Quoted 4 lines
Without knowing the grouping and a nonexistence device we shouldn't complicate the commands.
There are enough opcodes (64K) that can define new structure for more complex devices.
Quoted 11 lines
I think you are asking for a bit much frankly, it's up to you to build the interface. Just like with code, if the design does not feel robust the bar is much higher even if one can not prove there's a locking problem. Same here, this interface design does not yet feel very robust yet - so either we build it in a way that seems robust based basically on our gut feeling, or actually spend time predicting and addressing future use-cases to prove they can be addressed. I dare say we've developed some intuition about what makes an extensible interface and where things are likely to go here at the TC, so I wouldn't discard all feedback as unnecessary complication even if it does not always come with concrete use-case examples.
Indeed. I'm not saying that my ideas are the right way to go, but we definitely need something that (a) covers the use cases we already know about, and (b) can accommodate future use cases with some confidence. Now there's always the chance that we have a requirement down the road that's so odd that our attempts to make this extensible are not enough, but we really need to be able to cover at least things we can imagine. Especially as the use cases in (a) are very specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
(And shoving everything into command-specific data seems too underspecified to me.)
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-dev] RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 6 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 8:29 PMvery specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
Why would one want to execute PCI VF MSI-X configuration command for a non PCI device?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-dev] RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 6 lines · Parav Pandit
Quoted 5 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 8:29 PMvery specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
Why would one want to execute PCI VF MSI-X configuration command for a non PCI device?
Well, that obviously does not make sense! But we also obviously still want to be able to target admin commands in general to non-PCI devices.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Parav PanditQuoted 7 lines · Parav Pandit
Quoted 6 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 8:29 PMvery specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
Why would one want to execute PCI VF MSI-X configuration command for a non PCI device?
E.g. there were proposals to add msi support to virtio-mmio.
Signature · 1 line
MST
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 8:58 PMQuoted 4 lines · Parav Pandit
Quoted 3 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 8:29 PM
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
very specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
Why would one want to execute PCI VF MSI-X configuration command for a
non PCI device?
E.g. there were proposals to add msi support to virtio-mmio.
What prevents mmio specific command say mmio_dev_interrupt_config()? Which fields do overlap with msix config of pci vf?
How is the mmio device identified?
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 10 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 8:58 PMQuoted 6 lines · Parav Pandit
Quoted 6 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 8:29 PMvery specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Why would one want to execute PCI VF MSI-X configuration command for a
non PCI device?
E.g. there were proposals to add msi support to virtio-mmio.
What prevents mmio specific command say
mmio_dev_interrupt_config()? Which fields do overlap with msix config of pci vf?
How is the mmio device identified?
if mmio starts supporting msi-x then # of vectors can reasonably be specified in exactly the same way.
Signature · 1 line
MST
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Cornelia HuckQuoted 91 lines · Cornelia Huck
Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Parav Pandit
Quoted 64 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 6:50 PMQuoted 39 lines · Parav Pandit
Quoted 39 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMQuoted 8 lines
Quoted 5 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 3 lines
We just cannot reserve some arbitrary bytes for unknown. You suggested in v2 to reserve 4 bytes for
sf_id, and I explained you that 4bytes may not be enough.
Quoted 3 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 2 lines
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 2 lines
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 2 lines
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 3 lines
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reservingbytes for different device id type, interrupt type and more.
Quoted 2 lines
We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iovand its configuration.
I have the feeling that we might be overcomplicating this. We have some groups of targets (a device, a group, that more complicated SF thingy), and we want to distinguish between them. That's easy enough to cover via some kind of enum-equivalent (0 == same dev, 1 == target a dev id, 2 == target a group id, 3 == multi-layer target) and some spec how 1 and 2 should look like (as I'd expect them to be common for many different things).
Quoted 4 lines
Quoted 1 line
Do we have a concrete example of a command that can be targeted for same device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
Yes, that would be an example.
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+
Quoted 4 lines
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
I am not sure why do we have both dev id and group id. They are never used together right?
Maybe just have an id length field if we can't agree on how much space to reserve.
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
The SF thingy can be covered
by 3, and we'll probably want to make that one command-specific, as the whole setup does not have enough things we can standardize on.This comment has underlying assumption that there are nested layers. And that assumption may not be true at all.
At least for iov extension of intel and SF of Mellanox (already open source for 1+ year) doesn't have the nested layers as today as far as I understand.Quoted 4 lines
Does that make sense? This standardizes the simpler setups, and gives enough flexibility for the more complicated ones. If we have another simpler setup in the future, it can become type 4, 5, etc.
I feel that without a use case we are over complicating the commands by introducing group/target id etc.
Quoted 8 lines
Quoted 2 lines
So better to first come up with a valid use case and a device that supports it, which needs a group.
Otherwise a target id can be a long string of PCI device = 0000:03:00.0 or a BDF or a VF number or a VF of a different PCI PF or a SF number 32-bit or SF UUID or a VF on a remote DPU system or PCI device on transparent bridge or something else.Well PASID is IIRC just 20 bit on express. I find it unlikely that we'll need more than 64 bit. Yes, it's hard to predict the future but just doing 16 bit here seems frankly like a premature optimization. UUID for a transient thing such as SF just seems unnecessary. 32 or 64 bit seem both acceptable.
Quoted 6 lines
However, if we want to be able to accommodate targets we have no idea about yet (and that may have nothing to do with PCI at all), we should maybe standardize on something that can fit a uuid -- if all else fails, you can identify anything with a uuid.
64 bits might be enough in practice, 32 bit seems a bit short.
Ok, 64 sounds reasonable, not too much.
Quoted 29 lines
Quoted 5 lines
Quoted 4 lines
Without knowing the grouping and a nonexistence device we shouldn't complicate the commands.
There are enough opcodes (64K) that can define new structure for more complex devices.
Quoted 11 lines
I think you are asking for a bit much frankly, it's up to you to build the interface. Just like with code, if the design does not feel robust the bar is much higher even if one can not prove there's a locking problem. Same here, this interface design does not yet feel very robust yet - so either we build it in a way that seems robust based basically on our gut feeling, or actually spend time predicting and addressing future use-cases to prove they can be addressed. I dare say we've developed some intuition about what makes an extensible interface and where things are likely to go here at the TC, so I wouldn't discard all feedback as unnecessary complication even if it does not always come with concrete use-case examples.
Indeed. I'm not saying that my ideas are the right way to go, but we definitely need something that (a) covers the use cases we already know about, and (b) can accommodate future use cases with some confidence. Now there's always the chance that we have a requirement down the road that's so odd that our attempts to make this extensible are not enough, but we really need to be able to cover at least things we can imagine. Especially as the use cases in (a) are very specific (essentially PCI-specific), we need to at least answer the question "how could this work for things that are not PCI?"
(And shoving everything into command-specific data seems too underspecified to me.)
[virtio-comment] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 92 lines · Michael S. Tsirkin
Quoted 91 lines · Cornelia Huck
Quoted 65 lines · Michael S. Tsirkin
Quoted 64 lines · Parav Pandit
Quoted 64 lines
From: Cornelia Huck <[email protected]>
Sent: Tuesday, February 8, 2022 6:50 PMQuoted 39 lines · Parav Pandit
Quoted 39 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 12:13 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, February 7, 2022 4:09 PMQuoted 8 lines
Quoted 5 lines
Next, trying to think about scalable iov extensions. So we will have groups of VFs and then SFs as the next level. How does one differentiate between the two?
Maybe reserve a field for "destination type"?We already discussed this in v2.
SF will have different identification than 16-bits. And no one knows whatthat would be.
Quoted 3 lines
We just cannot reserve some arbitrary bytes for unknown. You suggested in v2 to reserve 4 bytes for
sf_id, and I explained you that 4bytes may not be enough.
Quoted 3 lines
Whether SFs are on top of VFs or SFs are on top of PFs or both is completely
different spec.
Quoted 2 lines
Whether PF will manage SFs of the VFs or it will be done nested manner by
VF etc is a completely different discussion than what is being proposed here.
Quoted 2 lines
Whether PF will manage the SF is yet another big question. We work with
users and they dislike this.
Quoted 2 lines
To address it, some OS has a different management interface (not visible to
PF) for SF life cycle even though SFs are anchored on a PF.
Quoted 3 lines
So SF/iov extension discussion has long way to go for community to first
understand the use cases before crafting some extension.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
So lets not complicate and mix things here for a blurry definition of scalable
iov/sf extension.
Some reserved bytes won't hurt. 2 bytes for type gives us 64k types, sounds like that should be enough.
It doesn't stop there.
Mentioning some destination type, interrupt type, etc also requires reservingbytes for different device id type, interrupt type and more.
Quoted 2 lines
We past this stage long ago after discussing this in v1 at [1].
It is just better and cleaner to define a different structure to describe SF/iovand its configuration.
I have the feeling that we might be overcomplicating this. We have some groups of targets (a device, a group, that more complicated SF thingy), and we want to distinguish between them. That's easy enough to cover via some kind of enum-equivalent (0 == same dev, 1 == target a dev id, 2 == target a group id, 3 == multi-layer target) and some spec how 1 and 2 should look like (as I'd expect them to be common for many different things).
Quoted 4 lines
Quoted 1 line
Do we have a concrete example of a command that can be targeted for same device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
Yes, that would be an example.
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+
Quoted 7 lines
Quoted 4 lines
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
I am not sure why do we have both dev id and group id. They are never used together right?
Right, we can certainly use a single field for both.
Quoted 2 lines
Maybe just have an id length field if we can't agree on how much space to reserve.
If we think that 64 bit should be able to accommodate everything, I'd say just go with that, no need to make it overly complicated.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-dev] RE: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Michael S. TsirkinQuoted 26 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 8:56 PMQuoted 17 lines
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+Quoted 4 lines
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
Quoted 4 lines
I am not sure why do we have both dev id and group id.
They are never used together right?
Maybe just have an id length field if we can't agree on how much space to reserve.
This is what I propose in a previous email.
A device id can be duplicate in different groups.
So to build hierarchy group id will be desired.
So id[] array can contain nested one or multiple fields.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 30 lines · Parav Pandit
Quoted 26 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 8:56 PMQuoted 17 lines
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+Quoted 4 lines
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
Quoted 4 lines
I am not sure why do we have both dev id and group id.
They are never used together right?
Maybe just have an id length field if we can't agree on how much space to reserve.
Quoted 5 lines
This is what I propose in a previous email.
A device id can be duplicate in different groups.
So to build hierarchy group id will be desired.So id[] array can contain nested one or multiple fields.
I guess nesting is for when there's like an SF within a VF?
Signature · 1 line
MST
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 36 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 9:08 PMQuoted 30 lines · Parav Pandit
Quoted 30 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 8:56 PMQuoted 17 lines
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+Quoted 4 lines
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
I am not sure why do we have both dev id and group id. They are never used together right?
Maybe just have an id length field if we can't agree on how much space to reserve.
Quoted 7 lines
Quoted 5 lines
This is what I propose in a previous email.
A device id can be duplicate in different groups.
So to build hierarchy group id will be desired.So id[] array can contain nested one or multiple fields.
I guess nesting is for when there's like an SF within a VF?
That is one case, but I imagine that if SF is on top of VF, than VF will directly manage it. PF should not get involved in nested management.
Nesting is for the case, where there is explicit group. A simple example is, A NIC has two PCI functions, both support SR-IOV and VFs.
But only one PF is managing the VFs for msix, mac etc configuration. In this case there are two groups g0, g1.
grp0 belongs to PF0.
grp1 belongs to PF1.
So a id definition looks like below.
struct pf_identifier {
u8 device; /* pci device of bdf */
u8 function; /* pci function of bdf */
} __attribute__(packed);
struct pci_vf_grouped_id {
struct pf_identifier pf_id;
u16 vf_num;
};struct device_identfier {
u8 id_type; /* below id type enum */
u16 id_len;
union {
u8 raw[0];
struct pci_vf_grouped_id gvf_id;
} id_data;
};enum id_type {
VIRTIO_PCI_VF = 0, /* implicitly grouped VF */
VIRTIO_PCI_REMOTE_VF = 1, /* part of hirechical group */ VIRTIO_SF,
VIRTIO_PCI_REMOTE_SF, /* part of hirechical group */ VIRTIO_MMIO,
/* more */
};
[virtio-comment] Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 40 lines · Parav Pandit
Quoted 40 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 9:08 PMQuoted 34 lines · Parav Pandit
Quoted 26 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 8:56 PMQuoted 17 lines
I might have been a bit too vague about what I had been thinking about. Let's do a sketch (intentionally without concrete sizes):
+-------------------------------------------------------+ | command | +-------------------------------------------------------+ | target type (0 - self, 1 - dev id, 2 - group id, ... | +-------------------------------------------------------+ | dev id | +-------------------------------------------------------+ | group id | +-------------------------------------------------------+ | command-specific data | +-------------------------------------------------------+ | response part | +-------------------------------------------------------+Quoted 4 lines
'dev id' would be valid for 'target type' == 1, 'group id' would be valid for 'target type' == 2. Alternatively, 'dev id' and 'group id' could be a single 'target id' field; if there's nothing better to use, it can simply contain a uuid.
Quoted 4 lines
I am not sure why do we have both dev id and group id. They are never used together right?
Maybe just have an id length field if we can't agree on how much space to reserve.This is what I propose in a previous email.
A device id can be duplicate in different groups.
So to build hierarchy group id will be desired.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
So id[] array can contain nested one or multiple fields.
I guess nesting is for when there's like an SF within a VF?
That is one case, but I imagine that if SF is on top of VF, than VF will directly manage it. PF should not get involved in nested management.
Nesting is for the case, where there is explicit group. A simple example is, A NIC has two PCI functions, both support SR-IOV and VFs.
But only one PF is managing the VFs for msix, mac etc configuration.
Or maybe that's the case of "it hurts when I do this - then don't do that".
Quoted 25 lines
In this case there are two groups g0, g1.
grp0 belongs to PF0.
grp1 belongs to PF1.So a id definition looks like below.
struct
pf_identifier{
u8 device; /* pci device of bdf */
u8 function; /* pci function of bdf */
}__attribute__(packed);struct pci_vf_grouped_id { struct pf_identifier pf_id; u16 vf_num; };struct device_identfier { u8 id_type; /* below id type enum */ u16 id_len; union { u8 raw[0]; struct pci_vf_grouped_id gvf_id; } id_data; };
Quoted 8 lines
enum
id_type{
VIRTIO_PCI_VF= 0, /* implicitly grouped VF */
VIRTIO_PCI_REMOTE_VF= 1, /* part of hirechical group */VIRTIO_SF,
VIRTIO_PCI_REMOTE_SF, /* part of hirechical group */VIRTIO_MMIO,
/* more */
};
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
In reply to Michael S. TsirkinQuoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 7:29 PMQuoted 1 line
Do we have a concrete example of a command that can be targeted for same
device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
A mac address of NIC can be programmed via the existing control VQ for the self. Lets come up with some other example.
Quoted 8 lines
Quoted 1 line
So better to first come up with a valid use case and a device that supports it,
which needs a group.
Quoted 1 line
Otherwise a target id can be a long string of PCI device = 0000:03:00.0 or a
BDF or a VF number or a VF of a different PCI PF or a SF number 32-bit or SF UUID or a VF on a remote DPU system or PCI device on transparent bridge or something else.
Well PASID is IIRC just 20 bit on express.
There are off line discussion and some on the mailing list too (I don't have a link to few months/year old email), That PASID is being overloaded to identify the SF and process both. And I tend to agree to it.
So I won't be surprised if a new SF function id takes different route than PASID. More below.
Quoted 5 lines
I find it unlikely that we'll need more
than 64 bit. Yes, it's hard to predict the future but just doing 16 bit here seems frankly like a premature optimization. UUID for a transient thing such as SF just seems unnecessary. 32 or 64 bit seem both acceptable.
Quoted 7 lines
Quoted 1 line
Without knowing the grouping and a nonexistence device we shouldn't
complicate the commands.
Quoted 2 lines
There are enough opcodes (64K) that can define new structure for more
complex devices.
I think you are asking for a bit much frankly, it's up to you to build the interface.
I likely didn't understand above point.
Quoted 8 lines
Just like with code, if the design does not feel robust the bar is much higher even if one can not prove there's a locking problem.
Same here, this interface
design does not yet feel very robust yet - so either we build it in a way that seems robust based basically on our gut feeling, or actually spend time predicting and addressing future use-cases to prove they can be addressed.
Quoted 4 lines
I dare say we've developed some intuition about what makes an extensible interface and where things are likely to go here at the TC, so I wouldn't discard all feedback as unnecessary complication even if it does not always come with concrete use-case examples.
I do not doubt your intuition.
I am open to feedback, but we haven't really established at least single explicit grouping example and ask is to add some arbitrary reserved bytes for it.
From my DPU experience, over last 3 years, we have built SF and VF device on parent PF-A, and their management interface on parent PF-B. On top of that it is expected to be uniquely indefinable in multi-host env, that brings the notion of multiple controllers by a single management device. And also make it work uniformly with same interface when parent and management device are same.
You can see an extendible interface at [1].
With this base line of [1], I do not agree that defining a u32 device_identifier (to contain 20-bit PASID) will be sufficient in future.
So if we really want to cover variety of cases like [1] and some more complex nested cases, we better define, Device identifier as below,
struct device_identifier {
u8 id_length;
u8 id[]; /* variable length field
};For implicitly grouped VFs of a PF, id can be 2 bytes.
For more advance cases it can be a structure consist one or more combination of (a) host id or controller id (b) PF BDF, (c) sf id (d) PASID and more.
[1] https://www.kernel.org/doc/Documentation/networking/devlink/devlink-port.rst
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 11 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 7:29 PMQuoted 1 line
Do we have a concrete example of a command that can be targeted for same
device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-defined use case.
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
A mac address of NIC can be programmed via the existing control VQ for the self.
Not if it's disabled for the guest.
Quoted 1 line
Lets come up with some other example.
Go ahead.
Quoted 43 lines
Quoted 8 lines
Quoted 1 line
So better to first come up with a valid use case and a device that supports it,
which needs a group.
Quoted 1 line
Otherwise a target id can be a long string of PCI device = 0000:03:00.0 or a
BDF or a VF number or a VF of a different PCI PF or a SF number 32-bit or SF UUID or a VF on a remote DPU system or PCI device on transparent bridge or something else.
Well PASID is IIRC just 20 bit on express.
There are off line discussion and some on the mailing list too (I don't have a link to few months/year old email), That PASID is being overloaded to identify the SF and process both. And I tend to agree to it.
So I won't be surprised if a new SF function id takes different route than PASID. More below.
Quoted 5 lines
I find it unlikely that we'll need more
than 64 bit. Yes, it's hard to predict the future but just doing 16 bit here seems frankly like a premature optimization. UUID for a transient thing such as SF just seems unnecessary. 32 or 64 bit seem both acceptable.Quoted 7 lines
Quoted 1 line
Without knowing the grouping and a nonexistence device we shouldn't
complicate the commands.
Quoted 2 lines
There are enough opcodes (64K) that can define new structure for more
complex devices.
I think you are asking for a bit much frankly, it's up to you to build the interface.
I likely didn't understand above point.
Quoted 8 lines
Just like with code, if the design does not feel robust the bar is much higher even if one can not prove there's a locking problem.
Same here, this interface
design does not yet feel very robust yet - so either we build it in a way that seems robust based basically on our gut feeling, or actually spend time predicting and addressing future use-cases to prove they can be addressed.Quoted 4 lines
I dare say we've developed some intuition about what makes an extensible interface and where things are likely to go here at the TC, so I wouldn't discard all feedback as unnecessary complication even if it does not always come with concrete use-case examples.
I do not doubt your intuition.
I am open to feedback, but we haven't really established at least single explicit grouping example and ask is to add some arbitrary reserved bytes for it.
Quoted 7 lines
Quoted 1 line
From my DPU experience, over last 3 years, we have built SF and VF device on parent PF-A, and their management interface on parent PF-B.
On top of that it is expected to be uniquely indefinable in multi-host env, that brings the notion of multiple controllers by a single management device. And also make it work uniformly with same interface when parent and management device are same.
You can see an extendible interface at [1].With this base line of [1], I do not agree that defining a u32
device_identifier(to contain 20-bit PASID) will be sufficient in future.
OK, and Cornelia also said she thinks 64 is necessary.
Quoted 10 lines
So if we really want to cover variety of cases like [1] and some more complex nested cases, we better define, Device identifier as below,
struct device_identifier { u8 id_length; u8 id[]; /* variable length field };For implicitly grouped VFs of a PF, id can be 2 bytes.
For more advance cases it can be a structure consist one or more combination of (a) host id or controller id (b) PF BDF, (c) sf id (d) PASID and more.[1] https://www.kernel.org/doc/Documentation/networking/devlink/devlink-port.rst
I'm fine with this too.
Signature · 1 line
MST
RE: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 9:10 PMQuoted 8 lines · Parav Pandit
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 7:29 PMQuoted 2 lines
Do we have a concrete example of a command that can be targeted for same
device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-
Quoted 9 lines
defined use case.
Quoted 4 lines
Quoted 3 lines
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
A mac address of NIC can be programmed via the existing control VQ for the
self.
Not if it's disabled for the guest.
Its unrelated.
The idea was to issue same command in same way by two devices = primary and secondary. And in case of primary, it will refer to secondary device.
And in second case secondary tells to self.
So if its disabled by guest, it doesn't matter how guest transports it, either via CVQ or AQ. Its disabled.
Why to re-invent command that exists on CVQ to AQ?
Quoted 10 lines
OK, and Cornelia also said she thinks 64 is necessary.
Quoted 8 lines
So if we really want to cover variety of cases like [1] and some more complex nested cases, we better define, Device identifier as below,
struct device_identifier { u8 id_length; u8 id[]; /* variable length field };For implicitly grouped VFs of a PF, id can be 2 bytes. For more advance cases it can be a structure consist one or more
Quoted 9 lines
combination of (a) host id or controller id (b) PF BDF, (c) sf id (d) PASID and more.
Quoted 4 lines
[1]
https://www.kernel.org/doc/Documentation/networking/devlink/devlink-port.rstI'm fine with this too.
Since we aim for future proofing and flexibility, variable length id is better than constant u64. It covers the u64 case anyway.
Quoted 2 lines
--
MST
Re: [PATCH v3 1/4] Add virtio Admin virtqueue
Quoted 22 lines · Parav Pandit
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 9:10 PMQuoted 8 lines · Parav Pandit
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, February 8, 2022 7:29 PMQuoted 2 lines
Do we have a concrete example of a command that can be targeted for same
device and a target device, which requires differentiating their destination? If so, lets discuss and then it make sense to add for the well-
Quoted 9 lines
defined use case.
Quoted 4 lines
Quoted 3 lines
So e.g. things like controlling NIC's MAC can reasonably be part of the same device.
A mac address of NIC can be programmed via the existing control VQ for the
self.
Not if it's disabled for the guest.
Quoted 8 lines
Its unrelated.
The idea was to issue same command in same way by two devices = primary and secondary. And in case of primary, it will refer to secondary device.
And in second case secondary tells to self.So if its disabled by guest, it doesn't matter how guest transports it, either via CVQ or AQ. Its disabled.
Why to re-invent command that exists on CVQ to AQ?
I can go into it but it's beside the point. I was just trying to help you come up with use-cases. Don't like it - come up with your own ones.
Quoted 23 lines
Quoted 10 lines
OK, and Cornelia also said she thinks 64 is necessary.
Quoted 8 lines
So if we really want to cover variety of cases like [1] and some more complex nested cases, we better define, Device identifier as below,
struct device_identifier { u8 id_length; u8 id[]; /* variable length field };For implicitly grouped VFs of a PF, id can be 2 bytes. For more advance cases it can be a structure consist one or more
Quoted 9 lines
combination of (a) host id or controller id (b) PF BDF, (c) sf id (d) PASID and more.
Quoted 4 lines
[1]
https://www.kernel.org/doc/Documentation/networking/devlink/devlink-port.rstI'm fine with this too.
Since we aim for future proofing and flexibility, variable length id is better than constant u64. It covers the u64 case anyway.
Quoted 2 lines
--
MST
PATCH v3 2/4 Add miscellaneous configuration structure for PCI
In reply to Max GurtovoyThis new structure will be used for adding new miscellaneous registers for a virtio device configuration layout.
For now, only admin_queue_index register is added. Admin virtqueue index does not depend on the device type. Hence, add a PCI capability to read the admin virtqueue index.
--- conformance.tex | 2 ++ content.tex | 25 +++++++++++++++++++++++++ 2 files changed, 27 insertions(+)
conformance.tex +2 -0
@@ -102,6 +102,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability} \item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration} \item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Miscellaneous configuration structure layout} \end{itemize} \conformance{\subsection}{MMIO Driver Conformance}\label{sec:Conformance / Driver Conformance / MMIO Driver Conformance}
@@ -363,6 +364,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration} \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications} \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Miscellaneous configuration structure layout} \end{itemize} \conformance{\subsection}{MMIO Device Conformance}\label{sec:Conformance / Device Conformance / MMIO Device Conformance}
content.tex +25 -0
@@ -712,6 +712,7 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option \item ISR Status \item Device-specific configuration (optional) \item PCI configuration access +\item Miscellaneous configuration \end{itemize} Each structure can be mapped by a Base Address register (BAR) belonging to
@@ -771,6 +772,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option #define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8 /* Vendor-specific data */ #define VIRTIO_PCI_CAP_VENDOR_CFG 9 +/* Miscellaneous configuration */ +#define VIRTIO_PCI_CAP_MISC_CFG 10 \end{lstlisting} Any other value is reserved for future use.
@@ -1352,6 +1355,28 @@ \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport O specified by some other Virtio Structure PCI Capability of type other than \field{VIRTIO_PCI_CAP_PCI_CFG}. +\subsubsection{Miscellaneous configuration structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Miscellaneous configuration structure layout} + +The miscellaneous configuration structure is found at the bar and offset within the VIRTIO_PCI_CAP_MISC_CFG capability. +Its layout is below. +\begin{lstlisting} +struct virtio_pci_misc_cfg { + le16 admin_queue_index; /* read-only for driver */ +}; +\end{lstlisting} + +\begin{description} +\item[\field{admin_queue_index}] + The device uses this to report the index of the admin virtqueue. + This field is valid only if VIRTIO_F_ADMIN_VQ is set. +\end{description} + +\devicenormative{\paragraph}{Miscellaneous configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Miscellaneous configuration structure layout} +The device MUST present a valid \field{admin_queue_index} when VIRTIO_F_ADMIN_VQ is set. + +\drivernormative{\paragraph}{Miscellaneous configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Miscellaneous configuration structure layout} +The driver MUST use the value of \field{admin_queue_index} to configure the admin virtqueue. For more details on virtqueue configuration see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration}. + \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout} Transitional devices MUST present part of configuration
Signature · 1 line
2.21.0
PATCH v3 3/4 Add device management facility
In reply to Max GurtovoyA virtio device may be capable of managing other virtio device features and configuration. These management operations are composed of one or more admin commands from the admin command set. One such management interface for issuing these commands can be the virtio admin virtqueue.
--- content.tex | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
content.tex +14 -0
@@ -451,6 +451,20 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo \input{admin.tex} +\section{Device management}\label{sec:Basic Facilities of a Virtio Device / Device management} + +Virtualized environments might be composed of one or more virtio devices. These devices might be +associated with each other. For example, virtio PCI PF and its VFS are composing one virtio group. +A PCI PF device, by default, is the primary device in the group, and its PCI VFs are the secondary devices in the group. +A sophisticated primary device might have capabilities to manage its secondary devices. This primary +device will be the virtio group manager and the secondary devices would be the managed devices in this group. + +The primary device will use the admin command set to manage secondary devices (see section +\ref{sec:Basic Facilities of a Virtio Device / Admin command set} for more details). The primary device +can use any management interface that is defined in the virtio specification for issuing admin commands. +For example, the admin virtqueue (see section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} for more details) +of the PF can be used to configure some attributes (such as MSI-X vectors) for its VFs in order to improve resource utilization. + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
Signature · 1 line
2.21.0
PATCH v3 4/4 Add support for MSI-X vectors configuration for PCI VFs
In reply to Max GurtovoyA typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a user requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way today in the spec to change the number of MSI-X vectors supported by a VF, although there are some operating systems that support this.
Introduce new admin commands for a generic interrupt vector management for PCI VFs. For now, this mechanism will manage the MSI-X interrupt vectors assignments of a VF by its parent PF.
These admin commands will be easily extended, if needed, for other types of interrupt vectors in the future with backward compatibility to old drivers and devices.
--- admin.tex | 162 +++++++++++++++++++++++++++++++++++++++++++++++- conformance.tex | 1 + content.tex | 72 +++++++++++++++++++++ 3 files changed, 233 insertions(+), 2 deletions(-)
admin.tex +160 -2
@@ -63,17 +63,33 @@ \section{Admin command set}\label{sec:Basic Facilities of a Virtio Device / Admi \hline \hline 0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ \hline -0001h - 7FFFh & Generic admin cmds & - \\ +0001h & VIRTIO_ADMIN_PCI_SRIOV_ATTRS & O \\ +\hline +0002h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET & O \\ +\hline +0003h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET & O \\ +\hline +0004h - 7FFFh & Generic admin cmds & - \\ \hline 8000h - FFFFh & Reserved & - \\ \hline \end{tabular} +\drivernormative{\subsection}{Admin command set}{Basic Facilities of a Virtio Device / Admin command set} +A driver SHOULD NOT issue VIRTIO_ADMIN_PCI_SRIOV_ATTRS command when VIRTIO_F_SR_IOV is not negotiated. + +A driver SHOULD NOT issue VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET +commands when VIRTIO_F_SR_IOV is not negotiated or when PCI Single Root I/O Virtualization is disabled. + \devicenormative{\subsection}{Admin command set}{Basic Facilities of a Virtio Device / Admin command set} A device that advertises VIRTIO_F_ADMIN_VQ capability MUST support all the mandatory admin commands. A device that advertises VIRTIO_F_ADMIN_VQ capability MAY support one or more optional admin commands. +A device MUST fail VIRTIO_ADMIN_PCI_SRIOV_ATTRS command when VIRTIO_F_SR_IOV is not negotiated. + +A device MUST fail VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET +commands when VIRTIO_F_SR_IOV is not negotiated or when PCI Single Root I/O Virtualization is disabled. \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN CAPS IDENTIFY command}
@@ -83,7 +99,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a \begin{lstlisting} struct virtio_admin_caps_identify_result { /* - * caps[0] - Bit 0 - Bit 7 are reserved + * caps[0] - Bit 0 - if set, VF MSI-X control supported + * Bit 1 - Bit 7 are reserved * caps[1] - Bit 0 - Bit 7 are reserved * caps[2] - Bit 0 - Bit 7 are reserved * ....
@@ -92,3 +109,144 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a u8 caps[8192]; }; \end{lstlisting} + +For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}. + +\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN PCI SRIOV ATTRS command} + +The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about PCI SRIOV +related capabilities and attributes for the device. This command can be supported only by +PCI devices that supports Single Root I/O Virtualization. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_sriov_attrs_result { + /* For compatibility - indicates which of the below fields are valid (1 means valid): + * Bit 0 - vfs_total_msix_count + * Bit 1 - vfs_assigned_msix_count + * Bit 2 - per_vf_max_msix_count + * Bits 3 - 63 - reserved for future fields + */ + le64 attrs_mask; + /* Total number of msix vectors for the total number of VFs */ + le32 vfs_total_msix_count; + /* Assigned number of msix vectors for the enabled VFs */ + le32 vfs_assigned_msix_count; + /* Max number of msix vectors that can be assigned for a single VF */ + le16 per_vf_max_msix_count; +}; +\end{lstlisting} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors +count for a PCI virtual function. The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_set_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * modified (1 means that field should be modified): + * Bit 0 - msix_count + * Bits 1 - 63 - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors */ + le16 msix_count; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED & MSI-X count exceed the max value per VF \\ +\hline +02h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_NO_RSC & Device don't have requested MSI-X count \\ +\hline +03h & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE & VF is already in use, operation failed \\ +\hline +04h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED +is returned when the amount of MSI-X to assign exceed the maximum value that can be +assigned to a single VF.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_NO_RSC +is returned when the device doesn't have requested number of MSI-X vectors free.} +\end{note} + +This command has no command specific result set by the device. Upon success, the device guarantees +that all the requested properties were modified to the given values. Otherwise, error will be returned. + +\begin{note} +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and MUST not be used by the driver. +Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.} +\end{note} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin command set / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs +interrupt vectors configuration. +The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * queried (1 means that field should be queried): + * Bit 0 - msix_count (The amount of MSI-X interrupt vectors) + * Bits 1 - 63 - reserved for future fields + */ + le64 attrs_mask; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +This command, upon success, returns a data buffer that describes the properties that were requested +and their values for the subject virtio VF device according to the given vf_number. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_result { + /* For compatibility - indicates which of the below fields were returned + * (1 means that field was returned): + * Bit 0 - msix_count + * Bits 1 - 63 - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors currently assigned to the VF */ + le16 msix_count; +}; +\end{lstlisting}
conformance.tex +1 -0
@@ -87,6 +87,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization} \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup} \item \ref{drivernormative:Reserved Feature Bits} +\item \ref{drivernormative:Basic Facilities of a Virtio Device / Admin command set} \end{itemize} \conformance{\subsection}{PCI Driver Conformance}\label{sec:Conformance / Driver Conformance / PCI Driver Conformance}
content.tex +72 -0
78 lines not on this page · raw message
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/