VIRTIO: Provision maximum MSI-X vectors for a VF
v2 cover VIRTIO: Provision maximum MSI-X vectors for a VF
Hi,
In a PCI SR-IOV configuration, MSI-X vectors of the device is precious device resource. Hence making efficient use of it based on the use case that aligns to the VM configuration is desired for best system performance.
For example, today's static assignment of the amount of MSI-X vectors doesn't allow sophisticated utilization of resources.
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. Each VM might have a different purpose and different amount of resources accordingly (e.g. number of CPUs). A common driver usage of device's MSI-X vectors is proportional to the number of CPUs in the VM. Since the system administrator might know the amount of CPUs in the requested VM, he can also configure the VF's MSI-X vectors amount proportional to the number of CPUs in the VM. In this way, the utilization of the physical hardware will be improved.
Today we have some operating systems that support provisioning MSI-X vectors for PCI VFs.
Update the specification to have a method to change the number of MSI-X vectors supported by a VF using the PF admin virtqueue interface. For that, create a generic infrastructure for managing PCI resources of the managed VF by its parent PF.
The admin virtqueue interface will be extended in the future with more and more features that some of them already in discussions. Some of these features don't fit to MMIO/config_space characteristics, therefore a queue is selected to address admin commands.
Motivation for choosing admin queue:
1. It is anticipated that admin queue will be used for managing and configuring many different type of resources. For example,
a. PCI PF configuring PCI VF attributes in [1].
b. virtio device creating/destroying/configuring subfunctions discussed in [2] c. composing device config space of VF or SF such as mac address, number of VQs, virtio features
Mapping all of them as configuration registers to MMIO will require large MMIO space, if done for each VF/SF. Such MMIO implementation in physical devices such as PCI PF and VF requires on-chip resources to complete within MMIO access latencies. Such resources are very expensive.
2. Such limitation can be overcome by having smaller MMIO register set to build a command request response interface. However, such MMIO based command interface will be limited to serve single outstanding command execution. Such limitation can resulting in high device creation and composing time which can affect VM startup time. Often device can queue and service multiple commands in parallel, such command interface cannot use parallelism offered by the device.
3. When a command wants to DMA data from one or more physical addresses, for example in the future a live migration command may need to fetch device state consist of config space, tens of VQs state, VLAN and MAC table, per VQ partial outstanding block IO list database and more. Packing one or more DMA addresses over new command interface will be burden some and continue to suffer single outstanding command execution latencies. Such limitation is not good for time sensitive live migration use cases.
4. A virtio queue overcomes all the above limitations. It also supports DMA and multiple outstanding descriptors. Similar mechanism exist today for device specific configuration - the control VQ.
[1] This proposal
[2] https://lists.oasis-open.org/archives/virtio-comment/202108/msg00025.html
Patches (1/4) introduce the admin virtqueue concept and feature bit. Patches (2/4)-(3/4) add the admin virtq to virtio-blk and virtio-net devices.
Patch (4/4) introduce MSI-X control support.
---
changes from V1:
1. Addressed below comments from MST:
- updated cover letter for admin queue motivation
- removed VIRTIO_F_ADMIN_PCI_VIRT_MANAGER
- simplified admin queue interface by removing VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER feature bits
- added a subsection for VF MSI-X control capability in PCI section
- re-designed interrupt vector management admin commands
- added a mandatory admin command to expose admin capabilities
- improved commit messages
- described error code for unsupported command
- described error code for errors on invalid VF
- described system software requirements for supporting MSI-X configuration
2. Addressed comments from Parav
- extended command opcode to 16-bit
- improve commit messages
3. Added more command specific error codes
---
Max Gurtovoy (4):
Add virtio Admin Virtqueue
virtio-blk: add support for VIRTIO_F_ADMIN_VQ
virtio-net: add support for VIRTIO_F_ADMIN_VQ Add support for MSI-X vectors configuration for PCI VFs
admin-virtq.tex | 235 ++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 67 +++++++++++++- 2 files changed, 299 insertions(+), 3 deletions(-) create mode 100644 admin-virtq.tex
Signature · 1 line
2.21.0
PATCH v2 1/4 Add virtio Admin Virtqueue
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.
Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-) create mode 100644 admin-virtq.tex
admin-virtq.tex new file +89 -0
@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers +are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued only to the admin +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also one or more +optional admin commands. All commands are of the following form: + +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command; + u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error; + u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported or invalid opcode \\ +\hline +\end{tabular} + +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driver only if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error} +holds the command specific error. If \field{status} is not set to VIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined. + +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular} + +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command} + +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about the supported +admin capabilities by the device. This information is of form: +\begin{lstlisting} +struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +}; +\end{lstlisting}
content.tex +7 -2
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 and above] Feature bits reserved for future extensions. +\item[42 and above] Feature bits reserved for future extensions. \end{description} \begin{note}
@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.21.0
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
admin-virtq.tex line 15
Quoted 2 lines · Max Gurtovoy
admin-virtq.tex new file +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers +are used by the device in out of order manner.
Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again, VIRTIO_F_PARTIAL_ORDER proposal as one, arguably cleaner and more generic way to address this. And if that's not adequate I'd like to address that as part of the PARTIAL_ORDER proposal, this kind of per-queue in order was definitely on the radar as it was formulated.
Signature · 1 line
MST
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 7 lines · Michael S. Tsirkin
admin-virtq.tex line 15
Quoted 2 lines · Max Gurtovoy
admin-virtq.tex new file +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers +are used by the device in out of order manner.Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again,
VIRTIO_F_PARTIAL_ORDERproposal as one, arguably cleaner and more generic way to address this.
We already suggested a special capability for IN_ORDER for AQ and you asked to drop it. We drop it and agreed that AQ will be OOO.
Why are we going back here ?
You also mentioned that this patch set is already big enough so why solve all the problems we can think of in this one ? Why mixing VIRTIO_F_PARTIAL_ORDER here ?
Quoted 4 lines
And if that's not adequate I'd like to address that as part of the
PARTIAL_ORDERproposal, this kind of per-queue in order was definitely on the radar as it was formulated.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 10 lines · Max Gurtovoy
Quoted 7 lines · Michael S. Tsirkin
admin-virtq.tex line 15
Quoted 2 lines · Max Gurtovoy
admin-virtq.tex new file +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers +are used by the device in out of order manner.Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again,
VIRTIO_F_PARTIAL_ORDERproposal as one, arguably cleaner and more generic way to address this.
Quoted 4 lines
We already suggested a special capability for
IN_ORDERfor AQ and you asked to drop it. We drop it and agreed that AQ will be OOO.Why are we going back here ?
That's a misunderstanding. Currently all VQs of a device follow same ordering. So I suggested making AQ just follow ordering of rest of VQs of the device.
Quoted 2 lines
You also mentioned that this patch set is already big enough so why solve all the problems we can think of in this one ?
Right. So just leave the ordering be for now, driver can just skip IN_ORDER and use AQ without it if it wants to.
Quoted 2 lines
Why mixing
VIRTIO_F_PARTIAL_ORDERhere ?
PARTIAL_ORDER has its own patch, no need to mix it in. Reusing that seems cleaner than special-casing here though, we do try to have reusable modules, otherwise things just get out of hand quickly.
Quoted 4 lines
Quoted 4 lines
And if that's not adequate I'd like to address that as part of the
PARTIAL_ORDERproposal, this kind of per-queue in order was definitely on the radar as it was formulated.
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 9 lines · Michael S. Tsirkin
Quoted 8 lines · Max Gurtovoy
Quoted 7 lines · Michael S. Tsirkin
admin-virtq.tex line 15
Quoted 2 lines · Max Gurtovoy
admin-virtq.tex new file +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers +are used by the device in out of order manner.Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again,
VIRTIO_F_PARTIAL_ORDERproposal as one, arguably cleaner and more generic way to address this.
Quoted 7 lines
Quoted 4 lines
We already suggested a special capability for
IN_ORDERfor AQ and you asked to drop it. We drop it and agreed that AQ will be OOO.Why are we going back here ?
That's a misunderstanding. Currently all VQs of a device follow same ordering. So I suggested making AQ just follow ordering of rest of VQs of the device.
Can we say the the AQ is by nature OOO so if AQ is set then IN_ORDER can't be set ?
Quoted 5 lines
Quoted 3 lines
You also mentioned that this patch set is already big enough so why solve all the problems we can think of in this one ?
Right. So just leave the ordering be for now, driver can just skip
IN_ORDERand use AQ without it if it wants to.
Without mentioning this in the spec ? can we control all the drivers in the world ?
Quoted 6 lines
Quoted 3 lines
Why mixing
VIRTIO_F_PARTIAL_ORDERhere ?
PARTIAL_ORDERhas its own patch, no need to mix it in. Reusing that seems cleaner than special-casing here though, we do try to have reusable modules, otherwise things just get out of hand quickly.
re-using modules that haven't yet merged ? This require mixing 2 proposals.
I think best we can do is to say:
"A device MUST NOT propose VIRTIO_F_IN_ORDER if VIRTIO_F_ADMIN_VQ is proposed."
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
And if that's not adequate I'd like to address that as part of the
PARTIAL_ORDERproposal, this kind of per-queue in order was definitely on the radar as it was formulated.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Michael S. TsirkinQuoted 21 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, January 26, 2022 8:34 PMQuoted 9 lines · Max Gurtovoy
Quoted 7 lines · Michael S. Tsirkin
Review
Quoted 2 lines · Max Gurtovoy
(quoted diff) +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue +command buffers are used by the device in out of order manner.Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again,
VIRTIO_F_PARTIAL_ORDERproposal as one, arguably cleaner and moregeneric way to address this.
Quoted 5 lines
We already suggested a special capability for
IN_ORDERfor AQ and you asked to drop it. We drop it and agreed that AQ will be OOO.Why are we going back here ?
Quoted 9 lines
That's a misunderstanding. Currently all VQs of a device follow same ordering. So I suggested making AQ just follow ordering of rest of VQs of the device.
Quoted 2 lines
You also mentioned that this patch set is already big enough so why solve all the problems we can think of in this one ?
Right. So just leave the ordering be for now, driver can just skip
IN_ORDERand use AQ without it if it wants to.
Ok. I read your same suggestion as what I thought now. It answers my previous email.
Thanks.
RE: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Michael S. TsirkinQuoted 5 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, January 26, 2022 8:11 PMOn Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
Quoted 10 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command +buffers are used by the device in out of order manner.Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again,
VIRTIO_F_PARTIAL_ORDERproposal as one, arguably cleaner and more generic way to address this.
And if that's not adequate I'd like to address that as part of thePARTIAL_ORDERproposal, this kind of per-queue in order was definitely on the radar as it was formulated.
As we dropped other less important items from this proposal because it was too big. I am going to keep the PARTIAL_ORDER also out of this one. It falls in same bucket.
So AQ follows same ordering rules as other queues. Are you ok with this in v3?
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 16 lines · Parav Pandit
Quoted 6 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, January 26, 2022 8:11 PMOn Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
Quoted 10 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command +buffers are used by the device in out of order manner.Instead of special-casing AQ I'd like to see a generic capability addressing this need. For example, TX for virtio net might benefit from this too. And I'd like to mention, again,
VIRTIO_F_PARTIAL_ORDERproposal as one, arguably cleaner and more generic way to address this.
And if that's not adequate I'd like to address that as part of thePARTIAL_ORDERproposal, this kind of per-queue in order was definitely on the radar as it was formulated.
Quoted 5 lines
As we dropped other less important items from this proposal because it was too big. I am going to keep the
PARTIAL_ORDERalso out of this one. It falls in same bucket.So AQ follows same ordering rules as other queues. Are you ok with this in v3?
I'm fine with this.
Thanks
Quoted 1 line
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Max GurtovoyQuoted 45 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +10
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} + +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit.
admin-virtq.tex line 20
Quoted 10 lines
admin-virtq.tex new file +10
-0+ +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers +are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued only to the admin +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ feature, MUST +support all the mandatory admin commands. A device MAY support also one or more +optional admin commands. All commands are of the following form:
Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex line 25
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;
Two questions:
1) It looks to me it's better to have a explicit device_id here other than embed it in each command
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
admin-virtq.tex line 30
Quoted 5 lines
admin-virtq.tex new file +5
-0+ u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error;
I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
Quoted 31 lines
admin-virtq.tex new file +31
-0+ u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported or invalid opcode \\ +\hline +\end{tabular} + +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driver only if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error} +holds the command specific error. If \field{status} is not set to VIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined. + +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|}
admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular}
See above, I'd suggest to use class/command as virtio-net cvq.
Quoted 7 lines
admin-virtq.tex new file +7
-0+ +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command} + +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about the supported +admin capabilities by the device. This information is of form: +\begin{lstlisting}
admin-virtq.tex line 88
Quoted 10 lines
admin-virtq.tex new file +10
-0+struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};
Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Thanks
Quoted 28 lines
admin-virtq.tex new file +1
-0+\end{lstlisting}content.tex +4
-2@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B \begin{description} \item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms -\item[41 and above] Feature bits reserved for future extensions. +\item[42 and above] Feature bits reserved for future extensions. \end{description} \begin{note}@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6857
Quoted 9 lines
content.tex +3
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 76 lines · Jason Wang
Quoted 54 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>---
admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++--2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +13
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue commandbuffers
admin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued onlyQuoted 6 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQfeature, MUST
+support all the mandatory admin commands. A device MAY support also one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex line 25
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 1 line
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each command
this was already discussed.
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
I'm not against putting such id but we don't have an exact idea how will SF device_id will look like.
Maybe it will be some 128 bit uuid ? maybe u64 ? how can we guess ?
Quoted 2 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ? are you open for changes or should we copy/paste from net cvq ?
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Quoted 10 lines
Quoted 7 lines
+ u8 command_specific_data[];
+
+ /* Device-writable part */
+ u8 status;
+ u8command_specific_error;I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
To make it generic we need to separate.
Quoted 42 lines
Quoted 42 lines
+ u8 command_specific_result[];
admin-virtq.tex new file +14
-0+}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported orinvalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +7
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driveronly if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +9
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline
Quoted 8 lines
Quoted 5 lines
+0001h - 7FFFh & Generic admin cmds & - \\
+\hline
+8000h - FFFFh & Reserved & - \\
+\hline
+\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
see my comment above.
Quoted 17 lines
Quoted 17 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific dataset by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +7
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting} +struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved
Quoted 8 lines
Quoted 5 lines
+ * ....
+ * caps[8191] - Bit0x0- Bit0x7are reserved + */
+ u8 caps[8192];
+};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
It doesn't scale. I tried it. It doesn't work.
Thanks.
Quoted 46 lines
Thanks
Quoted 26 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilitiesof a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions. +\item[42 and above] Feature bits reserved for future extensions. \end{description}
\begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4
UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
+\input{admin-virtq.tex}
+
\chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}Quoted 7 lines
that the driver can reset a queue individually.
See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
+ the device supports administration virtqueue negotiation.
+
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SW%2FM98QZ7Av2wsv1NoekveDdQmlfWMYcvZQiFvpLcAE%3D&reserved=0
Feedback License: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=C5D9qmFqbNrx7CoTzsx%2B9qalts534XPaImHlcszwm1M%3D&reserved=0
List Guidelines: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bi7OJDuH9CvoKdxtA01dn5SpZwDp3Bna4%2B00bVOuz2o%3D&reserved=0
Committee: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SdGNrxTUZVIe2WJwb%2BdbT2UxagM9AdEzDWXccFvRE3s%3D&reserved=0
Join OASIS: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ynKmp30vKJMw%2FL4IHbBFY9EebT%2BsVX2%2Boz7IIcO%2F1xg%3D&reserved=0
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 87 lines · Max Gurtovoy
Quoted 79 lines · Jason Wang
Quoted 55 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +13
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue commandbuffers
admin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued onlyQuoted 6 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQfeature, MUST
+support all the mandatory admin commands. A device MAY support also one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex line 25
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 2 lines
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each commandthis was already discussed.
Quoted 7 lines
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
I'm not against putting such id but we don't have an exact idea how will SF
device_idwill look like.Maybe it will be some 128 bit uuid ? maybe u64 ?
How do you know u16 is sufficient for command?
Quoted 1 line
how can we guess ?
You know it's you but not me that tries to propose this solution, right?
Quoted 7 lines
Quoted 5 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ?
Where did you see I say you can't use u16?
Quoted 2 lines
are you open for changes or
should we copy/paste from net cvq ?
Net cvq comes first, it's natural to ask why you do things differently.
Quoted 3 lines
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Don't you see my reply above?
"
commands belongs to MSI could be grouped into the same class "
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
admin-virtq.tex new file +5
-0+ u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error;
Quoted 7 lines
Quoted 3 lines
I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
We're talking about the command format, not its semantics, right?
It's command format is pretty general.
Quoted 84 lines
To make it generic we need to separate.
Quoted 42 lines
Quoted 42 lines
admin-virtq.tex new file +15
-0+ u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported orinvalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +7
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driveronly if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +9
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hlineQuoted 8 lines
Quoted 5 lines
+0001h - 7FFFh & Generic admin cmds & - \\
+\hline
+8000h - FFFFh & Reserved & - \\
+\hline
+\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
see my comment above.
Quoted 19 lines
Quoted 19 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific dataset by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +8
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting} +struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * ....Quoted 8 lines
admin-virtq.tex line 88
Quoted 4 lines
admin-virtq.tex new file +4
-0+ * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Quoted 1 line
It doesn't scale. I tried it. It doesn't work.
What prevents you from improving the scalability instead of trying to revinting a duplicated mechanism?
Thanks
Quoted 69 lines
Thanks.
Quoted 64 lines
Thanks
Quoted 36 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilitiesof a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions.
+\item[42 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex}
+
\chapter{General Initialization And Device
Operation}\label{sec:General Initialization And Device Operation}
We start with an overview of device initialization, then expand
on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /Virtqueue Reset}.
+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates thatcontent.tex +2
-0+ the device supports administration virtqueue negotiation. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{ReservedQuoted 1 line
Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected] List archive:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SW%2FM98QZ7Av2wsv1NoekveDdQmlfWMYcvZQiFvpLcAE%3D&reserved=0
Feedback License:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=C5D9qmFqbNrx7CoTzsx%2B9qalts534XPaImHlcszwm1M%3D&reserved=0
List Guidelines:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bi7OJDuH9CvoKdxtA01dn5SpZwDp3Bna4%2B00bVOuz2o%3D&reserved=0
Committee:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SdGNrxTUZVIe2WJwb%2BdbT2UxagM9AdEzDWXccFvRE3s%3D&reserved=0
Join OASIS:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ynKmp30vKJMw%2FL4IHbBFY9EebT%2BsVX2%2Boz7IIcO%2F1xg%3D&reserved=0
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 83 lines · Jason Wang
Quoted 82 lines · Max Gurtovoy
Quoted 76 lines · Jason Wang
Quoted 55 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +13
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue commandbuffers
admin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued onlyQuoted 6 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQfeature, MUST
+support all the mandatory admin commands. A device MAY support also one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex line 25
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 2 lines
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each commandthis was already discussed.
Quoted 8 lines
Quoted 7 lines
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
I'm not against putting such id but we don't have an exact idea how will SF
device_idwill look like.Maybe it will be some 128 bit uuid ? maybe u64 ?
How do you know u16 is sufficient for command?
it's sufficient for VFs and this is the subject of the proposed commands.
When you'll add SF to virtio spec we'll create new commands for it with SF_id.
Again, this was discussed with Parav in V1 and was close. Please don't re-open.
Quoted 3 lines
Quoted 2 lines
how can we guess ?
You know it's you but not me that tries to propose this solution, right?
I'm proposing a solution but you ask questions that we answer but somehow they are asked again.
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ?
Where did you see I say you can't use u16?
you suggest to replace it to class + command.
I don't understand why.
If it's not mandatory, please ack.
Quoted 4 lines
Quoted 3 lines
are you open for changes or
should we copy/paste from net cvq ?Net cvq comes first, it's natural to ask why you do things differently.
I answered why.
Admin command set has a different purpose.
Quoted 8 lines
Quoted 3 lines
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Don't you see my reply above?
"
commands belongs to MSI could be grouped into the same class "
It's doesn't explain why class A + commands 1, 2 is better than opcodes 1 + 2 without a class.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
admin-virtq.tex line 30
Quoted 7 lines
admin-virtq.tex new file +5
-0+ u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error;I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
Quoted 6 lines
Quoted 3 lines
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
We're talking about the command format, not its semantics, right?
It's command format is pretty general.
in the Admin command set we would like to have better format that will allow users/drivers to better understand device error.
It is very common in many HW devices and specs out there.
I hope it doesn't critical for you as a concept.
Quoted 51 lines
Quoted 51 lines
To make it generic we need to separate.
Quoted 48 lines
Quoted 36 lines
admin-virtq.tex new file +15
-0+ u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported orinvalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +7
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driveronly if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +4
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|}admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
Quoted 1 line
Quoted 1 line
see my comment above.
I answered above.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 12 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific dataset by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +2
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting}admin-virtq.tex line 88
Quoted 10 lines
admin-virtq.tex new file +10
-0+struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Quoted 3 lines
Quoted 1 line
It doesn't scale. I tried it. It doesn't work.
What prevents you from improving the scalability instead of trying to revinting a duplicated mechanism?
It was already discussed.
Spec doesn't scale, too many constrains between command-A,B,C and feature 44,45, 46 and for transport X, Y, Z.
Quoted 91 lines
Thanks
Quoted 65 lines
Thanks.
Quoted 62 lines
Thanks
Quoted 36 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilitiesof a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions.
+\item[42 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex}
+
\chapter{General Initialization And Device
Operation}\label{sec:General Initialization And Device Operation}
We start with an overview of device initialization, then expand
on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /Virtqueue Reset}.
+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates thatcontent.tex +2
-0+ the device supports administration virtqueue negotiation. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{ReservedQuoted 1 line
Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected] List archive:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hkU2cd8o0F%2FTH%2F%2BHA8GQzVKZHPIDpuHBLMgVbjQSvN4%3D&reserved=0
Feedback License:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jqtoxmxY5oMIJ8X4R%2FCADdAwYHFNP%2Fix%2Frue8xqf8%2F0%3D&reserved=0
List Guidelines:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=a3bGp1iZqbK8dy2zMV3Umypft1kIL%2Bugjc8sZRZaX1M%3D&reserved=0
Committee:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=H5cMIMFYWJT1AVO9%2FLrJ8KDidyuFhuE%2Bb7xFTlKNDLI%3D&reserved=0
Join OASIS:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RY2TmLGYxJLQkX6GQVUxXOL7GeoWR6EqL8kStesrmD0%3D&reserved=0This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 89 lines · Max Gurtovoy
Quoted 86 lines · Jason Wang
Quoted 85 lines · Max Gurtovoy
Quoted 76 lines · Jason Wang
Quoted 55 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +13
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue commandbuffers
admin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued onlyQuoted 6 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQfeature, MUST
+support all the mandatory admin commands. A device MAY support also one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex line 25
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 2 lines
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each commandthis was already discussed.
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
I'm not against putting such id but we don't have an exact idea how will SF
device_idwill look like.Maybe it will be some 128 bit uuid ? maybe u64 ?
How do you know u16 is sufficient for command?
it's sufficient for VFs and this is the subject of the proposed commands.
So VIRTIO_ADMIN_CAPS_IDENTIFY maps to 3 commands:
VIRTIO_ADMIN_PCI_SRIOV_ATTRS,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET
Unless you do 1:1 mapping between cap bit and command, you will be short of commands if you're using u16.
Quoted 6 lines
When you'll add SF to virtio spec we'll create new commands for it with
SF_id.Again, this was discussed with Parav in V1 and was close. Please don't re-open.
I don't think so, you can reserve a special device id for the PF itself.
Quoted 32 lines
Quoted 4 lines
Quoted 3 lines
how can we guess ?
You know it's you but not me that tries to propose this solution, right?
I'm proposing a solution but you ask questions that we answer but somehow they are asked again.
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ?
Where did you see I say you can't use u16?
you suggest to replace it to class + command.
I don't understand why.
If it's not mandatory, please ack.
Quoted 4 lines
Quoted 3 lines
are you open for changes or
should we copy/paste from net cvq ?Net cvq comes first, it's natural to ask why you do things differently.
I answered why.
Admin command set has a different purpose.
Quoted 3 lines
Quoted 3 lines
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Quoted 8 lines
Quoted 5 lines
Don't you see my reply above?
"
commands belongs to MSI could be grouped into the same class "It's doesn't explain why class A + commands 1, 2 is better than opcodes 1 + 2 without a class.
Well, if you read how cvq work it's not hard to get the conclusion,
It's easier to be mapped to a feature bit. You just map a class and that's all.
Otherwise, you need to feature X: command A,B,C,D ....
Quoted 105 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
admin-virtq.tex line 30
Quoted 9 lines
admin-virtq.tex new file +5
-0+ u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error;I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
Quoted 6 lines
Quoted 3 lines
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
We're talking about the command format, not its semantics, right?
It's command format is pretty general.
in the Admin command set we would like to have better format that will allow users/drivers to better understand device error.
It is very common in many HW devices and specs out there.
I hope it doesn't critical for you as a concept.
Quoted 51 lines
Quoted 51 lines
To make it generic we need to separate.
Quoted 48 lines
Quoted 36 lines
admin-virtq.tex new file +15
-0+ u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported orinvalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +7
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driveronly if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +4
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|}admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
Quoted 1 line
Quoted 1 line
see my comment above.
I answered above.
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 12 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific dataset by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +2
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting}admin-virtq.tex line 88
Quoted 10 lines
admin-virtq.tex new file +10
-0+struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
It doesn't scale. I tried it. It doesn't work.
What prevents you from improving the scalability instead of trying to revinting a duplicated mechanism?
It was already discussed.
Spec doesn't scale, too many constrains between command-A,B,C and feature 44,45, 46 and for transport X, Y, Z.
What I meant is, if you feel the current feature negotiation doesn't scale, you can improve it instead of reinventing it partially (e.g no negotiation) .
Thanks
Quoted 92 lines
Quoted 92 lines
Thanks
Quoted 65 lines
Thanks.
Quoted 62 lines
Thanks
Quoted 36 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilitiesof a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions.
+\item[42 and above] Feature bits reserved for future extensions.
\end{description}\begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex}
+
\chapter{General Initialization And Device
Operation}\label{sec:General Initialization And Device Operation}
We start with an overview of device initialization, then expand
on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /Virtqueue Reset}.
+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates thatcontent.tex +2
-0+ the device supports administration virtqueue negotiation. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{ReservedQuoted 1 line
Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected] List archive:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hkU2cd8o0F%2FTH%2F%2BHA8GQzVKZHPIDpuHBLMgVbjQSvN4%3D&reserved=0
Feedback License:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jqtoxmxY5oMIJ8X4R%2FCADdAwYHFNP%2Fix%2Frue8xqf8%2F0%3D&reserved=0
List Guidelines:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=a3bGp1iZqbK8dy2zMV3Umypft1kIL%2Bugjc8sZRZaX1M%3D&reserved=0
Committee:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=H5cMIMFYWJT1AVO9%2FLrJ8KDidyuFhuE%2Bb7xFTlKNDLI%3D&reserved=0
Join OASIS:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RY2TmLGYxJLQkX6GQVUxXOL7GeoWR6EqL8kStesrmD0%3D&reserved=0This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 9 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 92 lines · Jason Wang
Quoted 85 lines · Max Gurtovoy
Quoted 83 lines · Jason Wang
Quoted 82 lines · Max Gurtovoy
Quoted 74 lines · Jason Wang
Quoted 55 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +13
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue commandbuffers
admin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued onlyQuoted 6 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQfeature, MUST
+support all the mandatory admin commands. A device MAY support also one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex line 25
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 2 lines
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each commandthis was already discussed.
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
I'm not against putting such id but we don't have an exact idea how will SF
device_idwill look like.Maybe it will be some 128 bit uuid ? maybe u64 ?
How do you know u16 is sufficient for command?
it's sufficient for VFs and this is the subject of the proposed commands.
Quoted 8 lines
So
VIRTIO_ADMIN_CAPS_IDENTIFYmaps to 3 commands:
VIRTIO_ADMIN_PCI_SRIOV_ATTRS,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GETUnless you do 1:1 mapping between cap bit and command, you will be short of commands if you're using u16.
2^16 admin commands are more than enough.
Quoted 7 lines
Quoted 6 lines
When you'll add SF to virtio spec we'll create new commands for it with
SF_id.Again, this was discussed with Parav in V1 and was close. Please don't re-open.
I don't think so, you can reserve a special device id for the PF itself.
can you propose something concrete ?
are you sure by 100% that sf_id is 32 bit ? can the TG ack on that ?
Quoted 30 lines
Quoted 30 lines
Quoted 3 lines
Quoted 2 lines
how can we guess ?
You know it's you but not me that tries to propose this solution, right?
I'm proposing a solution but you ask questions that we answer but somehow they are asked again.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ?
Where did you see I say you can't use u16?
you suggest to replace it to class + command.
I don't understand why.
If it's not mandatory, please ack.
Quoted 3 lines
Quoted 2 lines
are you open for changes or
should we copy/paste from net cvq ?Net cvq comes first, it's natural to ask why you do things differently.
I answered why.
Admin command set has a different purpose.
Quoted 7 lines
Quoted 2 lines
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Don't you see my reply above?
"
commands belongs to MSI could be grouped into the same class "
Quoted 7 lines
Quoted 2 lines
It's doesn't explain why class A + commands 1, 2 is better than opcodes 1 + 2 without a class.
Well, if you read how cvq work it's not hard to get the conclusion,
It's easier to be mapped to a feature bit. You just map a class and that's all.
Otherwise, you need to feature X: command A,B,C,D ....
But I just added one feature bit: the existence of the AQ.
in your suggestion I'll need to add for each feature:
Feature X can be negotiated only if AQ feature is negotiated.
And if feature X will have a small extension so a new feature will be needed and it will say:
Feature Y can be negotiated in feature X and feature AQ are negotiated.
or if feature X and feature Y are mutual exclusive then:
feature X can't be negotiated if feature Y is negotiated.
feature X can be negotiated only if feature AQ is negotiated.
and this is a small example. We will have more and more Admin functionality and this will be impossible to scale.
A simple identify command solve this.
Quoted 99 lines
Quoted 99 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
admin-virtq.tex line 30
Quoted 7 lines
admin-virtq.tex new file +5
-0+ u8 command_specific_data[]; + + /* Device-writable part */ + u8 status; + u8 command_specific_error;I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
Quoted 6 lines
Quoted 3 lines
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
We're talking about the command format, not its semantics, right?
It's command format is pretty general.
in the Admin command set we would like to have better format that will allow users/drivers to better understand device error.
It is very common in many HW devices and specs out there.
I hope it doesn't critical for you as a concept.
Quoted 48 lines
Quoted 48 lines
To make it generic we need to separate.
Quoted 46 lines
Quoted 35 lines
admin-virtq.tex new file +15
-0+ u8 command_specific_result[]; +}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported orinvalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +7
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the \field{command_specific_result}, +if needed. + +The \field{command_specific_error} should be inspected by the driveronly if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +4
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|}admin-virtq.tex line 71
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
Quoted 1 line
Quoted 1 line
see my comment above.
I answered above.
Quoted 26 lines
Quoted 23 lines
Quoted 15 lines
Quoted 15 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific dataset by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +6
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting} +struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reservedQuoted 8 lines
admin-virtq.tex line 88
Quoted 6 lines
admin-virtq.tex new file +6
-0+ * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Quoted 1 line
It doesn't scale. I tried it. It doesn't work.
What prevents you from improving the scalability instead of trying to revinting a duplicated mechanism?
Quoted 7 lines
Quoted 4 lines
It was already discussed.
Spec doesn't scale, too many constrains between command-A,B,C and feature 44,45, 46 and for transport X, Y, Z.
What I meant is, if you feel the current feature negotiation doesn't scale, you can improve it instead of reinventing it partially (e.g no negotiation) .
We're adding an identify command for the admin command set.
did you review it ?
Quoted 121 lines
Thanks
Quoted 87 lines
Quoted 87 lines
Thanks
Quoted 63 lines
Thanks.
Quoted 61 lines
Thanks
Quoted 36 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilitiesof a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions.
+\item[42 and above] Feature bits reserved for future extensions.\end{description} \begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex}+
\chapter{General Initialization And Device
Operation}\label{sec:General Initialization And Device Operation}
We start with an overview of device initialization, then expand
on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /Virtqueue Reset}.
+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates thatcontent.tex +2
-0+ the device supports administration virtqueue negotiation. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{ReservedQuoted 1 line
Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected] List archive:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&reserved=0
Feedback License:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&reserved=0
List Guidelines:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&reserved=0
Committee:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&reserved=0
Join OASIS:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&reserved=0This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 8 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 104 lines · Max Gurtovoy
Quoted 102 lines · Jason Wang
Quoted 93 lines · Max Gurtovoy
Quoted 91 lines · Jason Wang
Quoted 90 lines · Max Gurtovoy
Quoted 82 lines · Jason Wang
Quoted 61 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>---
admin-virtq.tex | 89
+++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 9 +++--2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +2
-0+ +Admin virtqueue is used to send administrative commands tomanipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs ofadmin-virtq.tex new file +2
-0+a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its adminvirtqueue.).
admin-virtq.tex new file +7
-0+ +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queuecommand
buffersadmin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issued onlyQuoted 8 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQ
feature, MUST
+support all the mandatory admin commands. A device MAY support also
one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 2 lines
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each commandthis was already discussed.
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
I'm not against putting such id but we don't have an exact idea how will SF
device_idwill look like.Maybe it will be some 128 bit uuid ? maybe u64 ?
How do you know u16 is sufficient for command?
it's sufficient for VFs and this is the subject of the proposed commands.
So
VIRTIO_ADMIN_CAPS_IDENTIFYmaps to 3 commands:
Quoted 8 lines
Quoted 6 lines
VIRTIO_ADMIN_PCI_SRIOV_ATTRS,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GETUnless you do 1:1 mapping between cap bit and command, you will be short of commands if you're using u16.
2^16 admin commands are more than enough.
You introduce 65536 cap bits via identify. And you use u16 for command.
But you map 3 commands to a single cap bit. You will be short of commands. Isn't it?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
When you'll add SF to virtio spec we'll create new commands for it with
SF_id.
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Again, this was discussed with Parav in V1 and was close. Please don't re-open.
I don't think so, you can reserve a special device id for the PF itself.
can you propose something concrete ?
are you sure by 100% that
sf_idis 32 bit ? can the TG ack on that ?
I meant for command without and destination. You can reserve a special ID for that.
Regarding the bit, if you think 32bit is not sufficient, let's just use 64bit, or reserving some bytes in the command.
Quoted 47 lines
Quoted 31 lines
Quoted 31 lines
Quoted 4 lines
Quoted 3 lines
how can we guess ?
You know it's you but not me that tries to propose this solution, right?
I'm proposing a solution but you ask questions that we answer but somehow they are asked again.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ?
Where did you see I say you can't use u16?
you suggest to replace it to class + command.
I don't understand why.
If it's not mandatory, please ack.
Quoted 3 lines
Quoted 2 lines
are you open for changes or
should we copy/paste from net cvq ?Net cvq comes first, it's natural to ask why you do things differently.
I answered why.
Admin command set has a different purpose.
Quoted 7 lines
Quoted 2 lines
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Don't you see my reply above?
"
commands belongs to MSI could be grouped into the same class "Quoted 7 lines
Quoted 2 lines
It's doesn't explain why class A + commands 1, 2 is better than opcodes 1 + 2 without a class.
Well, if you read how cvq work it's not hard to get the conclusion,
It's easier to be mapped to a feature bit. You just map a class and that's all.
Otherwise, you need to feature X: command A,B,C,D ....
But I just added one feature bit: the existence of the AQ.
in your suggestion I'll need to add for each feature:
Feature X can be negotiated only if AQ feature is negotiated.
And if feature X will have a small extension so a new feature will be needed and it will say:
Quoted 7 lines
Feature Y can be negotiated in feature X and feature AQ are negotiated.
or if feature X and feature Y are mutual exclusive then:
feature X can't be negotiated if feature Y is negotiated.
feature X can be negotiated only if feature AQ is negotiated.
This is how virtqueue works for all the existing device. It's as simple as depending the feature to another feature which indicates the existing of a specific type of virtqueue.
Quoted 114 lines
and this is a small example. We will have more and more Admin functionality and this will be impossible to scale.
A simple identify command solve this.
Quoted 108 lines
Quoted 108 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
+ u8 command_specific_data[];
+
+ /* Device-writable part */
+ u8 status;
+ u8command_specific_error;I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
Quoted 6 lines
Quoted 3 lines
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
We're talking about the command format, not its semantics, right?
It's command format is pretty general.
in the Admin command set we would like to have better format that will allow users/drivers to better understand device error.
It is very common in many HW devices and specs out there.
I hope it doesn't critical for you as a concept.
Quoted 52 lines
Quoted 52 lines
To make it generic we need to separate.
Quoted 50 lines
Quoted 39 lines
+ u8 command_specific_result[];
admin-virtq.tex new file +14
-0+}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specific error \\ +\hline +02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED &unsupported or
invalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +4
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the\field{command_specific_result},
admin-virtq.tex new file +3
-0+if needed. + +The \field{command_specific_error} should be inspected by thedriver
only if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content
of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +4
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|}admin-virtq.tex
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
Quoted 1 line
Quoted 1 line
see my comment above.
I answered above.
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 12 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specificdata
set by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +2
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting}admin-virtq.tex
Quoted 10 lines
admin-virtq.tex new file +10
-0+struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Quoted 3 lines
Quoted 1 line
It doesn't scale. I tried it. It doesn't work.
What prevents you from improving the scalability instead of trying to revinting a duplicated mechanism?
It was already discussed.
Spec doesn't scale, too many constrains between command-A,B,C and feature 44,45, 46 and for transport X, Y, Z.
Quoted 7 lines
Quoted 3 lines
What I meant is, if you feel the current feature negotiation doesn't scale, you can improve it instead of reinventing it partially (e.g no negotiation) .
We're adding an identify command for the admin command set.
did you review it ?
Yes, but what I meant is that we've already had a feature negotiation mechanism. We should improve it instead of reinventing another one.
Thanks
Quoted 129 lines
Quoted 129 lines
Thanks
Quoted 94 lines
Quoted 94 lines
Thanks
Quoted 70 lines
Thanks.
Quoted 68 lines
Thanks
Quoted 31 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:BasicFacilities
of a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions. +\item[42 and above] Feature bits reserved for future extensions. \end{description}
\begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex}
+
\chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}that the driver can reset a queue individually.
Quoted 8 lines
See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation. +
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected] List archive:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&reserved=0This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 8 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 9 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 114 lines · Jason Wang
Quoted 112 lines · Max Gurtovoy
Quoted 109 lines · Jason Wang
Quoted 94 lines · Max Gurtovoy
Quoted 92 lines · Jason Wang
Quoted 91 lines · Max Gurtovoy
Quoted 83 lines · Jason Wang
Quoted 62 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>---
admin-virtq.tex | 89
+++++++++++++++++++++++++++++++++++++++++++++++++
content.tex | 9 +++--2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +1
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a VirtioDevice / Admin Virtqueues}
admin-virtq.tex new file +2
-0+ +Admin virtqueue is used to send administrative commands tomanipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs ofadmin-virtq.tex new file +2
-0+a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its adminvirtqueue.).
admin-virtq.tex new file +7
-0+ +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types. + +Regardless of device offering VIRTIO_F_IN_ORDER, admin queuecommand
buffersadmin-virtq.tex new file +3
-0+are used by the device in out of order manner. + +The Admin command set defines the commands that may be issuedonly
Quoted 8 lines
to the admin
+virtqueue. Each virtio device that advertises theVIRTIO_F_ADMIN_VQ
feature, MUST
+support all the mandatory admin commands. A device MAY support also
one or more
+optional admin commands. All commands are of the following form:Do we need a way for advertising the supported optional commands (or features bits)? Otherwise dealing with the compatibility will result of per command probing.
admin-virtq.tex
Quoted 5 lines
admin-virtq.tex new file +5
-0+ +\begin{lstlisting} +struct virtio_admin_cmd { + /* Device-readable part */ + u16 command;Two questions:
Quoted 2 lines
1) It looks to me it's better to have a explicit
device_idhere other than embed it in each commandthis was already discussed.
We agreed that some commands are not referring to different device so no, we don't need to put it in generic structure.
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
I'm not against putting such id but we don't have an exact idea how will SF
device_idwill look like.Maybe it will be some 128 bit uuid ? maybe u64 ?
How do you know u16 is sufficient for command?
it's sufficient for VFs and this is the subject of the proposed commands.
So
VIRTIO_ADMIN_CAPS_IDENTIFYmaps to 3 commands:
VIRTIO_ADMIN_PCI_SRIOV_ATTRS,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GETUnless you do 1:1 mapping between cap bit and command, you will be short of commands if you're using u16.
Quoted 6 lines
Quoted 1 line
2^16 admin commands are more than enough.
You introduce 65536 cap bits via identify. And you use u16 for command.
But you map 3 commands to a single cap bit. You will be short of commands. Isn't it?
No we just don't use all the bits in the identify. Identify has 2^16 bits to be able to cover all we need.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 10 lines
When you'll add SF to virtio spec we'll create new commands for it with
SF_id.Again, this was discussed with Parav in V1 and was close. Please don't re-open.
I don't think so, you can reserve a special device id for the PF itself.
Quoted 8 lines
Quoted 3 lines
can you propose something concrete ?
are you sure by 100% that
sf_idis 32 bit ? can the TG ack on that ?I meant for command without and destination. You can reserve a special ID for that.
Regarding the bit, if you think 32bit is not sufficient, let's just use 64bit, or reserving some bytes in the command.
So 64bit size is acked by TG for vdev_id ?
Quoted 51 lines
Quoted 51 lines
Quoted 33 lines
Quoted 33 lines
Quoted 6 lines
Quoted 5 lines
how can we guess ?
You know it's you but not me that tries to propose this solution, right?
I'm proposing a solution but you ask questions that we answer but somehow they are asked again.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
2) virtio-net cvq use class/command which seems more elegant, e.g all commands belongs to MSI could be grouped into the same class
why can't we do it in u16 command opcode ?
Where did you see I say you can't use u16?
you suggest to replace it to class + command.
I don't understand why.
If it's not mandatory, please ack.
Quoted 3 lines
Quoted 2 lines
are you open for changes or
should we copy/paste from net cvq ?Net cvq comes first, it's natural to ask why you do things differently.
I answered why.
Admin command set has a different purpose.
Quoted 7 lines
Quoted 2 lines
Let's ask differently. Why we need class/command structure ? why is it more elegant ?
Don't you see my reply above?
"
commands belongs to MSI could be grouped into the same class "Quoted 7 lines
Quoted 2 lines
It's doesn't explain why class A + commands 1, 2 is better than opcodes 1 + 2 without a class.
Well, if you read how cvq work it's not hard to get the conclusion,
It's easier to be mapped to a feature bit. You just map a class and that's all.
Otherwise, you need to feature X: command A,B,C,D ....
But I just added one feature bit: the existence of the AQ.
in your suggestion I'll need to add for each feature:
Feature X can be negotiated only if AQ feature is negotiated.
And if feature X will have a small extension so a new feature will be needed and it will say:
Feature Y can be negotiated in feature X and feature AQ are negotiated.
Quoted 8 lines
Quoted 5 lines
or if feature X and feature Y are mutual exclusive then:
feature X can't be negotiated if feature Y is negotiated.
feature X can be negotiated only if feature AQ is negotiated.
This is how virtqueue works for all the existing device. It's as simple as depending the feature to another feature which indicates the existing of a specific type of virtqueue.
It is not simple and not scale.
Quoted 121 lines
Quoted 121 lines
and this is a small example. We will have more and more Admin functionality and this will be impossible to scale.
A simple identify command solve this.
Quoted 112 lines
Quoted 102 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
+ u8 command_specific_data[];
+
+ /* Device-writable part */
+ u8 status;
+ u8command_specific_error;I wonder the reason why not using a single field for the above two fields. (Or any value for separating command specific error out, we don't do that for virtio-net cvq).
Quoted 6 lines
Quoted 3 lines
each command have its own specific errors.
virtio net cvq is not generic - it's a net device cvqueue.
We're talking about the command format, not its semantics, right?
It's command format is pretty general.
in the Admin command set we would like to have better format that will allow users/drivers to better understand device error.
It is very common in many HW devices and specs out there.
I hope it doesn't critical for you as a concept.
Quoted 53 lines
Quoted 53 lines
To make it generic we need to separate.
Quoted 51 lines
Quoted 40 lines
+ u8 command_specific_result[];
admin-virtq.tex new file +12
-0+}; +\end{lstlisting} + +The following table describes the generic Admin status codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_STATUS_OK & successful completion \\ +\hline +01h & VIRTIO_ADMIN_STATUS_CS_ERR & command specificerror \\
+\hline
+02h & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED & unsupported or
invalid opcode \\
+\hline
+\end{tabular}admin-virtq.tex new file +4
-0+ +The \field{command} and \field{command_specific_data} are +set by the driver, and the device sets the \field{status}, the +\field{command_specific_error} and the\field{command_specific_result},
admin-virtq.tex new file +3
-0+if needed. + +The \field{command_specific_error} should be inspected by thedriver
only if \field{status} is set to +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content
of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set toVIRTIO_ADMIN_STATUS_CS_ERR, the +\field{command_specific_error} value is undefined.admin-virtq.tex new file +4
-0+ +The following table describes the Admin command set: + +\begin{tabular}{|l|l|l|}admin-virtq.tex
Quoted 10 lines
admin-virtq.tex new file +10
-0+\hline +Opcode & Command & M/O \\ +\hline \hline +0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ +\hline +0001h - 7FFFh & Generic admin cmds & - \\ +\hline +8000h - FFFFh & Reserved & - \\ +\hline +\end{tabular}See above, I'd suggest to use class/command as virtio-net cvq.
Quoted 1 line
Quoted 1 line
see my comment above.
I answered above.
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 12 lines
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}admin-virtq.tex new file +2
-0+ +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specificdata
set by the driver.
+This command upon success, returns a data buffer that describes information about the supportedadmin-virtq.tex new file +2
-0+admin capabilities by the device. This information is of form: +\begin{lstlisting}admin-virtq.tex
Quoted 10 lines
admin-virtq.tex new file +10
-0+struct virtio_admin_caps_identify_result { + /* + * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[1] - Bit 0x0 - Bit 0x7 are reserved + * caps[2] - Bit 0x0 - Bit 0x7 are reserved + * .... + * caps[8191] - Bit 0x0 - Bit 0x7 are reserved + */ + u8 caps[8192]; +};Ok, so I see the identify command. But I wonder if we can do that via the feature bits?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
It doesn't scale. I tried it. It doesn't work.
What prevents you from improving the scalability instead of trying to revinting a duplicated mechanism?
It was already discussed.
Spec doesn't scale, too many constrains between command-A,B,C and feature 44,45, 46 and for transport X, Y, Z.
What I meant is, if you feel the current feature negotiation doesn't scale, you can improve it instead of reinventing it partially (e.g no negotiation) .
Quoted 6 lines
Quoted 3 lines
We're adding an identify command for the admin command set.
did you review it ?
Yes, but what I meant is that we've already had a feature negotiation mechanism. We should improve it instead of reinventing another one.
I've explained the scale issue.
Quoted 143 lines
Thanks
Quoted 138 lines
Quoted 137 lines
Thanks
Quoted 94 lines
Quoted 94 lines
Thanks
Quoted 70 lines
Thanks.
Quoted 68 lines
Thanks
Quoted 31 lines
+\end{lstlisting}
content.tex +0
-0@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:BasicFacilities
of a Virtio Device / Feature B
\begin{description}
\item[0 to 23] Feature bits for the specific device type -\item[24 to 40] Feature bits reserved for extensions to the queue and +\item[24 to 41] Feature bits reserved for extensions to the queue and
feature negotiation mechanisms
-\item[41 and above] Feature bits reserved for future extensions. +\item[42 and above] Feature bits reserved for future extensions. \end{description}
\begin{note}content.tex +0
-0@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:BasicFacilities of a Virtio Device / Expo
types. It is RECOMMENDED that devices generate version 4 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}. +\input{admin-virtq.tex}
+
\chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on thecontent.tex +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}that the driver can reset a queue individually.
Quoted 8 lines
See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation. +
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected] List archive:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pJ5nr6uphZoehnGnA0VSMX5eu2x%2Fgy%2BGp6eYNz6Pi0g%3D&reserved=0This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
Quoted 8 lines
Quoted 8 lines
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Max GurtovoyQuoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-) create mode 100644 admin-virtq.texadmin-virtq.tex new file +2
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} +
admin-virtq.tex line 12
Quoted 10 lines
admin-virtq.tex new file +10
-0+Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.
So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiated
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
Anyway, I'd reword the two sentences above:
"An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is negotiated. The index of the admin virtqueue is device type specific."
(...)
admin-virtq.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6853
Quoted 6 lines
admin-virtq.tex new file +2
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation.
Maybe
"This feature indicates that an administration virtqueue is supported." ?
admin-virtq.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6857
Quoted 4 lines
admin-virtq.tex new file +1
-0+ \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 55 lines · Cornelia Huck
Quoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-) create mode 100644 admin-virtq.texadmin-virtq.tex new file +2
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} +admin-virtq.tex line 12
Quoted 10 lines
admin-virtq.tex new file +10
-0+Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiated
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 22 lines
Anyway, I'd reword the two sentences above:
"An admin virtqueue exists for a certain device if
VIRTIO_F_ADMIN_VQis negotiated. The index of the admin virtqueue is device type specific."(...)
admin-virtq.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6853
Quoted 6 lines
admin-virtq.tex new file +2
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation.Maybe
"This feature indicates that an administration virtqueue is supported." ?
Quoted 4 lines
admin-virtq.tex new file +1
-0+ \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiated
Quoted 7 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 30 lines · Cornelia Huck
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 7 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number. Another advantage to this approach is that we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.
Signature · 1 line
MST
[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 39 lines · Michael S. Tsirkin
Quoted 30 lines · Cornelia Huck
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 7 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Quoted 7 lines
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number. Another advantage to this approach is that we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.
I only meant to make it the last one _now_ :)
But admin-vq-gets-its-own-page is a good point.
Maybe pci gets a new entry in common_cfg, mmio gets a new register, and ccw gets a new command? (Although I'd prefer to be conservative with new commands for ccw, maybe it's time to introduce a "get misc config" type command that can be reused for other things. There's a generic ccw for that, but adding new stuff to it would require an s390 architecture change AFAIK, so I had decided not to go down that path for virtio.)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 49 lines · Cornelia Huck
Quoted 39 lines · Michael S. Tsirkin
Quoted 30 lines · Cornelia Huck
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 7 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Quoted 7 lines
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number. Another advantage to this approach is that we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.
Quoted 6 lines
I only meant to make it the last one _now_ :)
But admin-vq-gets-its-own-page is a good point.
Maybe pci gets a new entry in
common_cfg, mmio gets a new register, and ccw gets a new command?
That's fine.
Quoted 5 lines
(Although I'd prefer to be conservative with new
commands for ccw, maybe it's time to introduce a "get misc config" type command that can be reused for other things. There's a generic ccw for that, but adding new stuff to it would require an s390 architecture change AFAIK, so I had decided not to go down that path for virtio.)
You mean a generic "get misc config" for all transports? That's fine by me too. I suspect nvidia guys will need help building something like this though ...
Signature · 1 line
MST
[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 12 lines · Michael S. Tsirkin
Quoted 2 lines · Cornelia Huck
Maybe pci gets a new entry in
common_cfg, mmio gets a new register, and ccw gets a new command?That's fine.
Quoted 5 lines
(Although I'd prefer to be conservative with new
commands for ccw, maybe it's time to introduce a "get misc config" type command that can be reused for other things. There's a generic ccw for that, but adding new stuff to it would require an s390 architecture change AFAIK, so I had decided not to go down that path for virtio.)
Quoted 3 lines
You mean a generic "get misc config" for all transports? That's fine by me too. I suspect nvidia guys will need help building something like this though ...
I was more thinking about ccw only... not sure if we can define something useful for all transports.
I'm willing to help with defining a ccw mechanism; I realize that this will be hard for most people.
(Let's get agreement on whether we want to go down that route first.)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Michael S. TsirkinQuoted 40 lines · Michael S. Tsirkin
Quoted 30 lines · Cornelia Huck
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 7 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Quoted 3 lines
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 36 lines · Jason Wang
Quoted 35 lines · Michael S. Tsirkin
Quoted 34 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Quoted 6 lines
Quoted 3 lines
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index 0xFFFF for admin virtqueue for compatibility.
Quoted 11 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 38 lines · Max Gurtovoy
Quoted 36 lines · Jason Wang
Quoted 35 lines · Michael S. Tsirkin
Quoted 34 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.
I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
Quoted 12 lines
Quoted 12 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new cfg_type (#define VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
that will look something like:
struct virtio_pci_admin_cfg {
le32 queue_index; /* read only for the driver */
le16 queue_size; /* read-write */
le16 queue_msix_vector; /* read-write */ le16 queue_enable; /* read-write */
le16 queue_notify_off; /* read-only for driver */ le64 queue_desc; /* read-write */
le64 queue_driver; /* read-write */
le64 queue_device; /* read-write */
le16 queue_notify_data; /* read-only for driver */ le16 queue_reset; /* read-write */
};
instead of re-using the struct virtio_pci_common_cfg ?
or do you prefer extending the struct virtio_pci_common_cfg with "le16 admin_queue_index; /* read only for the driver */ ?
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 69 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */
Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?
The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a misc_config structure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16 admin_queue_index register.
For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
The point here is that we are making it easier to add more fields just like admin queue index in the future.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.
Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define VIRTIO_PCI_CAP_MISC_CFG 10
and
struct virtio_pci_misc_cfg {
le16 admin_queue_index; /* read-only for driver */
};
Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 96 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10
Quoted 7 lines
and
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
This is Cornelia's idea, we'll need her response.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 97 lines · Michael S. Tsirkin
Quoted 96 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 8 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
This is Cornelia's idea, we'll need her response.
Ok, Cornelia please review the above.
For V3 I'll prepare the PCI transport and we can work off list on the MMIO and CCW patches for V4.
sounds reasonable ?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Thanks
Quoted 6 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
--
MST
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 106 lines · Max Gurtovoy
Quoted 104 lines · Michael S. Tsirkin
Quoted 94 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueue is used to send administrative commands to manipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs of
+a parent PCI PF device are grouped together. These devices can be
+optionally managed by its parent PCI PF using its admin virtqueue.).
+
+Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10Quoted 7 lines
and
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
Quoted 8 lines
Quoted 1 line
This is Cornelia's idea, we'll need her response.
Ok, Cornelia please review the above.
For V3 I'll prepare the PCI transport and we can work off list on the MMIO and CCW patches for V4.
sounds reasonable ?
Yes except pls work on list.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Michael S. TsirkinQuoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define CCW_CMD_READ_MISC_CONF 0x82
struct virtio_misc_conf {
be16 admin_queue_index;
};bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 119 lines · Cornelia Huck
Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueue is used to send administrative commands to manipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs of
+a parent PCI PF device are grouped together. These devices can be
+optionally managed by its parent PCI PF using its admin virtqueue.).
+
+Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };
Quoted 6 lines
bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
So something like:
AdminQueueIndex/0x0c4/R
If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register returns the queue index of the administration virtqueue.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.
No, I mean a register that switches 100+ between device specific and misc space.
Signature · 1 line
MST
[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 44 lines · Michael S. Tsirkin
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 8 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Do you want to make that dependent on an extra feature bit, then? We certainly want to make that facility available if the admin vq is not negotiated but another future feature using misc is. (We probably don't want to bump the device version.)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 52 lines · Cornelia Huck
Quoted 50 lines · Michael S. Tsirkin
Quoted 48 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.
Quoted 7 lines
Quoted 2 lines
No, I mean a register that switches 100+ between device specific and misc space.
Do you want to make that dependent on an extra feature bit, then? We certainly want to make that facility available if the admin vq is not negotiated but another future feature using misc is. (We probably don't want to bump the device version.)
I guess we could do that, yes. It's unfortunate that value of unused registers is undefined for MMIO.
Signature · 1 line
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Michael S. TsirkinQuoted 44 lines · On Mon, 31 Jan 2022 09:52:54 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 9 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Maybe adding a register that tells us where does the "misc config start" is another option. I don't think we need an open ended device-config in practice. I have no idea if there are any upper limits on MMIO address space though. If we are constrained there, the switching is certainly more efficient. Otherwise, I think having the misc config somewhere after device specific config is simpler.
BTW I don't really like this "misc" as the name. FWIW it is less miscellaneous that the device specific config, since it is common for all devices. I don't have a good name for it, but I would prefer "common" over "misc"
Regards,
Halil
[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 57 lines · Halil Pasic
Quoted 44 lines · On Mon, 31 Jan 2022 09:52:54 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 9 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Quoted 6 lines
Maybe adding a register that tells us where does the "misc config start" is another option. I don't think we need an open ended device-config in practice. I have no idea if there are any upper limits on MMIO address space though. If we are constrained there, the switching is certainly more efficient. Otherwise, I think having the misc config somewhere after device specific config is simpler.
I think we first need to agree what the "misc" thing is actually supposed to be. My idea was that we don't have an unlimited supply of ccws to use for new features, so introducing one for reading "misc" configuration would be a way to keep things extensible (it also might make the config/register space for other transports less cluttered). The same idea (save on ccws) would apply to the multiplexing "action" ccw I mentioned in my other mail.
So, for the case here (simply relaying the location of the admin vq), we don't really need a "misc" mechanism for pci/mmio, but I'd like to introduce one for ccw. If we agree that it would be useful for pci/mmio as well, we should introduce it now.
Quoted 5 lines
BTW I don't really like this "misc" as the name. FWIW it is less miscellaneous that the device specific config, since it is common for all devices. I don't have a good name for it, but I would prefer "common" over "misc"
I'm not sure whether "common" would be more confusing, as pci already has a "common" config (with things that are covered in the mmio registers, and via various ccw commands.) Does anyone have a better idea for a bikeshed colour?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 74 lines · Cornelia Huck
Quoted 57 lines · Halil Pasic
Quoted 44 lines · On Mon, 31 Jan 2022 09:52:54 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 9 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Quoted 6 lines
Maybe adding a register that tells us where does the "misc config start" is another option. I don't think we need an open ended device-config in practice. I have no idea if there are any upper limits on MMIO address space though. If we are constrained there, the switching is certainly more efficient. Otherwise, I think having the misc config somewhere after device specific config is simpler.
I think we first need to agree what the "misc" thing is actually supposed to be. My idea was that we don't have an unlimited supply of ccws to use for new features, so introducing one for reading "misc" configuration would be a way to keep things extensible (it also might make the config/register space for other transports less cluttered). The same idea (save on ccws) would apply to the multiplexing "action" ccw I mentioned in my other mail.
Quoted 4 lines
So, for the case here (simply relaying the location of the admin vq), we don't really need a "misc" mechanism for pci/mmio, but I'd like to introduce one for ccw. If we agree that it would be useful for pci/mmio as well, we should introduce it now.
OK, my idea was that it's a config that is just live device config but both device and transport independent. Adding it in the misc structure would allow defining it in a single place.
Quoted 10 lines
Quoted 5 lines
BTW I don't really like this "misc" as the name. FWIW it is less miscellaneous that the device specific config, since it is common for all devices. I don't have a good name for it, but I would prefer "common" over "misc"
I'm not sure whether "common" would be more confusing, as pci already has a "common" config (with things that are covered in the mmio registers, and via various ccw commands.) Does anyone have a better idea for a bikeshed colour?
I'd say "shared" maybe but unfortunately pci already gained "shared memory" so I think it would be confusing.
"transport independent" is too wordy...
Signature · 1 line
MST
Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Cornelia HuckQuoted 66 lines · On Mon, 31 Jan 2022 18:10:39 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 57 lines · Halil Pasic
Quoted 44 lines · On Mon, 31 Jan 2022 09:52:54 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 9 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Quoted 6 lines
Maybe adding a register that tells us where does the "misc config start" is another option. I don't think we need an open ended device-config in practice. I have no idea if there are any upper limits on MMIO address space though. If we are constrained there, the switching is certainly more efficient. Otherwise, I think having the misc config somewhere after device specific config is simpler.
Quoted 8 lines
I think we first need to agree what the "misc" thing is actually supposed to be. My idea was that we don't have an unlimited supply of ccws to use for new features, so introducing one for reading "misc" configuration would be a way to keep things extensible (it also might make the config/register space for other transports less cluttered). The same idea (save on ccws) would apply to the multiplexing "action" ccw I mentioned in my other mail.
I agree with not wasting CCWs.
Quoted 4 lines
So, for the case here (simply relaying the location of the admin vq), we don't really need a "misc" mechanism for pci/mmio, but I'd like to introduce one for ccw. If we agree that it would be useful for pci/mmio as well, we should introduce it now.
Please see Michael's response. My understanding was also that what we want is something like config space for the virtio protocol stuff. The current config space is entirely device specific, so if we would want a common thing in _config space_, like _the index of the administration vq_ then each device would have to define it separately, in a device specific location. Which is not nice.
AFAIU having a this protocol config could be sufficient in the sense that we probably don't need another misc. My line of thinking is: with this we basically get a read-write interface for exposing stuff. The only other thing I can think of is _transport specific fields_. That is if we needed something that ain't specific to the device, but ain't common to all virtio (i.e. the virtio protocol). One idea, which would allow us to remain flexible is to a make this new thing not only this new _protocol config_ but state in some sort of a header that _protocol config_ is a given range of addresses within the space on which what you called MISC_CONF operates on.
This multiplexing "action" ccw sounds like an interesting idea to explore. Maybe we only need that, and can integrate misc config or protocol config into that interface.
Do you have a proposal somewhere? I do remember the other email you mentioned it in, but I don't remember seeing anything akin to an interface specification.
Regards,
Halil
Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 85 lines · Halil Pasic
Quoted 66 lines · On Mon, 31 Jan 2022 18:10:39 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 57 lines · Halil Pasic
Quoted 44 lines · On Mon, 31 Jan 2022 09:52:54 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 9 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Quoted 6 lines
Maybe adding a register that tells us where does the "misc config start" is another option. I don't think we need an open ended device-config in practice. I have no idea if there are any upper limits on MMIO address space though. If we are constrained there, the switching is certainly more efficient. Otherwise, I think having the misc config somewhere after device specific config is simpler.
Quoted 8 lines
I think we first need to agree what the "misc" thing is actually supposed to be. My idea was that we don't have an unlimited supply of ccws to use for new features, so introducing one for reading "misc" configuration would be a way to keep things extensible (it also might make the config/register space for other transports less cluttered). The same idea (save on ccws) would apply to the multiplexing "action" ccw I mentioned in my other mail.
I agree with not wasting CCWs.
Quoted 4 lines
So, for the case here (simply relaying the location of the admin vq), we don't really need a "misc" mechanism for pci/mmio, but I'd like to introduce one for ccw. If we agree that it would be useful for pci/mmio as well, we should introduce it now.
Quoted 6 lines
Please see Michael's response. My understanding was also that what we want is something like config space for the virtio protocol stuff. The current config space is entirely device specific, so if we would want a common thing in
_configspace_, like_theindex of the administrationvq_then each device would have to define it separately, in a device specific location. Which is not nice.
Ok, so for ccw, it will be more of a secondary "config space", and we should reuse all the infrastructure that we already use for our normal "config space" (put into quotes because it isn't really a config space.)
Maybe we can call this "protocol config space"?
Quoted 11 lines
AFAIU having a this protocol config could be sufficient in the sense that we probably don't need another misc. My line of thinking is: with this we basically get a read-write interface for exposing stuff. The only other thing I can think of is
_transportspecificfields_. That is if we needed something that ain't specific to the device, but ain't common to all virtio (i.e. the virtio protocol). One idea, which would allow us to remain flexible is to a make this new thing not only this new_protocolconfig_but state in some sort of a header that_protocolconfig_is a given range of addresses within the space on which what you calledMISC_CONFoperates on.
Yeah, we can make transport config a subset of protocol config... possibly just a command + address setup, and the transport can multiplex on that?
Quoted 4 lines
This multiplexing "action" ccw sounds like an interesting idea to explore. Maybe we only need that, and can integrate misc config or protocol config into that interface.
It would probably be better to try and integrate it into the protocol config, if that's going to be read/write anyway. We can introduce like queue reset we're still missing on ccw into the transport config and already test out whether that makes actual sense.
Quoted 4 lines
Do you have a proposal somewhere? I do remember the other email you mentioned it in, but I don't remember seeing anything akin to an interface specification.
No, nothing formal. I was thinking of a ccw with a command/length/data payload, with successful conclusion of the I/O signifying successful triggering of the action (or maybe conclusion, depending on the command.) That might be more effective than the "config space" read/write dance, but I'm not sure whether it's worth deviating from the other transports.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 94 lines · Cornelia Huck
Quoted 85 lines · Halil Pasic
Quoted 66 lines · On Mon, 31 Jan 2022 18:10:39 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 57 lines · Halil Pasic
Quoted 44 lines · On Mon, 31 Jan 2022 09:52:54 -0500 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 43 lines · Cornelia Huck
Quoted 29 lines · Michael S. Tsirkin
Quoted 28 lines · Cornelia Huck
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Max Gurtovoy
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 3 lines
Halil, do you think that would work?
For mmio, I'd need to think a bit more. Any mmio experts around?
Not an expert but I think we can rely on a feature bit to be acked since admin vq is only needed after feature negotiation is complete.
You mean a register that is valid conditionally? I don't see an easy way to add some kind of "misc" interface for mmio, unlike for the other transports.
Quoted 9 lines
Quoted 5 lines
So something like:
AdminQueueIndex/0x0c4/R
IfVIRTIO_F_ADMIN_VQhas been negotiated, reading from this register returns the queue index of the administration virtqueue.No, I mean a register that switches 100+ between device specific and misc space.
Quoted 6 lines
Maybe adding a register that tells us where does the "misc config start" is another option. I don't think we need an open ended device-config in practice. I have no idea if there are any upper limits on MMIO address space though. If we are constrained there, the switching is certainly more efficient. Otherwise, I think having the misc config somewhere after device specific config is simpler.
Quoted 8 lines
I think we first need to agree what the "misc" thing is actually supposed to be. My idea was that we don't have an unlimited supply of ccws to use for new features, so introducing one for reading "misc" configuration would be a way to keep things extensible (it also might make the config/register space for other transports less cluttered). The same idea (save on ccws) would apply to the multiplexing "action" ccw I mentioned in my other mail.
I agree with not wasting CCWs.
Quoted 4 lines
So, for the case here (simply relaying the location of the admin vq), we don't really need a "misc" mechanism for pci/mmio, but I'd like to introduce one for ccw. If we agree that it would be useful for pci/mmio as well, we should introduce it now.
Quoted 6 lines
Please see Michael's response. My understanding was also that what we want is something like config space for the virtio protocol stuff. The current config space is entirely device specific, so if we would want a common thing in
_configspace_, like_theindex of the administrationvq_then each device would have to define it separately, in a device specific location. Which is not nice.
Quoted 5 lines
Ok, so for ccw, it will be more of a secondary "config space", and we should reuse all the infrastructure that we already use for our normal "config space" (put into quotes because it isn't really a config space.)
Maybe we can call this "protocol config space"?
Well config is not an english word ;)
We don't have many mentions of config space, those that are there should probably be converted to device configuration.
I am fine with protocol configuration if others like it, though I'm not 100% sure which "protocol" does this refer to, but there are instances in the spec where "protocol" means "virtio spec".
Quoted 12 lines
Quoted 11 lines
AFAIU having a this protocol config could be sufficient in the sense that we probably don't need another misc. My line of thinking is: with this we basically get a read-write interface for exposing stuff. The only other thing I can think of is
_transportspecificfields_. That is if we needed something that ain't specific to the device, but ain't common to all virtio (i.e. the virtio protocol). One idea, which would allow us to remain flexible is to a make this new thing not only this new_protocolconfig_but state in some sort of a header that_protocolconfig_is a given range of addresses within the space on which what you calledMISC_CONFoperates on.
Quoted 3 lines
Yeah, we can make transport config a subset of protocol config... possibly just a command + address setup, and the transport can multiplex on that?
OK but we need size too? I'd like to avoid a ton of exits if possible, people already complain about pci discovery causing a ton of exits, an exit per value is possible for pci after all.
Quoted 9 lines
Quoted 4 lines
This multiplexing "action" ccw sounds like an interesting idea to explore. Maybe we only need that, and can integrate misc config or protocol config into that interface.
It would probably be better to try and integrate it into the protocol config, if that's going to be read/write anyway. We can introduce like queue reset we're still missing on ccw into the transport config and already test out whether that makes actual sense.
The annoying thing is we added queue reset to pci already, ugh. There are two ways to do it for pci then? Oh well ...
Quoted 11 lines
Quoted 4 lines
Do you have a proposal somewhere? I do remember the other email you mentioned it in, but I don't remember seeing anything akin to an interface specification.
No, nothing formal. I was thinking of a ccw with a command/length/data payload, with successful conclusion of the I/O signifying successful triggering of the action (or maybe conclusion, depending on the command.) That might be more effective than the "config space" read/write dance, but I'm not sure whether it's worth deviating from the other transports.
Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Cornelia HuckQuoted 115 lines · On Mon, 31 Jan 2022 10:16:43 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueue is used to send administrative commands to manipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs of
+a parent PCI PF device are grouped together. These devices can be
+optionally managed by its parent PCI PF using its admin virtqueue.).
+
+Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82
Quoted 8 lines
struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Halil, do you think that would work?
I think so. But I would like to review the actual proposal :)
Some of the questions I have in mind are:
* Are fields in this config protected with feature bits
* Do we ever need to write this config? Adding a writable field to pci is easier than to ccw (no need to invent a new ccw)
* Looks like we will have an addressability mismatch with ccw compared to pci and mmio: ccw can only read a prefix of this new config, while mmio and pci have access to the individual fields
* Is this static stuff only? If not, do we need notifications for config change like for the device specific config?
Quoted 8 lines
For mmio, I'd need to think a bit more. Any mmio experts around?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 130 lines · Halil Pasic
Quoted 115 lines · On Mon, 31 Jan 2022 10:16:43 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 99 lines · Michael S. Tsirkin
Quoted 98 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
On Mon, Jan 24 2022, Max Gurtovoy <[email protected]> wrote:
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueue is used to send administrative commands to manipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs of
+a parent PCI PF device are grouped together. These devices can be
+optionally managed by its parent PCI PF using its admin virtqueue.).
+
+Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.
So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it
also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the
admin vq (if some use case comes along that will require all devices to
participate, for example?)I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10and
Quoted 7 lines
Quoted 5 lines
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
That seems ok for pci.
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82Quoted 8 lines
struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Halil, do you think that would work?
I think so. But I would like to review the actual proposal :)
Quoted 9 lines
Some of the questions I have in mind are:
* Are fields in this config protected with feature bits
* Do we ever need to write this config? Adding a writable field to pci is easier than to ccw (no need to invent a new ccw)
* Looks like we will have an addressability mismatch with ccw compared to pci and mmio: ccw can only read a prefix of this new config, while mmio and pci have access to the individual fields
* Is this static stuff only? If not, do we need notifications for config change like for the device specific config?
I'd just reuse the generic config change if we ever add something non-static.
Donnu about other questions.
Quoted 8 lines
Quoted 8 lines
For mmio, I'd need to think a bit more. Any mmio experts around?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Halil PasicQuoted 13 lines · Halil Pasic
Quoted 11 lines · On Mon, 31 Jan 2022 10:16:43 +0100 Cornelia Huck <[email protected]> wrote:
For ccw, I'd do something like
#define
CCW_CMD_READ_MISC_CONF0x82struct virtio_misc_conf { be16 admin_queue_index; };bound to revision 3, which gets a payload data containing the length of this structure (for future expansions).
Quoted 7 lines
Quoted 1 line
Halil, do you think that would work?
I think so. But I would like to review the actual proposal :)
Some of the questions I have in mind are:
* Are fields in this config protected with feature bits
I'd say they exist, as long as they are contained in the length provided via the revision data. Whether the data in the field is valid depends on feature bits.
Quoted 2 lines
* Do we ever need to write this config? Adding a writable field to pci is easier than to ccw (no need to invent a new ccw)
If we need that, we need to define a companion write command. If we already think we might need to write some fields in the future, we could introduce that command already right now.
Quoted 3 lines
* Looks like we will have an addressability mismatch with ccw compared to pci and mmio: ccw can only read a prefix of this new config, while mmio and pci have access to the individual fields
Yeah, that's basically like the config space access, which we're also modelling with reading up to the requested field. I'm not sure whether there's a better way to model this.
Quoted 2 lines
* Is this static stuff only? If not, do we need notifications for config change like for the device specific config?
This needs more thought :) To the others reading this thread: do you think "a mechanism for the device to make various extra configuration information available to the driver" (i.e. static and read-only) is enough, or do you think we need more? I think "initiate an action" like the queue reset is not neccessarily served best by an interface like this (I had thought about a multiplexing "initiate action" ccw for that purpose.) The different transports likely have different preferences here.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Michael S. TsirkinQuoted 95 lines · Michael S. Tsirkin
Quoted 94 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Review
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
(quoted diff) +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10
Quoted 8 lines
Quoted 7 lines
and
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
I wonder how much useful is this.
E.g for PCI we have an equation to calculate the queue notify address, if device choose to use dedicated notify for each queue it will probably end up with the last queue.
And I think the admin_queue_index should be stable regardless of the feature that has been negotiated?
Thanks
Quoted 16 lines
This is Cornelia's idea, we'll need her response.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 109 lines · Jason Wang
Quoted 95 lines · Michael S. Tsirkin
Quoted 94 lines · Max Gurtovoy
Quoted 85 lines · Michael S. Tsirkin
Quoted 67 lines · Max Gurtovoy
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Max Gurtovoy
Quoted 39 lines · Jason Wang
Quoted 38 lines · Michael S. Tsirkin
Quoted 27 lines · Cornelia Huck
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Cornelia Huck
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueue is used to send administrative commands to manipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs of
+a parent PCI PF device are grouped together. These devices can be
+optionally managed by its parent PCI PF using its admin virtqueue.).
+
+Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 6 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Or we can use index
0xFFFFfor admin virtqueue for compatibility.I think I'd prefer a register with the #. For example we might want to limit the # of VQs in order to pass extra data with the kick write.
So you are suggesting adding a new
cfg_type(#defineVIRTIO_PCI_CAP_ADMIN_CFG10) ?that will look something like:
struct
virtio_pci_admin_cfg{le32
queue_index; /* read only for the driver */le16
queue_size; /* read-write */
le16queue_msix_vector; /* read-write */ le16queue_enable; /* read-write */
le16queue_notify_off; /* read-only for driver */ le64queue_desc; /* read-write */
le64queue_driver; /* read-write */
le64queue_device; /* read-write */
le16queue_notify_data; /* read-only for driver */ le16queue_reset; /* read-write */Quoted 7 lines
};
instead of re-using the struct
virtio_pci_common_cfg?or do you prefer extending the struct
virtio_pci_common_cfgwith "le16admin_queue_index; /* read only for the driver */ ?The later. Other transports will need this too.
Cornelia has another idea which is that instead of adding just the admin queue register to all transports, we instead add a
misc_configstructure to all transports. Working basically like device specific config, but being device independent. For now it will only have a single le16admin_queue_indexregister.Quoted 4 lines
For PCI we would thus add it with
VIRTIO_PCI_CAP_MISC_CFGThe point here is that we are making it easier to add more fields just like admin queue index in the future.
OK.
#define
VIRTIO_PCI_CAP_MISC_CFG10Quoted 8 lines
Quoted 7 lines
and
struct
virtio_pci_misc_cfg{
le16admin_queue_index; /* read-only for driver */
};Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
We need to add it to MMIO and CCW I guess too.
I wonder how much useful is this.
Quoted 8 lines
E.g for PCI we have an equation to calculate the queue notify address, if device choose to use dedicated notify for each queue it will probably end up with the last queue.
And I think the
admin_queue_indexshould be stable regardless of the feature that has been negotiated?Thanks
there's only one last queue though. I wouldn't just use it and hope we don't need it for anything else.
Quoted 17 lines
Quoted 17 lines
This is Cornelia's idea, we'll need her response.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Thanks
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Jason WangQuoted 45 lines · Jason Wang
Quoted 40 lines · Michael S. Tsirkin
Quoted 30 lines · Cornelia Huck
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Cornelia Huck
admin-virtq.tex line 12
Quoted 12 lines · Max Gurtovoy
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
admin-virtq.tex new file +11
-0+ +Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedQuoted 7 lines
Quoted 3 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
I suspect yes. And that's one of the reasons why I'd rather we had a device independent way to locate the admin queue. There are less transports than device types.
Quoted 7 lines
So, do we want to bite the bullet now and simply say that every device type has the admin vq as the last vq if the feature is negotiated? Should be straightforward for the device types that have a fixed number of vqs, and doable for those that have a variable amount (two device types are covered by this series anyway.) I think we need to put it with the device types, as otherwise the numbering of virtqueues could change in unpredictable ways with the admin vq off/on.
Quoted 3 lines
Well that only works once. The next thing we'll need we won't be able to make the last one ;) So I am inclined to add a per-transport field that gives the admin queue number.
Quoted 5 lines
Technically, there's no need to use the same namespace for admin virtqueue if it has a dedicated notification area. If we go this way, we can simply use 0 as queue index for admin virtqueue.
Thanks
Problem is 0 and 1 use the same page for notification unless there's a page per vq (which is possible but expensive in memory space usage term).
Quoted 9 lines
Quoted 9 lines
Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if we want to allow access to regular vqs but not to the admin queue to guest) even if regular vqs share a page. Will help devices use less memory space.--
MST
[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
In reply to Cornelia HuckQuoted 54 lines · Cornelia Huck
Quoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +2
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} +admin-virtq.tex line 12
Quoted 10 lines
admin-virtq.tex new file +10
-0+Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiated
Quoted 8 lines
Do we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
Anyway, I'd reword the two sentences above:
"An admin virtqueue exists for a certain device if
VIRTIO_F_ADMIN_VQis negotiated. The index of the admin virtqueue is device type specific."
Ok. I'll use this in V3, unless we'll find some other mechanism that is not so clear now.
Quoted 7 lines
(...)
Quoted 4 lines
admin-virtq.tex new file +0
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
Quoted 5 lines
admin-virtq.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6853
Quoted 2 lines
admin-virtq.tex new file +2
-0+ \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation.Maybe
"This feature indicates that an administration virtqueue is supported." ?
Ok.
Quoted 5 lines
admin-virtq.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6857
Quoted 5 lines
admin-virtq.tex new file +1
-0+ \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Re: [PATCH v2 1/4] Add virtio Admin Virtqueue
Quoted 60 lines · Max Gurtovoy
Quoted 58 lines · Cornelia Huck
Quoted 37 lines · Max Gurtovoy
In one of the many use cases a user wants to manipulate features and configuration of the virtio devices regardless of the device type (
net/block/console). Some of this configuration is generic enough. i.e Number of MSI-X vectors of a virtio PCI VF device. There is a need to do such features query and manipulation by its parent PCI PF.Currently virtio specification defines control virtqueue to manipulate features and configuration of the device it operates on. However, control virtqueue commands are device type specific, which makes it very difficult to extend for device agnostic commands.
To support this requirement in elegant way, this patch introduces a new admin virtqueue. Admin virtqueue will use the same command format for all types of virtio devices.
Manipulate features via admin virtqueue is asynchronous, scalable, easy to extend and doesn't require additional and expensive on-die resources to be allocated for every new feature that will be added in the future.
Subsequent patches make use of this admin virtqueue.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ content.tex | 9 +++-- 2 files changed, 96 insertions(+), 2 deletions(-)create mode 100644 admin-virtq.tex
admin-virtq.tex new file +2
-0@@ -0,0 +1,89 @@ +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues} +admin-virtq.tex line 12
Quoted 10 lines
admin-virtq.tex new file +10
-0+Admin virtqueue is used to send administrative commands to manipulate +various features of the device and/or to manipulate various features, +if possible, of another device within the same group (e.g. PCI VFs of +a parent PCI PF device are grouped together. These devices can be +optionally managed by its parent PCI PF using its admin virtqueue.). + +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ +feature bit. + +Admin virtqueue index may vary among different device types.So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it also needs to specify where it shows up when the feature is negotiatedDo we expect that eventually all device types will need to support the admin vq (if some use case comes along that will require all devices to participate, for example?)
Quoted 7 lines
Quoted 4 lines
Anyway, I'd reword the two sentences above:
"An admin virtqueue exists for a certain device if
VIRTIO_F_ADMIN_VQis negotiated. The index of the admin virtqueue is device type specific."Ok. I'll use this in V3, unless we'll find some other mechanism that is not so clear now.
I suggest a transport specific register for the admin queue index. What do others think?
Quoted 19 lines
Quoted 11 lines
(...)
admin-virtq.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 6852
Quoted 5 lines
admin-virtq.tex new file +2
-0@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} that the driver can reset a queue individually. See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}. + \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that + the device supports administration virtqueue negotiation.Maybe
"This feature indicates that an administration virtqueue is supported." ?
Ok.
Quoted 4 lines
Quoted 4 lines
admin-virtq.tex new file +1
-0+ \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
PATCH v2 2/4 virtio-blk: add support for VIRTIO_F_ADMIN_VQ
In reply to Max GurtovoySet the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation.
--- content.tex | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
content.tex +10 -1
@@ -4517,10 +4517,19 @@ \subsection{Device ID}\label{sec:Device Types / Block Device / Device ID} 2 \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues} + if VIRTIO_F_ADMIN_VQ is not negotiated, the request queues layout is as follows: \begin{description} \item[0] requestq1 \item[\ldots] \item[N-1] requestqN +\end{description} + + If VIRTIO_F_ADMIN_VQ is negotiated, the queues layout is as follows: +\begin{description} +\item[0] requestq1 +\item[\ldots] +\item[N-1] requestqN +\item[N] adminq \end{description} N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
@@ -4589,7 +4598,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Block Device / bits as indicated above. The field \field{num_queues} only exists if VIRTIO_BLK_F_MQ is set. This field specifies -the number of queues. +the number of request queues. This field doesn't account admin virtqueue. The parameters in the configuration space of the device \field{max_discard_sectors} \field{discard_sector_alignment} are expressed in 512-byte units if the
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
PATCH v2 3/4 virtio-net: add support for VIRTIO_F_ADMIN_VQ
In reply to Max GurtovoySet the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation.
--- content.tex | 3 +++ 1 file changed, 3 insertions(+)
content.tex +3 -0
@@ -3033,6 +3033,7 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues} \item[2(N-1)] receiveqN \item[2(N-1)+1] transmitqN \item[2N] controlq +\item[2N + 1] adminq (or \textbf{2N} in case VIRTIO_NET_F_CTRL_VQ is not set) \end{description} N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
@@ -3040,6 +3041,8 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues} controlq only exists if VIRTIO_NET_F_CTRL_VQ set. + adminq only exists if VIRTIO_F_ADMIN_VQ set. + \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits} \begin{description}
Signature · 1 line
2.21.0
PATCH v2 4/4 Add support for MSI-X vectors configuration for PCI VFs
In reply to Max GurtovoyA typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a user requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way today in the spec to change the number of MSI-X vectors supported by a VF, although there are some operating systems that support this.
Introduce new admin commands for a generic interrupt vector management for PCI VFs. For now, this mechanism will manage the MSI-X interrupt vectors assignments of a VF by its parent PF.
These admin commands will be easily extended, if needed, for other types of interrupt vectors in the future with backward compatibility to old drivers and devices.
--- admin-virtq.tex | 150 +++++++++++++++++++++++++++++++++++++++++++++++- content.tex | 44 ++++++++++++++ 2 files changed, 192 insertions(+), 2 deletions(-)
admin-virtq.tex +148 -2
@@ -64,7 +64,13 @@ \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin \hline \hline 0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ \hline -0001h - 7FFFh & Generic admin cmds & - \\ +0001h & VIRTIO_ADMIN_PCI_SRIOV_ATTRS & O \\ +\hline +0002h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET & O \\ +\hline +0003h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET & O \\ +\hline +0004h - 7FFFh & Generic admin cmds & - \\ \hline 8000h - FFFFh & Reserved & - \\ \hline
@@ -78,7 +84,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a \begin{lstlisting} struct virtio_admin_caps_identify_result { /* - * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[0] - Bit 0x0 - if set, VF MSI-X control supported + * Bit 0x1 - Bit 0x7 are reserved * caps[1] - Bit 0x0 - Bit 0x7 are reserved * caps[2] - Bit 0x0 - Bit 0x7 are reserved * ....
@@ -87,3 +94,142 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a u8 caps[8192]; }; \end{lstlisting} + +For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}. + +\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} + +The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about PCI SRIOV +related capabilities and attributes for the device. This command can be supported only by +PCI devices that supports Single Root I/O Virtualization. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_sriov_attrs_result { + /* For compatibility - indicates which of the below fields are valid (1 means valid): + * Bit 0x0 - vfs_free_msix_count + * Bit 0x1 - per_vf_max_msix_count + * Bits 0x2 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* Number of free msix vectors in the global msix pool for VFs */ + le32 vfs_free_msix_count; + /* Max number of msix vectors that can be assigned for a single VF */ + le16 per_vf_max_msix_count; +}; +\end{lstlisting} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors +count for a PCI virtual function. The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_set_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * modified (1 means that field should be modified): + * Bit 0x0 - msix_count + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors */ + le16 msix_count; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED & MSI-X count exceed the max value per VF \\ +\hline +02h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC & MSI-X count exceed the free pool resources \\ +\hline +03h & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE & VF is already in use, operation failed \\ +\hline +04h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED +is returned when the amount of MSI-X to assign exceed the maximum value that can be +assigned to a single VF.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC +is returned when the amount of MSI-X to assign exceed the amount of free MSI-X +vectors in the global pool.} +\end{note} + +This command has no command specific result set by the device. Upon success, the device guarantees +that all the requested properties were modified to the given values. Otherwise, error will be returned. + +\begin{note} +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver. +Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.} +\end{note} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs +interrupt vectors configuration. +The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * queried (1 means that field should be queried): + * Bit 0x0 - msix_count (The amount of MSI-X interrupt vectors) + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +This command, upon success, returns a data buffer that describes the properties that were requested +and their values for the subject virtio VF device according to the given vf_number. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_result { + /* For compatibility - indicates which of the below fields were returned + * (1 means that field was returned): + * Bit 0x0 - msix_count + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors currently assigned to the VF */ + le16 msix_count; +}; +\end{lstlisting}
content.tex +44 -0
50 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 226 lines · Max Gurtovoy
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a user requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way today in the spec to change the number of MSI-X vectors supported by a VF, although there are some operating systems that support this.
Introduce new admin commands for a generic interrupt vector management for PCI VFs. For now, this mechanism will manage the MSI-X interrupt vectors assignments of a VF by its parent PF.
These admin commands will be easily extended, if needed, for other types of interrupt vectors in the future with backward compatibility to old drivers and devices.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 150 +++++++++++++++++++++++++++++++++++++++++++++++- content.tex | 44 ++++++++++++++ 2 files changed, 192 insertions(+), 2 deletions(-)admin-virtq.tex +148
-2@@ -64,7 +64,13 @@ \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin \hline \hline 0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ \hline -0001h - 7FFFh & Generic admin cmds & - \\ +0001h & VIRTIO_ADMIN_PCI_SRIOV_ATTRS & O \\ +\hline +0002h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET & O \\ +\hline +0003h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET & O \\ +\hline +0004h - 7FFFh & Generic admin cmds & - \\ \hline 8000h - FFFFh & Reserved & - \\ \hline@@ -78,7 +84,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a \begin{lstlisting} struct virtio_admin_caps_identify_result { /* - * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[0] - Bit 0x0 - if set, VF MSI-X control supported + * Bit 0x1 - Bit 0x7 are reserved * caps[1] - Bit 0x0 - Bit 0x7 are reserved * caps[2] - Bit 0x0 - Bit 0x7 are reserved * ....@@ -87,3 +94,142 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a u8 caps[8192]; }; \end{lstlisting} + +For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}. + +\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} + +The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about PCI SRIOV +related capabilities and attributes for the device. This command can be supported only by +PCI devices that supports Single Root I/O Virtualization. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_sriov_attrs_result { + /* For compatibility - indicates which of the below fields are valid (1 means valid): + * Bit 0x0 - vfs_free_msix_count + * Bit 0x1 - per_vf_max_msix_count + * Bits 0x2 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* Number of free msix vectors in the global msix pool for VFs */ + le32 vfs_free_msix_count; + /* Max number of msix vectors that can be assigned for a single VF */ + le16 per_vf_max_msix_count; +}; +\end{lstlisting} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors +count for a PCI virtual function. The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_set_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * modified (1 means that field should be modified): + * Bit 0x0 - msix_count + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors */ + le16 msix_count; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED & MSI-X count exceed the max value per VF \\ +\hline +02h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC & MSI-X count exceed the free pool resources \\ +\hline +03h & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE & VF is already in use, operation failed \\ +\hline +04h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED +is returned when the amount of MSI-X to assign exceed the maximum value that can be +assigned to a single VF.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC +is returned when the amount of MSI-X to assign exceed the amount of free MSI-X +vectors in the global pool.} +\end{note} + +This command has no command specific result set by the device. Upon success, the device guarantees +that all the requested properties were modified to the given values. Otherwise, error will be returned. + +\begin{note} +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver. +Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.} +\end{note} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs +interrupt vectors configuration. +The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * queried (1 means that field should be queried): + * Bit 0x0 - msix_count (The amount of MSI-X interrupt vectors) + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +This command, upon success, returns a data buffer that describes the properties that were requested +and their values for the subject virtio VF device according to the given vf_number. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_result { + /* For compatibility - indicates which of the below fields were returned + * (1 means that field was returned): + * Bit 0x0 - msix_count + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors currently assigned to the VF */ + le16 msix_count; +}; +\end{lstlisting}content.tex +23
-0@@ -1734,6 +1734,50 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options / \end{itemize} \end{itemize} +\subsection{PCI-specific Admin capabilities}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin capabilities} + +This documents the group of Admin capabilities for PCI virtio devices. Each capability is +implemented using one or more Admin queue commands. + +\subsubsection{VF MSI-X control}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control} + +This capability enables a virtio PCI PF device to control the assignment of MSI-X interrupt vectors +for its managed VFs. Capable devices will need to set bit 0x0 of caps[0] in the result of VIRTIO_ADMIN_CAPS_IDENTIFY +admin command. See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command} +for more details. + +Capable devices will also need to implement VIRTIO_ADMIN_PCI_SRIOV_ATTRS, VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and +VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin commands. + +In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a capable device should return the number of free msix vectors +in the global msix pool for its VFs in \field{vfs_free_msix_count} field and also the maximal number of msix vectors that +can be assigned for a single VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1 should be set +to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more +details. + +A PCI PF device may assign default number of MSI-X vectors to the
Quoted 1 line
enabled PCI VFs.
MAY SHIULD etc all need to be capitalized and moved to a conformance sections. Free text just using simple tense, e.g. "bit 0 is set".
So with a vague statement like this, how can the vectors be managed? I guess one basically has to start by querying all VFs?
content.tex \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
Quoted 5 lines
Also, using
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SETcontent.tex +4
-0+it is possible to update the default MSI-X assignment for a specific VF. +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, a driver set the virtual function number in +\field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}. +In addition, bit 0x0 in the \field{attrs_mask} field
this use of hex for bit numbers is confusing. it's ok to use them for values but bit numbers are clearer decimail imho.
content.tex \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options / · line 1768
Quoted 6 lines
should be set. A successful operation guarantees that the requested
content.tex +5
-0+amount of MSI-X interrupt vectors was assigned to the subject virtual function. +Also, a successful operation guarantees that the MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect +the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control +register should reflect a value of 7. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
Lifetime rules for these commands need to be discussed.
I am guessing this command is not allowed when driver
is accessing the VF?
VFs have lots of other resources besides MSI-X.
SRIOV is quite primitive in requiring knowing # of VFs enabled up front. If we are going to this length to manage resources per VF should we not start with a more basic thing, and allow enabling/disabling individual VFs?
All this looks kind of messy.
I think Jason's previous comment where he suggested enabling the VF with a single command begins to look more attractive.
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
And naturally driver is supposed to attach only to enabled VFs.
Quoted 5 lines
content.tex +2
-0+ +It is beyond the scope of the virtio specification to definenecessary synchronization in system software to ensure that a virtio PCI VF device +interrupt configuration modification is reflected in the PCI device.
I don't see why this is beyond the scope of the specification, seems quite pertinent. It's probably hard to do, so we can say that we don't, but let's not pretend we are doing anyone a favor here ;)
Quoted 16 lines
However, it is expected that any modern system
software implementing virtio +drivers and PCI subsystem should ensure that any changes occurring in the VF interrupt configuration is either updated in the PCI VF device or +such configuration fails.content.tex +7
-0+ +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin command, a driver should set the virtual function number in +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} field should be set to indicate requested output fields in +the result from the device. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently +assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit 0x0 in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details. + \section{Virtio Over MMIO}\label{sec:Virtio Transport Options / Virtio Over MMIO} Virtual environments without PCI support (a common situation inSignature · 1 line
2.21.0
This needs driver and device conformance sections.
RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 25, 2022 5:23 PMReview
Quoted 4 lines · Max Gurtovoy
(quoted diff) +4
-0+In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a +capable device should return the number of free msix vectors in the +global msix pool for its VFs in \field{vfs_free_msix_count} field and +also the maximal number of msix vectors that can be assigned for a singleVF in \field{per_vf_max_msix_count} field. In addition, bit
0x0and bit0x1should be set to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer.
Quoted 9 lines
Quoted 5 lines
(quoted diff) +4
-0+See section \ref{sec:Basic Facilities of a Virtio Device / Admin +Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more details. + +A PCI PF device may assign default number of MSI-X vectors to theenabled PCI VFs.
MAY SHIULD etc all need to be capitalized and moved to a conformance sections.
Above line is an informative detail. It is not related to conformance. Is current place better? Or?
Quoted 2 lines
Free text just using simple tense, e.g. "bit 0 is set".
Ok.
Quoted 2 lines
So with a vague statement like this, how can the vectors be managed? I guess one basically has to start by querying all VFs?
Need not be.
When deploying a VF, just pick a first free VF, assign the required vectors if not same, and use it. But yes, first time query can be done too get all VFs info.
Quoted 7 lines
Review
Quoted 5 lines
Also, using
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET(quoted diff) +3
-0+it is possible to update the default MSI-X assignment for a specific VF. +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin +command, a driver set the virtual function number in \field{vf_number} andthe amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
Quoted 5 lines
Quoted 1 line
+In addition, bit
0x0in the \field{attrs_mask} fieldthis use of hex for bit numbers is confusing. it's ok to use them for values but bit numbers are clearer decimail imho.
Ok.
Need to update in v3.
Quoted 10 lines
Quoted 2 lines
should be set. A successful operation guarantees that the requested +amount of MSI-X interrupt vectors was assigned to the subject virtual
function.
Review
Quoted 3 lines
(quoted diff) +3
-0+Also, a successful operation guarantees that the MSI-X capability +access by the subject PCI VF defined by the PCI specification must +reflect the new configuration in all relevant fields. For example, by default ifthe PCI VF has been assigned 4 MSI-X vectors, and
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SETincreases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control register should reflect a value of 7.
Quoted 4 lines
Quoted 1 line
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
Lifetime rules for these commands need to be discussed.
It is described below
Quoted 6 lines
I am guessing this command is not allowed when driver is accessing the VF? VFs have lots of other resources besides MSI-X.
SRIOV is quite primitive in requiring knowing # of VFs enabled up front. If we are going to this length to manage resources per VF should we not start with a more basic thing, and allow enabling/disabling individual VFs?
Sure, It is.
This is why we and others created scalable functions/subfunctions/adi to overcome some of these limitations over last 2-3 years. Even though SIOV exists in some form, it has own deployment limitations from platform side.
However, this primitive SRIOV technology is exposed by the virtio spec using VIRTIO_F_SR_IOV for a while and also in use by users. :) Here is an attempt to improve it.
Quoted 6 lines
All this looks kind of messy.
I think Jason's previous comment where he suggested enabling the VF with a single command begins to look more attractive.
The device vendor who actually implements it can propose it extension and plumbing in the spec.
I have explored this more than 2 years ago before upstreaming the mlx5 SF solution. And its fairly complicated beast to achieve in PCI subsystem. To our knowledge, there is no such device, PCI subsystem, and OS in existence that can actually do it.
And when that occurs, it will be able to make use of msix config command presented here on existing VFs. It will be able to use it without disabling the whole VF on redeploying the VF second time.
Quoted 3 lines
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
I wish best to those who will attempt this.
Devices that we experience are able to handle SR-IOV scale for several hundreds of VFs without need to undergo enable-disable individual VFs.
Quoted 11 lines
And naturally driver is supposed to attach only to enabled VFs.
Quoted 5 lines
(quoted diff) +2
-0+ +It is beyond the scope of the virtio specification to definenecessary synchronization in system software to ensure that a virtio PCI VF device +interrupt configuration modification is reflected in the PCI device.
I don't see why this is beyond the scope of the specification, seems quite pertinent.
It is beyond the scope of specification, because virtio specification do not define what all system software modules to do. It is very OS specific.
virtio specification doesn't know what those modules are either.
It can only provide the guidance. Cannot define how/what to do.
And this hasn't changed from our discussion in v1.
Quoted 20 lines
Review
Quoted 9 lines
However, it is expected that any modern system software implementing virtio +drivers and PCI subsystem should ensure that any changes occurring in the VF interrupt configuration is either updated in the PCI VF device or +such configuration fails.
(quoted diff) +5
-0+ +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin +command, a driver should set the virtual function number in +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} +field should be set to indicate requested output fields in the result from thedevice. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit
0x0in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field.Quoted 1 line
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
Quoted 4 lines
+
\section{Virtio Over MMIO}\label{sec:Virtio Transport Options /
Virtio Over MMIO}
Quoted 5 lines
Quoted 3 lines
Virtual environments without PCI support (a common situation in
--
2.21.0This needs driver and device conformance sections.
Ok. v3 to cover this too.
[virtio-comment] Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 22 lines · Parav Pandit
Quoted 13 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 25, 2022 5:23 PMReview
Quoted 4 lines · Max Gurtovoy
(quoted diff) +4
-0+In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a +capable device should return the number of free msix vectors in the +global msix pool for its VFs in \field{vfs_free_msix_count} field and +also the maximal number of msix vectors that can be assigned for a singleVF in \field{per_vf_max_msix_count} field. In addition, bit
0x0and bit0x1should be set to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer.Quoted 9 lines
Quoted 5 lines
(quoted diff) +4
-0+See section \ref{sec:Basic Facilities of a Virtio Device / Admin +Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more details. + +A PCI PF device may assign default number of MSI-X vectors to theenabled PCI VFs.
MAY SHIULD etc all need to be capitalized and moved to a conformance sections.
Quoted 2 lines
Above line is an informative detail. It is not related to conformance. Is current place better? Or?
the way we do it generally is this. write an informative text avoiding use of MAY/SHOULD etc. Just describe the way it works. Add conformance statements somewhat repeating the informative text using MAY/SHOULD/MUST etc.
Quoted 7 lines
Quoted 2 lines
Free text just using simple tense, e.g. "bit 0 is set".
Ok.
Quoted 2 lines
So with a vague statement like this, how can the vectors be managed? I guess one basically has to start by querying all VFs?
Need not be.
When deploying a VF, just pick a first free VF, assign the required vectors
management tools need to balance requirements versus availability
Quoted 1 line
if not same,
same as what?
Quoted 28 lines
and use it.
But yes, first time query can be done too get all VFs info.Quoted 7 lines
Review
Quoted 5 lines
Also, using
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET(quoted diff) +3
-0+it is possible to update the default MSI-X assignment for a specific VF. +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin +command, a driver set the virtual function number in \field{vf_number} andthe amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
Quoted 5 lines
Quoted 1 line
+In addition, bit
0x0in the \field{attrs_mask} fieldthis use of hex for bit numbers is confusing. it's ok to use them for values but bit numbers are clearer decimail imho.
Ok.
Need to update in v3.Quoted 10 lines
Quoted 2 lines
should be set. A successful operation guarantees that the requested +amount of MSI-X interrupt vectors was assigned to the subject virtual
function.
Review
Quoted 3 lines
(quoted diff) +3
-0+Also, a successful operation guarantees that the MSI-X capability +access by the subject PCI VF defined by the PCI specification must +reflect the new configuration in all relevant fields. For example, by default ifthe PCI VF has been assigned 4 MSI-X vectors, and
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SETincreases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control register should reflect a value of 7.
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
Lifetime rules for these commands need to be discussed.
It is described below
Couldn't find it sorry. Adding a reference might be helpful.
Quoted 6 lines
Quoted 6 lines
I am guessing this command is not allowed when driver is accessing the VF? VFs have lots of other resources besides MSI-X.
SRIOV is quite primitive in requiring knowing # of VFs enabled up front. If we are going to this length to manage resources per VF should we not start with a more basic thing, and allow enabling/disabling individual VFs?
Quoted 6 lines
Sure, It is.
This is why we and others created scalable functions/subfunctions/adi to overcome some of these limitations over last 2-3 years. Even though SIOV exists in some form, it has own deployment limitations from platform side.However, this primitive SRIOV technology is exposed by the virtio spec using
VIRTIO_F_SR_IOVfor a while and also in use by users. :) Here is an attempt to improve it.
Right. So how about enabling/disabling individual VFs then?
Quoted 6 lines
Quoted 6 lines
All this looks kind of messy.
I think Jason's previous comment where he suggested enabling the VF with a single command begins to look more attractive.
Quoted 6 lines
The device vendor who actually implements it can propose it extension and plumbing in the spec.
I have explored this more than 2 years ago before upstreaming the mlx5 SF solution. And its fairly complicated beast to achieve in PCI subsystem. To our knowledge, there is no such device, PCI subsystem, and OS in existence that can actually do it.And when that occurs, it will be able to make use of msix config command presented here on existing VFs. It will be able to use it without disabling the whole VF on redeploying the VF second time.
I'm not sure we are speaking about the same thing.
The way you propose is each VF gets some "default" config and then the commands can be used to tweak that. Which seems pretty complex for management software if it ever does need to tweak things.
And alternative is to declare that VF is not set up automatically, and ask driver to set it up before use.
It's strictly less work for the device so I find it hard to believe it's impossible to implement.
Quoted 4 lines
Quoted 3 lines
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
I wish best to those who will attempt this.
It's possible that you will end up being that person, we need to have the interface that's generic and clear enough even if your hardware does not benefit.
Quoted 1 line
Devices that we experience are able to handle SR-IOV scale for several hundreds of VFs without need to undergo enable-disable individual VFs.
OK... and is it true that at the same time they do not support enough MSI-X vectors per VF such that we need complex machinery to control them?
What is the motivation for this proposal again?
Quoted 12 lines
Quoted 11 lines
And naturally driver is supposed to attach only to enabled VFs.
Quoted 5 lines
(quoted diff) +2
-0+ +It is beyond the scope of the virtio specification to definenecessary synchronization in system software to ensure that a virtio PCI VF device +interrupt configuration modification is reflected in the PCI device.
I don't see why this is beyond the scope of the specification, seems quite pertinent.
Quoted 6 lines
It is beyond the scope of specification, because virtio specification do not define what all system software modules to do. It is very OS specific.
virtio specification doesn't know what those modules are either.
It can only provide the guidance. Cannot define how/what to do.And this hasn't changed from our discussion in v1.
Well we define a bunch of PCI things like BARs, device ID etc. How is this one different? I think basically it's a lot of work and we do not think it's srictly necessary to define it, but it's not because of scope.
Quoted 20 lines
Quoted 20 lines
Review
Quoted 9 lines
However, it is expected that any modern system software implementing virtio +drivers and PCI subsystem should ensure that any changes occurring in the VF interrupt configuration is either updated in the PCI VF device or +such configuration fails.
(quoted diff) +5
-0+ +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin +command, a driver should set the virtual function number in +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} +field should be set to indicate requested output fields in the result from thedevice. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit
0x0in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field.Quoted 1 line
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
Quoted 4 lines
+
\section{Virtio Over MMIO}\label{sec:Virtio Transport Options /
Virtio Over MMIO}
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Virtual environments without PCI support (a common situation in
--
2.21.0This needs driver and device conformance sections.
Ok. v3 to cover this too.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, January 26, 2022 7:39 PMQuoted 4 lines
Quoted 4 lines
Quoted 2 lines
+A PCI PF device may assign default number of MSI-X vectors to the enabled PCI VFs.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
MAY SHIULD etc all need to be capitalized and moved to a conformance sections.
Above line is an informative detail. It is not related to conformance. Is current place better? Or?
the way we do it generally is this. write an informative text avoiding use of MAY/SHOULD etc.
Ok.
Quoted 4 lines
Just describe the way it works. Add conformance statements somewhat repeating the informative text using MAY/SHOULD/MUST etc.
Got it. Will do in v3.
Quoted 10 lines
Quoted 9 lines
Quoted 3 lines
Free text just using simple tense, e.g. "bit 0 is set".
Ok.
Quoted 2 lines
So with a vague statement like this, how can the vectors be managed? I guess one basically has to start by querying all VFs?
Need not be.
When deploying a VF, just pick a first free VF, assign the required vectors
Quoted 6 lines
management tools need to balance requirements versus availability
Quoted 1 line
if not same,
same as what?
Basically the example is,
If say all the VFs has default of 4 vectors, and if system admin wants to assign 8 vectors to a VF, It issues msix_config command for a VF A to change from 4 to 8, and uses the VF.
If the system admin wants to assign 4 vectors to a VF, the current value in VF and desired value are same. So no need to assign.
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
Lifetime rules for these commands need to be discussed.
It is described below
Below is the snippet from the patch. Please let us know if its missing something for lifetime rules.
Quoted 1 line
Couldn't find it sorry. Adding a reference might be helpful.
A PCI PF device may assign default number of MSI-X vectors to the
+enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
+it is possible to update the default MSI-X assignment for a specific VF.
+In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command,
+a driver set the virtual function number in \field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
+In addition, bit 0x0 in the \field{attrs_mask} field should be set. A
+successful operation guarantees that the requested amount of MSI-X interrupt vectors was assigned to the subject virtual function.
+Also, a successful operation guarantees that the MSI-X capability
+access by the subject PCI VF defined by the PCI specification must
+reflect the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control register should reflect a value of 7. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
+
Quoted 21 lines
Quoted 2 lines
However, this primitive SRIOV technology is exposed by the virtio spec using
VIRTIO_F_SR_IOVfor a while and also in use by users. :) Here is anattempt to improve it.
Right. So how about enabling/disabling individual VFs then?
Quoted 7 lines
Quoted 6 lines
All this looks kind of messy.
I think Jason's previous comment where he suggested enabling the VF with a single command begins to look more attractive.
The device vendor who actually implements it can propose it extension and
plumbing in the spec.
Quoted 1 line
I have explored this more than 2 years ago before upstreaming the mlx5 SF
solution. And its fairly complicated beast to achieve in PCI subsystem.
Quoted 1 line
To our knowledge, there is no such device, PCI subsystem, and OS in
existence that can actually do it.
Quoted 2 lines
And when that occurs, it will be able to make use of msix config command
Quoted 9 lines
presented here on existing VFs.
Quoted 1 line
It will be able to use it without disabling the whole VF on redeploying the VF
second time.
I'm not sure we are speaking about the same thing.
The way you propose is each VF gets some "default" config and then the commands can be used to tweak that. Which seems pretty complex for management software if it ever does need to tweak things.
A default configured vector can be zero, or device may have given some default value.
It is like a netdevice comes up default with some X queues and if user wants to modify it, it does ethtool and use the new value. It is not that complex.
Quoted 3 lines
And alternative is to declare that VF is not set up automatically, and ask driver to set it up before use.
Yes. This is equally good idea.
May be a pci_msix_config on PF device will be good idea indicating that, always have all the VFs with zero default vector. And user will setup the vector before giving the VF to the driver.
Quoted 3 lines
It's strictly less work for the device so I find it hard to believe it's impossible to implement.
It is less work. This is fine.
Enabling/disabling individual VF is a completely different mechanics.
Quoted 10 lines
Quoted 5 lines
Quoted 4 lines
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
I wish best to those who will attempt this.
It's possible that you will end up being that person, we need to have the interface that's generic and clear enough even if your hardware does not benefit.
Quoted 6 lines
Quoted 1 line
Devices that we experience are able to handle SR-IOV scale for several
hundreds of VFs without need to undergo enable-disable individual VFs.
OK... and is it true that at the same time they do not support enough MSI-X vectors per VF such that we need complex machinery to control them?
Yes.
Quoted 2 lines
What is the motivation for this proposal again?
Efficiently distribute device resources among large number of VFs.
Quoted 15 lines
Quoted 13 lines
Quoted 4 lines
And naturally driver is supposed to attach only to enabled VFs.
Quoted 8 lines
Quoted 5 lines
(quoted diff) +2
-0+ +It is beyond the scope of the virtio specification to definenecessary synchronization in system software to ensure that a virtio PCI VF device +interrupt configuration modification is reflected in the PCI device.
I don't see why this is beyond the scope of the specification, seems quite pertinent.
Quoted 1 line
It is beyond the scope of specification, because virtio specification do not
define what all system software modules to do.
Quoted 8 lines
Quoted 5 lines
It is very OS specific.
virtio specification doesn't know what those modules are either. It can only provide the guidance. Cannot define how/what to do.And this hasn't changed from our discussion in v1.
Well we define a bunch of PCI things like BARs, device ID etc. How is this one different?
It is different in following way.
When to pci capabilities, where to store in driver structure, when to update them, when to mark those capabilities dirty and re-read it. All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can give guidance of what to expect so that OS can implement above for msix config update.
Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 36 lines · Parav Pandit
Quoted 8 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, January 26, 2022 7:39 PMQuoted 4 lines
Quoted 4 lines
Quoted 2 lines
+A PCI PF device may assign default number of MSI-X vectors to the enabled PCI VFs.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
MAY SHIULD etc all need to be capitalized and moved to a conformance sections.
Above line is an informative detail. It is not related to conformance. Is current place better? Or?
the way we do it generally is this. write an informative text avoiding use of MAY/SHOULD etc.
Ok.
Quoted 4 lines
Just describe the way it works. Add conformance statements somewhat repeating the informative text using MAY/SHOULD/MUST etc.
Got it. Will do in v3.
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
Free text just using simple tense, e.g. "bit 0 is set".
Ok.
Quoted 2 lines
So with a vague statement like this, how can the vectors be managed? I guess one basically has to start by querying all VFs?
Need not be.
When deploying a VF, just pick a first free VF, assign the required vectorsmanagement tools need to balance requirements versus availability
Quoted 1 line
if not same,
Quoted 8 lines
Quoted 2 lines
same as what?
Basically the example is,
If say all the VFs has default of 4 vectors, and if system admin wants to assign 8 vectors to a VF, It issuesmsix_configcommand for a VF A to change from 4 to 8, and uses the VF.If the system admin wants to assign 4 vectors to a VF, the current value in VF and desired value are same. So no need to assign.
But there's no way to know how many vectors there are by default, or even whether the default is same for all VFs, without first issuing a ton of commands.
Quoted 11 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
Lifetime rules for these commands need to be discussed.
It is described below
Below is the snippet from the patch. Please let us know if its missing something for lifetime rules.
Quoted 1 line
Couldn't find it sorry. Adding a reference might be helpful.
A PCI PF device may assign default number of MSI-X vectors to the
(quoted diff) +1
-0+enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
Review
Quoted 10 lines
(quoted diff) +10
-0+it is possible to update the default MSI-X assignment for a specific VF. +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, +a driver set the virtual function number in \field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}. +In addition, bit 0x0 in the \field{attrs_mask} field should be set. A +successful operation guarantees that the requested amount of MSI-X interrupt vectors was assigned to the subject virtual function. +Also, a successful operation guarantees that the MSI-X capability +access by the subject PCI VF defined by the PCI specification must +reflect the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control register should reflect a value of 7. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details. +
Nothing here specifies the lifetime, e.g. nothing precludes tweaking the number while driver is using the VF. Which will just be a mess to support.
Quoted 30 lines
Quoted 21 lines
Quoted 2 lines
However, this primitive SRIOV technology is exposed by the virtio spec using
VIRTIO_F_SR_IOVfor a while and also in use by users. :) Here is anattempt to improve it.
Right. So how about enabling/disabling individual VFs then?
Quoted 7 lines
Quoted 6 lines
All this looks kind of messy.
I think Jason's previous comment where he suggested enabling the VF with a single command begins to look more attractive.
The device vendor who actually implements it can propose it extension and
plumbing in the spec.
Quoted 1 line
I have explored this more than 2 years ago before upstreaming the mlx5 SF
solution. And its fairly complicated beast to achieve in PCI subsystem.
Quoted 1 line
To our knowledge, there is no such device, PCI subsystem, and OS in
existence that can actually do it.
Quoted 2 lines
And when that occurs, it will be able to make use of msix config command
Quoted 9 lines
presented here on existing VFs.
Quoted 1 line
It will be able to use it without disabling the whole VF on redeploying the VF
second time.
I'm not sure we are speaking about the same thing.
The way you propose is each VF gets some "default" config and then the commands can be used to tweak that. Which seems pretty complex for management software if it ever does need to tweak things.
Quoted 4 lines
A default configured vector can be zero, or device may have given some default value.
It is like a netdevice comes up default with some X queues and if user wants to modify it, it does ethtool and use the new value. It is not that complex.
A ton of commands will be needed to first query all VFs, then configure them all the way management wants to.
Quoted 7 lines
Quoted 3 lines
And alternative is to declare that VF is not set up automatically, and ask driver to set it up before use.
Yes. This is equally good idea.
May be apci_msix_configon PF device will be good idea indicating that, always have all the VFs with zero default vector. And user will setup the vector before giving the VF to the driver.
Quoted 5 lines
Quoted 3 lines
It's strictly less work for the device so I find it hard to believe it's impossible to implement.
It is less work. This is fine.
Enabling/disabling individual VF is a completely different mechanics.
Well. you seem to already allow allocating 0 vectors to a VF, which will break most drivers effectively disabling it. I just this it's best to formalize this.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Quoted 4 lines
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
I wish best to those who will attempt this.
It's possible that you will end up being that person, we need to have the interface that's generic and clear enough even if your hardware does not benefit.
Quoted 1 line
Devices that we experience are able to handle SR-IOV scale for several
hundreds of VFs without need to undergo enable-disable individual VFs.
Quoted 8 lines
Quoted 3 lines
OK... and is it true that at the same time they do not support enough MSI-X vectors per VF such that we need complex machinery to control them?
Yes.
Quoted 2 lines
What is the motivation for this proposal again?
Efficiently distribute device resources among large number of VFs.
So what's the total # of vectors supported and what's the # of VFs then?
Quoted 19 lines
Quoted 19 lines
Quoted 13 lines
Quoted 4 lines
And naturally driver is supposed to attach only to enabled VFs.
Quoted 8 lines
Quoted 5 lines
(quoted diff) +2
-0+ +It is beyond the scope of the virtio specification to definenecessary synchronization in system software to ensure that a virtio PCI VF device +interrupt configuration modification is reflected in the PCI device.
I don't see why this is beyond the scope of the specification, seems quite pertinent.
Quoted 1 line
It is beyond the scope of specification, because virtio specification do not
define what all system software modules to do.
Quoted 4 lines
It is very OS specific.
virtio specification doesn't know what those modules are either. It can only provide the guidance. Cannot define how/what to do.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
And this hasn't changed from our discussion in v1.
Well we define a bunch of PCI things like BARs, device ID etc. How is this one different?
It is different in following way.
When to pci capabilities, where to store in driver structure, when to update them, when to mark those capabilities dirty and re-read it. All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can give guidance of what to expect so that OS can implement above for msix config update.
All I'm saying it let's drop the "beyond the scope" thing and in fact this whole paragraph does not seem to add anything useful to the spec.
Maybe you are trying to say that the driver or systems software needs to do some kind of synchronization - is that it? It is not clear from this description what are you talking about, what is synched with what, or
what does "reflected in the PCI device" mean.
Signature · 1 line
MST
RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, January 27, 2022 6:10 PMQuoted 1 line
If the system admin wants to assign 4 vectors to a VF, the current value in VF
and desired value are same.
Quoted 1 line
So no need to assign.
But there's no way to know how many vectors there are by default, or even whether the default is same for all VFs, without first issuing a ton of commands.
Either query all and build database (likely sane thing to do) or Query the first free VF, configure the value to it and use it.
Quoted 3 lines
Nothing here specifies the lifetime, e.g. nothing precludes tweaking the number while driver is using the VF. Which will just be a mess to support.
It is described in the v2.
+{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.
Above should be changed to "must not be used by the driver."
Quoted 6 lines
Quoted 1 line
It is like a netdevice comes up default with some X queues and if user wants
to modify it, it does ethtool and use the new value.
Quoted 1 line
It is not that complex.
A ton of commands will be needed to first query all VFs, then configure them all the way management wants to.
So, it that a problem? To configure some value, query is needed.
If user doesn't want to issue too many commands it can pick the free VF and configure it.
Other option is below, where HV says, ah I don't like to query all the VFs. I want to manage each VF individually.
So HV does one command even before enabling SR-IOV to give 0 vectors by default to each VF. So sequence will be like,
1. reset all vf vectors, count = 0.
2. query total vectors of the device
3. enable sriov (num vfs = N)
3. assign #A vectors to VF-A.
4. assign VF to VM
5. use VM
6. release VM and VF
7. assign #B vectors to VF-A.
8. Repeat 5 to 6
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
And alternative is to declare that VF is not set up automatically, and ask driver to set it up before use.
Yes. This is equally good idea.
May be apci_msix_configon PF device will be good idea indicating that,always have all the VFs with zero default vector.
Quoted 2 lines
And user will setup the vector before giving the VF to the driver.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
It's strictly less work for the device so I find it hard to believe it's impossible to implement.
It is less work. This is fine.
Enabling/disabling individual VF is a completely different mechanics.Well. you seem to already allow allocating 0 vectors to a VF, which will break most drivers effectively disabling it.
Allocating 0 vectors is not to break driver. Not sure why you use the term break.
Quoted 1 line
I just this it's best to formalize this.
I liked the idea of above 8 steps example, this will be efficient for the user.
Quoted 17 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
I wish best to those who will attempt this.
It's possible that you will end up being that person, we need to have the interface that's generic and clear enough even if your hardware does not benefit.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Devices that we experience are able to handle SR-IOV scale for several
hundreds of VFs without need to undergo enable-disable individual VFs.
OK... and is it true that at the same time they do not support enough MSI-X vectors per VF such that we need complex machinery to
control them?
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Yes.
Quoted 2 lines
What is the motivation for this proposal again?
Efficiently distribute device resources among large number of VFs.
So what's the total # of vectors supported and what's the # of VFs then?
One example, 256 VFs. total of 2K MSI-X vectors among these 256 VFs.
Quoted 9 lines
Quoted 1 line
When to pci capabilities, where to store in driver structure, when to update
them, when to mark those capabilities dirty and re-read it.
Quoted 1 line
All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can
give guidance of what to expect so that OS can implement above for msix config update.
All I'm saying it let's drop the "beyond the scope" thing and in fact this whole paragraph does not seem to add anything useful to the spec.
Oh, you suggested to describe this details in v1 in the spec.
Quoted 2 lines
Maybe you are trying to say that the driver or systems software needs to do some kind of synchronization - is that it?
Yes.
Quoted 2 lines
It is not clear from this description what are you talking about, what is synched with what, or what does "reflected in the PCI device" mean.
Below is what is written in proposed patch:
"MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect the new configuration in all relevant fields."
All the configuration relevant to the MSI-X should reflect in necessary registers defined by PCI spec.
Probably it should be written as "All MSI-X related registers, capabilities and fields must reflect the new configuration". Is this better?
Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 4 lines · Parav Pandit
Quoted 3 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, January 27, 2022 6:10 PM
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
If the system admin wants to assign 4 vectors to a VF, the current value in VF
and desired value are same.
Quoted 1 line
So no need to assign.
But there's no way to know how many vectors there are by default, or even whether the default is same for all VFs, without first issuing a ton of commands.
Either query all and build database (likely sane thing to do) or Query the first free VF, configure the value to it and use it.
And not only that. Let's say you have 256 vectors split between 256 VFs equally. If you want to increase vectors for VF0, you need to start by reducing vectors for other VFs.
It's really a ton of round trips to the card just for startup.
Quoted 9 lines
Quoted 3 lines
Nothing here specifies the lifetime, e.g. nothing precludes tweaking the number while driver is using the VF. Which will just be a mess to support.
It is described in the v2.
+{Before setting
msix_countproperty the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.Above should be changed to "must not be used by the driver."
ok so there's an implicit uninitialized state.
This is why I felt that instead, we should make it implicit calling the command INITIALIZE, with flags specifying what is being initialized.
And an UINITIALIZE command for symmetry.
Alternatively - VFs should start in
a known state, with 0 vectors, and uninitialized - driver can not bind. Following this, we'd do commands to assign resources to the VFs. Finally, we make the VF initialized - after this point driver can bind, but configuration can no longer be changed.
It looks like in your proposal, just assigning msix count makes VF initialized, so this is implicit. One also can not put VF back in uninitialized state which seems asymmetrical.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
It is like a netdevice comes up default with some X queues and if user wants
to modify it, it does ethtool and use the new value.
Quoted 1 line
It is not that complex.
A ton of commands will be needed to first query all VFs, then configure them all the way management wants to.
So, it that a problem? To configure some value, query is needed.
yes because it's not even one command. It can be like 200 commands. That is excessive.
Quoted 11 lines
If user doesn't want to issue too many commands it can pick the free VF and configure it.
Other option is below, where HV says, ah I don't like to query all the VFs. I want to manage each VF individually.
So HV does one command even before enabling SR-IOV to give 0 vectors by default to each VF.So sequence will be like,
1. reset all vf vectors, count = 0.
This one small step is like 200 commands though.
Quoted 21 lines
2. query total vectors of the device
3. enable sriov (num vfs = N)
3. assign #A vectors to VF-A.
4. assign VF to VM
5. use VM
6. release VM and VF
7. assign #B vectors to VF-A.
8. Repeat 5 to 6Quoted 12 lines
Quoted 6 lines
Quoted 4 lines
And alternative is to declare that VF is not set up automatically, and ask driver to set it up before use.
Yes. This is equally good idea.
May be apci_msix_configon PF device will be good idea indicating that,always have all the VFs with zero default vector.
Quoted 5 lines
And user will setup the vector before giving the VF to the driver.
Quoted 3 lines
It's strictly less work for the device so I find it hard to believe it's impossible to implement.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
It is less work. This is fine.
Enabling/disabling individual VF is a completely different mechanics.Well. you seem to already allow allocating 0 vectors to a VF, which will break most drivers effectively disabling it.
Allocating 0 vectors is not to break driver. Not sure why you use the term break.
just that most drivers won't work if device can not drive any interrupt.
Quoted 2 lines
Quoted 1 line
I just this it's best to formalize this.
I liked the idea of above 8 steps example, this will be efficient for the user.
step 1 is problematic though.
Quoted 19 lines
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 3 lines
IOW we would have two commands, enable and disable, an enabled VFs has to be supplied with all of its configuration.
I wish best to those who will attempt this.
It's possible that you will end up being that person, we need to have the interface that's generic and clear enough even if your hardware does not benefit.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Devices that we experience are able to handle SR-IOV scale for several
hundreds of VFs without need to undergo enable-disable individual VFs.
OK... and is it true that at the same time they do not support enough MSI-X vectors per VF such that we need complex machinery to
control them?
Quoted 3 lines
Quoted 1 line
Yes.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
What is the motivation for this proposal again?
Efficiently distribute device resources among large number of VFs.
So what's the total # of vectors supported and what's the # of VFs then?
One example, 256 VFs. total of 2K MSI-X vectors among these 256 VFs.
OK so 128 per VF by default. So pls think of a set of commands where we can manage most reasonable tasks without resorting to hundreds of commands.
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
When to pci capabilities, where to store in driver structure, when to update
them, when to mark those capabilities dirty and re-read it.
Quoted 1 line
All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can
give guidance of what to expect so that OS can implement above for msix config update.
All I'm saying it let's drop the "beyond the scope" thing and in fact this whole paragraph does not seem to add anything useful to the spec.
Oh, you suggested to describe this details in v1 in the spec.
Wall this does is try to say that you won't though.
Quoted 10 lines
Quoted 2 lines
Maybe you are trying to say that the driver or systems software needs to do some kind of synchronization - is that it?
Yes.
Quoted 2 lines
It is not clear from this description what are you talking about, what is synched with what, or what does "reflected in the PCI device" mean.
Below is what is written in proposed patch:
"MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect the new configuration in all relevant fields."
MUST I guess and do the conformance paragraph dance then.
and I think you mean "after setting the msix_count field" right?
otherwise this is ok, but I was referring to the beyond scope part, not to this sentence at all.
Quoted 5 lines
All the configuration relevant to the MSI-X should reflect in necessary registers defined by PCI spec.
Probably it should be written as "All MSI-X related registers, capabilities and fields must reflect the new configuration". Is this better?
no, that sentence was mostly ok.
I think this thread has outlived its usefulness.
Please just post a new version and we can restart as at this point it's no longer clear who is responding to what.
Signature · 1 line
MST
Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
In reply to Max GurtovoyQuoted 53 lines · Max Gurtovoy
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a user requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way today in the spec to change the number of MSI-X vectors supported by a VF, although there are some operating systems that support this.
Introduce new admin commands for a generic interrupt vector management for PCI VFs. For now, this mechanism will manage the MSI-X interrupt vectors assignments of a VF by its parent PF.
These admin commands will be easily extended, if needed, for other types of interrupt vectors in the future with backward compatibility to old drivers and devices.
Reviewed-by Parav Pandit <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- admin-virtq.tex | 150 +++++++++++++++++++++++++++++++++++++++++++++++- content.tex | 44 ++++++++++++++ 2 files changed, 192 insertions(+), 2 deletions(-)admin-virtq.tex +9
-2@@ -64,7 +64,13 @@ \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin \hline \hline 0000h & VIRTIO_ADMIN_CAPS_IDENTIFY & M \\ \hline -0001h - 7FFFh & Generic admin cmds & - \\ +0001h & VIRTIO_ADMIN_PCI_SRIOV_ATTRS & O \\ +\hline +0002h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET & O \\ +\hline +0003h & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET & O \\ +\hline +0004h - 7FFFh & Generic admin cmds & - \\ \hline 8000h - FFFFh & Reserved & - \\ \hline@@ -78,7 +84,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a \begin{lstlisting} struct virtio_admin_caps_identify_result { /* - * caps[0] - Bit 0x0 - Bit 0x7 are reserved + * caps[0] - Bit 0x0 - if set, VF MSI-X control supported + * Bit 0x1 - Bit 0x7 are reserved * caps[1] - Bit 0x0 - Bit 0x7 are reserved * caps[2] - Bit 0x0 - Bit 0x7 are reserved * ....
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 101
Quoted 8 lines
admin-virtq.tex +5
-0@@ -87,3 +94,142 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a u8 caps[8192]; }; \end{lstlisting} + +For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}. + +\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} +
So this is all for PCI transport, do we need to
1) move this to PCI transport
or
2) use non PCI specific termiology (PF/VF) here
?
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 113
Quoted 12 lines
admin-virtq.tex +12
-0+The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver. +This command upon success, returns a data buffer that describes information about PCI SRIOV +related capabilities and attributes for the device. This command can be supported only by +PCI devices that supports Single Root I/O Virtualization. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_sriov_attrs_result { + /* For compatibility - indicates which of the below fields are valid (1 means valid): + * Bit 0x0 - vfs_free_msix_count + * Bit 0x1 - per_vf_max_msix_count + * Bits 0x2 - 0x3F - reserved for future fields + */
So we have several layers of compatibilities:
1) feature bits
2) cap bitmap / identifiy
3) attrs_mask (no negotiation so it's unclear that how the compatibility can work here).
This looks like a burden for the management.
Is it really helpful to have a device that can only return free_msix_count but not per_vf_max_msix_count?
And I think it's better to explicit split the commands then we can have dedicated commands for MSI instead of trying to mix all SRIOV related attributes into the same command.
Quoted 16 lines
admin-virtq.tex +16
-0+ le64 attrs_mask; + /* Number of free msix vectors in the global msix pool for VFs */ + le32 vfs_free_msix_count; + /* Max number of msix vectors that can be assigned for a single VF */ + le16 per_vf_max_msix_count; +}; +\end{lstlisting} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors +count for a PCI virtual function. The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_set_data { + /* The virtual function number */ + le16 vf_number;
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 139
Quoted 10 lines
admin-virtq.tex +10
-0+ /* For compatibility - indicates which of the below properties should be + * modified (1 means that field should be modified): + * Bit 0x0 - msix_count + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors */ + le16 msix_count; +}; +\end{lstlisting}
I think it's better to have a dedicated command to set the msix_vectors, then there's no need for the attrs_mask.
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 147
Quoted 8 lines
admin-virtq.tex +8
-0+ +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline +Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\
Does it mean it exceeds the totalVFs? We need an accurate definition on "invalid" here.
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 149
Quoted 2 lines
admin-virtq.tex +2
-0+\hline +01h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED & MSI-X count exceed the max value per VF \\
I guess it's actually per_vf_max_msix_count? What if the device doesn't give us per_vf_max_msix_count (it's odd but it's allowed in this propsal if I was not wrong).
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 152
Quoted 3 lines
admin-virtq.tex +3
-0+\hline +02h & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC & MSI-X count exceed the free pool resources \\ +\hline
Do we need to define "pool" here?
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 153
Quoted 1 line
admin-virtq.tex +1
-0+03h & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE & VF is already in use, operation failed \\
How do you define "VF is already in use"? Is this command only applicable when sriov_numvfs in the capability is zero? How about the initial associated VF via initial_vfs?
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 162
Quoted 9 lines
admin-virtq.tex +9
-0+\hline +04h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.}
Should be TotalVFs in the SR-IOV extended capability.
Quoted 10 lines
admin-virtq.tex +10
-0+\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED +is returned when the amount of MSI-X to assign exceed the maximum value that can be +assigned to a single VF.} +\end{note} + +\begin{note} +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 182
Quoted 10 lines
admin-virtq.tex +10
-0+is returned when the amount of MSI-X to assign exceed the amount of free MSI-X +vectors in the global pool.} +\end{note} + +This command has no command specific result set by the device. Upon success, the device guarantees +that all the requested properties were modified to the given values. Otherwise, error will be returned. + +\begin{note} +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver. +Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.}
How did the device know whether or not VF is not used by any driver?
Quoted 24 lines
admin-virtq.tex +24
-0+\end{note} + +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} + +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs +interrupt vectors configuration. +The command specific data set by the driver is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_data { + /* The virtual function number */ + le16 vf_number; + /* For compatibility - indicates which of the below properties should be + * queried (1 means that field should be queried): + * Bit 0x0 - msix_count (The amount of MSI-X interrupt vectors) + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; +}; +\end{lstlisting} + +The following table describes the command specific error codes codes: + +\begin{tabular}{|l|l|l|} +\hline
admin-virtq.tex \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a · line 216
Quoted 10 lines
admin-virtq.tex +10
-0+Opcode & Status & Description \\ +\hline \hline +00h & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID & invalid VF number \\ +\hline +01h - FFh & Reserved & - \\ +\hline +\end{tabular} + +\begin{note} +{vf_number can't be greater than NumVFs value as defined in the PCI specification
I guess for NumVFs you meant the TotalVFs in SR-IOV extended capability.
Quoted 61 lines
admin-virtq.tex +19
-0+or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID +is returned when vf_number is out of the range.} +\end{note} + +This command, upon success, returns a data buffer that describes the properties that were requested +and their values for the subject virtio VF device according to the given vf_number. +This information is of form: +\begin{lstlisting} +struct virtio_admin_pci_vf_interrupt_config_get_result { + /* For compatibility - indicates which of the below fields were returned + * (1 means that field was returned): + * Bit 0x0 - msix_count + * Bits 0x1 - 0x3F - reserved for future fields + */ + le64 attrs_mask; + /* The amount of MSI-X interrupt vectors currently assigned to the VF */ + le16 msix_count; +}; +\end{lstlisting}content.tex +34
-0@@ -1734,6 +1734,50 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options / \end{itemize} \end{itemize} +\subsection{PCI-specific Admin capabilities}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin capabilities} + +This documents the group of Admin capabilities for PCI virtio devices. Each capability is +implemented using one or more Admin queue commands. + +\subsubsection{VF MSI-X control}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control} + +This capability enables a virtio PCI PF device to control the assignment of MSI-X interrupt vectors +for its managed VFs. Capable devices will need to set bit 0x0 of caps[0] in the result of VIRTIO_ADMIN_CAPS_IDENTIFY +admin command. See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command} +for more details. + +Capable devices will also need to implement VIRTIO_ADMIN_PCI_SRIOV_ATTRS, VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and +VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin commands. + +In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a capable device should return the number of free msix vectors +in the global msix pool for its VFs in \field{vfs_free_msix_count} field and also the maximal number of msix vectors that +can be assigned for a single VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1 should be set +to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more +details. + +A PCI PF device may assign default number of MSI-X vectors to the enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET +it is possible to update the default MSI-X assignment for a specific VF. +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, a driver set the virtual function number in +\field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}. +In addition, bit 0x0 in the \field{attrs_mask} field should be set. A successful operation guarantees that the requested +amount of MSI-X interrupt vectors was assigned to the subject virtual function. +Also, a successful operation guarantees that the MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect +the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control +register should reflect a value of 7. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details. + +It is beyond the scope of the virtio specification to define necessary synchronization in system software to ensure that a virtio PCI VF device
content.tex \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options / · line 1780
Quoted 10 lines
content.tex +10
-0+interrupt configuration modification is reflected in the PCI device. However, it is expected that any modern system software implementing virtio +drivers and PCI subsystem should ensure that any changes occurring in the VF interrupt configuration is either updated in the PCI VF device or +such configuration fails. + +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin command, a driver should set the virtual function number in +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} field should be set to indicate requested output fields in +the result from the device. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently +assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit 0x0 in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field. +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details. +
I think we should have a guidance on how to provision #msix_vectors step by step. E.g:
1) Do we need to do that before or after the VF provisioning
2) Can the value survive when 0 is wrote to sriov_numvfs
etc.
Thanks
Quoted 3 lines
content.tex +0
-0\section{Virtio Over MMIO}\label{sec:Virtio Transport Options / Virtio Over MMIO} Virtual environments without PCI support (a common situation in
RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 2 lines
From: Jason Wang <[email protected]>
Sent: Thursday, January 27, 2022 9:06 AM
[...]
Quoted 4 lines
And I think it's better to explicit split the commands then we can have dedicated commands for MSI instead of trying to mix all SRIOV related attributes into the same command.
It is not mixed. Its clubbed together so that driver-device communication can be minimal and still flexible.
For example in [1] it has, command like,
ip link add vx0 type vxlan id 100 local 1.1.1.1 remote 2.2.2.2 dev eth0 dstport 4789
it would be inefficient do it as,
$ ip link add vx0 type vxlan id 100 $ ip link vx0 local 1.1.1.1 $ ip link vx0 remote 2.2.2.2 $ ip link vx0 dev eth0 $ ip link vx0 dstport 4789
Similarly, we prefer the ability and flexibility to set more fields in one command.
Quoted 3 lines
I think it's better to have a dedicated command to set the
msix_vectors, then there's no need for theattrs_mask.
Main design principle in having attr_mask is to avoid issuing multiple commands for related/adjacent attributes. Lesser the number of commands serviced, more efficient the overall system is.
Hence the attribute mask gives the flexibility to set one or more.
[virtio-comment] Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 16 lines · Parav Pandit
Quoted 3 lines
From: Jason Wang <[email protected]>
Sent: Thursday, January 27, 2022 9:06 AM[...]
Quoted 4 lines
And I think it's better to explicit split the commands then we can have dedicated commands for MSI instead of trying to mix all SRIOV related attributes into the same command.
It is not mixed. Its clubbed together so that driver-device communication can be minimal and still flexible.
For example in [1] it has, command like,
ip link add vx0 type vxlan id 100 local 1.1.1.1 remote 2.2.2.2 dev eth0 dstport 4789
Quoted 8 lines
it would be inefficient do it as,
$ ip link add vx0 type vxlan id 100 $ ip link vx0 local 1.1.1.1 $ ip link vx0 remote 2.2.2.2 $ ip link vx0 dev eth0 $ ip link vx0 dstport 4789
vxlan part actually.
Quoted 2 lines
Similarly, we prefer the ability and flexibility to set more fields in one command.
I understood this motivation. But what I want to say is:
struct virtio_cmd {
u64 attr_mask;
field_X;
field_Y;
field_Z;
}What's the default value if one of the X,Y, Z is not specified? How to get those default values?
And suppose we add a new field
struct virtio_cmd {
u64 attr_mask;
field_X;
field_Y;
field_Z;
field_M;
}How can the driver know it can use field_M? And if field_M is supported in src but not dst, how can we keep the migration compatibility?
Would it be simple to just mandate all the fields in this case?
Thanks
Quoted 7 lines
Quoted 4 lines
I think it's better to have a dedicated command to set the
msix_vectors, then there's no need for theattrs_mask.Main design principle in having
attr_maskis to avoid issuing multiple commands for related/adjacent attributes. Lesser the number of commands serviced, more efficient the overall system is.
Hence the attribute mask gives the flexibility to set one or more.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 4 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 28, 2022 8:54 AM
vxlan part actually.
Right.
Quoted 7 lines
Quoted 3 lines
Similarly, we prefer the ability and flexibility to set more fields in one
command.
I understood this motivation. But what I want to say is:
Quoted 8 lines
struct virtio_cmd { u64 attr_mask; field_X; field_Y; field_Z; }What's the default value if one of the X,Y, Z is not specified?
If minimum default values are not provided, HV cannot proceed to do configuration and ENOSUPPORT error code is returned by kernel to user.
Quoted 2 lines
How to get those default values?
By doing identify or _get respective command.
Quoted 11 lines
And suppose we add a new field
struct virtio_cmd { u64 attr_mask; field_X; field_Y; field_Z; field_M; }How can the driver know it can use
field_M?
If attribute mask is present for field_M, driver can use it.
Quoted 2 lines
And if
field_Mis supported in src
but not dst, how can we keep the migration compatibility?
This is HV level command to query and provision a VF.
Migration compatibility is high level check where it will identify on which HV to migrate where I can provision X vectors for a VF.
Quoted 3 lines
Would it be simple to just mandate all the fields in this case?
May be, attribute mask is basically to let expand structure without inventing new fields.
For a given command few minimal attributes to be set and driver can verify, that if they are not, a given functionality is unsupported.
Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 16 lines · Parav Pandit
Quoted 3 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 28, 2022 8:54 AM
vxlan part actually.Right.
Quoted 12 lines
Quoted 1 line
Similarly, we prefer the ability and flexibility to set more fields in one
command.
I understood this motivation. But what I want to say is:
struct virtio_cmd { u64 attr_mask; field_X; field_Y; field_Z; }
Quoted 2 lines
Quoted 1 line
What's the default value if one of the X,Y, Z is not specified?
If minimum default values are not provided, HV cannot proceed to do configuration and ENOSUPPORT error code is returned by kernel to user.
Do we need to document the minimum default value?
Quoted 4 lines
Quoted 3 lines
How to get those default values?
By doing identify or
_getrespective command.
This can only work if there's no attr_mask in get command. If the device can selectively provides us partial information we can't do compatibility check since we don't know its value.
Quoted 11 lines
Quoted 11 lines
And suppose we add a new field
struct virtio_cmd { u64 attr_mask; field_X; field_Y; field_Z; field_M; }
Quoted 2 lines
Quoted 1 line
How can the driver know it can use
field_M?If attribute mask is present for
field_M, driver can use it.
So the driver need first use get then try to use set?
Quoted 5 lines
Quoted 3 lines
And if
field_Mis supported in src
but not dst, how can we keep the migration compatibility?This is HV level command to query and provision a VF.
Migration compatibility is high level check where it will identify on which HV to migrate where I can provision X vectors for a VF.
By probing capability of attr_mask in each possible commands?
Quoted 4 lines
Quoted 3 lines
Would it be simple to just mandate all the fields in this case?
May be, attribute mask is basically to let expand structure without inventing new fields.
It could be done via new command.
Thanks
Quoted 2 lines
For a given command few minimal attributes to be set and driver can verify, that if they are not, a given functionality is unsupported.
RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs
Quoted 21 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 28, 2022 9:06 AMQuoted 16 lines · Parav Pandit
Quoted 2 lines
From: Jason Wang <[email protected]>
Sent: Friday, January 28, 2022 8:54 AM vxlan part actually.Right.
Quoted 13 lines
Quoted 2 lines
Similarly, we prefer the ability and flexibility to set more fields in one
command.
I understood this motivation. But what I want to say is:
struct virtio_cmd { u64 attr_mask; field_X; field_Y; field_Z; }
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
What's the default value if one of the X,Y, Z is not specified?
If minimum default values are not provided, HV cannot proceed to do
configuration and ENOSUPPORT error code is returned by kernel to user.
Do we need to document the minimum default value?
I guess, not because attr_mask is expected to grow and there may be unrelated attributes to it.
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines
How to get those default values?
By doing identify or
_getrespective command.This can only work if there's no
attr_maskin get command. If the device can selectively provides us partial information we can't do compatibility check since we don't know its value.
Device will follow some sane implementation.
If device which wants to support msix config for a VF, it will report right values in generic identify and get cmd of device. A simple error check of bit mask will be enough in driver.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
And suppose we add a new field
struct virtio_cmd { u64 attr_mask; field_X; field_Y; field_Z; field_M; }
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
How can the driver know it can use
field_M?If attribute mask is present for
field_M, driver can use it.So the driver need first use get then try to use set?
No. the flow is:
1. AQ feature bit is negotiated
2. Create AQ
3. Do identify cmd and cache the device capabilities
A bit says that VF msix modification supported.
4. user queried all VFs current vector distribution (optimize this bit later, but consider this for now) If attr.mask.bitX not set, fail the user request.
4. user requests to modify VF-A vectors from (8 to 4) uses attr.mask.bitX.
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
And if
field_Mis supported in src
but not dst, how can we keep the migration compatibility?This is HV level command to query and provision a VF.
Migration compatibility is high level check where it will identify on which HVto migrate where I can provision X vectors for a VF.
By probing capability of
attr_maskin each possible commands?
No. not in each possible commands.
Identify command gives the capabilities supported in step #3 above to see if PFs on source and destination HV support it or not.
Quoted 9 lines
Quoted 5 lines
Quoted 4 lines
Would it be simple to just mandate all the fields in this case?
May be, attribute mask is basically to let expand structure without inventing
new fields.
It could be done via new command.
Sure, it can be.
We want to avoid too many commands, if they can be clubbed in single command when starting a VF.