transport-pci: Introduce legacy registers access using AQ

This short series introduces legacy registers access commands for the owner group member PCI PF to access the legacy registers of the member VFs.

If in future any SIOV devices to support legacy registers, they can be easily supported using same commands by using the group member identifiers of the future SIOV devices.

More details as overview, motivation, use case are further described below.

Patch summary:
--------------
patch-1 split rows of admin opcode tables by a line patch-2 adds administrative virtuqueue commands patch-3 adds its conformance section and links

It uses the newly introduced administrative virtqueue facility with 4 new commands which uses the existing virtio_admin_cmd.

Usecase:
--------
1. A hypervisor/system needs to provide transitional virtio devices to the guest VM at scale of thousands, typically, one to eight devices per VM.

2. A hypervisor/system needs to provide such devices using a vendor agnostic driver in the hypervisor system.

3. A hypervisor system prefers to have single stack regardless of virtio device type (net/blk) and be future compatible with a single vfio stack using SR-IOV or other scalable device virtualization technology to map PCI devices to the guest VM. (as transitional or otherwise)

Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for PCI SR-IOV based devices. Currently it does not work beyond PCI PF, or as software emulated device in reality. Currently it has below cited system level limitations:

[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.

[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from the physical-memory address space. The I/O address space consists of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.

[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be aligned to a 4 KB boundary.

Overview:
---------
Above usecase requirements is solved by PCI PF group owner accessing its group member PCI VFs legacy registers using an admin virtqueue of the group owner PCI PF.

Two new admin virtqueue commands are added which read/write PCI VF registers.

Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver framework in Linux kernel.

                +----------------------+
                |pci_dev_id = 0x100X   |
+---------------|pci_rev_id = 0x0      |-----+
|vfio device    |BAR0 = I/O region     |     |
|               |Other attributes      |     |
|               +----------------------+     |
|                                            |
+   +--------------+     +-----------------+ |
|   |I/O BAR to AQ |     | Other vfio      | |
|   |rd/wr mapper  |     | functionalities | |
|   +--------------+     +-----------------+ |
|                                            |
+------+-------------------------+-----------+
       |                         |
       |                    Driver notification
       |                         |
       |                         |
  +----+------------+       +----+------------+
  | +-----+         |       | PCI VF device A |
  | | AQ  |-------------+---->+-------------+ |
  | +-----+         |   |   | | legacy regs | |
  | PCI PF device   |   |   | +-------------+ |
  +-----------------+   |   +-----------------+
                        |
                        |   +----+------------+
                        |   | PCI VF device N |
                        +---->+-------------+ |
                            | | legacy regs | |
                            | +-------------+ |
                            +-----------------+

2. Virtio pci driver to bind to the listed device id and use it as native device in the host.

3. Use it in a light weight hypervisor to run bare-metal OS.

Please review.

Alternatives considered:
========================
1. Exposing BAR0 as MMIO BAR that follows legacy registers template Pros:
a. Kind of works with legacy drivers as some of them have used API
which is agnostic to MMIO vs IOBAR.
b. Does not require hypervisor intervantion
Cons:
a. Device reset is extremely hard to implement in device at scale as
driver does not wait for device reset completion
b. Device register width related problems persist that hypervisor if
wishes, it cannot be fixed.

2. Accessing VF registers by tunneling it through new legacy PCI capability Pros:
a. Self contained, but cannot work with future PCI SIOV devices Cons:
a. Equally slow as AQ access
b. Still requires new capability for notification access
c. Requires hardware to build low level registers access which is not worth
for long term future

3. Accessing VF notification region using new PF BAR
Cons:
a. Requires hardware to build new PCI steering logic per PF to forward
notification from the PF to VF, requires double the amount of logic compared to today
b. Requires very large additional PF BAR whose size must be max_Vfs * BAR size.

conclusion for picking AQ approach:
==================================
1. Overall AQ based access is simpler to implement with combination of best from software and device so that legacy registers do not get baked in the device hardware
2. AQ allows hypervisor software to intercept legacy registers and make corrections if needed
3. Provides trade-off between performance, device complexity vs spec, while still maintaining passthrough mode for the VFs with minimal hypervisor intercepts only for legacy registers access
4. AQ mechanism is designed for accessing other member devices registers as noted in AQ submission, it utilizes the existing infrastructure over other alternatives.
5. Uses existing driver notification region similar to legacy notification saving hardware resources

---
changelog:
v3->v4:
- moved noted to the conformance section details in next patch
- removed queue notify address query AQ command on Michael's suggestion, though it is fine. Instead replaced with extending virtio_pci_notify_cap to indicate that legacy queue notifications can be done on the notification location
- fixed spelling errors
- replaced administrative virtqueue to administration virtqueue
- moved legacy interface normative references to legacy conformance section
v2->v3:
- added new patch to split raws of admin vq opcode table
- adddressed Jason and Michael's comment to split single register access command to common config and device specific commands.
- dropped the suggetion to introduce enable/disable command as
admin command cap bit already covers it.
- added other alternative design considered and discussed in detail in v0, v1 and v2 v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
v0->v1:
- addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
- far more simpler design than MMR access
- removed complexities of MMR device ids
- removed complexities of MMR registers and extended capabilities
- dropped adding new extended capabilities because if if they are added, a pci device still needs to have existing capabilities
in the legacy configuration space and hypervisor driver do not
need to access them

Parav Pandit (3):
admin: Split opcode table rows with a line
transport-pci: Introduce legacy registers access commands
transport-pci: Add legacy register access conformance section

 admin.tex                     |  12 ++-
 conformance.tex               |   2 +
 transport-pci-legacy-regs.tex | 155 ++++++++++++++++++++++++++++++++++
 transport-pci.tex             |  19 +++++
 4 files changed, 187 insertions(+), 1 deletion(-)
 create mode 100644 transport-pci-legacy-regs.tex

2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Currently all opcode appears to be in a single row. Separate them with a line similar to other tables.

---
changelog:
v2->v3:
- new patch

---
 admin.tex | 2 ++
 1 file changed, 2 insertions(+)

index 2efd4d7..e51f9e6 100644
@@ -114,7 +114,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 opcode & Name & Command Description \\
 \hline \hline
 0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
+\hline
 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
+\hline
 0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\

2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This patch introduces legacy registers access commands for the owner group member PCI PF to access the legacy registers of the member VFs.

If in future any SIOV devices to support legacy registers, they can be easily supported using same commands by using the group member identifiers of the future SIOV devices.

More details as overview, motivation, use case are further described below.

Usecase:
--------
1. A hypervisor/system needs to provide transitional virtio devices to the guest VM at scale of thousands, typically, one to eight devices per VM.

2. A hypervisor/system needs to provide such devices using a vendor agnostic driver in the hypervisor system.

3. A hypervisor system prefers to have single stack regardless of virtio device type (net/blk) and be future compatible with a single vfio stack using SR-IOV or other scalable device virtualization technology to map PCI devices to the guest VM. (as transitional or otherwise)

Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for PCI SR-IOV based devices. Currently it does not work beyond PCI PF, or as software emulated device in reality. Currently it has below cited system level limitations:

[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.

[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from the physical-memory address space. The I/O address space consists of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.

[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be aligned to a 4 KB boundary.

Overview:
---------
Above usecase requirements can be solved by PCI PF group owner accessing its group member PCI VFs legacy registers using an admin virtqueue of the group owner PCI PF.

Two new admin virtqueue commands are added which read/write PCI VF registers.

The third command suggested by Jason queries the VF device's driver notification region.

Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver framework in Linux kernel.

                +----------------------+
                |pci_dev_id = 0x100X   |
+---------------|pci_rev_id = 0x0      |-----+
|vfio device    |BAR0 = I/O region     |     |
|               |Other attributes      |     |
|               +----------------------+     |
|                                            |
+   +--------------+     +-----------------+ |
|   |I/O BAR to AQ |     | Other vfio      | |
|   |rd/wr mapper  |     | functionalities | |
|   +--------------+     +-----------------+ |
|                                            |
+------+-------------------------+-----------+
       |                         |
       |                         |
       |                  Driver notifications
       |                         |
  +----+------------+       +----+------------+
  | +-----+         |       | PCI VF device A |
  | | AQ  |-------------+---->+-------------+ |
  | +-----+         |   |   | | legacy regs | |
  | PCI PF device   |   |   | +-------------+ |
  +-----------------+   |   +-----------------+
                        |
                        |   +----+------------+
                        |   | PCI VF device N |
                        +---->+-------------+ |
                            | | legacy regs | |
                            | +-------------+ |
                            +-----------------+

2. Virtio pci driver to bind to the listed device id and use it as native device in the host.

3. Use it in a light weight hypervisor to run bare-metal OS.

Please review.

---
changelog:
v3->v4:
- moved noted to the conformance section details in next patch
- removed queue notify address query AQ command on Michael's suggestion, though it is fine. Instead replaced with extending virtio_pci_notify_cap to indicate that legacy queue notifications can be done on the notification location.
- fixed spelling errors.
- replaced administrative virtqueue to administration virtqueue
- added queue notification capability register to indicate legacy q notification supported
v2->v3:
- adddressed Jason and Michael's comment to split single register access command to common config and device specific commands.
- dropped the suggetion to introduce enable/disable command as admin command cap bit already covers it.
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix

- added overview in commit log
- renamed subsection to reflect command

---
 admin.tex                     |  10 ++-
 transport-pci-legacy-regs.tex | 122 ++++++++++++++++++++++++++++++++++
 transport-pci.tex             |  19 ++++++
 3 files changed, 150 insertions(+), 1 deletion(-)
 create mode 100644 transport-pci-legacy-regs.tex

index e51f9e6..8f55bf4 100644
@@ -117,7 +117,15 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline
 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
 \hline
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
+0x0002 & VIRTIO_ADMIN_CMD_LCC_REG_WRITE & Write legacy common configuration registers of a member device    \\
+\hline
+0x0003 & VIRTIO_ADMIN_CMD_LCC_REG_READ & Read legacy common configuration registers of a member device    \\
+\hline
+0x0004 & VIRTIO_ADMIN_CMD_LD_REG_WRITE & Write legacy device registers of a member device    \\
+\hline
+0x0005 & VIRTIO_ADMIN_CMD_LD_REG_READ & Read legacy device registers of a member device    \\
+\hline
+0x0006 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
 \hline

index a5c6719..42bc104 100644
@@ -541,6 +541,7 @@ \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options
 struct virtio_pci_notify_cap {
         struct virtio_pci_cap cap;
         le32 notify_off_multiplier; /* Multiplier for queue_notify_off. */
+        u8 legacy_q_notify_supported;
 };
 \end{lstlisting}
 

@@ -560,6 +561,14 @@ \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options
 the same Queue Notify address for all queues.
 \end{note}
 
+\field{legacy_q_notify_supported} when set to 1, indicates that the device
+supports legacy queue notifications at this notification location. Legacy Queue
+Notification address is derived within a BAR for a virtqueue:
+
+\begin{lstlisting}
+        cap.offset
+\end{lstlisting}
+
 \devicenormative{\paragraph}{Notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 The device MUST present at least one notification capability.
 

@@ -596,6 +605,14 @@ \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options
 cap.length >= queue_notify_off * notify_off_multiplier + 4
 \end{lstlisting}
 
+\paragraph{Legacy Interfaces: Notification capability}\label{par:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: Notification capability}
+
+The device SHOULD set \field{legacy_q_notify_supported} when the device
+notification location supports legacy driver notifications.
+
+The driver in the hypervisor MAY use notification location for accessing
+legacy queue notify address when the device supports it.
+
 \subsubsection{ISR status capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 
 The VIRTIO_PCI_CAP_ISR_CFG capability

@@ -1212,3 +1229,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
         re-examine the configuration space to see what changed.
     \end{itemize}
 \end{itemize}
+
+\input{transport-pci-legacy-regs.tex}

2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Add device and driver conformanace section for legacy registers access commands interface.

---
changelog:
v3->v4:
- moved legacy interface normative references to legacy conformance section
v2->v3:
- added normative lines for two additional commands

---
 conformance.tex               |  2 ++
 transport-pci-legacy-regs.tex | 33 +++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+)

index 01ccd69..be0bf89 100644
@@ -265,6 +265,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
 \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection / Legacy Interface: A Note on Device Layout Detection}
 \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
+\item Section \ref{par:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces Requirements: SR-IOV VFs Registers Access}
+\item Section \ref{par:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: Notification capability}
 \item Section \ref{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface}
 \item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision / Legacy Interfaces: A Note on Setting the Virtio Revision}
 \item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue / Legacy Interface: A Note on Configuring a Virtqueue}

index 2a464da..ef6a33c 100644
@@ -120,3 +120,36 @@ \subsubsection{Legacy Device Registers Read Command}\label{sec:Virtio Transport
 	u8 registers[];
 };
 \end{lstlisting}
+
+\paragraph{Legacy Interfaces Requirements: SR-IOV VFs Legacy Registers Access}\label{par:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces Requirements: SR-IOV VFs Registers Access}
+
+If the PCI PF device supports legacy registers access for its group members,
+the device MUST set all corresponding bits for commands VIRTIO_ADMIN_CMD_LCC_REG_WRITE,
+VIRTIO_ADMIN_CMD_LCC_REG_READ, VIRTIO_ADMIN_CMD_LD_REG_WRITE,
+VIRTIO_ADMIN_CMD_LD_REG_READ in the command result of
+VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
+
+The device MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LCC_REG_WRITE, VIRTIO_ADMIN_CMD_LCC_REG_READ
+commands for the invalid offset which is outside the legacy common configuration
+register's address range.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LD_REG_WRITE, VIRTIO_ADMIN_CMD_LD_REG_READ
+commands for the invalid offset which is outside the legacy device specific
+register's address range.
+
+The PCI VF device SHOULD NOT expose PCI BAR 0 when it prefers to support
+legacy interface registers access.
+
+The driver MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The driver SHOULD send VIRTIO_ADMIN_CMD_LCC_REG_WRITE and
+VIRTIO_ADMIN_CMD_LCC_REG_READ commands with a valid offset which is in the
+legacy common configuration registers address range.
+
+The driver SHOULD send commands VIRTIO_ADMIN_CMD_LD_REG_WRITE and
+VIRTIO_ADMIN_CMD_LD_REG_READ with a valid offset which is in the legacy
+device specific registers address range.

2.26.2


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Thanks for the patches! Getting ready for travel until June 20th, expect delays.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]