Skip to contentSkip to conversation

virtio-pci: management device

RFC PATCH New Jason Wang 1 participant 4 messages Aug 20, 2021

lore
Share & tools
lore

Jason Wang

author
#0 lore raw

cover virtio-pci: management device

Hi all:

As promised, I post this series to demonstrate the possibility of using a transport specific way for implementing the management device. The main motivation is to achieve hyper scalability and flexibility. This is an alternative way of transporting the managed(virtual) device via the admin virtqueue (most of the admin virtqueue commands proposed there were just a duplication of the PCI capability).

The idea is simple, introduce a device selector and try to reuse as much existing virtio-pci capabilities as possible.

In order to make this work, two independent capabilities are introduced. A new capability for storing MSI-X message is introduced for per-device MSI-X configuration for the managed device. PASID capability is also introduced for DMA isolation among the managed devices.

To be simple, I don't propose device MMU interface in this series. It could be added in the future since it's not a must if PASID is supported.

Sending this for early feedback and comments.

Thanks

Jason Wang (3):
virtio-pci: introduce device specific MSI-X messages store
virtio-pci: PASID support
virtio-pci: introduce the management device capability

 content.tex | 202 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 202 insertions(+)
Signature · 18 lines
2.24.3 (Apple Git-128)


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
author
#1 lore raw

RFC PATCH 1/3 virtio-pci: introduce device specific MSI-X messages store

This patch introduce a new capability that allows the MSI-X messages to be stored and setup through a dedicated capability. This will be useful for the cases when:

- the device is required to have more than 2048 MSI-X entries, one example is a device have more than 2048 queues
- the managed device without a transport specific way to configure the MSI-X entries

---
 content.tex | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
content.tex +73 -0
index 620c0e2..50c0523 100644
@@ -705,6 +705,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 #define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
 /* Vendor-specific data */
 #define VIRTIO_PCI_CAP_VENDOR_CFG        9
+/* MSI-X configuration */
+#define VIRTIO_CPI_CAP_MSIX_CFG          10
 \end{lstlisting}
 
         Any other value is reserved for future use.
@@ -1261,6 +1263,77 @@ \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport O
 specified by some other Virtio Structure PCI Capability
 of type other than \field{VIRTIO_PCI_CAP_PCI_CFG}.
 
+\subsubsection{MSI-X configuration capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout /  MSI-X configuration capability}
+
+The VIRTIO_PCI_CAP_MSIX_CFG capability allows the MSI-X vectors to be
+stored in a virtio-pci device specific store instead of the PCI-E
+MSI-X table.
+
+The capability is immediately followed by an additional field like so:
+
+\begin{lstlisting}
+struct virtio_pci_msix_cap {
+        struct virtio_pci_cap cap;
+        u8 enable;                  /* read-write */
+        le16 num_vectors;           /* read-only for driver */
+        le16 vector_select;         /* read-write */
+        le32 msix_address_high;     /* read-write */
+        le32 msix_address_low;      /* read-write */
+        le32 msix_data;             /* read-write */
+        u8 mask;                    /* read-write */
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{enable}]
+The driver uses this to enable (writing 1) or disable (writing 0) the
+virtio-pci specific MSI-X messages storing. When enabled, the device
+will use the MSI-X messages that is setup through this
+capability instead of the PCI-E MSI-X table. When disabled, the
+device will use the MSI-X messages that is setup through the PCI-E
+MSI-X table.
+
+\item[\field{num_vectors}]
+The device specifies the maximum number of MSI-X entries supported here.
+
+\item[\field{vector_select}]
+Vector Select. The driver selects which virtqueues the following
+fields refer to.
+
+\item[\field{msix_address_high}]
+The driver writes the high address of a specific MSI-X entry here.
+
+\item[\field{msix_address_lo}]
+The driver writes the low address of a specific MSI-X entry here.
+
+\item[\field{msix_data}]
+The driver writes the MSI-X data of a specific MSI-X entry here.
+
+\item[\field{mask}]
+The driver use this to mask (write 1) or unmask (write 0) a specific
+MSI-X entry indexed via \field{vector_select} here.
+\end{description}
+
+\devicenormative{\paragraph}{MSI-X configuration capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / MSI-X configuration capability}
+
+The device MUST disable virtio-pci device specific MSI-X messages
+store and presenting 0 to \field{enable} upon reset.
+
+The device MUST mask all the vectors and present 0 to \field{mask}
+upon reset.
+
+The device MUST record the pending MSI-X vectors and generate the
+MSI-X interrupt upon unmasking.
+
+The device MUST ignore the value that is wrote to
+\field{vector_select} which is equal or greater than
+\field{num_vectors}.
+
+\drivernormative{\paragraph}{MSI-X configuration capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / MSI-X configuration capability}
+
+The driver MUST mask a specific vector before trying to setup the
+MSI-X addresses and MSI-X data.
+
 \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
 
 Transitional devices MUST present part of configuration
Signature · 1 line
2.24.3 (Apple Git-128)
author
#2 lore raw

RFC PATCH 2/3 virtio-pci: PASID support

In reply to Jason Wang

This patch introduce a new virtio-pci capability for assigning PASID for a specific virtqueue.

---
 content.tex | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
content.tex +54 -0
index 50c0523..5266965 100644
@@ -707,6 +707,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 #define VIRTIO_PCI_CAP_VENDOR_CFG        9
 /* MSI-X configuration */
 #define VIRTIO_CPI_CAP_MSIX_CFG          10
+/* PASID configuration */
+#define VIRTIO_PCI_CAP_PASID             11
 \end{lstlisting}
 
         Any other value is reserved for future use.
@@ -1334,6 +1336,54 @@ \subsubsection{MSI-X configuration capability}\label{sec:Virtio Transport Option
 The driver MUST mask a specific vector before trying to setup the
 MSI-X addresses and MSI-X data.
 
+\subsubsection{PASID configuration}\label{MSI-X configuration capability}{PASID / Virtio Over PCI Bus / PCI Device Layout / PASID configuration}
+
+The VIRTIO_PCI_CAP_RING_PASID capability is for assigning PASID for a
+specific virtqueue.
+
+The capability is immediately followed by an additional field like so:
+
+\begin{lstlisting}
+struct virtio_pci_cfg_cap {
+        struct virtio_pci_cap cap;
+        le16 queue_select;       /* read-write */
+        le32 {
+          queue_pasid : 20;      /* read-write */
+          reserved : 12;         /* read-write */
+        };
+        u8 pasid_enable;         /* read-write */
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{queue_select}]
+        Queue Select. The driver selects which virtqueue the following
+        fields refer to.
+\item[\field{queue_pasid}]
+        PASID for the virtqueue.
+\item[\field{reserved}]
+        Reserved for the future use.
+\item[\field{pasid_enable}]
+        The driver use this to enable (write 1) or disable (write 0)
+        PASID for the virtqueue.
+\end{description}
+
+\devicenormative{\paragraph}{PASID configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PASID configuration}
+
+The device MUST present at least one VIRTIO_PCI_CAP_PASID capability
+if it presents the PASID Extended Capability.
+
+The device MUST NOT present VIRTIO_PCI_CAP_PASID capability if PASID
+Extended Capability is not presented.
+
+The device MUST disable PASID for all virtqueues and presenting 0 to
+\field{enable} upon reset.
+
+\drivernormative{\paragraph}{PASID configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PASID configuration}
+
+The driver MUST configure the PASID for each virtqueue if PASID is
+enabled via PASID Extended Capability.
+
 \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
 
 Transitional devices MUST present part of configuration
@@ -1590,6 +1640,10 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
   corresponding to this vector into \field{queue_msix_vector}. Read
   \field{queue_msix_vector}: on success, previously written value is
   returned; on failure, NO_VECTOR value is returned.
+
+\item Optionally, if PASID Extended Capability is present and enabled
+  on the device, select a PASID to use for the TLP of a
+  virtqueue. Write the PASID value into \field{pasid}.
 \end{enumerate}
 
 \subparagraph{Legacy Interface: A Note on Virtqueue Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
Signature · 1 line
2.24.3 (Apple Git-128)
author
#3 lore raw

RFC PATCH 3/3 virtio-pci: introduce the management device capability

In reply to Jason Wang

This patch introduces the management device capability. The device that offers this capability is the management device. The management device driver can use this capability to create managed device who doesn't have a dedicated transport specific configuration interface.

The idea is simple, introduce a device_select field then we can reuse almost all the existing capabilities (except for the shared memory one) to configure the managed device.

A simple managed device provision interface is also proposed.

To be simple, I tie this capability with the PASID capability instead of introducing device MMU functions.

---
 content.tex | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
content.tex +75 -0
index 5266965..d3ebd76 100644
@@ -709,6 +709,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 #define VIRTIO_CPI_CAP_MSIX_CFG          10
 /* PASID configuration */
 #define VIRTIO_PCI_CAP_PASID             11
+/* Managed device configuration */
+#define VIRTIO_PCI_CAP_MANAGED_DEV_CFG   12
 \end{lstlisting}
 
         Any other value is reserved for future use.
@@ -1384,6 +1386,79 @@ \subsubsection{PASID configuration}\label{MSI-X configuration capability}{PASID
 The driver MUST configure the PASID for each virtqueue if PASID is
 enabled via PASID Extended Capability.
 
+\subsubsection{Managed device configuration}\label{Managed device configuration}{PASID / Virtio Over PCI Bus / PCI Device Layout / Managed device configuration}
+
+The VIRTIO_PCI_CAP_MANAGED_DEV_CFG capability allows the one device to
+manage other (usually virtual) virtio devices. The device that offers
+VIRTIO_PCI_CAP_MANAGED_DEV_CFG is called management device. The
+device that is managed by the management device is called managed
+device.
+
+The capability is immediately followed by an additional field like so:
+
+\begin{lstlisting}
+struct virtio_pci_managed_dev_cap {
+        struct virtio_pci_cap cap;
+        u32 num_devices;            /* read-only for driver */
+        le32 device_select;         /* read-write */
+        le32 config_select;         /* read-write */
+        u8 config[256];             /* read-write */
+        u8 create;                  /* read-write */
+        u8 create_ok;               /* read-only for driver */
+};
+
+/* create_ok values */
+#define VIRTIO_MGMT_OK          0
+#define VIRTIO_MGMT_ID_DUP      1
+#define VIRTIO_MGMT_INVALID_CFG 2
+\end{lstlisting}
+
+\begin{description}
+\item[\field{num_devices}]
+        The device specifies the maximum number of managed devices
+        here.
+\item[\field{device_select}]
+        Device Select. The driver selects which device the following
+        fields and other virtio-pci capabilities refer to.
+\item[\field{config_select}]
+        The driver uses this to select which 256 bytes chunk of
+        \field{config} shows.
+\item[field{config}]
+        The metadata that is used to create a managed device. The
+        actual format is device specific.
+\item[\field{create}]
+        The driver use this to create (write 1) or destroy (write 0) a
+        specific managed device.
+\item[\field{create_ok}]
+        The device specifies whether the managed device is
+        successfully created or destroyed. 0 means success, others
+        means failure.
+\end{description}
+
+\devicenormative{\paragraph}{Managed device configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Managed device configuration}
+
+The management device MUST reserved device index 0 as itself.
+
+The management device MUST present 0 to \field{device_select} upon
+reset.
+
+The management device MUST fail the write to \field{create} if the
+\field{device_select} is zero.
+
+The management device MUST fail the creation of a managed device if
+the filed\{device_select} is duplicated with the id of an existing
+managed device.
+
+The device MUST NOT present VIRTIO_PCI_CAP_MANAGED_DEV_CFG if
+VIRTIO_PCI_CAP_PASID is not presented.
+
+\drivernormative{\paragraph}{Managed device configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Managed device configuration}
+
+The management driver MUST create a managed device by allocating a
+non-zero device id for \field{device_select} and use that for the
+following steps for this managed device if the creating is succeed
+(read \field{create_ok} as zero).
+
 \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
 
 Transitional devices MUST present part of configuration
Signature · 1 line
2.24.3 (Apple Git-128)