Skip to contentSkip to conversation

Large shared memory regions

PATCH Superseded Dr. David Alan Gilbert (git) 2 participants 9 messages Jul 10, 2019 – Jul 12, 2019

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v6 cover Large shared memory regions

From: "Dr. David Alan Gilbert" <[email protected]>

This series formalises the idea of shared memory regions I'd
previously discussed and is intended for use with virtio-fs.

I've split it into five parts:

a) A general definition of the idea

b) An addition of an 'id' field to allow multiple distinct capabilities of the same type.

c) An addition of a 64bit capability type.

d) A definition on virtio-pci using the layout previously
discussed.

e) A definition on virtio-mmio.

v6:
Modifications from Cornelia's review

Dr. David Alan Gilbert (5):
shared memory: Define shared memory regions
pci: Define id field
pci: Define virtio_pci_cap64
shared memory: Define PCI capability
shared memory: Define mmio registers

 conformance.tex |  1 +
 content.tex     | 83 +++++++++++++++++++++++++++++++++++++++++++++++--
 shared-mem.tex  | 42 +++++++++++++++++++++++++
 3 files changed, 124 insertions(+), 2 deletions(-)
 create mode 100644 shared-mem.tex
Signature · 18 lines
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

PATCH v6 1/5 shared memory: Define shared memory regions

From: "Dr. David Alan Gilbert" <[email protected]>

Define the requirements and idea behind shared memory regions.

---
 conformance.tex |  1 +
 content.tex     |  2 ++
 shared-mem.tex  | 42 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+)
 create mode 100644 shared-mem.tex
conformance.tex +1 -0
index 42f702a..4524237 100644
@@ -197,6 +197,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring}
 \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression}
+\item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions}
 \item \ref{devicenormative:Reserved Feature Bits}
 \end{itemize}
 
content.tex +2 -0
index 8f0498e..6433226 100644
@@ -371,6 +371,8 @@ \subsection{Driver notifications} \label{sec:Virtqueues / Driver notifications}
 has been negotiated, these notifications would then have
 identical \field{next_off} and \field{next_wrap} values.
 
+\input{shared-mem.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
shared-mem.tex new file +42 -0

42 lines not on this page · raw message

Signature · 18 lines
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#2 lore raw

PATCH v6 2/5 pci: Define id field

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

For the virtio-fs device we require multiple large shared memory regions. Differentiate these by an 'id' field in the base capability.

---
 content.tex | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
content.tex +10 -2
index 6433226..fafa532 100644
@@ -651,7 +651,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
         u8 cap_len;     /* Generic PCI field: capability length */
         u8 cfg_type;    /* Identifies the structure. */
         u8 bar;         /* Where to find it. */
-        u8 padding[3];  /* Pad to full dword. */
+        u8 id;          /* Multiple capabilities of the same type */
+        u8 padding[2];  /* Pad to full dword. */
         le32 offset;    /* Offset within bar. */
         le32 length;    /* Length of the structure, in bytes. */
 };
@@ -697,7 +698,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
         The device MAY offer more than one structure of any type - this makes it
         possible for the device to expose multiple interfaces to drivers.  The order of
         the capabilities in the capability list specifies the order of preference
-        suggested by the device.
+        suggested by the device.  A device may specify that this ordering mechanism be
+        overridden by the use of the \field{id} field.
         \begin{note}
           For example, on some hypervisors, notifications using IO accesses are
         faster than memory accesses. In this case, the device would expose two
@@ -716,6 +718,12 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 
         Any other value is reserved for future use.
 
+\item[\field{id}]
+        Used by some device types to uniquely identify multiple capabilities
+        of a certain type. If the device type does not specify the meaning of
+        this field, its contents are undefined.
+
+
 \item[\field{offset}]
         indicates where the structure begins relative to the base address associated
         with the BAR.  The alignment requirements of \field{offset} are indicated
Signature · 18 lines
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Reviewed-by on v6 2/5
#3 lore raw

[virtio-comment] Re: [PATCH v6 2/5] pci: Define id field

Quoted 9 lines · On Wed, 10 Jul 2019 19:32:09 +0100 "Dr. David Alan Gilbert (git)"…

From: "Dr. David Alan Gilbert" <[email protected]>

For the virtio-fs device we require multiple large shared memory regions. Differentiate these by an 'id' field in the base capability.

---
 content.tex | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#4 lore raw

PATCH v6 3/5 pci: Define virtio_pci_cap64

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Define 'virtio_pci_cap64' to allow capabilities to describe memory regions larger than, or with an offset larger than 4GiB.

This will be used by the shared memory region capability.

---
 content.tex | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
content.tex +17 -0
index fafa532..d481359 100644
@@ -746,6 +746,23 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
         \end{note}
 \end{description}
 
+A variant of this type, struct virtio_pci_cap64, is defined for
+those capaibilites that require offsets or lengths larger than
+4GiB:
+
+\begin{lstlisting}
+struct virtio_pci_cap64 {
+        struct virtio_pci_cap cap;
+        u32 offset_hi;
+        u32 length_hi;
+};
+\end{lstlisting}
+
+Given that the \field{cap.length} and \field{cap.offset} fields
+are only 32 bit, the additional \field{offset_hi} and \field {length_hi}
+fields provide the most significant 32 bits of a total 64 bit offset and
+length within the bar specified by \field{cap.bar}.
+
 \drivernormative{\subsubsection}{Virtio Structure PCI Capabilities}{Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}
 
 The driver MUST ignore any vendor-specific capability structure which has
Signature · 18 lines
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#5 lore raw

PATCH v6 4/5 shared memory: Define PCI capability

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Define the PCI capability used for enumerating shared memory regions.

---
 content.tex | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
content.tex +22 -0
index d481359..1476b0e 100644
@@ -689,6 +689,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 #define VIRTIO_PCI_CAP_DEVICE_CFG        4
 /* PCI configuration access */
 #define VIRTIO_PCI_CAP_PCI_CFG           5
+/* Shared memory region */
+#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
 \end{lstlisting}
 
         Any other value is reserved for future use.
@@ -1077,6 +1079,26 @@ \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options
 
 The \field{offset} for the device-specific configuration MUST be 4-byte aligned.
 
+\subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
+
+Shared memory regions \ref{sec:Basic Facilities of a Virtio
+Device / Shared Memory Regions} are enumerated on the PCI transport
+as a sequence of VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capabilities, one per region.
+
+The capability is defined by a struct virtio_pci_cap64 and
+utilises the \field{cap.id} to allow multiple shared memory
+regions per device.
+The identifier in \field{cap.id} does not denote a certain order of
+preference; it is only used to uniquely identify a region.
+
+\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
+
+The region defined by the combination of the \field {cap.offset},
+\field {cap.offset_hi}, and \field {cap.length}, \field
+{cap.length_hi} fields MUST be contained within the declared bar.
+
+The \field{cap.id} MUST be unique for any one device instance.
+
 \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 
 The VIRTIO_PCI_CAP_PCI_CFG capability
Signature · 18 lines
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#6 lore raw

[virtio-comment] Re: [PATCH v6 4/5] shared memory: Define PCI capability

Quoted 9 lines · On Wed, 10 Jul 2019 19:32:11 +0100 "Dr. David Alan Gilbert (git)"…

From: "Dr. David Alan Gilbert" <[email protected]>

Define the PCI capability used for enumerating shared memory regions.

---
 content.tex | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

(...)

Quoted 1 line

+\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}

Sorry about not noticing that before, but doesn't that need to be referenced in the conformance section somewhere?

content.tex \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options · line 1104

Quoted 10 lines
content.tex +7 -0
+
+The region defined by the combination of the \field {cap.offset},
+\field {cap.offset_hi}, and \field {cap.length}, \field
+{cap.length_hi} fields MUST be contained within the declared bar.
+
+The \field{cap.id} MUST be unique for any one device instance.
+
 \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 
 The VIRTIO_PCI_CAP_PCI_CFG capability

Otherwise, looks good.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#7 lore raw

[virtio-comment] Re: [PATCH v6 4/5] shared memory: Define PCI capability

Quoted 13 lines · * Cornelia Huck ([email protected]) wrote:
Quoted 9 lines · On Wed, 10 Jul 2019 19:32:11 +0100 "Dr. David Alan Gilbert (git)"…

From: "Dr. David Alan Gilbert" <[email protected]>

Define the PCI capability used for enumerating shared memory regions.

---
 content.tex | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
Quoted 6 lines

(...)

Quoted 1 line

+\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}

Sorry about not noticing that before, but doesn't that need to be referenced in the conformance section somewhere?

OK, new version coming up.

Dave

Quoted 9 lines
Quoted 9 lines
content.tex +7 -0
+
+The region defined by the combination of the \field {cap.offset},
+\field {cap.offset_hi}, and \field {cap.length}, \field
+{cap.length_hi} fields MUST be contained within the declared bar.
+
+The \field{cap.id} MUST be unique for any one device instance.
+
 \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
Quoted 3 lines
Quoted 1 line

The VIRTIO_PCI_CAP_PCI_CFG capability

Otherwise, looks good.

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw

PATCH v6 5/5 shared memory: Define mmio registers

In reply to Dr. David Alan Gilbert (git)

From: "Dr. David Alan Gilbert" <[email protected]>

Define an MMIO interface to discover and map shared memory regions.

---
 content.tex | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
content.tex +30 -0
index 1476b0e..ee0d7c9 100644
@@ -1736,6 +1736,33 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     selected by writing to \field{QueueSel}.
   }
   \hline 
+  \mmioreg{SHMSel}{Shared memory id}{0x0ac}{W}{%
+    Writing to this register selects the shared memory region \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions}
+    following operations on \field{SHMLenLow}, \field{SHMLenHigh},
+    \field{SHMBaseLow} and \field{SHMBaseHigh} apply to.
+  }
+  \hline 
+  \mmiodreg{SHMLenLow}{SHMLenHigh}{Shared memory region 64 bit long length}{0x0b0}{0x0xb4}{R}{%
+    These registers return the length of the shared memory
+    region in bytes, as defined by the device for the region selected by
+    the \field{SHMSel} register.  The lower 32 bits of the length
+    are read from \field{SHMLenLow} and the higher 32 bits from
+    \field{SHMLenHigh}.  Reading from a non-existent
+    region (i.e. where the ID written to \field{SHMSel} is unused)
+    results in a length of -1.
+  }
+  \hline 
+  \mmiodreg{SHMBaseLow}{SHMBaseHigh}{Shared memory region 64 bit long physical address}{0x0b8}{0x0xbc}{R}{%
+    The driver reads these registers to discover the base address
+    of the region in physical address space.  This address is
+    chosen by the device (or other part of the VMM).
+    The lower 32 bits of the address are read from \field{SHMBaseLow}
+    with the higher 32 bits from \field{SHMBaseHigh}.  Reading
+    from a non-existent region (i.e. where the ID written to
+    \field{SHMSel} is unused) results in a base address of
+    0xffffffffffffffff.
+  }
+  \hline 
   \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{
     Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}).
     The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
@@ -1825,6 +1852,9 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 and if its value is zero (0x0) MUST abort initialization and
 MUST NOT access any other register.
 
+Drivers not expecting shared memory MUST NOT use the shared
+memory registers.
+
 Further initialization MUST follow the procedure described in
 \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}.
 
Signature · 18 lines
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/