Large shared memory regions
v6 cover Large shared memory regions
From: "Dr. David Alan Gilbert" <[email protected]>
This series formalises the idea of shared memory regions I'd
previously discussed and is intended for use with virtio-fs.
I've split it into five parts:
a) A general definition of the idea
b) An addition of an 'id' field to allow multiple distinct capabilities of the same type.
c) An addition of a 64bit capability type.
d) A definition on virtio-pci using the layout previously
discussed.
e) A definition on virtio-mmio.
v6:
Modifications from Cornelia's review
Dr. David Alan Gilbert (5):
shared memory: Define shared memory regions
pci: Define id field
pci: Define virtio_pci_cap64
shared memory: Define PCI capability
shared memory: Define mmio registers
conformance.tex | 1 + content.tex | 83 +++++++++++++++++++++++++++++++++++++++++++++++-- shared-mem.tex | 42 +++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 2 deletions(-) create mode 100644 shared-mem.tex
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
PATCH v6 1/5 shared memory: Define shared memory regions
From: "Dr. David Alan Gilbert" <[email protected]>
Define the requirements and idea behind shared memory regions.
--- conformance.tex | 1 + content.tex | 2 ++ shared-mem.tex | 42 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+) create mode 100644 shared-mem.tex
conformance.tex +1 -0
@@ -197,6 +197,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / The Virtqueue Used Ring} \item \ref{devicenormative:Basic Facilities of a Virtio Device / Virtqueues / Available Buffer Notification Suppression} +\item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions} \item \ref{devicenormative:Reserved Feature Bits} \end{itemize}
content.tex +2 -0
@@ -371,6 +371,8 @@ \subsection{Driver notifications} \label{sec:Virtqueues / Driver notifications} has been negotiated, these notifications would then have identical \field{next_off} and \field{next_wrap} values. +\input{shared-mem.tex} + \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation} We start with an overview of device initialization, then expand on the
shared-mem.tex new file +42 -0
42 lines not on this page · raw message
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
PATCH v6 2/5 pci: Define id field
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
For the virtio-fs device we require multiple large shared memory regions. Differentiate these by an 'id' field in the base capability.
--- content.tex | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
content.tex +10 -2
@@ -651,7 +651,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option u8 cap_len; /* Generic PCI field: capability length */ u8 cfg_type; /* Identifies the structure. */ u8 bar; /* Where to find it. */ - u8 padding[3]; /* Pad to full dword. */ + u8 id; /* Multiple capabilities of the same type */ + u8 padding[2]; /* Pad to full dword. */ le32 offset; /* Offset within bar. */ le32 length; /* Length of the structure, in bytes. */ };
@@ -697,7 +698,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option The device MAY offer more than one structure of any type - this makes it possible for the device to expose multiple interfaces to drivers. The order of the capabilities in the capability list specifies the order of preference - suggested by the device. + suggested by the device. A device may specify that this ordering mechanism be + overridden by the use of the \field{id} field. \begin{note} For example, on some hypervisors, notifications using IO accesses are faster than memory accesses. In this case, the device would expose two
@@ -716,6 +718,12 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option Any other value is reserved for future use. +\item[\field{id}] + Used by some device types to uniquely identify multiple capabilities + of a certain type. If the device type does not specify the meaning of + this field, its contents are undefined. + + \item[\field{offset}] indicates where the structure begins relative to the base address associated with the BAR. The alignment requirements of \field{offset} are indicated
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v6 2/5] pci: Define id field
Quoted 9 lines · On Wed, 10 Jul 2019 19:32:09 +0100 "Dr. David Alan Gilbert (git)"…
From: "Dr. David Alan Gilbert" <[email protected]>
For the virtio-fs device we require multiple large shared memory regions. Differentiate these by an 'id' field in the base capability.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- content.tex | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
PATCH v6 3/5 pci: Define virtio_pci_cap64
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Define 'virtio_pci_cap64' to allow capabilities to describe memory regions larger than, or with an offset larger than 4GiB.
This will be used by the shared memory region capability.
--- content.tex | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
content.tex +17 -0
@@ -746,6 +746,23 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option \end{note} \end{description} +A variant of this type, struct virtio_pci_cap64, is defined for +those capaibilites that require offsets or lengths larger than +4GiB: + +\begin{lstlisting} +struct virtio_pci_cap64 { + struct virtio_pci_cap cap; + u32 offset_hi; + u32 length_hi; +}; +\end{lstlisting} + +Given that the \field{cap.length} and \field{cap.offset} fields +are only 32 bit, the additional \field{offset_hi} and \field {length_hi} +fields provide the most significant 32 bits of a total 64 bit offset and +length within the bar specified by \field{cap.bar}. + \drivernormative{\subsubsection}{Virtio Structure PCI Capabilities}{Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities} The driver MUST ignore any vendor-specific capability structure which has
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
PATCH v6 4/5 shared memory: Define PCI capability
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Define the PCI capability used for enumerating shared memory regions.
--- content.tex | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
content.tex +22 -0
@@ -689,6 +689,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option #define VIRTIO_PCI_CAP_DEVICE_CFG 4 /* PCI configuration access */ #define VIRTIO_PCI_CAP_PCI_CFG 5 +/* Shared memory region */ +#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8 \end{lstlisting} Any other value is reserved for future use.
@@ -1077,6 +1079,26 @@ \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options The \field{offset} for the device-specific configuration MUST be 4-byte aligned. +\subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability} + +Shared memory regions \ref{sec:Basic Facilities of a Virtio +Device / Shared Memory Regions} are enumerated on the PCI transport +as a sequence of VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capabilities, one per region. + +The capability is defined by a struct virtio_pci_cap64 and +utilises the \field{cap.id} to allow multiple shared memory +regions per device. +The identifier in \field{cap.id} does not denote a certain order of +preference; it is only used to uniquely identify a region. + +\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability} + +The region defined by the combination of the \field {cap.offset}, +\field {cap.offset_hi}, and \field {cap.length}, \field +{cap.length_hi} fields MUST be contained within the declared bar. + +The \field{cap.id} MUST be unique for any one device instance. + \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability} The VIRTIO_PCI_CAP_PCI_CFG capability
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v6 4/5] shared memory: Define PCI capability
Quoted 9 lines · On Wed, 10 Jul 2019 19:32:11 +0100 "Dr. David Alan Gilbert (git)"…
From: "Dr. David Alan Gilbert" <[email protected]>
Define the PCI capability used for enumerating shared memory regions.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- content.tex | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
(...)
Quoted 1 line
+\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
Sorry about not noticing that before, but doesn't that need to be referenced in the conformance section somewhere?
content.tex \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options · line 1104
Quoted 10 lines
content.tex +7
-0+ +The region defined by the combination of the \field {cap.offset}, +\field {cap.offset_hi}, and \field {cap.length}, \field +{cap.length_hi} fields MUST be contained within the declared bar. + +The \field{cap.id} MUST be unique for any one device instance. + \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability} The VIRTIO_PCI_CAP_PCI_CFG capability
Otherwise, looks good.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
[virtio-comment] Re: [PATCH v6 4/5] shared memory: Define PCI capability
Quoted 13 lines · * Cornelia Huck ([email protected]) wrote:
Quoted 9 lines · On Wed, 10 Jul 2019 19:32:11 +0100 "Dr. David Alan Gilbert (git)"…
From: "Dr. David Alan Gilbert" <[email protected]>
Define the PCI capability used for enumerating shared memory regions.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- content.tex | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
Quoted 6 lines
(...)
Quoted 1 line
+\devicenormative{\paragraph}{Device-specific configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
Sorry about not noticing that before, but doesn't that need to be referenced in the conformance section somewhere?
OK, new version coming up.
Dave
Quoted 9 lines
Quoted 9 lines
content.tex +7
-0+ +The region defined by the combination of the \field {cap.offset}, +\field {cap.offset_hi}, and \field {cap.length}, \field +{cap.length_hi} fields MUST be contained within the declared bar. + +The \field{cap.id} MUST be unique for any one device instance. + \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
Quoted 3 lines
Quoted 1 line
The
VIRTIO_PCI_CAP_PCI_CFGcapabilityOtherwise, looks good.
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
PATCH v6 5/5 shared memory: Define mmio registers
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Define an MMIO interface to discover and map shared memory regions.
--- content.tex | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
content.tex +30 -0
@@ -1736,6 +1736,33 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi selected by writing to \field{QueueSel}. } \hline + \mmioreg{SHMSel}{Shared memory id}{0x0ac}{W}{% + Writing to this register selects the shared memory region \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions} + following operations on \field{SHMLenLow}, \field{SHMLenHigh}, + \field{SHMBaseLow} and \field{SHMBaseHigh} apply to. + } + \hline + \mmiodreg{SHMLenLow}{SHMLenHigh}{Shared memory region 64 bit long length}{0x0b0}{0x0xb4}{R}{% + These registers return the length of the shared memory + region in bytes, as defined by the device for the region selected by + the \field{SHMSel} register. The lower 32 bits of the length + are read from \field{SHMLenLow} and the higher 32 bits from + \field{SHMLenHigh}. Reading from a non-existent + region (i.e. where the ID written to \field{SHMSel} is unused) + results in a length of -1. + } + \hline + \mmiodreg{SHMBaseLow}{SHMBaseHigh}{Shared memory region 64 bit long physical address}{0x0b8}{0x0xbc}{R}{% + The driver reads these registers to discover the base address + of the region in physical address space. This address is + chosen by the device (or other part of the VMM). + The lower 32 bits of the address are read from \field{SHMBaseLow} + with the higher 32 bits from \field{SHMBaseHigh}. Reading + from a non-existent region (i.e. where the ID written to + \field{SHMSel} is unused) results in a base address of + 0xffffffffffffffff. + } + \hline \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{ Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}). The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
@@ -1825,6 +1852,9 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti and if its value is zero (0x0) MUST abort initialization and MUST NOT access any other register. +Drivers not expecting shared memory MUST NOT use the shared +memory registers. + Further initialization MUST follow the procedure described in \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}.
Signature · 18 lines
2.21.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/