shared-mem: introduce page alignment restrictions
cover shared-mem: introduce page alignment restrictions
There's an incresing number of machines supporting multiple page sizes and, on these machines, the host and a guest can be running with different pages sizes.
In addition to this, there might be devices that have a required and/or preferred page size for mapping memory.
In this series we extend the "Shared Memory Regions" with a subsection explaining the posible existence of page alignment restrictions.
For the device to provide the page size information to the driver, we need to extend the PCI and MMIO transports. For the former, we borrow 8 bits from the 16 bit padding in virtio_pci_cap to hold a page_shift field which can be used to derive the page size by using the following
formula: (page_size = 1 << (page_shift + 12)).
For MMIO, we add a the SHMPageShift register at offset 0x0c4, also holding the page_shift value. Since MMIO registers are 32 bit wide, we could have asked the device to directly provide page_size instead of page_shift, but seems reasonable to be consistent across transports.
Since accessing a new MMIO register could be interpreted by the device as an error, this behavior is gated behind the newly introduced VIRTIO_F_SHM_PAGE_SIZE feature bit.
An implementation of the changes proposed in this series has been published as an RFC to the LKML, to be used as a reference:
https://lore.kernel.org/all/[email protected]/
Sergio Lopez (3):
shared-mem: introduce page alignment restrictions
transport-pci: VIRTIO_F_SHM_PAGE_SIZE support
transport-mmio: VIRTIO_F_SHM_PAGE_SIZE
content.tex | 3 +++ shared-mem.tex | 7 +++++++ transport-mmio.tex | 8 ++++++++ transport-pci.tex | 10 +++++++++- 4 files changed, 27 insertions(+), 1 deletion(-)
Signature · 1 line
2.48.1
PATCH 1/3 shared-mem: introduce page alignment restrictions
Add a subsection for page alignment restrictions and introduce the VIRTIO_F_SHM_PAGE_SIZE feature bit.
--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)
content.tex +3 -0
@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for handling features reserved for future use. + \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that the device + transport provides information about the supported page size. + \end{description} \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
shared-mem.tex +7 -0
@@ -34,6 +34,13 @@ \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De The \field{shmid} may be explicit or may be inferred from the context of the reference. +\subsection{Page alignment restrictions}\label{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Page alignment restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting +the device to map memory into a shared memory region, the driver +MUST obtain the page size information from the transport and honor +the page alignment constrains derived from that page size. + \devicenormative{\subsection}{Shared Memory Regions}{Basic Facilities of a Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which
Signature · 1 line
2.48.1
Quoted 21 lines
From: Sergio Lopez <[email protected]>
Sent: Monday, February 17, 2025 5:22 PMAdd a subsection for page alignment restrictions and introduce the
VIRTIO_F_SHM_PAGE_SIZEfeature bit.Signed-off-by Sergio Lopez <[email protected]>--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)content.tex +0
-0@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}for
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 877
Quoted 5 lines
content.tex +3
-0handling features reserved for future use. + \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that the + device transport provides information about the supported page size. +
Please extend the description around "Feature Bits" section to increase 24 to 41 -> 24 to 42.
Quoted 11 lines
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} diff
--git a/shared-mem.tex b/shared-mem.tex index 6e6f6c4..dd90cb7 100644shared-mem.tex +0
-0@@ -34,6 +34,13 @@ \subsection{Addressing within regions}\label{sec:BasicFacilities of a Virtio De The \field{shmid} may be explicit or may be inferred from the context of the reference.
+\subsection{Page alignment restrictions}\label{sec:Basic Facilities of
shared-mem.tex
Quoted 7 lines
shared-mem.tex +7
-0+a Virtio Device / Shared Memory Regions / Page alignment restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting the +device to map memory into a shared memory region, the driver MUST +obtain the page size information from the transport and honor the page +alignment constrains derived from that page size. +
A device requirement is also needed to indicate that pci capability field is only valid after/when the feature is negotiated.
Quoted 6 lines
\devicenormative{\subsection}{Shared Memory Regions}{Basic Facilities of a
Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which
--
2.48.1
1. Shared memory cap is already defined as VIRTIO_PCI_CAP_SHARED_MEMORY_CFG. This capability should be extended instead of polluting generic structure field of padding.
If its done in generic way like proposed, you need to call out that page_size field is only valid for VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.
Overall extending pci cap structure is not good idea even though it may appear as small change for below reasons.
1. PCI spec already ran out of total bytes that can be stored in the capability section. Extending it will not help. A virtio level extended capability is not good either due to below guidance from PCI-SIG.
2. PCI spec highly discouraged putting vendor specific bits like this in the capability section.
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability structures architected by applicable PCI specifications."
So, you should consider discovering the shared memory page size and more such fields using device administration commands.
It may sound overkill to use administration commands for 8-bits of page size,
but as more generic and device specific things evolve, it is likely useful vehicle where you don't have to search hard to squeeze things in some existing structure like pci capability.
I skipped reviewing patch 2 and 3, and would like to understand your thoughts for above considerations.
Quoted 22 lines · Parav Pandit
Quoted 22 lines
From: Sergio Lopez <[email protected]>
Sent: Monday, February 17, 2025 5:22 PMAdd a subsection for page alignment restrictions and introduce the
VIRTIO_F_SHM_PAGE_SIZEfeature bit.Signed-off-by Sergio Lopez <[email protected]>--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)content.tex +0
-0@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}for
handling features reserved for future use.
Quoted 5 lines
content.tex
Quoted 3 lines
content.tex +3
-0+ \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that the + device transport provides information about the supported page size. +Please extend the description around "Feature Bits" section to increase 24 to 41 -> 24 to 42.
Ack.
Quoted 10 lines
Quoted 10 lines
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} diff
--git a/shared-mem.tex b/shared-mem.tex index 6e6f6c4..dd90cb7 100644shared-mem.tex +0
-0@@ -34,6 +34,13 @@ \subsection{Addressing within regions}\label{sec:BasicFacilities of a Virtio De The \field{shmid} may be explicit or may be inferred from the context of the reference.
Quoted 9 lines
shared-mem.tex
Quoted 8 lines
+\subsection{Page alignment restrictions}\label{sec:Basic Facilities of
shared-mem.tex +7
-0+a Virtio Device / Shared Memory Regions / Page alignment restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting the +device to map memory into a shared memory region, the driver MUST +obtain the page size information from the transport and honor the page +alignment constrains derived from that page size. +A device requirement is also needed to indicate that pci capability field is only valid after/when the feature is negotiated.
Ack.
Quoted 11 lines
Quoted 6 lines
\devicenormative{\subsection}{Shared Memory Regions}{Basic Facilities of a
Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which
--
2.48.11. Shared memory cap is already defined as
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG. This capability should be extended instead of polluting generic structure field of padding.If its done in generic way like proposed, you need to call out that
page_sizefield is only valid forVIRTIO_PCI_CAP_SHARED_MEMORY_CFG.
Quoted 8 lines
Overall extending pci cap structure is not good idea even though it may appear as small change for below reasons.
1. PCI spec already ran out of total bytes that can be stored in the capability section. Extending it will not help. A virtio level extended capability is not good either due to below guidance from PCI-SIG.
2. PCI spec highly discouraged putting vendor specific bits like this in the capability section.
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability structures architected by applicable PCI specifications."
Honestly, making it clear that the field is only valid for VIRTIO_PCI_CAP_SHARED_MEMORY_CFG _and_ when VIRTIO_F_SHM_PAGE_SIZE has been negotiated (I can extend the comment of the field itself to clarify that, otherwise, it just acts as padding), I fail to see which kind of problems might bring, but admitedly I'm not a PCI expert by any means.
If using the padding is really not an option, would it be reasonable to introduce a different variant of "virtio_pci_cap64" struct, including the page_size, to be used when VIRTIO_F_SHM_PAGE_SIZE has been negotiated?
IIUC, "virtio_pci_cap64" is VIRTIO-specific, so this shouldn't have impact anywhere else.
Quoted 4 lines
So, you should consider discovering the shared memory page size and more such fields using device administration commands.
It may sound overkill to use administration commands for 8-bits of page size,
but as more generic and device specific things evolve, it is likely useful vehicle where you don't have to search hard to squeeze things in some existing structure like pci capability.
Do you mean using an adminitration queue (VIRTIO_F_ADMIN_VQ)? If so, yes, I think it's extremely overkill for both the device and the driver. If possible, I would like explore every other option before resorting to this one.
Thanks!
Sergio.
Quoted 56 lines
From: Sergio Lopez Pascual <[email protected]>
Sent: Wednesday, February 26, 2025 11:20 PMQuoted 21 lines · Parav Pandit
Quoted 21 lines
From: Sergio Lopez <[email protected]>
Sent: Monday, February 17, 2025 5:22 PMAdd a subsection for page alignment restrictions and introduce the
VIRTIO_F_SHM_PAGE_SIZEfeature bit.Signed-off-by Sergio Lopez <[email protected]>--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)content.tex +0
-0@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / FeatureBits} for
handling features reserved for future use.Quoted 5 lines
content.tex
Quoted 3 lines
content.tex +3
-0+ \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that the + device transport provides information about the supported page size. +Please extend the description around "Feature Bits" section to increase 24
to 41 -> 24 to 42.
Ack.
Quoted 11 lines
Quoted 11 lines
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature
Bits} diff --git a/shared-mem.tex b/shared-mem.tex index 6e6f6c4..dd90cb7 100644shared-mem.tex +0
-0@@ -34,6 +34,13 @@ \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De The \field{shmid}may be explicit or may be inferred from the context of the reference.
Quoted 10 lines
shared-mem.tex
Quoted 9 lines
+\subsection{Page alignment restrictions}\label{sec:Basic Facilities
shared-mem.tex +8
-0+of a Virtio Device / Shared Memory Regions / Page alignment +restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting the +device to map memory into a shared memory region, the driver MUST +obtain the page size information from the transport and honor the +page alignment constrains derived from that page size. +A device requirement is also needed to indicate that pci capability field is
Quoted 3 lines
only valid after/when the feature is negotiated.
Ack.
I don’t think this is necessary because you say "driver obtains the information".
So to obtain a free information, feature bit is not necessary, this can be self-described in the pci capability itself. If a pci capability padding_bit_0 is set, it can tell rest of the bits have abc meaning.
This way pci capabilities are self-contained.
Quoted 27 lines
Quoted 8 lines
Quoted 7 lines
\devicenormative{\subsection}{Shared Memory Regions}{Basic
Facilities of a Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which --
2.48.11. Shared memory cap is already defined as
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Quoted 1 line
This capability should be extended instead of polluting generic structure
field of padding.
Quoted 2 lines
If its done in generic way like proposed, you need to call out that
page_sizefield is only valid for
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Quoted 2 lines
Overall extending pci cap structure is not good idea even though it may
appear as small change for below reasons.
Quoted 2 lines
1. PCI spec already ran out of total bytes that can be stored in the capability
section. Extending it will not help.
Quoted 1 line
A virtio level extended capability is not good either due to below guidance
from PCI-SIG.
Quoted 2 lines
2. PCI spec highly discouraged putting vendor specific bits like this in the
capability section.
Quoted 2 lines
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability
Quoted 9 lines
structures architected by applicable PCI specifications."
Honestly, making it clear that the field is only valid for
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG_and_ whenVIRTIO_F_SHM_PAGE_SIZEhas been negotiated (I can extend the comment of the field itself to clarify that, otherwise, it just acts as padding), I fail to see which kind of problems might bring, but admitedly I'm not a PCI expert by any means.
Feature negotiation for information exposed by the device is not needed.
Quoted 4 lines
If using the padding is really not an option, would it be reasonable to introduce a different variant of "
virtio_pci_cap64" struct, including thepage_size, to be used whenVIRTIO_F_SHM_PAGE_SIZEhas been negotiated?
It can be extended. It just not right way to do so in pci specs boundary.
Quoted 8 lines
IIUC, "
virtio_pci_cap64" is VIRTIO-specific, so this shouldn't have impact anywhere else.Quoted 1 line
So, you should consider discovering the shared memory page size and more
such fields using device administration commands.
Quoted 3 lines
It may sound overkill to use administration commands for 8-bits of page size, but as more generic and device specific things evolve, it is likely
Quoted 8 lines
useful vehicle where you don't have to search hard to squeeze things in some existing structure like pci capability.
Do you mean using an adminitration queue (
VIRTIO_F_ADMIN_VQ)? If so, yes, I think it's extremely overkill for both the device and the driver. If possible, I would like explore every other option before resorting to this one.
Right. I agree that exposing this information via admin command is overkill.
Do you agree that shared memory interface wont grow anymore further, only thing it needs now or in any near term is just one or two bytes of page size information? If yes, packing inside the pci cap seems fine.
If one needs to start expanding this more, than more mature interface like admin cmd will be useful.
Quoted 123 lines · Parav Pandit
Quoted 55 lines
From: Sergio Lopez Pascual <[email protected]>
Sent: Wednesday, February 26, 2025 11:20 PMQuoted 21 lines · Parav Pandit
Quoted 21 lines
From: Sergio Lopez <[email protected]>
Sent: Monday, February 17, 2025 5:22 PMAdd a subsection for page alignment restrictions and introduce the
VIRTIO_F_SHM_PAGE_SIZEfeature bit.Signed-off-by Sergio Lopez <[email protected]>--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)content.tex +0
-0@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / FeatureBits} for
handling features reserved for future use.Quoted 5 lines
content.tex
Quoted 3 lines
content.tex +3
-0+ \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that the + device transport provides information about the supported page size. +Please extend the description around "Feature Bits" section to increase 24
to 41 -> 24 to 42.
Ack.
Quoted 11 lines
Quoted 11 lines
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature
Bits} diff --git a/shared-mem.tex b/shared-mem.tex index 6e6f6c4..dd90cb7 100644shared-mem.tex +0
-0@@ -34,6 +34,13 @@ \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De The \field{shmid}may be explicit or may be inferred from the context of the reference.
Quoted 10 lines
shared-mem.tex
Quoted 9 lines
+\subsection{Page alignment restrictions}\label{sec:Basic Facilities
shared-mem.tex +8
-0+of a Virtio Device / Shared Memory Regions / Page alignment +restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting the +device to map memory into a shared memory region, the driver MUST +obtain the page size information from the transport and honor the +page alignment constrains derived from that page size. +A device requirement is also needed to indicate that pci capability field is
Quoted 3 lines
only valid after/when the feature is negotiated.
Ack.
I don’t think this is necessary because you say "driver obtains the information".
So to obtain a free information, feature bit is not necessary, this can be self-described in the pci capability itself. If a pci capabilitypadding_bit_0is set, it can tell rest of the bits have abc meaning.
This way pci capabilities are self-contained.Quoted 27 lines
Quoted 8 lines
Quoted 7 lines
\devicenormative{\subsection}{Shared Memory Regions}{Basic
Facilities of a Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which --
2.48.11. Shared memory cap is already defined as
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Quoted 1 line
This capability should be extended instead of polluting generic structure
field of padding.
Quoted 2 lines
If its done in generic way like proposed, you need to call out that
page_sizefield is only valid for
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Quoted 2 lines
Overall extending pci cap structure is not good idea even though it may
appear as small change for below reasons.
Quoted 2 lines
1. PCI spec already ran out of total bytes that can be stored in the capability
section. Extending it will not help.
Quoted 1 line
A virtio level extended capability is not good either due to below guidance
from PCI-SIG.
Quoted 2 lines
2. PCI spec highly discouraged putting vendor specific bits like this in the
capability section.
Quoted 2 lines
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability
Quoted 9 lines
structures architected by applicable PCI specifications."
Honestly, making it clear that the field is only valid for
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG_and_ whenVIRTIO_F_SHM_PAGE_SIZEhas been negotiated (I can extend the comment of the field itself to clarify that, otherwise, it just acts as padding), I fail to see which kind of problems might bring, but admitedly I'm not a PCI expert by any means.Feature negotiation for information exposed by the device is not needed.
Quoted 4 lines
If using the padding is really not an option, would it be reasonable to introduce a different variant of "
virtio_pci_cap64" struct, including thepage_size, to be used whenVIRTIO_F_SHM_PAGE_SIZEhas been negotiated?It can be extended. It just not right way to do so in pci specs boundary.
Quoted 8 lines
IIUC, "
virtio_pci_cap64" is VIRTIO-specific, so this shouldn't have impact anywhere else.Quoted 1 line
So, you should consider discovering the shared memory page size and more
such fields using device administration commands.
Quoted 3 lines
It may sound overkill to use administration commands for 8-bits of page size, but as more generic and device specific things evolve, it is likely
Quoted 8 lines
useful vehicle where you don't have to search hard to squeeze things in some existing structure like pci capability.
Do you mean using an adminitration queue (
VIRTIO_F_ADMIN_VQ)? If so, yes, I think it's extremely overkill for both the device and the driver. If possible, I would like explore every other option before resorting to this one.
Quoted 4 lines
Right. I agree that exposing this information via admin command is overkill.
Do you agree that shared memory interface wont grow anymore further, only thing it needs now or in any near term is just one or two bytes of page size information? If yes, packing inside the pci cap seems fine.
If one needs to start expanding this more, than more mature interface like admin cmd will be useful.
I can't think of any more information that would be needed to be added to the SHM interface beyond the page size. In a certain sense, it could be argued that it should have been there when shared memory regions were first introduced, since knowing the alignment restrictions for the maps is a requirement to able to make use of the region. It worked so far just because 4K page systems/VMs were prevalent.
Thanks!
Sergio.
Quoted 155 lines
From: Sergio Lopez Pascual <[email protected]>
Sent: Thursday, March 6, 2025 12:39 AMQuoted 26 lines · Parav Pandit
Quoted 26 lines
From: Sergio Lopez Pascual <[email protected]>
Sent: Wednesday, February 26, 2025 11:20 PM Parav PanditQuoted 22 lines · <[email protected]> writes:
Quoted 22 lines
From: Sergio Lopez <[email protected]>
Sent: Monday, February 17, 2025 5:22 PMAdd a subsection for page alignment restrictions and introduce the
VIRTIO_F_SHM_PAGE_SIZEfeature bit.Signed-off-by Sergio Lopez <[email protected]>--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)content.tex index c17ffa6..575065d renamed from content.tex +0
-0100644
content.tex +0
-0@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device /Feature Bits} for
handling features reserved for future use.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
+ \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that
+ the device transport provides information about the supported pagesize.
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
Quoted 1 line
+
Please extend the description around "Feature Bits" section to increase 24
to 41 -> 24 to 42.
Ack.
Quoted 2 lines
Quoted 2 lines
\end{description}
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
\drivernormative{\section}{Reserved Feature Bits}{Reserved
Feature Bits} diff --git a/shared-mem.tex b/shared-mem.tex index 6e6f6c4..dd90cb7 100644shared-mem.tex +0
-0@@ -34,6 +34,13 @@ \subsection{Addressing withinregions}\label{sec:Basic Facilities of a Virtio De The \field{shmid} may be explicit or may be inferred from the context of the
reference.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
shared-mem.tex \subsection{Addressing within regions}\label{sec:Basic Facilities of a Virtio De · line 43
Quoted 10 lines
+\subsection{Page alignment restrictions}\label{sec:Basic
shared-mem.tex +8
-0+Facilities of a Virtio Device / Shared Memory Regions / Page +alignment restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting +the device to map memory into a shared memory region, the driver +MUST obtain the page size information from the transport and +honor the page alignment constrains derived from that page size. +A device requirement is also needed to indicate that pci capability field is
Quoted 4 lines
Quoted 3 lines
only valid after/when the feature is negotiated.
Ack.
I don’t think this is necessary because you say "driver obtains the
information".
Quoted 1 line
So to obtain a free information, feature bit is not necessary, this can be self-
described in the pci capability itself.
Quoted 1 line
If a pci capability
padding_bit_0is set, it can tell rest of the bits have abcmeaning.
Quoted 40 lines
This way pci capabilities are self-contained.
Quoted 38 lines
Quoted 8 lines
Quoted 7 lines
\devicenormative{\subsection}{Shared Memory Regions}{Basic
Facilities of a Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which --
2.48.11. Shared memory cap is already defined as
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Quoted 2 lines
This capability should be extended instead of polluting generic structure
field of padding.
Quoted 3 lines
If its done in generic way like proposed, you need to call out that
page_sizefield is only valid for
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Quoted 3 lines
Overall extending pci cap structure is not good idea even though it may
appear as small change for below reasons.
Quoted 3 lines
1. PCI spec already ran out of total bytes that can be stored in the capability
section. Extending it will not help.
Quoted 2 lines
A virtio level extended capability is not good either due to below guidance
from PCI-SIG.
Quoted 3 lines
2. PCI spec highly discouraged putting vendor specific bits like this in the
capability section.
Quoted 3 lines
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability
structures architected by applicable PCI specifications."
Honestly, making it clear that the field is only valid for
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG_and_ when
VIRTIO_F_SHM_PAGE_SIZEQuoted 10 lines
Quoted 5 lines
has been negotiated (I can extend the comment of the field itself to clarify that, otherwise, it just acts as padding), I fail to see which kind of problems might bring, but admitedly I'm not a PCI expert by any means.
Feature negotiation for information exposed by the device is not needed.
Quoted 3 lines
If using the padding is really not an option, would it be reasonable to introduce a different variant of "
virtio_pci_cap64" struct, including thepage_size, to be used whenVIRTIO_F_SHM_PAGE_SIZEhasbeen negotiated?
Quoted 16 lines
Quoted 1 line
It can be extended. It just not right way to do so in pci specs boundary.
Quoted 13 lines
IIUC, "
virtio_pci_cap64" is VIRTIO-specific, so this shouldn't have impact anywhere else.Quoted 2 lines
So, you should consider discovering the shared memory page size and more
such fields using device administration commands.
Quoted 4 lines
It may sound overkill to use administration commands for 8-bits of page size, but as more generic and device specific things evolve, it is likely
useful vehicle where you don't have to search hard to squeeze things in some existing structure like pci capability.
Quoted 7 lines
Quoted 5 lines
Do you mean using an adminitration queue (
VIRTIO_F_ADMIN_VQ)? If so, yes, I think it's extremely overkill for both the device and the driver. If possible, I would like explore every other option before resorting to this one.Right. I agree that exposing this information via admin command is overkill. Do you agree that shared memory interface wont grow anymore further,
only thing it needs now or in any near term is just one or two bytes of page size information?
Quoted 2 lines
If yes, packing inside the pci cap seems fine.
If one needs to start expanding this more, than more mature interface like
Quoted 9 lines
admin cmd will be useful.
I can't think of any more information that would be needed to be added to the SHM interface beyond the page size. In a certain sense, it could be argued that it should have been there when shared memory regions were first introduced, since knowing the alignment restrictions for the maps is a requirement to able to make use of the region. It worked so far just because 4K page systems/VMs were prevalent.
Sounds good. So device sharing the info without the feature bit in this field looks ok to me.
Quoted 2 lines
Thanks!
Sergio.
Quoted 65 lines · Parav Pandit
Quoted 21 lines
From: Sergio Lopez <[email protected]>
Sent: Monday, February 17, 2025 5:22 PMAdd a subsection for page alignment restrictions and introduce the
VIRTIO_F_SHM_PAGE_SIZEfeature bit.Signed-off-by Sergio Lopez <[email protected]>--- content.tex | 3 +++ shared-mem.tex | 7 +++++++ 2 files changed, 10 insertions(+)content.tex +0
-0@@ -872,6 +872,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits}for
content.tex \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} · line 877
Quoted 5 lines
content.tex +3
-0handling features reserved for future use. + \item[VIRTIO_F_SHM_PAGE_SIZE(42)] This feature indicates that the + device transport provides information about the supported page size. +Please extend the description around "Feature Bits" section to increase 24 to 41 -> 24 to 42.
Quoted 11 lines
\end{description}
\drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits} diff
--git a/shared-mem.tex b/shared-mem.tex index 6e6f6c4..dd90cb7 100644shared-mem.tex +0
-0@@ -34,6 +34,13 @@ \subsection{Addressing within regions}\label{sec:BasicFacilities of a Virtio De The \field{shmid} may be explicit or may be inferred from the context of the reference.
+\subsection{Page alignment restrictions}\label{sec:Basic Facilities of
shared-mem.tex
Quoted 7 lines
shared-mem.tex +7
-0+a Virtio Device / Shared Memory Regions / Page alignment restrictions} + +If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, when requesting the +device to map memory into a shared memory region, the driver MUST +obtain the page size information from the transport and honor the page +alignment constrains derived from that page size. +A device requirement is also needed to indicate that pci capability field is only valid after/when the feature is negotiated.
Quoted 6 lines
\devicenormative{\subsection}{Shared Memory Regions}{Basic Facilities of a
Virtio Device / Shared Memory Regions} Shared memory regions MUST NOT expose shared memory regions which
--
2.48.11. Shared memory cap is already defined as
VIRTIO_PCI_CAP_SHARED_MEMORY_CFG. This capability should be extended instead of polluting generic structure field of padding.If its done in generic way like proposed, you need to call out that
page_sizefield is only valid forVIRTIO_PCI_CAP_SHARED_MEMORY_CFG.Overall extending pci cap structure is not good idea even though it may appear as small change for below reasons.
1. PCI spec already ran out of total bytes that can be stored in the capability section. Extending it will not help. A virtio level extended capability is not good either due to below guidance from PCI-SIG.
Quoted 8 lines
2. PCI spec highly discouraged putting vendor specific bits like this in the capability section.
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability structures architected by applicable PCI specifications."So, you should consider discovering the shared memory page size and more such fields using device administration commands.
It may sound overkill to use administration commands for 8-bits of page size,
but as more generic and device specific things evolve, it is likely useful vehicle where you don't have to search hard to squeeze things in some existing structure like pci capability.
Or just in config space, really.
Quoted 1 line
I skipped reviewing patch 2 and 3, and would like to understand your thoughts for above considerations.
There's actually a demand from embedded folks to allow the capabilities in a fixed offset in a BAR instead of in config space.
Quoted 8 lines · Parav Pandit
Overall extending pci cap structure is not good idea even though it may appear as small change for below reasons.
1. PCI spec already ran out of total bytes that can be stored in the capability section. Extending it will not help. A virtio level extended capability is not good either due to below guidance from PCI-SIG.
2. PCI spec highly discouraged putting vendor specific bits like this in the capability section.
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability structures architected by applicable PCI specifications."
We can argue about new capabilities, but this is not doing it - it is merely extending the existing one.
So I do not really get any of these arguments.
I'd be open to an alternative way to discover capabilities, as long as we don't do that, I don't really see a reason to block minor tweaks like this one.
At the same time, I am not sure why this is a capability at all. why not in config space?
Signature · 1 line
MST
Quoted 15 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 4, 2025 4:23 PMQuoted 1 line · Parav Pandit
Overall extending pci cap structure is not good idea even though it may
appear as small change for below reasons.
Quoted 2 lines
1. PCI spec already ran out of total bytes that can be stored in the capability
section. Extending it will not help.
Quoted 1 line
A virtio level extended capability is not good either due to below guidance
from PCI-SIG.
Quoted 2 lines
2. PCI spec highly discouraged putting vendor specific bits like this in the
Quoted 9 lines
capability section.
Quoted 2 lines
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability
structures architected by applicable PCI specifications."
We can argue about new capabilities, but this is not doing it - it is merely extending the existing one.
This patch is not even extending it. it is reusing the pad field, which is good. Extending capability is problematic.
Quoted 2 lines
So I do not really get any of these arguments.
I asked him to explore what all options did he consider.
Quoted 4 lines
I'd be open to an alternative way to discover capabilities, as long as we don't do that, I don't really see a reason to block minor tweaks like this one.
I am not blocking it. would like to know what other options were considered if the shared memory is going to grow or its just one-off field that can live in PCI capability field. I suggest, that this should be done without negotiating a feature bit. Because the device is just telling the restrictions...
Quoted 3 lines
At the same time, I am not sure why this is a capability at all. why not in config space?
When you say "config space", you meant "virtio config space" and not "pci config space".
Assuming yes, I believe the problem is, Sergio needs to add this field in ALL 19+ devices config space, which is not good.
Quoted 2 lines
--
MST
Quoted 29 lines · Parav Pandit
Quoted 15 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, March 4, 2025 4:23 PMQuoted 1 line · Parav Pandit
Overall extending pci cap structure is not good idea even though it may
appear as small change for below reasons.
Quoted 2 lines
1. PCI spec already ran out of total bytes that can be stored in the capability
section. Extending it will not help.
Quoted 1 line
A virtio level extended capability is not good either due to below guidance
from PCI-SIG.
Quoted 2 lines
2. PCI spec highly discouraged putting vendor specific bits like this in the
Quoted 9 lines
capability section.
Quoted 2 lines
Citation: "It is strongly recommended that PCI Express devices place no registers in Configuration Space other than those in headers or Capability
structures architected by applicable PCI specifications."
We can argue about new capabilities, but this is not doing it - it is merely extending the existing one.
This patch is not even extending it. it is reusing the pad field, which is good. Extending capability is problematic.
Quoted 2 lines
So I do not really get any of these arguments.
Quoted 7 lines
I asked him to explore what all options did he consider.
Quoted 4 lines
I'd be open to an alternative way to discover capabilities, as long as we don't do that, I don't really see a reason to block minor tweaks like this one.
I am not blocking it.
Sorry if it sounded I implied you do. I just said we shouldn't :)
Quoted 4 lines
would like to know what other options were
considered if the shared memory is going to grow or its just one-off
field that can live in PCI capability field.
I suggest, that this should be done without negotiating a feature bit. Because the device is just telling the restrictions...
True. But the feature is helpful for device to discover whether driver follows them. If not it can fail FEATURES_OK if it wants to.
Quoted 5 lines
Quoted 3 lines
At the same time, I am not sure why this is a capability at all. why not in config space?
When you say "config space", you meant "virtio config space" and not "pci config space".
Assuming yes, I believe the problem is, Sergio needs to add this field in ALL 19+ devices config space, which is not good.
Aha.
Quoted 2 lines
Quoted 2 lines
--
MST
PATCH 2/3 transport-pci: VIRTIO_F_SHM_PAGE_SIZE support
In reply to Sergio LopezIntroduce VIRTIO_F_SHM_PAGE_SIZE support by borrowing 8 bits from the 16 bit padding in virtio_pci_cap to hold a page_shift field which can be used to derive the page size supported by the device.
--- transport-pci.tex | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
transport-pci.tex +9 -1
@@ -147,7 +147,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option u8 cfg_type; /* Identifies the structure. */ u8 bar; /* Where to find it. */ u8 id; /* Multiple capabilities of the same type */ - u8 padding[2]; /* Pad to full dword. */ + u8 page_shift; /* Page shift for this shared memory region. */ + u8 padding[1]; /* Pad to full dword. */ le32 offset; /* Offset within bar. */ le32 length; /* Length of the structure, in bytes. */ };
@@ -222,6 +223,13 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option of a certain type. If the device type does not specify the meaning of this field, its contents are undefined. +\item[\field{page_shift}] + If VIRTIO_F_SHM_PAGE_SIZE has been negotiated and \field{cfg_type} is set + to VIRTIO_PCI_CAP_SHARED_MEMORY_CFG, the device MUST provide the page + shift derived from the supported page size for this shared memory region. + The page shift is derived from the page size by using this formula: + $page size = 1 << (page shift + 12)$. + See also \ref {sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Page alignment restrictions}. \item[\field{offset}] indicates where the structure begins relative to the base address associated
Signature · 1 line
2.48.1
PATCH 3/3 transport-mmio: VIRTIO_F_SHM_PAGE_SIZE
In reply to Sergio LopezIntroduce VIRTIO_F_SHM_PAGE_SIZE by adding a the new register "SHMPageShift" to hold the value that can be used to derive the page size value supported by the device.
--- transport-mmio.tex | 8 ++++++++ 1 file changed, 8 insertions(+)
transport-mmio.tex +8 -0
@@ -242,6 +242,14 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi apply to the queue selected by writing to \field{QueueSel}. } \hline + \mmioreg{SHMPageShift}{Shared memory region page shift 8 bit long}{0x0c4}{R}{% + If VIRTIO_F_SHM_PAGE_SIZE has been negotiated, this register returns + the page shift derived from the supported page size for this shared + memory region. The page shift is derived from the page size by using + this formula: $page size = 1 << (page shift + 12)$. + See also \ref {sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Page alignment restrictions}. + } + \hline \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{ Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}). The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
Signature · 1 line
2.48.1
Quoted 35 lines · Sergio Lopez
There's an incresing number of machines supporting multiple page sizes and, on these machines, the host and a guest can be running with different pages sizes.
In addition to this, there might be devices that have a required and/or preferred page size for mapping memory.
In this series we extend the "Shared Memory Regions" with a subsection explaining the posible existence of page alignment restrictions.
For the device to provide the page size information to the driver, we need to extend the PCI and MMIO transports. For the former, we borrow 8 bits from the 16 bit padding in
virtio_pci_capto hold apage_shiftfield which can be used to derive the page size by using the following
formula: (page_size= 1 << (page_shift+ 12)).For MMIO, we add a the SHMPageShift register at offset
0x0c4, also holding thepage_shiftvalue. Since MMIO registers are 32 bit wide, we could have asked the device to directly providepage_sizeinstead ofpage_shift, but seems reasonable to be consistent across transports.Since accessing a new MMIO register could be interpreted by the device as an error, this behavior is gated behind the newly introduced
VIRTIO_F_SHM_PAGE_SIZEfeature bit.An implementation of the changes proposed in this series has been published as an RFC to the LKML, to be used as a reference:
https://lore.kernel.org/all/[email protected]/
Sergio Lopez (3):
shared-mem: introduce page alignment restrictions
transport-pci:VIRTIO_F_SHM_PAGE_SIZEsupport
transport-mmio:VIRTIO_F_SHM_PAGE_SIZE
Quoted 6 lines
content.tex | 3 +++ shared-mem.tex | 7 +++++++ transport-mmio.tex | 8 ++++++++ transport-pci.tex | 10 +++++++++- 4 files changed, 27 insertions(+), 1 deletion(-)
Looks good
Signature · 2 lines
Best regards, Dmitry