Skip to contentSkip to conversation

virtio-balloon: add an untrusted device feature

PATCH Michael S. Tsirkin 3 participants 6 messages Aug 3, 2022 – Aug 9, 2022 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 2 lines · Keir Fraser

Add a feature bit to indicate that guest memory is protected from the host by the Trusted Compyuting Base (TCB).

Typo.

Quoted 3 lines

This flag indicates to the
driver that memory ownership must be relinquished via the TCB, by platform-specific means, before reporting that memory to the device.

Thanks for the proposal! I feel it needs to be more detailed to be useful. In particular we need a bit more info about TCB and memory ownership here I feel, so people can interpret this. Is there a spec link?

Quoted 11 lines
---
 content.tex | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
content.tex +0 -0
index e863709..1575ab7 100644

content.tex \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5548

Quoted 6 lines
content.tex +3 -0
@@ -5543,6 +5543,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
     Configuration field \field{poison_val} is valid.
 \item[ VIRTIO_BALLOON_F_PAGE_REPORTING(5) ] The device has support for free
     page reporting. A virtqueue for reporting free guest memory is present.
+\item[ VIRTIO_BALLOON_F_UNTRUSTED_DEVICE(6) ] The driver must
+    relinquish memory ownership via the Trusted Computing Base (TCB) before
+    notifying the device.

The name does not seem to match what it does.

Quoted 8 lines
content.tex +1 -0
 \end{description}
 
@@ -5558,11 +5561,22 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
 it MUST NOT accept VIRTIO_BALLOON_F_PAGE_REPORTING unless it also
 negotiates VIRTIO_BALLOON_F_PAGE_POISON.
 
+The driver SHOULD accept the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE

content.tex \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5574

Quoted 10 lines
content.tex +5 -0
+feature if offered by the device, and relinquish memory ownership via
+the TCB by platform-specific means.
+
 \devicenormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits}
 If the device offers the VIRTIO_BALLOON_F_MUST_TELL_HOST feature
 bit, and if the driver did not accept this feature bit, the
 device MAY signal failure by failing to set FEATURES_OK
 \field{device status} bit when the driver writes it.
+
+If guest memory is protected from the host,

This does not make it clear how this is different from VIRTIO_F_ACCESS_PLATFORM which says:

\item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
the device can be used on a platform where device access to data in memory is limited and/or translated.

Quoted 1 line

the device must offer the

MUST

Quoted 19 lines
content.tex +8 -0
+VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature bit. If the driver does
+not accept this feature bit, the device MUST signal failure by failing
+to set FEATURES_OK \field{device status} bit when the driver writes
+it.
+
 \subparagraph{Legacy Interface: Feature bits}\label{sec:Device
 Types / Memory Balloon Device / Feature bits / Legacy Interface:
 Feature bits}
@@ -5573,6 +5587,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
 allow guest to use memory before notifying host if
 VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated.
 
+The legacy interface cannot support VIRTIO_BALLOON_F_UNTRUSTED_DEVICE
+since there is no way to gracefully report feature negotiation failure.
+
 \subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout}
   \field{num_pages} and \field{actual} are always available.
 

content.tex \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5673

Quoted 10 lines
content.tex +4 -0
@@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
     pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
 } and the descriptor
     describing the resulting 32-bit array is added to the inflateq.
+  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
+  negotiated, the driver MUST relinquish memory ownership via the TCB
+  before adding it to the inflateq.
+
   \end{enumerate}
 
 \item To remove memory from the balloon (aka. deflate):

Don't we need to take it back before deflate?

content.tex \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi · line 6133

Quoted 9 lines
content.tex +3 -0
@@ -6105,6 +6126,9 @@ \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi
 driver MUST initialize all free pages with \field{poison_val} before
 reporting them.
 
+If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been negotiated, the
+driver MUST notify free pages to the TCB before reporting them.
+
 The driver MUST NOT use the reported pages until the device has
 acknowledged the reporting request.

Should this affect hinting too?

Is "notify" same as "relinquish ownership"?

Are these terms defined in some spec?

Quoted 3 lines
Signature · 2 lines
-- 
2.37.1.455.g008518b4e5-goog

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#2 lore raw
Quoted 1 line · Michael S. Tsirkin

On Wed, Aug 03, 2022 at 08:44:41AM +0000, Keir Fraser wrote:

[hmm... I don't see the original mail on virtio-comment]

Quoted 56 lines
Quoted 2 lines

Add a feature bit to indicate that guest memory is protected from the host by the Trusted Compyuting Base (TCB).

Typo.

Quoted 3 lines

This flag indicates to the
driver that memory ownership must be relinquished via the TCB, by platform-specific means, before reporting that memory to the device.

Thanks for the proposal! I feel it needs to be more detailed to be useful. In particular we need a bit more info about TCB and memory ownership here I feel, so people can interpret this. Is there a spec link?

Quoted 11 lines
---
 content.tex | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
content.tex +0 -0
index e863709..1575ab7 100644

content.tex \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5548

Quoted 6 lines
content.tex +3 -0
@@ -5543,6 +5543,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
     Configuration field \field{poison_val} is valid.
 \item[ VIRTIO_BALLOON_F_PAGE_REPORTING(5) ] The device has support for free
     page reporting. A virtqueue for reporting free guest memory is present.
+\item[ VIRTIO_BALLOON_F_UNTRUSTED_DEVICE(6) ] The driver must
+    relinquish memory ownership via the Trusted Computing Base (TCB) before
+    notifying the device.

The name does not seem to match what it does.

Quoted 8 lines
content.tex +1 -0
 \end{description}
 
@@ -5558,11 +5561,22 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
 it MUST NOT accept VIRTIO_BALLOON_F_PAGE_REPORTING unless it also
 negotiates VIRTIO_BALLOON_F_PAGE_POISON.
 
+The driver SHOULD accept the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE

content.tex \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5574

Quoted 10 lines
content.tex +5 -0
+feature if offered by the device, and relinquish memory ownership via
+the TCB by platform-specific means.
+
 \devicenormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits}
 If the device offers the VIRTIO_BALLOON_F_MUST_TELL_HOST feature
 bit, and if the driver did not accept this feature bit, the
 device MAY signal failure by failing to set FEATURES_OK
 \field{device status} bit when the driver writes it.
+
+If guest memory is protected from the host,

This does not make it clear how this is different from VIRTIO_F_ACCESS_PLATFORM which says:

Quoted 3 lines

\item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
the device can be used on a platform where device access to data in memory is limited and/or translated.

I'm also wondering how this relates to existing secure/existing techniques that basically mandate ACCESS_PLATFORM. Does this new feature require ACCESS_PLATFORM as well?

Quoted 4 lines
Quoted 8 lines
Quoted 1 line

the device must offer the

MUST

content.tex

Quoted 4 lines
content.tex +4 -0
+VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature bit. If the driver does
+not accept this feature bit, the device MUST signal failure by failing
+to set FEATURES_OK \field{device status} bit when the driver writes
+it.

So, the bit is mandatory if offered?

Quoted 10 lines

content.tex \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5591

Quoted 10 lines
content.tex +3 -0
+
 \subparagraph{Legacy Interface: Feature bits}\label{sec:Device
 Types / Memory Balloon Device / Feature bits / Legacy Interface:
 Feature bits}
@@ -5573,6 +5587,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
 allow guest to use memory before notifying host if
 VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated.
 
+The legacy interface cannot support VIRTIO_BALLOON_F_UNTRUSTED_DEVICE
+since there is no way to gracefully report feature negotiation failure.

UNTRUSTED_DEVICE requires VERSION_1, then?

Quoted 11 lines

content.tex \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5669

Quoted 11 lines
content.tex +4 -0
+
 \subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout}
   \field{num_pages} and \field{actual} are always available.
 
@@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
     pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
 } and the descriptor
     describing the resulting 32-bit array is added to the inflateq.
+  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
+  negotiated, the driver MUST relinquish memory ownership via the TCB
+  before adding it to the inflateq.

Do we need a definition of "relinquish memory ownership"?

Quoted 20 lines

content.tex \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5673

Quoted 4 lines
content.tex +1 -0
+
   \end{enumerate}
 
 \item To remove memory from the balloon (aka. deflate):

Don't we need to take it back before deflate?

content.tex \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi · line 6133

Quoted 9 lines
content.tex +3 -0
@@ -6105,6 +6126,9 @@ \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi
 driver MUST initialize all free pages with \field{poison_val} before
 reporting them.
 
+If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been negotiated, the
+driver MUST notify free pages to the TCB before reporting them.
+
 The driver MUST NOT use the reported pages until the device has
 acknowledged the reporting request.

Should this affect hinting too?

Quoted 3 lines

Is "notify" same as "relinquish ownership"?

Are these terms defined in some spec?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw
Quoted 5 lines · Cornelia Huck
Quoted 1 line · Michael S. Tsirkin

On Wed, Aug 03, 2022 at 08:44:41AM +0000, Keir Fraser wrote:

[hmm... I don't see the original mail on virtio-comment]

Sorry about that. My email bounced even after I subscribed to the list. MST replied to my Cc. My list membership seems to have settled now.

Quoted 10 lines
Quoted 8 lines
Quoted 1 line

the device must offer the

MUST

Review

Quoted 4 lines
(quoted diff) +4 -0
+VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature bit. If the driver does
+not accept this feature bit, the device MUST signal failure by failing
+to set FEATURES_OK \field{device status} bit when the driver writes
+it.

So, the bit is mandatory if offered?

In ARM PKVM the host is untrusted, and so the device cannot regain ownership of memory pages. That has to be done via the TCB (Hypervisor). If the driver isn't aware of that, it is impossible for memory to be transferred back to the host, and ballooning cannot work.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
(quoted diff) +1 -0
+
 \subparagraph{Legacy Interface: Feature bits}\label{sec:Device
 Types / Memory Balloon Device / Feature bits / Legacy Interface:
 Feature bits}
@@ -5573,6 +5587,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
 allow guest to use memory before notifying host if
 VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated.
 
Quoted 4 lines
Quoted 2 lines

Review \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5591

Quoted 2 lines
(quoted diff) +2 -0
+The legacy interface cannot support VIRTIO_BALLOON_F_UNTRUSTED_DEVICE
+since there is no way to gracefully report feature negotiation failure.

UNTRUSTED_DEVICE requires VERSION_1, then?

I think so. I don't see any benefit to supporting it on legacy interfaces.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +1 -0
+
 \subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout}
   \field{num_pages} and \field{actual} are always available.
 
Quoted 9 lines
Quoted 7 lines

Review \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5669

Quoted 7 lines
(quoted diff) +3 -0
@@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
     pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
 } and the descriptor
     describing the resulting 32-bit array is added to the inflateq.
+  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
+  negotiated, the driver MUST relinquish memory ownership via the TCB
+  before adding it to the inflateq.

Do we need a definition of "relinquish memory ownership"?

Maybe. Though I would say that the mechanism is platform specific and out of scope for virtio. But perhaps a short description of what this term means, in more detail, would make sense? And then tighten up usage of this term throughout the spec.

Regards,
Keir

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#4 lore raw
Quoted 10 lines · Keir Fraser
Quoted 5 lines · Cornelia Huck
Quoted 1 line · Michael S. Tsirkin

On Wed, Aug 03, 2022 at 08:44:41AM +0000, Keir Fraser wrote:

[hmm... I don't see the original mail on virtio-comment]

Sorry about that. My email bounced even after I subscribed to the list. MST replied to my Cc. My list membership seems to have settled now.

Yes, seems to work now.

Quoted 12 lines
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines

the device must offer the

MUST

Review

Quoted 4 lines
(quoted diff) +4 -0
+VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature bit. If the driver does
+not accept this feature bit, the device MUST signal failure by failing
+to set FEATURES_OK \field{device status} bit when the driver writes
+it.

So, the bit is mandatory if offered?

Quoted 4 lines

In ARM PKVM the host is untrusted, and so the device cannot regain ownership of memory pages. That has to be done via the TCB (Hypervisor). If the driver isn't aware of that, it is impossible for memory to be transferred back to the host, and ballooning cannot work.

Nod. I see that the Linux driver also explicitly clears the ACCESS_PLATFORM feature bit; I'm wondering what the overlap between the two features would be.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
(quoted diff) +1 -0
+
 \subparagraph{Legacy Interface: Feature bits}\label{sec:Device
 Types / Memory Balloon Device / Feature bits / Legacy Interface:
 Feature bits}
@@ -5573,6 +5587,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu
 allow guest to use memory before notifying host if
 VIRTIO_BALLOON_F_MUST_TELL_HOST is not negotiated.
 
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Review \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu · line 5591

Quoted 2 lines
(quoted diff) +2 -0
+The legacy interface cannot support VIRTIO_BALLOON_F_UNTRUSTED_DEVICE
+since there is no way to gracefully report feature negotiation failure.

UNTRUSTED_DEVICE requires VERSION_1, then?

I think so. I don't see any benefit to supporting it on legacy interfaces.

This dependency should be made explicit then, as balloon also supports legacy.

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
(quoted diff) +1 -0
+
 \subsection{Device configuration layout}\label{sec:Device Types / Memory Balloon Device / Device configuration layout}
   \field{num_pages} and \field{actual} are always available.
 
Quoted 9 lines
Quoted 7 lines

Review \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5669

Quoted 7 lines
(quoted diff) +3 -0
@@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
     pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
 } and the descriptor
     describing the resulting 32-bit array is added to the inflateq.
+  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
+  negotiated, the driver MUST relinquish memory ownership via the TCB
+  before adding it to the inflateq.

Do we need a definition of "relinquish memory ownership"?

Quoted 4 lines

Maybe. Though I would say that the mechanism is platform specific and out of scope for virtio. But perhaps a short description of what this term means, in more detail, would make sense? And then tighten up usage of this term throughout the spec.

I guess other protected memory technologies have similar concepts; would it make sense to define "memory ownership" as "exclusive access"? I.e. the device cannot access the memory, and the driver has to give up exclusive access before adding it to the inflateq.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
In reply to Michael S. Tsirkin
Quoted 9 lines · Michael S. Tsirkin
Quoted 3 lines · Keir Fraser

This flag indicates to the
driver that memory ownership must be relinquished via the TCB, by platform-specific means, before reporting that memory to the device.

Thanks for the proposal! I feel it needs to be more detailed to be useful. In particular we need a bit more info about TCB and memory ownership here I feel, so people can interpret this. Is there a spec link?

The specific motivation for this is PKVM on ARM, where protected VM memory is inaccessible to the host kernel. Ownership is tracked by a small hypervisor, and that hypervisor needs to be aware that the pVM is willingly returning memory to the host. There's some old description of ARM PKVM here https://lwn.net/Articles/836693/

So the purpose of the new flag is:

1. To inform the pVM that it must notify the hypervisor (although there could be out-of-band mechanisms to detect/probe this)

2. If the flag is not accepted by the driver, device can abort balloon setup (since pVM ballooning cannot work without hypervisor notifications).

Quoted 5 lines

Review

Quoted 3 lines

+\item[ VIRTIO_BALLOON_F_UNTRUSTED_DEVICE(6) ] The driver must

(quoted diff) +2 -0
+    relinquish memory ownership via the Trusted Computing Base (TCB) before
+    notifying the device.

The name does not seem to match what it does.

Maybe something more direct, like VIRTIO_BALLOON_F_NOTIFY_HYP?

Quoted 8 lines
Quoted 1 line

+If guest memory is protected from the host,

This does not make it clear how this is different from VIRTIO_F_ACCESS_PLATFORM which says:

\item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
the device can be used on a platform where device access to data in memory is limited and/or translated.

Using this existing flag could possibly work, assuming there are no conflicting legacy semantics for this flag in relation to balloon driver. At the moment the Linux virtio-balloon driver unconditionally clears this feature flag, so perhaps there are no legacy semantics at all.

We could instead accept this existing flag and use it to turn on the hypervisor notifications as well as using DMA API for vring setup (we do need both anyway)? If the flag is cleared, as with the existing kernel driver, the device knows for sure that the driver is not ARM PKVM aware.

"Turn on hypervisor notifications" here is platform specific: In ARM PKVM it means probe for existence of the new notification hypercall and send notifications if the hypercall exists.

I'd be more than happy to use this flag rather than introduce another. Multiplying these sorts of flags more than absolutely necessary is for sure not a good idea.

Quoted 10 lines
Quoted 10 lines
(quoted diff) +4 -0
@@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
     pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
 } and the descriptor
     describing the resulting 32-bit array is added to the inflateq.
+  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
+  negotiated, the driver MUST relinquish memory ownership via the TCB
+  before adding it to the inflateq.
+
   \end{enumerate}
 
Quoted 3 lines

Review \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5673

Quoted 1 line
(quoted diff) +0 -0
 \item To remove memory from the balloon (aka. deflate):

Don't we need to take it back before deflate?

No, giving memory to a protected VM does not need the VM's explicit permission.

If such permission were required then we would have a problem implementing free page reporting (and maybe hinting too).

Quoted 11 lines

Review \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi · line 6133

Quoted 9 lines
(quoted diff) +3 -0
@@ -6105,6 +6126,9 @@ \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi
 driver MUST initialize all free pages with \field{poison_val} before
 reporting them.
 
+If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been negotiated, the
+driver MUST notify free pages to the TCB before reporting them.
+
 The driver MUST NOT use the reported pages until the device has
 acknowledged the reporting request.

Should this affect hinting too?

Yes you are probbaly right. Though given the main usage of hinting is for live migration, it's unlikely to be enabled for ARM PKVM any time soon. I don't know how we'd live migrate a VM with confidential memory.

Quoted 1 line

Is "notify" same as "relinquish ownership"?

Yes, sorry, I should tighten up the wording.

Quoted 1 line

Are these terms defined in some spec?

A spec for relinquishing memory ownership in ARM PKVM? I could certainly put something together if it's helpful. Also, concretely, there are patches to the Linux kernel.

I assumed it would make sense to address the virtio spec before floating implementation patches, but perhaps it makes sense to consider it all in the round?

Also, if we can agree that VIRTIO_F_ACCESS_PLATFORM can be pressed into service for our needs, perhaps the spec changes become a bit more straightforward. Though I assume you would still want many of the above additions, to make it clear how a protected VM should interact with ballooning?

Quoted 12 lines
Quoted 3 lines
Signature · 2 lines
-- 
2.37.1.455.g008518b4e5-goog

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#6 lore raw
Quoted 55 lines · Keir Fraser
Quoted 9 lines · Michael S. Tsirkin
Quoted 3 lines · Keir Fraser

This flag indicates to the
driver that memory ownership must be relinquished via the TCB, by platform-specific means, before reporting that memory to the device.

Thanks for the proposal! I feel it needs to be more detailed to be useful. In particular we need a bit more info about TCB and memory ownership here I feel, so people can interpret this. Is there a spec link?

The specific motivation for this is PKVM on ARM, where protected VM memory is inaccessible to the host kernel. Ownership is tracked by a small hypervisor, and that hypervisor needs to be aware that the pVM is willingly returning memory to the host. There's some old description of ARM PKVM here https://lwn.net/Articles/836693/

So the purpose of the new flag is:

1. To inform the pVM that it must notify the hypervisor (although there could be out-of-band mechanisms to detect/probe this)

2. If the flag is not accepted by the driver, device can abort balloon setup (since pVM ballooning cannot work without hypervisor notifications).

Quoted 5 lines

Review

Quoted 3 lines

+\item[ VIRTIO_BALLOON_F_UNTRUSTED_DEVICE(6) ] The driver must

(quoted diff) +2 -0
+    relinquish memory ownership via the Trusted Computing Base (TCB) before
+    notifying the device.

The name does not seem to match what it does.

Maybe something more direct, like VIRTIO_BALLOON_F_NOTIFY_HYP?

Quoted 8 lines
Quoted 1 line

+If guest memory is protected from the host,

This does not make it clear how this is different from VIRTIO_F_ACCESS_PLATFORM which says:

\item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
the device can be used on a platform where device access to data in memory is limited and/or translated.

Using this existing flag could possibly work, assuming there are no conflicting legacy semantics for this flag in relation to balloon driver. At the moment the Linux virtio-balloon driver unconditionally clears this feature flag, so perhaps there are no legacy semantics at all.

We could instead accept this existing flag and use it to turn on the hypervisor notifications as well as using DMA API for vring setup (we do need both anyway)? If the flag is cleared, as with the existing kernel driver, the device knows for sure that the driver is not ARM PKVM aware.

Quoted 7 lines

"Turn on hypervisor notifications" here is platform specific: In ARM PKVM it means probe for existence of the new notification hypercall and send notifications if the hypercall exists.

I'd be more than happy to use this flag rather than introduce another. Multiplying these sorts of flags more than absolutely necessary is for sure not a good idea.

Now, a question: are these hypervisor notifications in particular exactly what the DMA API does?

Is it true that Linux will issue these notifications simply by mapping memory for DMA? Or is there some other trickery involved?

Quoted 48 lines
Quoted 10 lines
Quoted 10 lines
(quoted diff) +4 -0
@@ -5647,6 +5664,10 @@ \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D
     pages. These addresses are divided by 4096\footnote{This is historical, and independent of the guest page size.
 } and the descriptor
     describing the resulting 32-bit array is added to the inflateq.
+  \item If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been
+  negotiated, the driver MUST relinquish memory ownership via the TCB
+  before adding it to the inflateq.
+
   \end{enumerate}
 
Quoted 3 lines

Review \subsection{Device Operation}\label{sec:Device Types / Memory Balloon Device / D · line 5673

Quoted 1 line
(quoted diff) +0 -0
 \item To remove memory from the balloon (aka. deflate):

Don't we need to take it back before deflate?

No, giving memory to a protected VM does not need the VM's explicit permission.

If such permission were required then we would have a problem implementing free page reporting (and maybe hinting too).

Quoted 11 lines

Review \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi · line 6133

Quoted 9 lines
(quoted diff) +3 -0
@@ -6105,6 +6126,9 @@ \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi
 driver MUST initialize all free pages with \field{poison_val} before
 reporting them.
 
+If the VIRTIO_BALLOON_F_UNTRUSTED_DEVICE feature has been negotiated, the
+driver MUST notify free pages to the TCB before reporting them.
+
 The driver MUST NOT use the reported pages until the device has
 acknowledged the reporting request.

Should this affect hinting too?

Yes you are probbaly right. Though given the main usage of hinting is for live migration, it's unlikely to be enabled for ARM PKVM any time soon. I don't know how we'd live migrate a VM with confidential memory.

Quoted 1 line

Is "notify" same as "relinquish ownership"?

Yes, sorry, I should tighten up the wording.

Quoted 1 line

Are these terms defined in some spec?

A spec for relinquishing memory ownership in ARM PKVM? I could certainly put something together if it's helpful. Also, concretely, there are patches to the Linux kernel.

I assumed it would make sense to address the virtio spec before floating implementation patches, but perhaps it makes sense to consider it all in the round?

Quoted 27 lines

Also, if we can agree that VIRTIO_F_ACCESS_PLATFORM can be pressed into service for our needs, perhaps the spec changes become a bit more straightforward. Though I assume you would still want many of the above additions, to make it clear how a protected VM should interact with ballooning?

Quoted 21 lines
Quoted 3 lines
Signature · 2 lines
-- 
2.37.1.455.g008518b4e5-goog

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/