balloon: DEVICE_INIT_REPORTED and DEVICE_INIT_ON_INFLATE
v3 cover balloon: DEVICE_INIT_REPORTED and DEVICE_INIT_ON_INFLATE
Once a page gets out of balloon (on deflate, or on access for reporting) the Linux guest inits it before giving it to app. It's a waste - the page is already inited by the host.
Add balloon support for skipping that, that provides significant performance gains.
This adds two new feature bits for the balloon device that allow the device to signal that it has initialized (zeroed or poison-filled) pages on behalf of the driver.
The variant with poison does not really seem useful, so I am still trying to figure out whether we should simply outlaw it.
Old implementations (new ones will be posted ASAP):
QEMU: https://lore.kernel.org/all/[email protected]
Linux: https://lore.kernel.org/all/[email protected]
Tested with QEMU + Linux guest: 232ms -> 51ms (-78%) for 256MB page allocation in a 2GB VM.
Changes since v2:
- I realized we can't really avoid a new feature bit:
poison is a promise from driver to initialize pages.
- lots of other changes
Changes v1->v2:
- reporting can actually just utilize used length to see what was initialized! so we do not need a bitmap for that.
- qemu really needs a way to skip init for some pages. Add that to inflate path.
Add security considerations (for coco).
Michael S. Tsirkin (2):
balloon: add VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED
balloon: add VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
device-types/balloon/description.tex | 147 ++++++++++++++++++++++++++- 1 file changed, 145 insertions(+), 2 deletions(-)
--
MST
PATCH v3 1/2 balloon: add VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED
Add VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED (bit 6): the device initializes reported pages and signals success via the used length of each reporting_vq element.
Add Security Considerations section covering information leakage, data integrity, and denial of service.
--- device-types/balloon/description.tex | 85 +++++++++++++++++++++++++++- 1 file changed, 83 insertions(+), 2 deletions(-)
device-types/balloon/description.tex +83 -2
@@ -49,6 +49,18 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu \item[ VIRTIO_BALLOON_F_PAGE_REPORTING(5) ] The device has support for free page reporting. A virtqueue for reporting free guest memory is present. +\item[ VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED(6) ] The device initializes + reported pages. + +\end{description} + +\subsubsection{Feature bit requirements}\label{sec:Device Types / Memory Balloon Device / Feature bits / Feature bit requirements} + +Some balloon feature bits require other balloon feature bits +(see \ref{drivernormative:Basic Facilities of a Virtio Device / Feature Bits}): + +\begin{description} +\item[VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED] Requires VIRTIO_BALLOON_F_PAGE_REPORTING. \end{description} \drivernormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits}
@@ -61,7 +73,8 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu If the driver is expecting the pages to retain some initialized value, it MUST NOT accept VIRTIO_BALLOON_F_PAGE_REPORTING unless it also -negotiates VIRTIO_BALLOON_F_PAGE_POISON. +negotiates VIRTIO_BALLOON_F_PAGE_POISON or +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED. \devicenormative{\subsubsection}{Feature bits}{Device Types / Memory Balloon Device / Feature bits} If the device offers the VIRTIO_BALLOON_F_MUST_TELL_HOST feature
@@ -625,10 +638,78 @@ \subsubsection{Free Page Reporting}\label{sec:Device Types / Memory Balloon Devi Normative statements in this section apply if the VIRTIO_BALLOON_F_PAGE_REPORTING feature has been negotiated. -If the VIRTIO_BALLOON_F_PAGE_POISON feature has not been negotiated, the +If neither the VIRTIO_BALLOON_F_PAGE_POISON nor +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED feature has been negotiated, the device MAY modify the contents of any page supplied in a report request before acknowledging that request by using the reporting_vq descriptor. If the VIRTIO_BALLOON_F_PAGE_POISON feature has been negotiated, the device MUST NOT modify the the content of a reported page to a value other than \field{poison_val}. + +\subsubsection{Device Initialized Reported Pages}\label{sec:Device Types / Memory Balloon Device / Device Operation / Device Initialized Reported Pages} + +When VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED is negotiated, the device +initializes reported pages by writing into each byte of the page. + +\drivernormative{\paragraph}{Device Initialized Reported Pages}{Device Types / Memory Balloon Device / Device Operation / Device Initialized Reported Pages} + +Normative statements in this section apply if the +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED feature has been negotiated. + +The driver MUST NOT treat a reported page as initialized unless +the used length covers it. + +\devicenormative{\paragraph}{Device Initialized Reported Pages}{Device Types / Memory Balloon Device / Device Operation / Device Initialized Reported Pages} + +Normative statements in this section apply if the +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED feature has been negotiated. + +If VIRTIO_BALLOON_F_PAGE_POISON is also negotiated, the device +MUST fill each initialized page with \field{poison_val}. + +If VIRTIO_BALLOON_F_PAGE_POISON is not negotiated, the device +MUST fill each initialized page with zeros. + +\subsection{Security Considerations}\label{sec:Device Types / Memory Balloon Device / Security Considerations} + +In environments where the device is not trusted, such as +confidential computing, the balloon device presents several +security concerns. + +\subsubsection{Information Leakage}\label{sec:Device Types / Memory Balloon Device / Security Considerations / Information Leakage} + +When the driver inflates the balloon, the addresses of the pages +are communicated to the device. On some systems, the page contents +themselves are also accessible to the device. These pages may +contain sensitive data from previous use by the guest. In systems +where the device is untrusted, the driver should sanitize pages +before inflating them. + +Free page hinting and free page reporting similarly expose +information about which pages are free, and in the case of +reporting, may make page contents accessible to the device. Memory +statistics reporting exposes guest memory usage patterns. + +\subsubsection{Data Integrity}\label{sec:Device Types / Memory Balloon Device / Security Considerations / Data Integrity} + +When the device is untrusted, the driver cannot rely on the device +to correctly initialize pages. An untrusted device can report a +non-zero used length for VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED +without actually initializing the corresponding pages. If the +driver treats such pages as initialized (e.g., marking them as +zeroed), this could lead to information leakage between guest +processes or other security violations. + +Drivers operating in environments with untrusted devices are +expected to avoid negotiating +VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED, and to initialize +all pages themselves. + +\subsubsection{Denial of Service}\label{sec:Device Types / Memory Balloon Device / Security Considerations / Denial of Service} + +A malicious device could set \field{num_pages} to an +unreasonably large value, causing the driver to surrender most +of the guest's memory. Drivers should apply reasonable limits +and not inflate the balloon to the point where the guest +becomes unable to function.
Signature · 1 line
MST
PATCH v3 2/2 balloon: add VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE
In reply to Michael S. TsirkinAdd VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE (bit 7): the device initializes inflated pages and returns a per-page bitmap indicating which pages were successfully initialized.
Update Security Considerations Data Integrity subsection to cover both DEVICE_INIT_REPORTED and DEVICE_INIT_ON_INFLATE.
--- device-types/balloon/description.tex | 66 +++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 2 deletions(-)
device-types/balloon/description.tex +64 -2
@@ -52,6 +52,9 @@ \subsection{Feature bits}\label{sec:Device Types / Memory Balloon Device / Featu \item[ VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED(6) ] The device initializes reported pages. +\item[ VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE(7) ] The device initializes + inflated pages. + \end{description} \subsubsection{Feature bit requirements}\label{sec:Device Types / Memory Balloon Device / Feature bits / Feature bit requirements}
@@ -671,6 +674,64 @@ \subsubsection{Device Initialized Reported Pages}\label{sec:Device Types / Memor If VIRTIO_BALLOON_F_PAGE_POISON is not negotiated, the device MUST fill each initialized page with zeros. +\subsubsection{Device Initialized Pages on Inflate}\label{sec:Device Types / Memory Balloon Device / Device Operation / Device Initialized Pages on Inflate} + +When VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE is negotiated, the +device initializes inflated pages and reports which pages were +successfully initialized via a per-page bitmap. + +The driver appends a device-writable bitmap buffer to each inflate +descriptor chain, after the PFN outbuf. The bitmap contains one +bit per balloon page (4KB), where bit N corresponds to the Nth PFN +in the inflate request (0-indexed). Bit N is stored as bit (N mod 8) +of byte (N / 8), with bit 0 being the least significant bit. +The device sets a bit to 1 if it successfully initialized the +corresponding page, or 0 if it did not. + +If VIRTIO_BALLOON_F_PAGE_POISON is also negotiated, the device fills +pages with \field{poison_val}; otherwise, the device fills pages +with zeros. + +When the driver later deflates these pages, it may skip +initialization for pages that were marked as initialized in the +inflate bitmap. + +\drivernormative{\paragraph}{Device Initialized Pages on Inflate}{Device Types / Memory Balloon Device / Device Operation / Device Initialized Pages on Inflate} + +Normative statements in this section apply if the +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE feature has been negotiated. + +The driver MUST append a device-writable bitmap buffer to each +inflate descriptor chain. The bitmap buffer MUST be at least +$\lceil N/8 \rceil$ bytes, where N is the number of PFNs in the +inflate request. + +The driver MUST NOT treat an inflated page as initialized unless +the corresponding bit in the bitmap is set to 1. + +The driver MAY skip initializing a deflated page if the +corresponding inflate bitmap bit was set to 1. This overrides +the VIRTIO_BALLOON_F_PAGE_POISON requirement to initialize +deflated pages with \field{poison_val} for such pages. + +\devicenormative{\paragraph}{Device Initialized Pages on Inflate}{Device Types / Memory Balloon Device / Device Operation / Device Initialized Pages on Inflate} + +Normative statements in this section apply if the +VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE feature has been negotiated. + +The device MUST set bit N in the bitmap to 1 if it successfully +initialized PFN N, or to 0 otherwise. + +The device MUST set all padding bits in the last byte of the bitmap +to 0. The device MAY write more bytes than $\lceil N/8 \rceil$ but +MUST set all extra bits to 0. + +If VIRTIO_BALLOON_F_PAGE_POISON is also negotiated, the device +MUST fill each successfully initialized page with \field{poison_val}. + +If VIRTIO_BALLOON_F_PAGE_POISON is not negotiated, the device +MUST fill each successfully initialized page with zeros. + \subsection{Security Considerations}\label{sec:Device Types / Memory Balloon Device / Security Considerations} In environments where the device is not trusted, such as
@@ -696,14 +757,15 @@ \subsubsection{Data Integrity}\label{sec:Device Types / Memory Balloon Device / When the device is untrusted, the driver cannot rely on the device to correctly initialize pages. An untrusted device can report a non-zero used length for VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED +or set bits in the VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE bitmap without actually initializing the corresponding pages. If the driver treats such pages as initialized (e.g., marking them as zeroed), this could lead to information leakage between guest processes or other security violations. Drivers operating in environments with untrusted devices are -expected to avoid negotiating -VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED, and to initialize +expected to avoid negotiating VIRTIO_BALLOON_F_DEVICE_INIT_REPORTED +or VIRTIO_BALLOON_F_DEVICE_INIT_ON_INFLATE, and to initialize all pages themselves. \subsubsection{Denial of Service}\label{sec:Device Types / Memory Balloon Device / Security Considerations / Denial of Service}
Signature · 1 line
MST