virtio-gpu: add shared resource feature
PATCH 1/1 virtio-gpu: add shared resource feature
This patch adds a new virtio feature for shared resources.
Shared resources are allocated by the guest from normal ram, like traditional resources. They can be used by the guest almost like traditional resources, the only exception is that shared resources can only be used with backing storage attached (the linux driver does that anyway so the workflow doesn't change). The host can map these resources and use them directly (using udmabuf), so any guest changes to these resources can be visible on the host without explicit transfer request. Transfer requests are still needed though as the host might still have syncronize the resource, for example in case importing the udmabuf failed for some reason.
guest/host interface: Two new commands are added to create 2D and 3D shared resources. They work exactly like the non-shared counterparts.
qemu patches:
https://git.kraxel.org/cgit/qemu/log/?h=sirius/virtio-gpu-feature-shared
linux patches:
https://git.kraxel.org/cgit/linux/log/?h=drm-virtio-feature-shared
--- virtio-gpu.tex | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
virtio-gpu.tex +27 -0
@@ -35,6 +35,7 @@ \subsection{Feature bits}\label{sec:Device Types / GPU Device / Feature bits} \begin{description} \item[VIRTIO_GPU_F_VIRGL (0)] virgl 3D mode is supported. \item[VIRTIO_GPU_F_EDID (1)] EDID is supported. +\item[VIRTIO_GPU_F_RESOURCE_SHARED 2)] shared resources are supported. \end{description} \subsection{Device configuration layout}\label{sec:Device Types / GPU Device / Device configuration layout}
@@ -103,6 +104,8 @@ \subsubsection{Device Operation: Create a framebuffer and configure scanout} \begin{itemize*} \item Create a host resource using VIRTIO_GPU_CMD_RESOURCE_CREATE_2D. + In case VIRTIO_GPU_F_RESOURCE_SHARED is negotiated the driver can + also use VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED. \item Allocate a framebuffer from guest ram, and attach it as backing storage to the resource just created, using VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING. Scatter lists are
@@ -166,6 +169,28 @@ \subsubsection{Device Operation: Configure mouse cursor} the pointer shape and position. To move the pointer without updating the shape use VIRTIO_GPU_CMD_MOVE_CURSOR instead. +\subsubsection{Device Operation: Shared resources} + +In case VIRTIO_GPU_F_RESOURCE_SHARED is negotiated the driver can use +the VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED command to create shared +resources. Normal resources have both a guest buffer and host buffer +buffer for the resource and the VIRTIO_GPU_CMD_TRANSFER_* commands are +used to transfer data between host and guest. Shared (guest +allocated) buffers CAN be used directly by the host, to remove or +reduce the data copies needed. + +The driver MUST attach backing storage to a shared resource before +using it. Any changes on the shared resource MAY be instantly visible +on the host. + +Otherwise the shared resources are used like normal resources. +Especially the driver must send explicit VIRTIO_GPU_CMD_TRANSFER_* +commands to the device for both normal and shared resources. Reason: +The device might have to flush caches. The device MAY also choose to +not create mapping for the shared resource. Especially for small +resources it might be more efficient to just copy the data instead of +establishing a shared mapping. + \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU Device / Device Operation / Device Operation: Request header} All requests and responses on the virt queues have a fixed header
@@ -186,6 +211,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_CMD_GET_CAPSET_INFO, VIRTIO_GPU_CMD_GET_CAPSET, VIRTIO_GPU_CMD_GET_EDID, + VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED, /* cursor commands */ VIRTIO_GPU_CMD_UPDATE_CURSOR = 0x0300,
@@ -205,6 +231,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_RESP_ERR_INVALID_RESOURCE_ID, VIRTIO_GPU_RESP_ERR_INVALID_CONTEXT_ID, VIRTIO_GPU_RESP_ERR_INVALID_PARAMETER, + VIRTIO_GPU_RESP_ERR_NO_BACKING_STORAGE, }; #define VIRTIO_GPU_FLAG_FENCE (1 << 0)
Signature · 6 lines
2.18.1 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Hi Gerd,
I really like this!
virtio-gpu.tex \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De · line 214
Quoted 5 lines · Gerd Hoffmann
virtio-gpu.tex +1
-0@@ -186,6 +211,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_CMD_GET_CAPSET_INFO, VIRTIO_GPU_CMD_GET_CAPSET, VIRTIO_GPU_CMD_GET_EDID, + VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED,
Shouldn't there also the RESOURCE_CREATE_3D_SHARED? I know that 3D isn't speced here but it looks awkward not to reserve it.
virtio-gpu.tex \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De · line 237
Quoted 12 lines
virtio-gpu.tex +1
-0/* cursor commands */ VIRTIO_GPU_CMD_UPDATE_CURSOR = 0x0300,@@ -205,6 +231,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_RESP_ERR_INVALID_RESOURCE_ID, VIRTIO_GPU_RESP_ERR_INVALID_CONTEXT_ID, VIRTIO_GPU_RESP_ERR_INVALID_PARAMETER, + VIRTIO_GPU_RESP_ERR_NO_BACKING_STORAGE, }; #define VIRTIO_GPU_FLAG_FENCE (1 << 0)
Cheers,
Matti
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 4 lines · Matti Moell
Hi Gerd,
I really like this!
Quoted 8 lines
virtio-gpu.tex \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De · line 214
Quoted 5 lines · Gerd Hoffmann
virtio-gpu.tex +1
-0@@ -186,6 +211,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_CMD_GET_CAPSET_INFO, VIRTIO_GPU_CMD_GET_CAPSET, VIRTIO_GPU_CMD_GET_EDID, + VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED,Shouldn't there also the
RESOURCE_CREATE_3D_SHARED? I know that 3D isn't speced here but it looks awkward not to reserve it.
Yep, none of the 3D commands is in the specs, thats why RESOURCE_CREATE_3D_SHARED isn't there too (but, yes, it exists).
Guess it might be a good idea to at least list the commands, but that is something for a separate patch ...
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Interesting; what's the use case for this? Do we already have a mechanism to import memory to virtio-gpu via importing a bo handle from somewhere else?
Quoted 30 lines · Gerd Hoffmann
Quoted 6 lines · Matti Moell
Hi Gerd,
I really like this!
Review \subsubsection{Device Operation: Request
Quoted 1 line · Gerd Hoffmann
(quoted diff) +0
-0@@ -186,6 +211,7 @@ \subsubsection{Device Operation: Requestheader}\label{sec:Device Types / GPU De
Quoted 6 lines
virtio-gpu.tex \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De · line 214
Quoted 4 lines
virtio-gpu.tex +1
-0VIRTIO_GPU_CMD_GET_CAPSET_INFO, VIRTIO_GPU_CMD_GET_CAPSET, VIRTIO_GPU_CMD_GET_EDID, + VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED,Shouldn't there also the
RESOURCE_CREATE_3D_SHARED? I know that 3D isn't speced here but it looks awkward not to reserve it.Yep, none of the 3D commands is in the specs, thats why
RESOURCE_CREATE_3D_SHAREDisn't there too (but, yes, it exists).Guess it might be a good idea to at least list the commands, but that is something for a separate patch ...
cheers,
Gerd---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 1 line · Frank Yang
Interesting; what's the use case for this?
Mostly reduce data copying.
Quoted 3 lines
Do we already have a mechanism
to import memory to virtio-gpu via importing a bo handle from somewhere else?
Inside the guest? Import is not working yet, that needs another update (blob resources) due to current virtio-gpu resources requiring a format at creation time and dmabufs not having one.
See https://www.kraxel.org/blog/2019/11/virtio-gpu-status-and-plans/ for some background.
Exporting virtio-gpu resources works in the drm-misc-next branch, should land in mainline this merge window (i.e. 5.5-rc1).
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 5 lines · Gerd Hoffmann
Quoted 1 line · Frank Yang
Interesting; what's the use case for this?
Mostly reduce data copying.
Quoted 9 lines
Quoted 3 lines
Do we already have a mechanism
to import memory to virtio-gpu via importing a bo handle from somewhere else?Inside the guest? Import is not working yet, that needs another update (blob resources) due to current virtio-gpu resources requiring a format at creation time and dmabufs not having one.
Gotcha, I was just poking around related code and it looked like I could use a resource w/ R8 format and width = bytes. Blob resources would be a more proper way then :)
Quoted 10 lines
See https://www.kraxel.org/blog/2019/11/virtio-gpu-status-and-plans/ for some background.
Exporting virtio-gpu resources works in the drm-misc-next branch, should land in mainline this merge window (i.e. 5.5-rc1).
cheers,
Gerd
Hi,
Quoted 11 lines
Quoted 9 lines
Quoted 3 lines
Do we already have a mechanism
to import memory to virtio-gpu via importing a bo handle from somewhere else?Inside the guest? Import is not working yet, that needs another update (blob resources) due to current virtio-gpu resources requiring a format at creation time and dmabufs not having one.
Gotcha, I was just poking around related code and it looked like I could use a resource w/ R8 format and width = bytes.
Yep, that trick might work for some use cases. But mapping such an imported dma-buf to a scanout isn't going to work for example ...
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor} · line 177
Quoted 2 lines · Gerd Hoffmann
virtio-gpu.tex +2
-0+resources. Normal resources have both a guest buffer and host buffer +buffer for the resource and the VIRTIO_GPU_CMD_TRANSFER_* commands are
typo: "buffer\nbuffer"
Thanks
Laszlo
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 56 lines · Gerd Hoffmann
This patch adds a new virtio feature for shared resources.
Shared resources are allocated by the guest from normal ram, like traditional resources. They can be used by the guest almost like traditional resources, the only exception is that shared resources can only be used with backing storage attached (the linux driver does that anyway so the workflow doesn't change). The host can map these resources and use them directly (using udmabuf), so any guest changes to these resources can be visible on the host without explicit transfer request. Transfer requests are still needed though as the host might still have syncronize the resource, for example in case importing the udmabuf failed for some reason.
guest/host interface: Two new commands are added to create 2D and 3D shared resources. They work exactly like the non-shared counterparts.
qemu patches:
https://git.kraxel.org/cgit/qemu/log/?h=sirius/virtio-gpu-feature-sharedlinux patches:
https://git.kraxel.org/cgit/linux/log/?h=drm-virtio-feature-sharedSigned-off-by Gerd Hoffmann <[email protected]>--- virtio-gpu.tex | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)virtio-gpu.tex +6
-0@@ -35,6 +35,7 @@ \subsection{Feature bits}\label{sec:Device Types / GPU Device / Feature bits} \begin{description} \item[VIRTIO_GPU_F_VIRGL (0)] virgl 3D mode is supported. \item[VIRTIO_GPU_F_EDID (1)] EDID is supported. +\item[VIRTIO_GPU_F_RESOURCE_SHARED 2)] shared resources are supported. \end{description} \subsection{Device configuration layout}\label{sec:Device Types / GPU Device / Device configuration layout}@@ -103,6 +104,8 @@ \subsubsection{Device Operation: Create a framebuffer and configure scanout} \begin{itemize*} \item Create a host resource using VIRTIO_GPU_CMD_RESOURCE_CREATE_2D. + In case VIRTIO_GPU_F_RESOURCE_SHARED is negotiated the driver can + also use VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED. \item Allocate a framebuffer from guest ram, and attach it as backing storage to the resource just created, using VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING. Scatter lists are@@ -166,6 +169,28 @@ \subsubsection{Device Operation: Configure mouse cursor} the pointer shape and position. To move the pointer without updating the shape use VIRTIO_GPU_CMD_MOVE_CURSOR instead. +\subsubsection{Device Operation: Shared resources} + +In case VIRTIO_GPU_F_RESOURCE_SHARED is negotiated the driver can use
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor} · line 184
Quoted 10 lines
virtio-gpu.tex +10
-0+the VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED command to create shared +resources. Normal resources have both a guest buffer and host buffer +buffer for the resource and the VIRTIO_GPU_CMD_TRANSFER_* commands are +used to transfer data between host and guest. Shared (guest +allocated) buffers CAN be used directly by the host, to remove or +reduce the data copies needed. + +The driver MUST attach backing storage to a shared resource before +using it. Any changes on the shared resource MAY be instantly visible +on the host.
If the following scenario happens:
1) Driver sends RESOURCE_CREATE_2D shared request
2) Driver sends ATTACH_BACKING request
3) Device creates a shared resource
4) Driver sends SET_SCANOUT request
5) Device sends shared resource to display
6) Driver sends DETACH_BACKING request
How do we synchronize between (5) and (6)? Is there a guarantee the shared pages won't go back to guest RAM, and will be owned by the display (like traditional dma-bufs)?
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor}
Quoted 5 lines
virtio-gpu.tex +5
-0+ +Otherwise the shared resources are used like normal resources. +Especially the driver must send explicit VIRTIO_GPU_CMD_TRANSFER_* +commands to the device for both normal and shared resources. Reason: +The device might have to flush caches.
Can we make this spec stronger to avoid to avoid transfers all the time (i.e, in virtio_gpu_update_dumb_bo)?
drm_gem_shmem_* helpers seem to use WC buffers, and dumb buffers are traditionally WC as well. If we mandate the host ("device?" here) must properly clean caches at creation time, it may be possible to avoid hypercalls for 2D_SHARED resources.
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor} · line 192
Quoted 4 lines
The device MAY also choose to
virtio-gpu.tex +3
-0+not create mapping for the shared resource. Especially for small +resources it might be more efficient to just copy the data instead of +establishing a shared mapping.
Should the device send a response code (OK_SHARED} to inform the driver that the resource is shared? If the device can choose not to create shared mappings, would the response code + VIRTIO_GPU_CMD_RESOURCE_CREATE_2D + the feature flag suffice (i.e, no VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED)?
Quoted 21 lines
virtio-gpu.tex +3
-1+ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU Device / Device Operation / Device Operation: Request header} All requests and responses on the virt queues have a fixed header@@ -186,6 +211,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_CMD_GET_CAPSET_INFO, VIRTIO_GPU_CMD_GET_CAPSET, VIRTIO_GPU_CMD_GET_EDID, + VIRTIO_GPU_CMD_RESOURCE_CREATE_2D_SHARED, /* cursor commands */ VIRTIO_GPU_CMD_UPDATE_CURSOR = 0x0300,@@ -205,6 +231,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De VIRTIO_GPU_RESP_ERR_INVALID_RESOURCE_ID, VIRTIO_GPU_RESP_ERR_INVALID_CONTEXT_ID, VIRTIO_GPU_RESP_ERR_INVALID_PARAMETER, + VIRTIO_GPU_RESP_ERR_NO_BACKING_STORAGE, }; #define VIRTIO_GPU_FLAG_FENCE (1 << 0) --
Quoted 2 lines
2.18.1
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Hi,
Quoted 8 lines
If the following scenario happens:
1) Driver sends
RESOURCE_CREATE_2Dshared request
2) Driver sendsATTACH_BACKINGrequest
3) Device creates a shared resource
4) Driver sendsSET_SCANOUTrequest
5) Device sends shared resource to display
6) Driver sendsDETACH_BACKINGrequest
Hmm, I think you can crash the current qemu code that way (didn't test though). I think the proper solution would be to return -EBUSY on the DETACH_BACKING request. Guess I'll add a new error code for that.
Quoted 8 lines
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor}
Quoted 4 lines
virtio-gpu.tex +4
-0+Otherwise the shared resources are used like normal resources. +Especially the driver must send explicit VIRTIO_GPU_CMD_TRANSFER_* +commands to the device for both normal and shared resources. Reason: +The device might have to flush caches.Can we make this spec stronger to avoid to avoid transfers all the time (i.e, in
virtio_gpu_update_dumb_bo)?
Quoted 4 lines
drm_gem_shmem_* helpers seem to use WC buffers, and dumb buffers are traditionally WC as well. If we mandate the host ("device?" here) must properly clean caches at creation time, it may be possible to avoid hypercalls for 2D_SHARED resources.
I think we can do 90% of that optimization without changing the protocol. Display updates typically involve multiple commands. A transfer, possibly a set-scanout (when page-flipping), finally flush. The driver can first queue all commands, then notify the host, so we will have only one vmexit per display update instead of one vmexit per command.
Quoted 7 lines
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor} · line 192
Quoted 4 lines
The device MAY also choose to
virtio-gpu.tex +3
-0+not create mapping for the shared resource. Especially for small +resources it might be more efficient to just copy the data instead of +establishing a shared mapping.Should the device send a response code (
OK_SHARED} to inform the driver that the resource is shared?
Is there a good reason why the driver should be aware of that?
I'd prefer to make that an internal implementation detail of the device and not inform the guest.
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 12 lines · Gerd Hoffmann
Hi,
Quoted 8 lines
If the following scenario happens:
1) Driver sends
RESOURCE_CREATE_2Dshared request
2) Driver sendsATTACH_BACKINGrequest
3) Device creates a shared resource
4) Driver sendsSET_SCANOUTrequest
5) Device sends shared resource to display
6) Driver sendsDETACH_BACKINGrequest
Quoted 3 lines
Hmm, I think you can crash the current qemu code that way (didn't test though). I think the proper solution would be to return
-EBUSYon theDETACH_BACKINGrequest. Guess I'll add a new error code for that.
That would add the extra complication of a "delayed detach list" in the guest kernel. I don't know if it's absolutely necessary -- because the open source implementation (udmabuf) seems to take a reference on all constituent pages like DRM gem does [a][b][c][d]. We should figure out if the extra reference is sufficient to prevent the pages from going back to guest RAM (the guest kernel takes references on those pages too -- I don't know if guest references are treated differently).
[a] https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/drm_gem.c#L578
[b] https://github.com/torvalds/linux/blob/master/drivers/dma-buf/udmabuf.c#L161
[c] https://github.com/torvalds/linux/blob/master/drivers/dma-buf/udmabuf.c#L203
[d] http://lkml.iu.edu/hypermail/linux/kernel/0907.3/02155.html
Quoted 14 lines
Quoted 9 lines
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor}
Quoted 5 lines
virtio-gpu.tex +4
-0+Otherwise the shared resources are used like normal resources. +Especially the driver must send explicit VIRTIO_GPU_CMD_TRANSFER_* +commands to the device for both normal and shared resources. Reason: +The device might have to flush caches.Can we make this spec stronger to avoid to avoid transfers all the time (i.e, in
virtio_gpu_update_dumb_bo)?Quoted 4 lines
drm_gem_shmem_* helpers seem to use WC buffers, and dumb buffers are traditionally WC as well. If we mandate the host ("device?" here) must properly clean caches at creation time, it may be possible to avoid hypercalls for 2D_SHARED resources.
Quoted 6 lines
I think we can do 90% of that optimization without changing the protocol. Display updates typically involve multiple commands. A transfer, possibly a set-scanout (when page-flipping), finally flush. The driver can first queue all commands, then notify the host, so we will have only one vmexit per display update instead of one vmexit per command.
We also want to prevent this scenario:
1) Guest creates a writecombine mapping (like drm_gem_shmem_* helpers do), but the pages were previously accessed and thus some data is in the cache.
2) Cache eviction
3) Non-zero data in buffer.
Quoted 10 lines
Quoted 8 lines
virtio-gpu.tex \subsubsection{Device Operation: Configure mouse cursor} · line 192
Quoted 5 lines
The device MAY also choose to
virtio-gpu.tex +3
-0+not create mapping for the shared resource. Especially for small +resources it might be more efficient to just copy the data instead of +establishing a shared mapping.Should the device send a response code (
OK_SHARED} to inform the driver that the resource is shared?Is there a good reason why the driver should be aware of that?
Technically TRANSFER_TO_HOST is legal for 2D resources (though rarely used). I suppose we can modify RESOURCE_INFO to inform userspace the resource is shared, and thus avoid hypercalls.
If we leave the decision to create the shared resource to the driver and not the device, *_2D_SHARED makes sense. Right now, it's *_2D_MAYBE_SHARED.
Quoted 7 lines
I'd prefer to make that an internal implementation detail of the device and not inform the guest.
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
1) Driver sends
RESOURCE_CREATE_2Dshared request
2) Driver sendsATTACH_BACKINGrequest
3) Device creates a shared resource
4) Driver sendsSET_SCANOUTrequest
5) Device sends shared resource to display
6) Driver sendsDETACH_BACKINGrequest
Quoted 6 lines
Quoted 3 lines
Hmm, I think you can crash the current qemu code that way (didn't test though). I think the proper solution would be to return
-EBUSYon theDETACH_BACKINGrequest. Guess I'll add a new error code for that.That would add the extra complication of a "delayed detach list" in the guest kernel.
Why? I can't see how that can happen with the linux guest driver. You need a drm framebuffer to map an gem object to a scanout. The drm framebuffer holds a reference of the gem object, so the linux kernel wouldn't try to delete the gem object (-> DETACH_BACKING + UNREF) while it is mapped to the scanout because the refcount wouldn't go down to zero.
Quoted 7 lines
I don't know if it's absolutely necessary --
because the open source implementation (udmabuf) seems to take a reference on all constituent pages like DRM gem does [a][b][c][d]. We should figure out if the extra reference is sufficient to prevent the pages from going back to guest RAM (the guest kernel takes references on those pages too -- I don't know if guest references are treated differently).
They are never taken away from guest ram. The guest is free to do whatever it wants with the pages. It's the job of the guest driver to make sure the pages are not released until the host stopped using them. So the host must drop all references before completing the DETACH_BACKING command.
Quoted 7 lines
Quoted 6 lines
I think we can do 90% of that optimization without changing the protocol. Display updates typically involve multiple commands. A transfer, possibly a set-scanout (when page-flipping), finally flush. The driver can first queue all commands, then notify the host, so we will have only one vmexit per display update instead of one vmexit per command.
Quoted 7 lines
We also want to prevent this scenario:
1) Guest creates a writecombine mapping (like drm_gem_shmem_* helpers do), but the pages were previously accessed and thus some data is in the cache.
2) Cache eviction
3) Non-zero data in buffer.
This can happen on arm I guess? I know caching is a more complicated on arm than on x86. Care to explaint this (or do you have a good link?).
Also: what is special in virtualization here? There are arm drivers using drm_gem_shmem_* helpers. How do they manage this?
Quoted 4 lines
Quoted 1 line
Is there a good reason why the driver should be aware of that?
Technically
TRANSFER_TO_HOSTis legal for 2D resources (though rarely used).
It's used for dumb gem objects.
Quoted 6 lines
I suppose we can modify
RESOURCE_INFOto inform userspace the resource is shared, and thus avoid hypercalls.If we leave the decision to create the shared resource to the driver and not the device,
*_2D_SHAREDmakes sense. Right now, it's*_2D_MAYBE_SHARED.
Yes, the guest can't force it.
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 15 lines · Gerd Hoffmann
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
1) Driver sends
RESOURCE_CREATE_2Dshared request
2) Driver sendsATTACH_BACKINGrequest
3) Device creates a shared resource
4) Driver sendsSET_SCANOUTrequest
5) Device sends shared resource to display
6) Driver sendsDETACH_BACKINGrequestQuoted 6 lines
Quoted 3 lines
Hmm, I think you can crash the current qemu code that way (didn't test though). I think the proper solution would be to return
-EBUSYon theDETACH_BACKINGrequest. Guess I'll add a new error code for that.That would add the extra complication of a "delayed detach list" in the guest kernel.
Quoted 6 lines
Why? I can't see how that can happen with the linux guest driver. You need a drm framebuffer to map an gem object to a scanout. The drm framebuffer holds a reference of the gem object, so the linux kernel wouldn't try to delete the gem object (->
DETACH_BACKING+ UNREF) while it is mapped to the scanout because the refcount wouldn't go down to zero.
The case I'm thinking about is:
1) Guest DRM framebuffer refcount goes to zero
2) Host DRM framebuffer refcount is non-zero
I prefer the guest kernel just complain loudly rather than doing complicated tricks in such instance.
Quoted 9 lines
Quoted 8 lines
I don't know if it's absolutely necessary --
because the open source implementation (udmabuf) seems to take a reference on all constituent pages like DRM gem does [a][b][c][d]. We should figure out if the extra reference is sufficient to prevent the pages from going back to guest RAM (the guest kernel takes references on those pages too -- I don't know if guest references are treated differently).
Quoted 5 lines
They are never taken away from guest ram. The guest is free to do whatever it wants with the pages. It's the job of the guest driver to make sure the pages are not released until the host stopped using them. So the host must drop all references before completing the
DETACH_BACKINGcommand.
Let's write that in the spec.
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
I think we can do 90% of that optimization without changing the protocol. Display updates typically involve multiple commands. A transfer, possibly a set-scanout (when page-flipping), finally flush. The driver can first queue all commands, then notify the host, so we will have only one vmexit per display update instead of one vmexit per command.
We also want to prevent this scenario:
Quoted 7 lines
Quoted 5 lines
1) Guest creates a writecombine mapping (like drm_gem_shmem_* helpers do), but the pages were previously accessed and thus some data is in the cache.
2) Cache eviction
3) Non-zero data in buffer.This can happen on arm I guess?
Exactly.
Quoted 2 lines
I know caching is a more complicated on
arm than on x86. Care to explaint this (or do you have a good link?).
Two differences:
- On x86, the guest can perform cache operations, while on ARM it's a privileged operation [1].
- On ARM, the guest attribute seems to override the host attribute[1]. I've verified this.
Even on x86, it seems it's possible to get WC buffers by changing the MTTR[2] (though I haven't verified this yet).
[1] https://events.static.linuxfound.org/sites/events/files/slides/slides_10.pdf
[2] https://lwn.net/Articles/646712/
We (you?) need to modify drm_gem_shmem_* helpers to always be cached for virtgpu for now, since the TTM removal changed that behavior.
Then, we should expose host properties to the guest, like [1] suggests (maybe something like VIRTIO_GPU_F_GUEST_ATTRIBUTE_OVERRIDE, VIRTIO_GPU_F_GUEST_CACHE_OPS). If there's already a method to do this, that would be awesome.
Then we can optimize away.
Quoted 3 lines
Also: what is special in virtualization here? There are arm drivers using drm_gem_shmem_* helpers. How do they manage this?
Panfrost, v3d. I'm surprised they haven't hit this issue yet.
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
Is there a good reason why the driver should be aware of that?
Technically
TRANSFER_TO_HOSTis legal for 2D resources (though rarely used).It's used for dumb gem objects.
Quoted 3 lines
I suppose we can modify
RESOURCE_INFOto inform userspace the resource is shared, and thus avoid hypercalls.
Quoted 9 lines
Quoted 3 lines
If we leave the decision to create the shared resource to the driver and not the device,
*_2D_SHAREDmakes sense. Right now, it's*_2D_MAYBE_SHARED.Yes, the guest can't force it.
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Hi,
Quoted 7 lines
Quoted 5 lines
They are never taken away from guest ram. The guest is free to do whatever it wants with the pages. It's the job of the guest driver to make sure the pages are not released until the host stopped using them. So the host must drop all references before completing the
DETACH_BACKINGcommand.Let's write that in the spec.
Yes, saying so explicitly that totally makes sense. I'll update the patch accordingly.
Quoted 2 lines
We (you?) need to modify drm_gem_shmem_* helpers to always be cached for virtgpu for now, since the TTM removal changed that behavior.
We can just roll our own mmap function to handle that. I'll prepare a patch (that is independent from F_SHARED anyway).
Quoted 4 lines
Then, we should expose host properties to the guest, like [1] suggests (maybe something like
VIRTIO_GPU_F_GUEST_ATTRIBUTE_OVERRIDE,VIRTIO_GPU_F_GUEST_CACHE_OPS). If there's already a method to do this, that would be awesome.
We'll need that for sure when mapping host resources into the guest, but we are not there yet with F_SHARED only.
Quoted 4 lines
Quoted 2 lines
Also: what is special in virtualization here? There are arm drivers using drm_gem_shmem_* helpers. How do they manage this?
Panfrost, v3d. I'm surprised they haven't hit this issue yet.
Maybe because all mappings are consistent. With virtio-gpu you could end up in a situation where the guest uses wc whereas the host uses cached and things go boom from there.
cheers,
Gerd
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]