Virtio-GPU S4 (hibernation) support
Subsystems (3)
v8 cover Virtio-GPU S4 (hibernation) support
From: Dongwon Kim <[email protected]>
This patch series implements the .freeze and .restore hooks for the virtio-gpu driver, alongside a PM notifier to manage object restoration during S4 (hibernation) cycles.
The series is structured as follows:
1. The first patch introduces virtgpu_freeze and virtgpu_restore. These functions handle the teardown of virtio queues before suspension and their subsequent recreation during the resume process.
2. The second patch implements a tracking and restoration mechanism for virtio_gpu_object instances. This is essential because the host (e.g., QEMU) destroys all associated graphics resources during the virtio-gpu reset triggered by the hibernation/resume cycle.
3. The third patch adds a PM notifier to handle the resubmission of these tracked virtio-gpu objects to the host once the guest restores its virtqueues.
Together, these changes ensure that the virtio-gpu driver can survive hibernation without resource loss or invalid resource errors.
v2: - Added 10ms sleep in virtgpu_freeze to prevent lockups during resumption.
v3: - Replaced the fixed 10ms delay with wait calls that trigger once the
virtio queue is empty.
(Dmitry Osipenko)
v4: - Limited the scope to S4 (hibernation). S3 resource loss can be avoided
via QEMU configurations (e.g., x-pcie-pm-no-soft-reset=true).
v5: - Implemented removal of objects from the restore list before freeing to
prevent a use-after-free situation.
(Nirmoy Das)
- Protected restore list operations with a spinlock. (Nirmoy Das)
- Moved the restore list node into the virtio_gpu_bo struct to optimize memory usage.
(Dmitry Osipenko)
- Removed unused header: drm_atomic_helper.h.
(Dmitry Osipenko)
v6: - Added support for objects backed by imported dma-bufs.
(Dmitry Osipenko)
- Excluded virgl 3D objects from the restore list as they are currently non-recoverable.
(Dmitry Osipenko)
- Renamed the list node to 'restore_node'.
(Nirmoy Das)
- Switched from spinlock to mutex for restore list updates. (Nirmoy Das)
- Initialized restore_node at creation to allow safe 'list_empty' checks.
- Introduced a 'hibernation' flag in the PM notifier to trigger restoration at the correct stage in virtgpu_restore.
v7: - Added helper: virtio_gpu_add_object_to_restore_list.
(Dmitry Osipenko)
- Implemented unreferencing of all host-side objects before hibernation to ensure a clean state upon resume.
(Dmitry Osipenko)
v8: - Introduced virtio_gpu_remove_from_restore_list helper and ensured it is
called across all object destruction paths (General, Prime, and VRAM) to prevent use-after-free.
(Dmitry Osipenko)
- Relocated restore list removal logic from the final cleanup stage to the initial free stage for improved synchronization. (Dmitry Osipenko)
- Added missing drm_print.h header.
Dongwon Kim (3):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Add support for saving and restoring virtio_gpu_objects drm/virtio: Add PM notifier to restore objects after hibernation
drivers/gpu/drm/virtio/virtgpu_drv.c | 74 ++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 24 ++++++- drivers/gpu/drm/virtio/virtgpu_kms.c | 54 +++++++++++++-- drivers/gpu/drm/virtio/virtgpu_object.c | 88 ++++++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_prime.c | 45 ++++++++++++- drivers/gpu/drm/virtio/virtgpu_vq.c | 13 +++- drivers/gpu/drm/virtio/virtgpu_vram.c | 5 +- 7 files changed, 289 insertions(+), 14 deletions(-)
Signature · 1 line
2.34.1
PATCH v8 1/3 drm/virtio: Freeze and restore hooks to support suspend and resume
From: Dongwon Kim <[email protected]>
virtio device needs to delete before VM suspend happens then reinitialize all virtqueues again upon resume
v2: 10ms sleep was added in virtgpu_freeze to avoid the situation
the driver is locked up during resumption.
v3: Plain 10ms delay was replaced with wait calls which wait until
the virtio queue is empty.
(Dmitry Osipenko)
v4: Change wait_event to wait_event_timeout to prevent permanent wait
(Nirmoy Das)
--- drivers/gpu/drm/virtio/virtgpu_drv.c | 62 +++++++++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 1 + drivers/gpu/drm/virtio/virtgpu_kms.c | 23 ++++++++--- 3 files changed, 79 insertions(+), 7 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +61 -1
@@ -164,6 +164,62 @@ static unsigned int features[] = {VIRTIO_GPU_F_RESOURCE_BLOB , VIRTIO_GPU_F_CONTEXT_INIT, }; + +#ifdef CONFIG_PM_SLEEP +static int virtgpu_freeze (struct virtio_device *vdev) +{ +struct drm_device *dev = vdev->priv; +struct virtio_gpu_device *vgdev = dev->dev_private; +int error; + + error =drm_mode_config_helper_suspend (dev); +if (error) { +DRM_ERROR ("suspend error %d\n" , error); +return error; + } + +flush_work (&vgdev->obj_free_work); +flush_work (&vgdev->ctrlq.dequeue_work); +flush_work (&vgdev->cursorq.dequeue_work); +flush_work (&vgdev->config_changed_work); + +wait_event_timeout (vgdev->ctrlq.ack_queue, + vgdev->ctrlq.vq->num_free == vgdev->ctrlq.vq->num_max, +5 *HZ ); + +wait_event_timeout (vgdev->cursorq.ack_queue, + vgdev->cursorq.vq->num_free == vgdev->cursorq.vq->num_max, +5 *HZ ); + + vdev->config->del_vqs (vdev); + +return 0 ; +} + +static int virtgpu_restore (struct virtio_device *vdev) +{ +struct drm_device *dev = vdev->priv; +struct virtio_gpu_device *vgdev = dev->dev_private; +int error; + + error =virtio_gpu_find_vqs (vgdev); +if (error) { +DRM_ERROR ("failed to find virt queues\n" ); +return error; + } + +virtio_device_ready (vdev); + + error =drm_mode_config_helper_resume (dev); +if (error) { +DRM_ERROR ("resume error %d\n" , error); +return error; + } + +return 0 ; +} +#endif +static struct virtio_driver virtio_gpu_driver = { .feature_table = features, .feature_table_size = ARRAY_SIZE(features),
@@ -172,7 +228,11 @@ static struct virtio_driver virtio_gpu_driver = { .probe = virtio_gpu_probe, .remove = virtio_gpu_remove, .shutdown = virtio_gpu_shutdown, - .config_changed = virtio_gpu_config_changed + .config_changed = virtio_gpu_config_changed, +#ifdef CONFIG_PM_SLEEP + .freeze = virtgpu_freeze, + .restore = virtgpu_restore, +#endif };static int __init virtio_gpu_driver_init (void )
drivers/gpu/drm/virtio/virtgpu_drv.h +1 -0
@@ -300,6 +300,7 @@ void virtio_gpu_deinit(struct drm_device *dev);void virtio_gpu_release (struct drm_device *dev);int virtio_gpu_driver_open (struct drm_device *dev,struct drm_file *file);void virtio_gpu_driver_postclose (struct drm_device *dev,struct drm_file *file); +int virtio_gpu_find_vqs (struct virtio_gpu_device *vgdev);/* virtgpu_gem.c */ int virtio_gpu_gem_object_open(struct drm_gem_object *obj,
drivers/gpu/drm/virtio/virtgpu_kms.c +17 -6
@@ -115,15 +115,28 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev, vgdev->num_capsets = num_capsets; } -int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev) +int virtio_gpu_find_vqs (struct virtio_gpu_device *vgdev) {struct virtqueue_info vqs_info[] = { {"control" , virtio_gpu_ctrl_ack }, {"cursor" , virtio_gpu_cursor_ack }, }; -struct virtio_gpu_device *vgdev; -/* this will expand later */ struct virtqueue *vqs[2 ]; +int ret; + + ret =virtio_find_vqs (vgdev->vdev,2 , vqs, vqs_info,NULL ); +if (ret) +return ret; + + vgdev->ctrlq.vq = vqs[0 ]; + vgdev->cursorq.vq = vqs[1 ]; + +return 0 ; +} + +int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev) +{ +struct virtio_gpu_device *vgdev;u32 num_scanouts, num_capsets;int ret =0 ;
@@ -207,13 +220,11 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)DRM_INFO ("features: %ccontext_init\n" , vgdev->has_context_init ?'+' :'-' ); - ret =virtio_find_vqs (vgdev->vdev,2 , vqs, vqs_info,NULL ); + ret =virtio_gpu_find_vqs (vgdev);if (ret) {DRM_ERROR ("failed to find virt queues\n" ); goto err_vqs; } - vgdev->ctrlq.vq = vqs[0 ]; - vgdev->cursorq.vq = vqs[1 ]; ret =virtio_gpu_alloc_vbufs (vgdev);if (ret) {DRM_ERROR ("failed to alloc vbufs\n" );
Signature · 1 line
2.34.1
Re: [PATCH v8 1/3] drm/virtio: Freeze and restore hooks to support suspend and resume
Quoted 12 lines · On Wed, 2026-04-29 at 13:47 -0700, [email protected] wrote:
From: Dongwon Kim <[email protected]>
virtio device needs to delete before VM suspend happens then reinitialize all virtqueues again upon resume
v2: 10ms sleep was added in
virtgpu_freezeto avoid the situation the driver is locked up during resumption.v3: Plain 10ms delay was replaced with wait calls which wait until the virtio queue is empty.
(Dmitry Osipenko)
Quoted 8 lines
v4: Change
wait_eventtowait_event_timeoutto prevent permanent wait (Nirmoy Das)Suggested-by Dmitry Osipenko <[email protected]>Tested-by Dmitry Osipenko <[email protected]>Cc Vivek Kasireddy <[email protected]>Cc Nirmoy Das <[email protected]>Signed-off-by Dongwon Kim <[email protected]>
Thanks for fixing this issue!
I cherry-picked this patch onto a kernel 6.1 and tested it with QNX hypervisor.
It works as expected and resolves the problem in our environment.
Best regards,
Morris
Dmitry Osipenko
maintainerRe: [PATCH v8 1/3] drm/virtio: Freeze and restore hooks to support suspend and resume
In reply to [email protected]29.04.2026 23:47, [email protected] пишет:
Quoted 14 lines
drivers/gpu/drm/virtio/virtgpu_drv.c +14
-0+static int virtgpu_freeze (struct virtio_device *vdev) +{ +struct drm_device *dev = vdev->priv; +struct virtio_gpu_device *vgdev = dev->dev_private; +int error; + + error =drm_mode_config_helper_suspend (dev); +if (error) { +DRM_ERROR ("suspend error %d\n" , error); +return error; + } + +flush_work (&vgdev->obj_free_work); +flush_work (&vgdev->ctrlq.dequeue_work);
drivers/gpu/drm/virtio/virtgpu_drv.c static unsigned int features[] = { · line 192
Quoted 10 lines
drivers/gpu/drm/virtio/virtgpu_drv.c +10
-0+flush_work (&vgdev->cursorq.dequeue_work); +flush_work (&vgdev->config_changed_work); + +wait_event_timeout (vgdev->ctrlq.ack_queue, + vgdev->ctrlq.vq->num_free == vgdev->ctrlq.vq->num_max, +5 *HZ ); + +wait_event_timeout (vgdev->cursorq.ack_queue, + vgdev->cursorq.vq->num_free == vgdev->cursorq.vq->num_max, +5 *HZ );
Return error on timeout or better wait indefinitely. On [1] I suggested to print warning messages on event timeout, could be a common virtio_gpu_wait_queue_event() helper.
[1]
https://lore.kernel.org/dri-devel/[email protected]/
Signature · 2 lines
Best regards, Dmitry
PATCH v8 2/3 drm/virtio: Add support for saving and restoring virtio_gpu_objects
In reply to [email protected]From: Dongwon Kim <[email protected]>
When the host KVM/QEMU resumes from hibernation, it loses all graphics resources previously submitted by the guest OS, as the QEMU process is terminated during the suspend-resume cycle. This leads to invalid resource errors when the guest OS attempts to interact with the host using those resources after resumption.
To resolve this, the virtio-gpu driver now tracks all active virtio_gpu_objects and provides a mechanism to restore them by re-submitting the objects to QEMU when needed (e.g., during resume from hibernation).
v2: - Attach backing is done if bo->attached was set before
v3: - Restoration is no longer triggered via .restore; instead, it is handled
by a PM notifier only during hibernation.
v4: - Remove virtio_gpu_object from the restore list before freeing the object
to prevent a use-after-free situation.
(Nirmoy Das)
- Protect restore list operations with a spinlock (Nirmoy Das)
- Initialize ret with 0 in virtio_gpu_object_restore_all (Nirmoy Das)
- Move restore list node into virtio_gpu_bo struct to reduce memory usage (Dmitry Osipenko)
v5: - Include object backed by imported dmabuf
(Dmitry Osipenko)
- Not storing virgl objects in the restore_list as virgl 3D objects are not recoverable.
(Dmitry Osipenko)
- Change the name 'list',a node in restore_list to 'restore_node' (Nirmoy Das)
- Use mutex instead of spinlock when updating restore_list (Nirmoy Das)
- Initialize restore_node when virtio_gpu_object is created - this is to determine whether the object should be removed from the restore_list with 'list_empty' function when it is time to free the object as not all objects will be added to the list.
v6: - Add a helper, virtio_gpu_add_object_to_restore_list
(Dmitry Osipenko)
v7: - Add drm_print.h
v8: - Introduce virtio_gpu_remove_from_restore_list helper and ensure its call in
all object destruction paths (General, Prime, and VRAM) to prevent use-after-free.
(Dmitry Osipenko)
- Relocate the restore list removal logic from the final cleanup stage to the initial free stage for better synchronization. (Dmitry Osipenko)
--- drivers/gpu/drm/virtio/virtgpu_drv.h | 15 ++++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 3 ++ drivers/gpu/drm/virtio/virtgpu_object.c | 70 +++++++++++++++++++++++++ drivers/gpu/drm/virtio/virtgpu_prime.c | 43 +++++++++++++++ drivers/gpu/drm/virtio/virtgpu_vram.c | 3 ++ 5 files changed, 134 insertions(+)
drivers/gpu/drm/virtio/virtgpu_drv.h +15 -0
@@ -98,6 +98,10 @@ struct virtio_gpu_object {int uuid_state;uuid_t uuid; + +/* for restoration of objects after hibernation */ +struct virtio_gpu_object_params params; +struct list_head restore_node; };#define gem_to_virtio_gpu_obj (gobj) \ container_of((gobj), struct virtio_gpu_object, base.base)
@@ -265,6 +269,8 @@ struct virtio_gpu_device {struct work_struct obj_free_work;spinlock_t obj_free_lock;struct list_head obj_free_list; +struct mutex obj_restore_lock; +struct list_head obj_restore_list;struct virtio_gpu_drv_capset *capsets;uint32_t num_capsets;
@@ -467,6 +473,7 @@ void virtio_gpu_fence_event_process(struct virtio_gpu_device *vdev, u64 fence_id);/* virtgpu_object.c */ +void virtio_gpu_remove_from_restore_list (struct virtio_gpu_object *bo);void virtio_gpu_cleanup_object (struct virtio_gpu_object *bo);struct drm_gem_object *virtio_gpu_create_object (struct drm_device *dev,size_t size);
@@ -479,6 +486,12 @@ bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid); + +void virtio_gpu_add_object_to_restore_list (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo); + +int virtio_gpu_object_restore_all (struct virtio_gpu_device *vgdev); +/* virtgpu_prime.c */ int virtio_gpu_resource_assign_uuid (struct virtio_gpu_device *vgdev,struct virtio_gpu_object *bo);
@@ -493,6 +506,8 @@ int virtgpu_dma_buf_import_sgt(struct virtio_gpu_mem_entry **ents,unsigned int *nents, struct virtio_gpu_object *bo, struct dma_buf_attachment *attach); +int virtgpu_dma_buf_obj_resubmit (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo);/* virtgpu_debugfs.c */ void virtio_gpu_debugfs_init (struct drm_minor *minor);
drivers/gpu/drm/virtio/virtgpu_kms.c +3 -0
@@ -171,6 +171,8 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) virtio_gpu_array_put_free_work);INIT_LIST_HEAD (&vgdev->obj_free_list);spin_lock_init (&vgdev->obj_free_lock); +INIT_LIST_HEAD (&vgdev->obj_restore_list); +mutex_init (&vgdev->obj_restore_lock);#ifdef __LITTLE_ENDIANif (virtio_has_feature (vgdev->vdev,VIRTIO_GPU_F_VIRGL ))
@@ -299,6 +301,7 @@ void virtio_gpu_deinit(struct drm_device *dev)flush_work (&vgdev->config_changed_work);virtio_reset_device (vgdev->vdev); vgdev->vdev->config->del_vqs (vgdev->vdev); +mutex_destroy (&vgdev->obj_restore_lock); }void virtio_gpu_release (struct drm_device *dev)
drivers/gpu/drm/virtio/virtgpu_object.c +70 -0
97 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_prime.c +43 -0
73 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_vram.c +3 -0
@@ -18,6 +18,7 @@ static void virtio_gpu_vram_free(struct drm_gem_object *obj)if (unmap)virtio_gpu_cmd_unmap (vgdev, bo); +virtio_gpu_remove_from_restore_list (bo);virtio_gpu_cmd_unref_resource (vgdev, bo);virtio_gpu_notify (vgdev);return ;
@@ -200,6 +201,8 @@ int virtio_gpu_vram_create(struct virtio_gpu_device *vgdev, obj = &vram->base.base.base; obj->funcs = &virtio_gpu_vram_funcs; +INIT_LIST_HEAD (&vram->base.restore_node); + params->size =PAGE_ALIGN (params->size);drm_gem_private_object_init (vgdev->ddev, obj, params->size);
Signature · 1 line
2.34.1
Dmitry Osipenko
maintainerRe: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
29.04.2026 23:47, [email protected] пишет:
drivers/gpu/drm/virtio/virtgpu_object.c static void virtio_gpu_free_object(struct drm_gem_object *obj) · line 107
Quoted 9 lines
void
virtio_gpu_cleanup_object(struct virtio_gpu_object *bo){drivers/gpu/drm/virtio/virtgpu_object.c +1
-0struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;@@ -94,6 +103,7 @@ static void virtio_gpu_free_object(struct drm_gem_object *obj)struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;if (bo->created) { +virtio_gpu_remove_from_restore_list (bo);virtio_gpu_cmd_unref_resource (vgdev, bo);
Can virtio_gpu_remove_from_restore_list() be moved to virtio_gpu_cmd_unref_resource()?
Signature · 2 lines
Best regards, Dmitry
RE: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
Hi Dmitry,
Quoted 9 lines
Subject: Re: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring
virtio_gpu_objects29.04.2026 23:47, [email protected] пишет:
Quoted 2 lines
void
virtio_gpu_cleanup_object(struct virtio_gpu_object *bo){
structvirtio_gpu_device*vgdev =bo->base.base.dev->dev_private;@@
Quoted 1 line
-94,6 +103,7 @@ static void virtio_gpu_free_object(struct
drm_gem_object*obj)
Quoted 8 lines
drivers/gpu/drm/virtio/virtgpu_object.c static void virtio_gpu_free_object(struct drm_gem_object *obj) · line 107
Quoted 5 lines
drivers/gpu/drm/virtio/virtgpu_object.c +1
-0struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;if (bo->created) { +virtio_gpu_remove_from_restore_list (bo);virtio_gpu_cmd_unref_resource (vgdev, bo);Can
virtio_gpu_remove_from_restore_list()be moved tovirtio_gpu_cmd_unref_resource()?
Yes, technically it can be done and we can remove some redundancy but don't you think adding it or removing it from the list would be better to be handled outside of vq functions (virtgpu_vq.c) where actual communication messages are processed?
Thanks,
DW
Quoted 4 lines
--
Best regards,
Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
Quoted 6 lines · Dongwon
Quoted 2 lines
Can
virtio_gpu_remove_from_restore_list()be moved tovirtio_gpu_cmd_unref_resource()?Yes, technically it can be done and we can remove some redundancy but don't you think adding it or removing it from the list would be better to be handled outside of vq functions (
virtgpu_vq.c) where actual communication messages are processed?
cmd_unref() indeed isn't a good fit
Could you please check if list node can now be removed from virtio_gpu_cleanup_object()? IIRC, there was a problem with that in older patch versions.
Signature · 2 lines
Best regards, Dmitry
RE: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
Hi Dmitry,
Quoted 11 lines
Subject: Re: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring
virtio_gpu_objectsQuoted 6 lines · Dongwon
Quoted 2 lines
Can
virtio_gpu_remove_from_restore_list()be moved tovirtio_gpu_cmd_unref_resource()?Yes, technically it can be done and we can remove some redundancy but don't you think adding it or removing it from the list would be better to be handled outside of vq functions (
virtgpu_vq.c) where actual communication messages are processed?
Quoted 5 lines
cmd_unref()indeed isn't a good fitCould you please check if list node can now be removed from
virtio_gpu_cleanup_object()? IIRC, there was a problem with that in older patch versions.
Yes, you are talking about patch v7 where removal of the node was done in virtio_gpu_cleanup_object(). You actually mentioned this at that time,
" Also, looks wrong to delete object's node from the list by thevirtio_gpu_cleanup_object() that is invoked from cmd-completion callback. You calling virtio_gpu_object_unref_all() that walks up obj_restore_list while obj already may be in process of async unref'ing?"
Your concern was valid so I moved node removal right before cmd unref call in v8 (the latest so far).
Thanks,
DW
Quoted 4 lines
--
Best regards,
Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
Quoted 19 lines · Dongwon
Hi Dmitry,
Quoted 11 lines
Subject: Re: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring
virtio_gpu_objectsQuoted 6 lines · Dongwon
Quoted 2 lines
Can
virtio_gpu_remove_from_restore_list()be moved tovirtio_gpu_cmd_unref_resource()?Yes, technically it can be done and we can remove some redundancy but don't you think adding it or removing it from the list would be better to be handled outside of vq functions (
virtgpu_vq.c) where actual communication messages are processed?Quoted 5 lines
cmd_unref()indeed isn't a good fitCould you please check if list node can now be removed from
virtio_gpu_cleanup_object()? IIRC, there was a problem with that in older patch versions.
Quoted 7 lines
Yes, you are talking about patch v7 where removal of the node was done in
virtio_gpu_cleanup_object(). You actually mentioned this at that time," Also, looks wrong to delete object's node from the list by the
virtio_gpu_cleanup_object()that is invoked from cmd-completion callback. You callingvirtio_gpu_object_unref_all()that walks upobj_restore_listwhile obj already may be in process of async unref'ing?"Your concern was valid so I moved node removal right before cmd unref call in v8 (the latest so far).
Thanks for the reminder, let's keep it as in this patch then.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
In reply to [email protected]29.04.2026 23:47, [email protected] пишет:
Quoted 62 lines
From: Dongwon Kim <[email protected]>
When the host KVM/QEMU resumes from hibernation, it loses all graphics resources previously submitted by the guest OS, as the QEMU process is terminated during the suspend-resume cycle. This leads to invalid resource errors when the guest OS attempts to interact with the host using those resources after resumption.
To resolve this, the virtio-gpu driver now tracks all active
virtio_gpu_objectsand provides a mechanism to restore them by re-submitting the objects to QEMU when needed (e.g., during resume from hibernation).v2: - Attach backing is done if
bo->attachedwas set beforev3: - Restoration is no longer triggered via .restore; instead, it is handled
by a PM notifier only during hibernation.v4: - Remove
virtio_gpu_objectfrom the restore list before freeing the object
to prevent a use-after-free situation.
(Nirmoy Das)- Protect restore list operations with a spinlock (Nirmoy Das)
- Initialize ret with 0 in
virtio_gpu_object_restore_all(Nirmoy Das)- Move restore list node into
virtio_gpu_bostruct to reduce memory usage (Dmitry Osipenko)v5: - Include object backed by imported dmabuf
(Dmitry Osipenko)- Not storing virgl objects in the
restore_listas virgl 3D objects are not recoverable.
(Dmitry Osipenko)- Change the name 'list',a node in
restore_listto 'restore_node' (Nirmoy Das)- Use mutex instead of spinlock when updating
restore_list(Nirmoy Das)- Initialize
restore_nodewhenvirtio_gpu_objectis created - this is to determine whether the object should be removed from therestore_listwith 'list_empty' function when it is time to free the object as not all objects will be added to the list.v6: - Add a helper,
virtio_gpu_add_object_to_restore_list
(Dmitry Osipenko)v7: - Add
drm_print.hv8: - Introduce
virtio_gpu_remove_from_restore_listhelper and ensure its call in
all object destruction paths (General, Prime, and VRAM) to prevent use-after-free.
(Dmitry Osipenko)- Relocate the restore list removal logic from the final cleanup stage to the initial free stage for better synchronization. (Dmitry Osipenko)
Quoted 5 lines
Cc Dmitry Osipenko <[email protected]>Cc Vivek Kasireddy <[email protected]>Cc Nirmoy Das <[email protected]>Signed-off-by Dongwon Kim <[email protected]>---
Nit: change log is preferred to put under '--' to not have it in a commit message
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v8 2/3] drm/virtio: Add support for saving and restoring virtio_gpu_objects
In reply to [email protected]29.04.2026 23:47, [email protected] пишет:
drivers/gpu/drm/virtio/virtgpu_object.c static void virtio_gpu_free_object(struct drm_gem_object *obj) · line 107
Quoted 9 lines
void
virtio_gpu_cleanup_object(struct virtio_gpu_object *bo){drivers/gpu/drm/virtio/virtgpu_object.c +1
-0struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;@@ -94,6 +103,7 @@ static void virtio_gpu_free_object(struct drm_gem_object *obj)struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private;if (bo->created) { +virtio_gpu_remove_from_restore_list (bo);virtio_gpu_cmd_unref_resource (vgdev, bo);
Can virtio_gpu_remove_from_restore_list() be moved to virtio_gpu_cmd_unref_resource()?
Signature · 2 lines
Best regards, Dmitry
PATCH v8 3/3 drm/virtio: Add PM notifier to restore objects after hibernation
In reply to [email protected]From: Dongwon Kim <[email protected]>
Register a PM notifier in virtio-gpu to handle suspend/hibernate events. On PM_POST_HIBERNATION, restore all GPU objects so that the driver can properly recover after resume.
v2: Remove unused header - drm_atomic_helper.h
(Dmitry Osipenko)
v3: Objects for virgl usecase can't be recovered after resume so
blocking S4 when virgl is enabled
(Dmitry Osipenko)
v4: Restoring objects in the PM notifier is too late, as virtio-gpu
message communication begins in virtgpu_restore once virtqueues are re-established. To address this, a 'hibernation' flag is set during the PM_HIBERNATION_PREPARE phase in the notifier. This flag is then used in virtgpu_restore to detect if the system is resuming from S4, allowing objects to be recovered immediately after virtqueues are reconfigured.
v5: Unreference all objects before hibernation so they can be removed
on the host side, since they will be fully restored anyway. This prevents the situation where host-side hibernation fails (leaving all associated resources still alive) while the virtio-gpu driver still attempts to restore those objects.
(Dmitry Osipenko)
--- drivers/gpu/drm/virtio/virtgpu_drv.c | 12 +++++++++++ drivers/gpu/drm/virtio/virtgpu_drv.h | 8 ++++++- drivers/gpu/drm/virtio/virtgpu_kms.c | 28 +++++++++++++++++++++++++ drivers/gpu/drm/virtio/virtgpu_object.c | 18 +++++++++++++++- drivers/gpu/drm/virtio/virtgpu_prime.c | 2 +- drivers/gpu/drm/virtio/virtgpu_vq.c | 13 +++++++++--- drivers/gpu/drm/virtio/virtgpu_vram.c | 2 +- 7 files changed, 76 insertions(+), 7 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +12 -0
@@ -178,6 +178,9 @@ static int virtgpu_freeze(struct virtio_device *vdev)return error; } +if (vgdev->hibernation) +virtio_gpu_object_unref_all (vgdev); +flush_work (&vgdev->obj_free_work);flush_work (&vgdev->ctrlq.dequeue_work);flush_work (&vgdev->cursorq.dequeue_work);
@@ -210,6 +213,15 @@ static int virtgpu_restore(struct virtio_device *vdev)virtio_device_ready (vdev); +if (vgdev->hibernation) { + vgdev->hibernation = false; + error =virtio_gpu_object_restore_all (vgdev); +if (error) { +DRM_ERROR ("Failed to recover virtio-gpu objects\n" ); +return error; + } + } + error =drm_mode_config_helper_resume (dev);if (error) {DRM_ERROR ("resume error %d\n" , error);
drivers/gpu/drm/virtio/virtgpu_drv.h +7 -1
@@ -261,6 +261,7 @@ struct virtio_gpu_device {bool has_resource_blob;bool has_host_visible;bool has_context_init; +bool hibernation;struct virtio_shm_region host_visible_region;struct drm_mm host_visible_mm;
@@ -277,6 +278,8 @@ struct virtio_gpu_device {uint64_t capset_id_mask;struct list_head cap_cache; +struct notifier_block pm_nb; +/* protects uuid state when exporting */ spinlock_t resource_export_lock;/* protects map state and host_visible_mm */
@@ -341,7 +344,8 @@ void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev,struct virtio_gpu_object_array *objs, struct virtio_gpu_fence *fence);void virtio_gpu_cmd_unref_resource (struct virtio_gpu_device *vgdev, -struct virtio_gpu_object *bo); +struct virtio_gpu_object *bo, +int no_cb); int virtio_gpu_panic_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev, uint64_t offset, uint32_t width, uint32_t height,
@@ -492,6 +496,8 @@ void virtio_gpu_add_object_to_restore_list(struct virtio_gpu_device *vgdev,int virtio_gpu_object_restore_all (struct virtio_gpu_device *vgdev); +void virtio_gpu_object_unref_all (struct virtio_gpu_device *vgdev); +/* virtgpu_prime.c */ int virtio_gpu_resource_assign_uuid (struct virtio_gpu_device *vgdev,struct virtio_gpu_object *bo);
drivers/gpu/drm/virtio/virtgpu_kms.c +28 -0
@@ -26,6 +26,8 @@#include <linux/virtio.h> #include <linux/virtio_config.h> #include <linux/virtio_ring.h> +#include <linux/suspend.h> +#include <linux/pm_runtime.h> #include <drm/drm_file.h> #include <drm/drm_managed.h>
@@ -134,6 +136,25 @@ int virtio_gpu_find_vqs(struct virtio_gpu_device *vgdev)return 0 ; } +static int virtio_gpu_pm_notifier (struct notifier_block *nb,unsigned long mode, +void *data) +{ +struct virtio_gpu_device *vgdev =container_of (nb, + struct virtio_gpu_device, + pm_nb); + +if (mode ==PM_HIBERNATION_PREPARE ) { +if (vgdev->has_virgl_3d) { +DRM_ERROR ("S4 not allowed when VIRGL is enabled\n" ); +return notifier_from_errno (-EPERM ); + } + + vgdev->hibernation = true; + } + +return NOTIFY_DONE ; +} +int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev) {struct virtio_gpu_device *vgdev;
@@ -270,6 +291,12 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)wait_event_timeout (vgdev->resp_wq, !vgdev->display_info_pending,5 *HZ ); } + + vgdev->pm_nb.notifier_call = virtio_gpu_pm_notifier; + ret =register_pm_notifier (&vgdev->pm_nb); +if (ret) + goto err_scanouts; +return 0 ; err_scanouts:
@@ -302,6 +329,7 @@ void virtio_gpu_deinit(struct drm_device *dev)virtio_reset_device (vgdev->vdev); vgdev->vdev->config->del_vqs (vgdev->vdev);mutex_destroy (&vgdev->obj_restore_lock); +unregister_pm_notifier (&vgdev->pm_nb); }void virtio_gpu_release (struct drm_device *dev)
drivers/gpu/drm/virtio/virtgpu_object.c +17 -1
@@ -104,7 +104,7 @@ static void virtio_gpu_free_object(struct drm_gem_object *obj)if (bo->created) {virtio_gpu_remove_from_restore_list (bo); -virtio_gpu_cmd_unref_resource (vgdev, bo); +virtio_gpu_cmd_unref_resource (vgdev, bo, false);virtio_gpu_notify (vgdev);/* completion handler calls virtio_gpu_cleanup_object() */ return ;
@@ -341,3 +341,19 @@ int virtio_gpu_object_restore_all(struct virtio_gpu_device *vgdev)return ret; } + +void virtio_gpu_object_unref_all (struct virtio_gpu_device *vgdev) +{ +struct virtio_gpu_object *bo, *tmp; + +mutex_lock (&vgdev->obj_restore_lock); +list_for_each_entry_safe (bo, tmp, &vgdev->obj_restore_list, + restore_node) +if (bo->created) { +virtio_gpu_cmd_unref_resource (vgdev, bo, true); +virtio_gpu_notify (vgdev); + } + +mutex_unlock (&vgdev->obj_restore_lock); +} +
drivers/gpu/drm/virtio/virtgpu_prime.c +1 -1
@@ -218,7 +218,7 @@ static void virtgpu_dma_buf_free_obj(struct drm_gem_object *obj)if (bo->created) {virtio_gpu_remove_from_restore_list (bo); -virtio_gpu_cmd_unref_resource (vgdev, bo); +virtio_gpu_cmd_unref_resource (vgdev, bo, false);virtio_gpu_notify (vgdev);return ; }
drivers/gpu/drm/virtio/virtgpu_vq.c +10 -3
@@ -626,14 +626,21 @@ static void virtio_gpu_cmd_unref_cb(struct virtio_gpu_device *vgdev, }void virtio_gpu_cmd_unref_resource (struct virtio_gpu_device *vgdev, -struct virtio_gpu_object *bo) +struct virtio_gpu_object *bo, +int no_cb) {struct virtio_gpu_resource_unref *cmd_p;struct virtio_gpu_vbuffer *vbuf;int ret; - cmd_p =virtio_gpu_alloc_cmd_cb (vgdev, &vbuf,sizeof (*cmd_p), - virtio_gpu_cmd_unref_cb); +if (no_cb) { + cmd_p =virtio_gpu_alloc_cmd_cb (vgdev, &vbuf,sizeof (*cmd_p), +NULL ); + }else { + cmd_p =virtio_gpu_alloc_cmd_cb (vgdev, &vbuf,sizeof (*cmd_p), + virtio_gpu_cmd_unref_cb); + } +memset (cmd_p,0 ,sizeof (*cmd_p)); cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_UNREF );
drivers/gpu/drm/virtio/virtgpu_vram.c +1 -1
@@ -19,7 +19,7 @@ static void virtio_gpu_vram_free(struct drm_gem_object *obj)virtio_gpu_cmd_unmap (vgdev, bo);virtio_gpu_remove_from_restore_list (bo); -virtio_gpu_cmd_unref_resource (vgdev, bo); +virtio_gpu_cmd_unref_resource (vgdev, bo, false);virtio_gpu_notify (vgdev);return ; }
Signature · 1 line
2.34.1
Dmitry Osipenko
maintainerHi,
29.04.2026 23:47, [email protected] пишет:
Quoted 88 lines
From: Dongwon Kim <[email protected]>
This patch series implements the .freeze and .restore hooks for the virtio-gpu driver, alongside a PM notifier to manage object restoration during S4 (hibernation) cycles.
The series is structured as follows:
1. The first patch introduces
virtgpu_freezeandvirtgpu_restore. These functions handle the teardown of virtio queues before suspension and their subsequent recreation during the resume process.2. The second patch implements a tracking and restoration mechanism for
virtio_gpu_objectinstances. This is essential because the host (e.g., QEMU) destroys all associated graphics resources during the virtio-gpu reset triggered by the hibernation/resume cycle.3. The third patch adds a PM notifier to handle the resubmission of these tracked virtio-gpu objects to the host once the guest restores its virtqueues.
Together, these changes ensure that the virtio-gpu driver can survive hibernation without resource loss or invalid resource errors.
v2: - Added 10ms sleep in
virtgpu_freezeto prevent lockups during resumption.v3: - Replaced the fixed 10ms delay with wait calls that trigger once the
virtio queue is empty.
(Dmitry Osipenko)v4: - Limited the scope to S4 (hibernation). S3 resource loss can be avoided
via QEMU configurations (e.g., x-pcie-pm-no-soft-reset=true).v5: - Implemented removal of objects from the restore list before freeing to
prevent a use-after-free situation.
(Nirmoy Das)- Protected restore list operations with a spinlock. (Nirmoy Das)
- Moved the restore list node into the
virtio_gpu_bostruct to optimize memory usage.
(Dmitry Osipenko)- Removed unused header:
drm_atomic_helper.h.
(Dmitry Osipenko)v6: - Added support for objects backed by imported dma-bufs.
(Dmitry Osipenko)- Excluded virgl 3D objects from the restore list as they are currently non-recoverable.
(Dmitry Osipenko)- Renamed the list node to '
restore_node'.
(Nirmoy Das)- Switched from spinlock to mutex for restore list updates. (Nirmoy Das)
- Initialized
restore_nodeat creation to allow safe 'list_empty' checks.- Introduced a 'hibernation' flag in the PM notifier to trigger restoration at the correct stage in
virtgpu_restore.v7: - Added helper:
virtio_gpu_add_object_to_restore_list.
(Dmitry Osipenko)- Implemented unreferencing of all host-side objects before hibernation to ensure a clean state upon resume.
(Dmitry Osipenko)v8: - Introduced
virtio_gpu_remove_from_restore_listhelper and ensured it is
called across all object destruction paths (General, Prime, and VRAM) to prevent use-after-free.
(Dmitry Osipenko)- Relocated restore list removal logic from the final cleanup stage to the initial free stage for improved synchronization. (Dmitry Osipenko)
- Added missing
drm_print.hheader.Dongwon Kim (3):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Add support for saving and restoringvirtio_gpu_objectsdrm/virtio: Add PM notifier to restore objects after hibernation
Quoted 9 lines
drivers/gpu/drm/virtio/virtgpu_drv.c | 74 ++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 24 ++++++- drivers/gpu/drm/virtio/virtgpu_kms.c | 54 +++++++++++++-- drivers/gpu/drm/virtio/virtgpu_object.c | 88 ++++++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_prime.c | 45 ++++++++++++- drivers/gpu/drm/virtio/virtgpu_vq.c | 13 +++- drivers/gpu/drm/virtio/virtgpu_vram.c | 5 +- 7 files changed, 289 insertions(+), 14 deletions(-)
v8 works well, left couple minor comments
Signature · 2 lines
Best regards, Dmitry
Hi Dmitry,
Quoted 103 lines
Subject: Re: [PATCH v8 0/3] Virtio-GPU S4 (hibernation) support
Hi,
29.04.2026 23:47, [email protected] пишет:
Quoted 11 lines
From: Dongwon Kim <[email protected]>
This patch series implements the .freeze and .restore hooks for the virtio-gpu driver, alongside a PM notifier to manage object restoration during S4
(hibernation) cycles.The series is structured as follows:
1. The first patch introduces
virtgpu_freezeandvirtgpu_restore. These functions handle the teardown of virtio queues before suspension andtheir
Quoted 4 lines
subsequent recreation during the resume process.
2. The second patch implements a tracking and restoration mechanism for
virtio_gpu_objectinstances. This is essential because the host (e.g.,QEMU)
Quoted 11 lines
destroys all associated graphics resources during the virtio-gpu reset triggered by the hibernation/resume cycle.
3. The third patch adds a PM notifier to handle the resubmission of these tracked virtio-gpu objects to the host once the guest restores its virtqueues.
Together, these changes ensure that the virtio-gpu driver can survive hibernation without resource loss or invalid resource errors.
v2: - Added 10ms sleep in
virtgpu_freezeto prevent lockups duringresumption.
Quoted 40 lines
v3: - Replaced the fixed 10ms delay with wait calls that trigger once the
virtio queue is empty.
(Dmitry Osipenko)v4: - Limited the scope to S4 (hibernation). S3 resource loss can be avoided
via QEMU configurations (e.g., x-pcie-pm-no-soft-reset=true).v5: - Implemented removal of objects from the restore list before freeing to
prevent a use-after-free situation.
(Nirmoy Das)- Protected restore list operations with a spinlock. (Nirmoy Das)
- Moved the restore list node into the
virtio_gpu_bostruct to optimize memory usage.
(Dmitry Osipenko)- Removed unused header:
drm_atomic_helper.h.
(Dmitry Osipenko)v6: - Added support for objects backed by imported dma-bufs.
(Dmitry Osipenko)- Excluded virgl 3D objects from the restore list as they are currently non-recoverable.
(Dmitry Osipenko)- Renamed the list node to '
restore_node'.
(Nirmoy Das)- Switched from spinlock to mutex for restore list updates. (Nirmoy Das)
- Initialized
restore_nodeat creation to allow safe 'list_empty' checks.- Introduced a 'hibernation' flag in the PM notifier to trigger restoration at the correct stage in
virtgpu_restore.Quoted 8 lines
v7: - Added helper:
virtio_gpu_add_object_to_restore_list.
(Dmitry Osipenko)- Implemented unreferencing of all host-side objects before hibernation to ensure a clean state upon resume.
(Dmitry Osipenko)v8: - Introduced
virtio_gpu_remove_from_restore_listhelper and ensuredit is
Quoted 15 lines
called across all object destruction paths (General, Prime, and VRAM) to prevent use-after-free.
(Dmitry Osipenko)- Relocated restore list removal logic from the final cleanup stage to the initial free stage for improved synchronization. (Dmitry Osipenko)
- Added missing
drm_print.hheader.Dongwon Kim (3):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Add support for saving and restoringvirtio_gpu_objectsdrm/virtio: Add PM notifier to restore objects after hibernationQuoted 5 lines
drivers/gpu/drm/virtio/virtgpu_drv.c | 74 ++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 24 ++++++- drivers/gpu/drm/virtio/virtgpu_kms.c | 54 +++++++++++++-- drivers/gpu/drm/virtio/virtgpu_object.c | 88++++++++++++++++++++++++-
drivers/gpu/drm/virtio/virtgpu_prime.c| 45
Quoted 7 lines
++++++++++++-
Quoted 4 lines
drivers/gpu/drm/virtio/virtgpu_vq.c | 13 +++- drivers/gpu/drm/virtio/virtgpu_vram.c | 5 +- 7 files changed, 289 insertions(+), 14 deletions(-)v8 works well, left couple minor comments
I will shortly come up with v9 with your suggestions applied.
Quoted 4 lines
--
Best regards,
Dmitry
Thanks,
DW