Virtio-GPU suspend and resume
Subsystems (3)
cover Virtio-GPU suspend and resume
From: Dongwon Kim <[email protected]>
This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:
First patch adds virtgpu_freeze and virtgpu_restore functions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.
Second patch implements a mechanism for restoring virtio_gpu_object instances. This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration process.
These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.
Dongwon Kim (2):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Implement save and restore for virtio_gpu_objects
drivers/gpu/drm/virtio/virtgpu_drv.c | 59 +++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 11 +++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 24 ++++++--- drivers/gpu/drm/virtio/virtgpu_object.c | 65 +++++++++++++++++++++++++ 4 files changed, 152 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1
RFC PATCH 1/2 drm/virtio: Freeze and restore hooks to support suspend and resume
From: Dongwon Kim <[email protected]>
virtio device needs to delete before VM suspend happens then reinitialize all virtqueues again upon resume
--- drivers/gpu/drm/virtio/virtgpu_drv.c | 53 +++++++++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 1 + drivers/gpu/drm/virtio/virtgpu_kms.c | 23 ++++++++---- 3 files changed, 70 insertions(+), 7 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +52 -1
@@ -155,6 +155,53 @@ static unsigned int features[] = {VIRTIO_GPU_F_RESOURCE_BLOB , VIRTIO_GPU_F_CONTEXT_INIT, }; + +#ifdef CONFIG_PM_SLEEP +static int virtgpu_freeze (struct virtio_device *vdev) +{ +struct drm_device *dev = vdev->priv; +struct virtio_gpu_device *vgdev = dev->dev_private; +int error; + + error =drm_mode_config_helper_suspend (dev); +if (error) { +DRM_ERROR ("suspend error %d\n" , error); +return error; + } + +flush_work (&vgdev->obj_free_work); +flush_work (&vgdev->ctrlq.dequeue_work); +flush_work (&vgdev->cursorq.dequeue_work); +flush_work (&vgdev->config_changed_work); + vdev->config->del_vqs (vdev); + +return 0 ; +} + +static int virtgpu_restore (struct virtio_device *vdev) +{ +struct drm_device *dev = vdev->priv; +struct virtio_gpu_device *vgdev = dev->dev_private; +int error; + + error =virtio_gpu_find_vqs (vgdev); +if (error) { +DRM_ERROR ("failed to find virt queues\n" ); +return error; + } + +virtio_device_ready (vdev); + + error =drm_mode_config_helper_resume (dev); +if (error) { +DRM_ERROR ("resume error %d\n" , error); +return error; + } + +return 0 ; +} +#endif +static struct virtio_driver virtio_gpu_driver = { .feature_table = features, .feature_table_size = ARRAY_SIZE(features),
@@ -162,7 +209,11 @@ static struct virtio_driver virtio_gpu_driver = { .id_table = id_table, .probe = virtio_gpu_probe, .remove = virtio_gpu_remove, - .config_changed = virtio_gpu_config_changed + .config_changed = virtio_gpu_config_changed, +#ifdef CONFIG_PM_SLEEP + .freeze = virtgpu_freeze, + .restore = virtgpu_restore, +#endif };static int __init virtio_gpu_driver_init (void )
drivers/gpu/drm/virtio/virtgpu_drv.h +1 -0
@@ -300,6 +300,7 @@ void virtio_gpu_deinit(struct drm_device *dev);void virtio_gpu_release (struct drm_device *dev);int virtio_gpu_driver_open (struct drm_device *dev,struct drm_file *file);void virtio_gpu_driver_postclose (struct drm_device *dev,struct drm_file *file); +int virtio_gpu_find_vqs (struct virtio_gpu_device *vgdev);/* virtgpu_gem.c */ int virtio_gpu_gem_object_open(struct drm_gem_object *obj,
drivers/gpu/drm/virtio/virtgpu_kms.c +17 -6
@@ -114,15 +114,28 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev, vgdev->num_capsets = num_capsets; } -int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev) +int virtio_gpu_find_vqs (struct virtio_gpu_device *vgdev) {struct virtqueue_info vqs_info[] = { {"control" , virtio_gpu_ctrl_ack }, {"cursor" , virtio_gpu_cursor_ack }, }; -struct virtio_gpu_device *vgdev; -/* this will expand later */ struct virtqueue *vqs[2 ]; +int ret; + + ret =virtio_find_vqs (vgdev->vdev,2 , vqs, vqs_info,NULL ); +if (ret) +return ret; + + vgdev->ctrlq.vq = vqs[0 ]; + vgdev->cursorq.vq = vqs[1 ]; + +return 0 ; +} + +int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev) +{ +struct virtio_gpu_device *vgdev;u32 num_scanouts, num_capsets;int ret =0 ;
@@ -206,13 +219,11 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)DRM_INFO ("features: %ccontext_init\n" , vgdev->has_context_init ?'+' :'-' ); - ret =virtio_find_vqs (vgdev->vdev,2 , vqs, vqs_info,NULL ); + ret =virtio_gpu_find_vqs (vgdev);if (ret) {DRM_ERROR ("failed to find virt queues\n" ); goto err_vqs; } - vgdev->ctrlq.vq = vqs[0 ]; - vgdev->cursorq.vq = vqs[1 ]; ret =virtio_gpu_alloc_vbufs (vgdev);if (ret) {DRM_ERROR ("failed to alloc vbufs\n" );
Signature · 1 line
2.34.1
RFC PATCH 2/2 drm/virtio: Implement save and restore for virtio_gpu_objects
In reply to [email protected]From: Dongwon Kim <[email protected]>
Host KVM/QEMU loses all graphics resources submitted by the guest OS upon resumption from sleep or hibernation. This results in invalid resource errors when the guest OS attempts to interact with the host regarding those resources.
To address this issue, the virtio-gpu driver now resubmits all existing resources upon resumption. A linked list has been introduced to maintain references to all created virtio_gpu_object instances and their parameters.
Whenever a new object is created and sent to the host, it is added to this list. During the .resume function, all backed-up objects are re-sent to the host using the 'create resource' virtio GPU command, ensuring the resources are restored seamlessly.
--- drivers/gpu/drm/virtio/virtgpu_drv.c | 6 +++ drivers/gpu/drm/virtio/virtgpu_drv.h | 10 ++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 1 + drivers/gpu/drm/virtio/virtgpu_object.c | 65 +++++++++++++++++++++++++ 4 files changed, 82 insertions(+)
drivers/gpu/drm/virtio/virtgpu_drv.c +6 -0
@@ -192,6 +192,12 @@ static int virtgpu_restore(struct virtio_device *vdev)virtio_device_ready (vdev); + error =virtio_gpu_object_restore_all (vgdev); +if (error) { +DRM_ERROR ("Failed to recover objects\n" ); +return error; + } + error =drm_mode_config_helper_resume (dev);if (error) {DRM_ERROR ("resume error %d\n" , error);
drivers/gpu/drm/virtio/virtgpu_drv.h +10 -0
@@ -126,6 +126,12 @@ struct virtio_gpu_object_array { struct drm_gem_object *objs[] __counted_by(total); }; +struct virtio_gpu_object_restore { +struct virtio_gpu_object *bo; +struct virtio_gpu_object_params params; +struct list_head node; +}; +struct virtio_gpu_vbuffer ;struct virtio_gpu_device ;
@@ -265,6 +271,7 @@ struct virtio_gpu_device {struct work_struct obj_free_work;spinlock_t obj_free_lock;struct list_head obj_free_list; +struct list_head obj_restore;struct virtio_gpu_drv_capset *capsets;uint32_t num_capsets;
@@ -479,6 +486,9 @@ bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid); + +int virtio_gpu_object_restore_all (struct virtio_gpu_device *vgdev); +/* virtgpu_prime.c */ int virtio_gpu_resource_assign_uuid (struct virtio_gpu_device *vgdev,struct virtio_gpu_object *bo);
drivers/gpu/drm/virtio/virtgpu_kms.c +1 -0
@@ -162,6 +162,7 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) vgdev->fence_drv.context =dma_fence_context_alloc (1 );spin_lock_init (&vgdev->fence_drv.lock);INIT_LIST_HEAD (&vgdev->fence_drv.fences); +INIT_LIST_HEAD (&vgdev->obj_restore);INIT_LIST_HEAD (&vgdev->cap_cache);INIT_WORK (&vgdev->config_changed_work, virtio_gpu_config_changed_work_func);
drivers/gpu/drm/virtio/virtgpu_object.c +65 -0
88 lines not on this page · raw message
Signature · 1 line
2.34.1
Dmitry Osipenko
maintainerQuoted 20 lines · On 4/19/25 02:29, [email protected] wrote:
From: Dongwon Kim <[email protected]>
This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:
First patch adds
virtgpu_freezeandvirtgpu_restorefunctions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.Second patch implements a mechanism for restoring
virtio_gpu_objectinstances. This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration process.These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.
Dongwon Kim (2):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Implement save and restore forvirtio_gpu_objects
Quoted 5 lines
drivers/gpu/drm/virtio/virtgpu_drv.c | 59 +++++++++++++++++++++- drivers/gpu/drm/virtio/virtgpu_drv.h | 11 +++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 24 ++++++--- drivers/gpu/drm/virtio/virtgpu_object.c | 65 +++++++++++++++++++++++++ 4 files changed, 152 insertions(+), 7 deletions(-)
Hi,
Something is wrong with your git send-mail setup, it's not first time your patches have an empty list of mail recipients. Please fix. No need to resend patches, I'll review them on the ML.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerQuoted 9 lines · On 4/19/25 02:29, [email protected] wrote:
From: Dongwon Kim <[email protected]>
This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:
First patch adds
virtgpu_freezeandvirtgpu_restorefunctions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.
Quoted 6 lines
Second patch implements a mechanism for restoring
virtio_gpu_objectinstances. This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration process.These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.
Resetting GPU by QEMU on suspend sounds like a wrong behaviour. Are you talking about upstream QEMU?
Could you please point at the QEMU code where it handles virtio-gpu suspend/resume? Don't see where it happens.
I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:
virtio_gpu_transfer_to_host_2d: no backing storage 2
[ 22.909454] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response 0x1200 (command 0x105) virtio_gpu_transfer_to_host_2d: no backing storage 2 [ 23.169138] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response
0x1200 (command 0x105)
Signature · 2 lines
Best regards, Dmitry
Hi Dmitry,
Quoted 18 lines
Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume
Quoted 10 lines · On 4/19/25 02:29, [email protected] wrote:
From: Dongwon Kim <[email protected]>
This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:
First patch adds
virtgpu_freezeandvirtgpu_restorefunctions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.Second patch implements a mechanism for restoring
virtio_gpu_objectinstances.
Quoted 2 lines
This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration
process.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.
Resetting GPU by QEMU on suspend sounds like a wrong behaviour. Are you talking about upstream QEMU?
Could you please point at the QEMU code where it handles virtio-gpu suspend/resume? Don't see where it happens.
[Kim, Dongwon] So I am using Ubuntu guest and whenever I suspend then resume, I see static void virtio_gpu_reset_bh(void *opaque) is being called. It is probably triggered by virtio layer but I didn't look into it. Check the video please:
(https://drive.google.com/file/d/1D_JF9CzgCjzQZ3RcLAaNnUORqGj24_I9/view?usp=sharing)
Quoted 10 lines
I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:
virtio_gpu_transfer_to_host_2d: no backing storage 2[ 22.909454] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response 0x1200 (command 0x105) virtio_gpu_transfer_to_host_2d: no backing storage 2 [ 23.169138] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response
0x1200(command0x105)
[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic enough...
Quoted 6 lines
Cc +Alex Bennée +Akihiko Odaki--
Best regards,
Dmitry
[Kim, Dongwon] Thanks!
Dmitry Osipenko
maintainerQuoted 9 lines · Dongwon
Quoted 9 lines
I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:
virtio_gpu_transfer_to_host_2d: no backing storage 2[ 22.909454] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response 0x1200 (command 0x105) virtio_gpu_transfer_to_host_2d: no backing storage 2 [ 23.169138] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response
0x1200(command0x105)
Quoted 3 lines
[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic enough...
I used virtio-gpu when got these errors.
Signature · 2 lines
Best regards, Dmitry
Hi Dmitry,
Quoted 12 lines
Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume
Quoted 9 lines · Dongwon
Quoted 9 lines
I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:
virtio_gpu_transfer_to_host_2d: no backing storage 2[ 22.909454] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response 0x1200 (command 0x105) virtio_gpu_transfer_to_host_2d: no backing storage 2 [ 23.169138] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response
0x1200(command0x105)
Quoted 6 lines
Quoted 3 lines
[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic
enough...
I used virtio-gpu when got these errors.
[Kim, Dongwon] Hmm I thought I would be able to replicate your problem if I remove blob guest framebuffer but I can't. In the code changes, we are attaching the backing storage for the resource during recovery process.
+ } else {
+ virtio_gpu_cmd_create_resource(vgdev, curr->bo, &curr->params,
+ NULL, NULL);
+ virtio_gpu_object_attach(vgdev, curr->bo, ents, nents);
+ }
I do not understand why it says it can't find it. Did you get any error while doing Virtio_gpu_object_attach?
What linux distribution do you use for the guest?
Are you using virgl driver or sw rasterizer?
Quoted 4 lines
--
Best regards,
Dmitry
[Kim, Dongwon] Thanks!
Hi Dmitry,
Can you share some details about your test setup?
How do I replicate the problem you saw? And is "suspend and resume" working by default in your case?
Thanks,
DW
Quoted 52 lines
-----Original Message-----
From: Kim, Dongwon
Sent: Tuesday, May 6, 2025 9:47 AM
To: Dmitry Osipenko <[email protected]>; dri- [email protected]; Alex Bennée <[email protected]>; Akihiko Odaki <[email protected]>
Subject: RE: [RFC PATCH 0/2] Virtio-GPU suspend and resumeHi Dmitry,
Quoted 12 lines
Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume
Quoted 9 lines · Dongwon
Quoted 9 lines
I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:
virtio_gpu_transfer_to_host_2d: no backing storage 2[ 22.909454] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response 0x1200 (command 0x105) virtio_gpu_transfer_to_host_2d: no backing storage 2 [ 23.169138] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response
0x1200(command0x105)Quoted 7 lines
Quoted 4 lines
[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic
enough...
I used virtio-gpu when got these errors.
[Kim, Dongwon] Hmm I thought I would be able to replicate your problem if I remove blob guest framebuffer but I can't. In the code changes, we are attaching the backing storage for the resource during recovery process.
(quoted diff) +2
-0+ } else { + virtio_gpu_cmd_create_resource(vgdev, curr->bo,
&curr->params,(quoted diff) +3
-0+ NULL, NULL); + virtio_gpu_object_attach(vgdev, curr->bo, ents, nents); + }I do not understand why it says it can't find it. Did you get any error while doing
Virtio_gpu_object_attach?What linux distribution do you use for the guest?
Are you using virgl driver or sw rasterizer?Quoted 4 lines
--
Best regards,
Dmitry[Kim, Dongwon] Thanks!
Dmitry Osipenko
maintainerQuoted 4 lines · Dongwon
Hi Dmitry,
Can you share some details about your test setup?
How do I replicate the problem you saw? And is "suspend and resume" working by default in your case?
My testing setup:
- QEMU staging tree
- Guest kernel 6.14.6 + your patches
- Using "-serial mon:stdio -device virtio-vga -display gtk" in QEMU cmdline
I'm booting guest into VT and running "rtcwake -mmem -s5" from serial console. On resume from suspend I see those driver error messages. Without your patches there are no errors.
After resume from suspend display not working in both cases with/without your patches.
Signature · 2 lines
Best regards, Dmitry
Hi Dmitry,
It turned out that attaching backing during restoring process was skipped due to obj->attach == true condition, which is the reason for the err messages you reported. I had tested my patch on slightly older virtio-gpu driver that didn't have this flag (was added by one of your commits that added detaching backing).
Then I added a line to set this false before attaching backing again for restoring and I see The display is now up when resuming but the whole kernel seems to hang afterwards.
I am debugging it now.
Quoted 25 lines
Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume
Quoted 4 lines · Dongwon
Hi Dmitry,
Can you share some details about your test setup?
How do I replicate the problem you saw? And is "suspend and resume"working by default in your case?
My testing setup:
- QEMU staging tree
- Guest kernel 6.14.6 + your patches
- Using "-serial mon:stdio -device virtio-vga -display gtk" in QEMU cmdlineI'm booting guest into VT and running "rtcwake -mmem -s5" from serial console. On resume from suspend I see those driver error messages. Without your patches there are no errors.
After resume from suspend display not working in both cases with/without your patches.
--
Best regards,
Dmitry