Skip to contentSkip to conversation

Virtio-GPU suspend and resume

RFC PATCH Superseded [email protected] 2 participants 11 messages Apr 18, 2025 – May 14, 2025

Subsystems (3)

v1 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

cover Virtio-GPU suspend and resume

From: Dongwon Kim <[email protected]>

This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:

First patch adds virtgpu_freeze and virtgpu_restore functions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.

Second patch implements a mechanism for restoring virtio_gpu_object instances. This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration process.

These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.

Dongwon Kim (2):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Implement save and restore for virtio_gpu_objects

 drivers/gpu/drm/virtio/virtgpu_drv.c    | 59 +++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_drv.h    | 11 +++++
 drivers/gpu/drm/virtio/virtgpu_kms.c    | 24 ++++++---
 drivers/gpu/drm/virtio/virtgpu_object.c | 65 +++++++++++++++++++++++++
 4 files changed, 152 insertions(+), 7 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH 1/2 drm/virtio: Freeze and restore hooks to support suspend and resume

From: Dongwon Kim <[email protected]>

virtio device needs to delete before VM suspend happens then reinitialize all virtqueues again upon resume

Cc Dmitry Osipenko <[email protected]>
Cc Vivek Kasireddy <[email protected]>
---
 drivers/gpu/drm/virtio/virtgpu_drv.c | 53 +++++++++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_drv.h |  1 +
 drivers/gpu/drm/virtio/virtgpu_kms.c | 23 ++++++++----
 3 files changed, 70 insertions(+), 7 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +52 -1
index 2d88e390feb4..1c0e47447cc0 100644
@@ -155,6 +155,53 @@ static unsigned int features[] = {
 	VIRTIO_GPU_F_RESOURCE_BLOB,
 	VIRTIO_GPU_F_CONTEXT_INIT,
 };
+
+#ifdef CONFIG_PM_SLEEP
+static int virtgpu_freeze(struct virtio_device *vdev)
+{
+	struct drm_device *dev = vdev->priv;
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	int error;
+
+	error = drm_mode_config_helper_suspend(dev);
+	if (error) {
+		DRM_ERROR("suspend error %d\n", error);
+		return error;
+	}
+
+	flush_work(&vgdev->obj_free_work);
+	flush_work(&vgdev->ctrlq.dequeue_work);
+	flush_work(&vgdev->cursorq.dequeue_work);
+	flush_work(&vgdev->config_changed_work);
+	vdev->config->del_vqs(vdev);
+
+	return 0;
+}
+
+static int virtgpu_restore(struct virtio_device *vdev)
+{
+	struct drm_device *dev = vdev->priv;
+	struct virtio_gpu_device *vgdev = dev->dev_private;
+	int error;
+
+	error = virtio_gpu_find_vqs(vgdev);
+	if (error) {
+		DRM_ERROR("failed to find virt queues\n");
+		return error;
+	}
+
+	virtio_device_ready(vdev);
+
+	error = drm_mode_config_helper_resume(dev);
+	if (error) {
+		DRM_ERROR("resume error %d\n", error);
+		return error;
+	}
+
+	return 0;
+}
+#endif
+
 static struct virtio_driver virtio_gpu_driver = {
 	.feature_table = features,
 	.feature_table_size = ARRAY_SIZE(features),
@@ -162,7 +209,11 @@ static struct virtio_driver virtio_gpu_driver = {
 	.id_table = id_table,
 	.probe = virtio_gpu_probe,
 	.remove = virtio_gpu_remove,
-	.config_changed = virtio_gpu_config_changed
+	.config_changed = virtio_gpu_config_changed,
+#ifdef CONFIG_PM_SLEEP
+	.freeze = virtgpu_freeze,
+	.restore = virtgpu_restore,
+#endif
 };
 
 static int __init virtio_gpu_driver_init(void)
drivers/gpu/drm/virtio/virtgpu_drv.h +1 -0
index f17660a71a3e..1279f998c8e0 100644
@@ -300,6 +300,7 @@ void virtio_gpu_deinit(struct drm_device *dev);
 void virtio_gpu_release(struct drm_device *dev);
 int virtio_gpu_driver_open(struct drm_device *dev, struct drm_file *file);
 void virtio_gpu_driver_postclose(struct drm_device *dev, struct drm_file *file);
+int virtio_gpu_find_vqs(struct virtio_gpu_device *vgdev);
 
 /* virtgpu_gem.c */
 int virtio_gpu_gem_object_open(struct drm_gem_object *obj,
drivers/gpu/drm/virtio/virtgpu_kms.c +17 -6
index 7dfb2006c561..6c1af77ea083 100644
@@ -114,15 +114,28 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev,
 	vgdev->num_capsets = num_capsets;
 }
 
-int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
+int virtio_gpu_find_vqs(struct virtio_gpu_device *vgdev)
 {
 	struct virtqueue_info vqs_info[] = {
 		{ "control", virtio_gpu_ctrl_ack },
 		{ "cursor", virtio_gpu_cursor_ack },
 	};
-	struct virtio_gpu_device *vgdev;
-	/* this will expand later */
 	struct virtqueue *vqs[2];
+	int ret;
+
+	ret = virtio_find_vqs(vgdev->vdev, 2, vqs, vqs_info, NULL);
+	if (ret)
+		return ret;
+
+	vgdev->ctrlq.vq = vqs[0];
+	vgdev->cursorq.vq = vqs[1];
+
+	return 0;
+}
+
+int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
+{
+	struct virtio_gpu_device *vgdev;
 	u32 num_scanouts, num_capsets;
 	int ret = 0;
 
@@ -206,13 +219,11 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 	DRM_INFO("features: %ccontext_init\n",
 		 vgdev->has_context_init ? '+' : '-');
 
-	ret = virtio_find_vqs(vgdev->vdev, 2, vqs, vqs_info, NULL);
+	ret = virtio_gpu_find_vqs(vgdev);
 	if (ret) {
 		DRM_ERROR("failed to find virt queues\n");
 		goto err_vqs;
 	}
-	vgdev->ctrlq.vq = vqs[0];
-	vgdev->cursorq.vq = vqs[1];
 	ret = virtio_gpu_alloc_vbufs(vgdev);
 	if (ret) {
 		DRM_ERROR("failed to alloc vbufs\n");
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH 2/2 drm/virtio: Implement save and restore for virtio_gpu_objects

In reply to [email protected]

From: Dongwon Kim <[email protected]>

Host KVM/QEMU loses all graphics resources submitted by the guest OS upon resumption from sleep or hibernation. This results in invalid resource errors when the guest OS attempts to interact with the host regarding those resources.

To address this issue, the virtio-gpu driver now resubmits all existing resources upon resumption. A linked list has been introduced to maintain references to all created virtio_gpu_object instances and their parameters.

Whenever a new object is created and sent to the host, it is added to this list. During the .resume function, all backed-up objects are re-sent to the host using the 'create resource' virtio GPU command, ensuring the resources are restored seamlessly.

Cc Dmitry Osipenko <[email protected]>
Cc Vivek Kasireddy <[email protected]>
---
 drivers/gpu/drm/virtio/virtgpu_drv.c    |  6 +++
 drivers/gpu/drm/virtio/virtgpu_drv.h    | 10 ++++
 drivers/gpu/drm/virtio/virtgpu_kms.c    |  1 +
 drivers/gpu/drm/virtio/virtgpu_object.c | 65 +++++++++++++++++++++++++
 4 files changed, 82 insertions(+)
drivers/gpu/drm/virtio/virtgpu_drv.c +6 -0
index 1c0e47447cc0..8267b6d61b22 100644
@@ -192,6 +192,12 @@ static int virtgpu_restore(struct virtio_device *vdev)
 
 	virtio_device_ready(vdev);
 
+	error = virtio_gpu_object_restore_all(vgdev);
+	if (error) {
+		DRM_ERROR("Failed to recover objects\n");
+		return error;
+	}
+
 	error = drm_mode_config_helper_resume(dev);
 	if (error) {
 		DRM_ERROR("resume error %d\n", error);
drivers/gpu/drm/virtio/virtgpu_drv.h +10 -0
index 1279f998c8e0..55f836378237 100644
@@ -126,6 +126,12 @@ struct virtio_gpu_object_array {
 	struct drm_gem_object *objs[] __counted_by(total);
 };
 
+struct virtio_gpu_object_restore {
+	struct virtio_gpu_object *bo;
+	struct virtio_gpu_object_params params;
+	struct list_head node;
+};
+
 struct virtio_gpu_vbuffer;
 struct virtio_gpu_device;
 
@@ -265,6 +271,7 @@ struct virtio_gpu_device {
 	struct work_struct obj_free_work;
 	spinlock_t obj_free_lock;
 	struct list_head obj_free_list;
+	struct list_head obj_restore;
 
 	struct virtio_gpu_drv_capset *capsets;
 	uint32_t num_capsets;
@@ -479,6 +486,9 @@ bool virtio_gpu_is_shmem(struct virtio_gpu_object *bo);
 
 int virtio_gpu_resource_id_get(struct virtio_gpu_device *vgdev,
 			       uint32_t *resid);
+
+int virtio_gpu_object_restore_all(struct virtio_gpu_device *vgdev);
+
 /* virtgpu_prime.c */
 int virtio_gpu_resource_assign_uuid(struct virtio_gpu_device *vgdev,
 				    struct virtio_gpu_object *bo);
drivers/gpu/drm/virtio/virtgpu_kms.c +1 -0
index 6c1af77ea083..17d182737910 100644
@@ -162,6 +162,7 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 	vgdev->fence_drv.context = dma_fence_context_alloc(1);
 	spin_lock_init(&vgdev->fence_drv.lock);
 	INIT_LIST_HEAD(&vgdev->fence_drv.fences);
+	INIT_LIST_HEAD(&vgdev->obj_restore);
 	INIT_LIST_HEAD(&vgdev->cap_cache);
 	INIT_WORK(&vgdev->config_changed_work,
 		  virtio_gpu_config_changed_work_func);
drivers/gpu/drm/virtio/virtgpu_object.c +65 -0

88 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#3 lore raw
In reply to [email protected]
Quoted 20 lines · On 4/19/25 02:29, [email protected] wrote:

From: Dongwon Kim <[email protected]>

This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:

First patch adds virtgpu_freeze and virtgpu_restore functions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.

Second patch implements a mechanism for restoring virtio_gpu_object instances. This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration process.

These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.

Dongwon Kim (2):
drm/virtio: Freeze and restore hooks to support suspend and resume drm/virtio: Implement save and restore for virtio_gpu_objects

Quoted 5 lines
 drivers/gpu/drm/virtio/virtgpu_drv.c    | 59 +++++++++++++++++++++-
 drivers/gpu/drm/virtio/virtgpu_drv.h    | 11 +++++
 drivers/gpu/drm/virtio/virtgpu_kms.c    | 24 ++++++---
 drivers/gpu/drm/virtio/virtgpu_object.c | 65 +++++++++++++++++++++++++
 4 files changed, 152 insertions(+), 7 deletions(-)

Hi,

Something is wrong with your git send-mail setup, it's not first time your patches have an empty list of mail recipients. Please fix. No need to resend patches, I'll review them on the ML.

Signature · 2 lines
Best regards,
Dmitry
maintainer
#4 lore raw
In reply to [email protected]
Quoted 9 lines · On 4/19/25 02:29, [email protected] wrote:

From: Dongwon Kim <[email protected]>

This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:

First patch adds virtgpu_freeze and virtgpu_restore functions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.

Quoted 6 lines

Second patch implements a mechanism for restoring virtio_gpu_object instances. This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration process.

These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.

Resetting GPU by QEMU on suspend sounds like a wrong behaviour. Are you talking about upstream QEMU?

Could you please point at the QEMU code where it handles virtio-gpu suspend/resume? Don't see where it happens.

I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:

virtio_gpu_transfer_to_host_2d: no backing storage 2

[   22.909454] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response
0x1200 (command 0x105)
virtio_gpu_transfer_to_host_2d: no backing storage 2
[   23.169138] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response

0x1200 (command 0x105)

Cc +Alex Bennée +Akihiko Odaki
Signature · 2 lines
Best regards,
Dmitry
#5 lore raw

Hi Dmitry,

Quoted 18 lines

Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume

Quoted 10 lines · On 4/19/25 02:29, [email protected] wrote:

From: Dongwon Kim <[email protected]>

This patch series introduces a freeze and restore mechanism for the virtio-gpu driver:

First patch adds virtgpu_freeze and virtgpu_restore functions. These functions handle the deletion of virtio queues before suspension and their recreation during the restoration process.

Second patch implements a mechanism for restoring virtio_gpu_object

instances.

Quoted 2 lines

This is necessary because the host (QEMU) deletes all associated resources during the virtio-gpu reset, which occurs as part of the restoration

process.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

These changes ensure that the virtio-gpu driver can properly handle suspend and resume scenarios without resource loss.

Resetting GPU by QEMU on suspend sounds like a wrong behaviour. Are you talking about upstream QEMU?

Could you please point at the QEMU code where it handles virtio-gpu suspend/resume? Don't see where it happens.

[Kim, Dongwon] So I am using Ubuntu guest and whenever I suspend then resume, I see static void virtio_gpu_reset_bh(void *opaque) is being called. It is probably triggered by virtio layer but I didn't look into it. Check the video please:
(https://drive.google.com/file/d/1D_JF9CzgCjzQZ3RcLAaNnUORqGj24_I9/view?usp=sharing)

Quoted 10 lines

I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:

virtio_gpu_transfer_to_host_2d: no backing storage 2

[   22.909454] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response
0x1200 (command 0x105)
virtio_gpu_transfer_to_host_2d: no backing storage 2
[   23.169138] [drm:virtio_gpu_dequeue_ctrl_func] *ERROR* response

0x1200 (command 0x105)

[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic enough...

Quoted 6 lines
Cc +Alex Bennée +Akihiko Odaki

--
Best regards,
Dmitry

[Kim, Dongwon] Thanks!

maintainer
#6 lore raw
Quoted 9 lines · Dongwon
Quoted 9 lines

I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:

virtio_gpu_transfer_to_host_2d: no backing storage 2

[   22.909454] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response
0x1200 (command 0x105)
virtio_gpu_transfer_to_host_2d: no backing storage 2
[   23.169138] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response

0x1200 (command 0x105)

Quoted 3 lines

[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic enough...

I used virtio-gpu when got these errors.

Signature · 2 lines
Best regards,
Dmitry
#7 lore raw

Hi Dmitry,

Quoted 12 lines

Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume

Quoted 9 lines · Dongwon
Quoted 9 lines

I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:

virtio_gpu_transfer_to_host_2d: no backing storage 2

[   22.909454] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response
0x1200 (command 0x105)
virtio_gpu_transfer_to_host_2d: no backing storage 2
[   23.169138] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response

0x1200 (command 0x105)

Quoted 6 lines
Quoted 3 lines

[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic

enough...

I used virtio-gpu when got these errors.

[Kim, Dongwon] Hmm I thought I would be able to replicate your problem if I remove blob guest framebuffer but I can't. In the code changes, we are attaching the backing storage for the resource during recovery process.

+ } else {
+ virtio_gpu_cmd_create_resource(vgdev, curr->bo, &curr->params,
+ NULL, NULL);
+ virtio_gpu_object_attach(vgdev, curr->bo, ents, nents);
+ }

I do not understand why it says it can't find it. Did you get any error while doing Virtio_gpu_object_attach?

What linux distribution do you use for the guest?
Are you using virgl driver or sw rasterizer?

Quoted 4 lines

--
Best regards,
Dmitry

[Kim, Dongwon] Thanks!

#8 lore raw
In reply to Dmitry Osipenko

Hi Dmitry,

Can you share some details about your test setup?
How do I replicate the problem you saw? And is "suspend and resume" working by default in your case?

Thanks,
DW

Quoted 52 lines

-----Original Message-----
From: Kim, Dongwon
Sent: Tuesday, May 6, 2025 9:47 AM
To: Dmitry Osipenko <[email protected]>; dri- [email protected]; Alex Bennée <[email protected]>; Akihiko Odaki <[email protected]>
Subject: RE: [RFC PATCH 0/2] Virtio-GPU suspend and resume

Hi Dmitry,

Quoted 12 lines

Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume

Quoted 9 lines · Dongwon
Quoted 9 lines

I tried to apply your kernel patches and then suspend/resume guest kernel, it doesn't work:

virtio_gpu_transfer_to_host_2d: no backing storage 2

[   22.909454] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response
0x1200 (command 0x105)
virtio_gpu_transfer_to_host_2d: no backing storage 2
[   23.169138] [drm:virtio_gpu_dequeue_ctrl_func] ERROR response

0x1200 (command 0x105)

Quoted 7 lines
Quoted 4 lines

[Kim, Dongwon] I will check your failing case more closely. As you know, we use virtio-gpu device only for display device, which means I tested this in that specific environment although I thought it is generic

enough...

I used virtio-gpu when got these errors.

[Kim, Dongwon] Hmm I thought I would be able to replicate your problem if I remove blob guest framebuffer but I can't. In the code changes, we are attaching the backing storage for the resource during recovery process.

(quoted diff) +2 -0
+		} else {
+			virtio_gpu_cmd_create_resource(vgdev, curr->bo,

&curr->params,

(quoted diff) +3 -0
+						       NULL, NULL);
+			virtio_gpu_object_attach(vgdev, curr->bo, ents, nents);
+		}

I do not understand why it says it can't find it. Did you get any error while doing Virtio_gpu_object_attach?

What linux distribution do you use for the guest?
Are you using virgl driver or sw rasterizer?

Quoted 4 lines

--
Best regards,
Dmitry

[Kim, Dongwon] Thanks!

maintainer
#9 lore raw
Quoted 4 lines · Dongwon

Hi Dmitry,

Can you share some details about your test setup?
How do I replicate the problem you saw? And is "suspend and resume" working by default in your case?

My testing setup:

- QEMU staging tree
- Guest kernel 6.14.6 + your patches
- Using "-serial mon:stdio -device virtio-vga -display gtk" in QEMU cmdline

I'm booting guest into VT and running "rtcwake -mmem -s5" from serial console. On resume from suspend I see those driver error messages. Without your patches there are no errors.

After resume from suspend display not working in both cases with/without your patches.

Signature · 2 lines
Best regards,
Dmitry
#10 lore raw

Hi Dmitry,

It turned out that attaching backing during restoring process was skipped due to obj->attach == true condition, which is the reason for the err messages you reported. I had tested my patch on slightly older virtio-gpu driver that didn't have this flag (was added by one of your commits that added detaching backing).

Then I added a line to set this false before attaching backing again for restoring and I see The display is now up when resuming but the whole kernel seems to hang afterwards.

I am debugging it now.

Quoted 25 lines

Subject: Re: [RFC PATCH 0/2] Virtio-GPU suspend and resume

Quoted 4 lines · Dongwon

Hi Dmitry,

Can you share some details about your test setup?
How do I replicate the problem you saw? And is "suspend and resume"

working by default in your case?

My testing setup:

- QEMU staging tree
- Guest kernel 6.14.6 + your patches
- Using "-serial mon:stdio -device virtio-vga -display gtk" in QEMU cmdline

I'm booting guest into VT and running "rtcwake -mmem -s5" from serial console. On resume from suspend I see those driver error messages. Without your patches there are no errors.

After resume from suspend display not working in both cases with/without your patches.

--
Best regards,
Dmitry