virtio_console: fix suspend/resume and hot-unplug races
Subsystems (3)
v3 cover virtio_console: fix suspend/resume and hot-unplug races
Hi all,
This patchset addresses several critical race conditions and memory lifecycle bugs in the virtio_console driver, primarily surrounding the TX path(__send_to_port, put_chars), device hot-unplug, and PM freeze/restore.
The initial motivation for this series was to fix a race where hvc_console writes (which can remain active during suspend if no_console_suspend is enabled) could enqueue buffers while virtcons_freeze was tearing down virtqueues, leading to a BUG_ON in virtqueue_detach_unused_buf_split().
During the investigation and testing of the freeze race, several related vulnerabilities were uncovered in how TX paths handled buffer ownership and stale portdev pointers during concurrent hot-unplugs. Because these fixes structurally overlap in __send_to_port() and put_chars(), they are submitted together as a single patchset to avoid merge conflicts.
Patch Summary
=============
1. virtio_console: refactor __send_to_port() buffer ownership
Refactors buffer ownership in __send_to_port(). Previously, put_chars() would allocate a raw buffer, pass it down, and kfree() it immediately upon return. If virtqueue_get_buf() returned an older completed buffer from a previous non-blocking write, the newly queued buffer could be freed while the host was still DMA-ing from it. By transferring ownership of a struct port_buffer to __send_to_port(), we guarantee only the exact buffer returned by the host is freed.
2. virtio_console: fix hot-unplug races in TX paths
Hardens the TX paths against concurrent hot-unplugs. It adds READ_ONCE(port->portdev) checks, bails out cleanly (returning count to prevent hvc infinite retries), and synchronizes portdev = NULL in unplug_port() using the outvq_lock.
3. virtio_console: fix control queue race during restore
Fixes a race during virtcons_restore() where filling the control receive queue (c_ivq) immediately after DRIVER_OK could trigger the control workqueue before the driver finished restoring port states, leading to list corruption.
4. virtio_console: fix race between hvc put_chars and virtqueue teardown
Addresses the original PM freeze race by introducing a pm_freezing state. TX paths now safely drop output while freezing.
A synchronization loop in virtcons_freeze() ensures all active TX threads have drained before virtio_reset_device() is called.
Testing
=======
Runtime-tested on arm64 systems with no_console_suspend enabled during S4 cycles.
Changes
=======
v3:
Split the original monolithic patch into a 4-part patchset for better logical grouping and bisectability.
Sungho Bae (4):
virtio_console: refactor __send_to_port() buffer ownership
virtio_console: fix hot-unplug races in TX paths
virtio_console: fix control queue race during restore
virtio_console: fix race between hvc put_chars and virtqueue teardown
on freeze
drivers/char/virtio_console.c | 177 ++++++++++++++++++++++++++-------- 1 file changed, 137 insertions(+), 40 deletions(-)
Signature · 1 line
2.34.1
PATCH v3 1/4 virtio_console: refactor __send_to_port() buffer ownership
From: Sungho Bae <[email protected]>
Modify __send_to_port() to take ownership of a struct port_buffer * instead of a void * raw buffer.
Previously, put_chars() would pass a raw kmemdup'd buffer and free it immediately after __send_to_port() returned. This caused a potential Use-After-Free and data corruption if the virtqueue was shared with nonblocking writers, as virtqueue_get_buf() might return an older completed buffer, causing the newly added buffer to be kfree'd while the host is still DMAing from it.
By transferring ownership of the allocated port_buffer to __send_to_port(), we ensure that the exact buffer returned by the host is the one that gets freed, resolving the memory lifecycle mismatch.
--- drivers/char/virtio_console.c | 69 +++++++++++++++++++---------------- 1 file changed, 37 insertions(+), 32 deletions(-)
drivers/char/virtio_console.c +37 -32
@@ -402,7 +402,7 @@ static void reclaim_dma_bufs(void) }static struct port_buffer *alloc_buf (struct virtio_device *vdev,size_t buf_size, -int pages) +int pages,gfp_t gfp) {struct port_buffer *buf;
@@ -436,11 +436,10 @@ static struct port_buffer *alloc_buf(struct virtio_device *vdev, size_t buf_size/* Increase device refcnt to avoid freeing it */ get_device (buf->dev); - buf->buf =dma_alloc_coherent (buf->dev, buf_size, &buf->dma, -GFP_KERNEL ); + buf->buf =dma_alloc_coherent (buf->dev, buf_size, &buf->dma, gfp); }else { buf->dev =NULL ; - buf->buf =kmalloc (buf_size,GFP_KERNEL ); + buf->buf =kmalloc (buf_size, gfp); }if (!buf->buf)
@@ -595,7 +594,7 @@ static void reclaim_consumed_buffers(struct port *port)static ssize_t __send_to_port (struct port *port,struct scatterlist *sg,int nents,size_t in_count, -void *data,bool nonblock) +struct port_buffer *buf,bool nonblock) {struct virtqueue *out_vq;int err;
@@ -608,14 +607,14 @@ static ssize_t __send_to_port(struct port *port, struct scatterlist *sg,reclaim_consumed_buffers (port); - err =virtqueue_add_outbuf (out_vq, sg, nents, data,GFP_ATOMIC ); + err =virtqueue_add_outbuf (out_vq, sg, nents, buf,GFP_ATOMIC );/* Tell Host to go! */ virtqueue_kick (out_vq);if (err) { in_count =0 ; - goto done; + goto free_and_done; }if (out_vq->num_free ==0 )
@@ -632,10 +631,19 @@ static ssize_t __send_to_port(struct port *port, struct scatterlist *sg,* buffer and relax the spinning requirement. The downside is * we need to kmalloc a GFP_ATOMIC buffer each time the * console driver writes something out. +* +* Spin until host returns the buffer. +* Capture the returned buf so we can free it. +* If broken, buf == NULL and buf stays in the vq; +* remove_vqs() will call virtqueue_detach_unused_buf() -> free_buf(). */ -while (!virtqueue_get_buf (out_vq, &len) +while (!(buf =virtqueue_get_buf (out_vq, &len)) && !virtqueue_is_broken (out_vq))cpu_relax (); + +free_and_done: +if (buf) +free_buf (buf, false); done:spin_unlock_irqrestore (&port->outvq_lock, flags);
@@ -816,14 +824,14 @@ static ssize_t port_fops_write(struct file *filp, const char __user *ubuf, count =min ((size_t )(32 *1024 ), count); - buf =alloc_buf (port->portdev->vdev, count,0 ); + buf =alloc_buf (port->portdev->vdev, count,0 ,GFP_KERNEL );if (!buf)return -ENOMEM ; ret =copy_from_user (buf->buf, ubuf, count);if (ret) { - ret = -EFAULT ; - goto free_buf; +free_buf (buf, true); +return -EFAULT ; }/*
@@ -835,15 +843,7 @@ static ssize_t port_fops_write(struct file *filp, const char __user *ubuf,*/ nonblock = true;sg_init_one (sg, buf->buf, count); - ret =__send_to_port (port, sg,1 , count, buf, nonblock); - -if (nonblock && ret >0 ) - goto out; - -free_buf: -free_buf (buf, true); -out: -return ret; +return __send_to_port (port, sg,1 , count, buf, nonblock); }struct sg_list {
@@ -932,7 +932,7 @@ static ssize_t port_fops_splice_write(struct pipe_inode_info *pipe, goto error_out; occupancy =pipe_buf_usage (pipe); - buf =alloc_buf (port->portdev->vdev,0 , occupancy); + buf =alloc_buf (port->portdev->vdev,0 , occupancy,GFP_KERNEL );if (!buf) { ret = -ENOMEM ;
@@ -946,11 +946,12 @@ static ssize_t port_fops_splice_write(struct pipe_inode_info *pipe,sg_init_table (sgl.sg, sgl.size); ret =__splice_from_pipe (pipe, &sd, pipe_to_sg);pipe_unlock (pipe); +if (likely (ret >0 )) ret =__send_to_port (port, buf->sg, sgl.n, sgl.len, buf, true); - -if (unlikely (ret <=0 )) +else free_buf (buf, true); +return ret; error_out:
@@ -1108,21 +1109,25 @@ static ssize_t put_chars(u32 vtermno, const u8 *buf, size_t count) {struct port *port;struct scatterlist sg[1 ]; -void *data; -int ret; +struct port_buffer *pbuf; port =find_port_by_vtermno (vtermno);if (!port)return -EPIPE ; - data =kmemdup (buf, count,GFP_ATOMIC ); -if (!data) + pbuf =alloc_buf (port->portdev->vdev, count,0 ,GFP_ATOMIC ); +if (!pbuf)return -ENOMEM ; -sg_init_one (sg, data, count); - ret =__send_to_port (port, sg,1 , count, data, false); -kfree (data); -return ret; +memcpy (pbuf->buf, buf, count); + pbuf->len = count; +sg_init_one (sg, pbuf->buf, count); + +/* +* Ownership of pbuf is transferred to __send_to_port(). +* Do not touch or free pbuf after this call. +*/ +return __send_to_port (port, sg,1 , count, pbuf, false); }/*
@@ -1295,7 +1300,7 @@ static int fill_queue(struct virtqueue *vq, spinlock_t *lock) nr_added_bufs =0 ; do { - buf = alloc_buf(vq->vdev, PAGE_SIZE, 0); + buf = alloc_buf(vq->vdev, PAGE_SIZE, 0, GFP_KERNEL); if (!buf) return -ENOMEM;
Signature · 1 line
2.34.1
PATCH v3 2/4 virtio_console: fix hot-unplug races in TX paths
In reply to Sungho BaeFrom: Sungho Bae <[email protected]>
When a port is hot-unplugged, unplug_port() nullifies port->portdev. However, concurrent TX paths (__send_to_port, put_chars) could read a stale pointer or encounter a NULL pointer dereference.
Add READ_ONCE(port->portdev) and NULL checks in the TX paths. In __send_to_port(), move the out_vq assignment inside the outvq_lock and check portdev under the lock. Correspondingly, update unplug_port() to NULL out port->portdev while holding the outvq_lock to serialize with __send_to_port().
In put_chars(), return count instead of 0 on unplug to prevent the hvc layer from spinning in an infinite retry loop.
--- drivers/char/virtio_console.c | 37 +++++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 6 deletions(-)
drivers/char/virtio_console.c +31 -6
@@ -600,11 +600,19 @@ static ssize_t __send_to_port(struct port *port, struct scatterlist *sg,int err;unsigned long flags;unsigned int len; - - out_vq = port->out_vq; +struct ports_device *portdev;spin_lock_irqsave (&port->outvq_lock, flags); + portdev =READ_ONCE (port->portdev); + +if (!portdev) { + in_count =0 ; + goto free_and_done; + } + + out_vq = port->out_vq; +reclaim_consumed_buffers (port); err =virtqueue_add_outbuf (out_vq, sg, nents, buf,GFP_ATOMIC );
@@ -1110,12 +1118,24 @@ static ssize_t put_chars(u32 vtermno, const u8 *buf, size_t count)struct port *port;struct scatterlist sg[1 ];struct port_buffer *pbuf; +struct ports_device *portdev; port =find_port_by_vtermno (vtermno);if (!port)return -EPIPE ; - pbuf =alloc_buf (port->portdev->vdev, count,0 ,GFP_ATOMIC ); +/* +* Silently drop output if device hot-unplug is in progress. +* portdev was NULLed by unplug_port() after hvc_remove() was +* already called, so the hvc layer will stop invoking put_chars() +* very soon. Returning count avoids a pointless retry loop in the +* interim. +*/ + portdev =READ_ONCE (port->portdev); +if (!portdev) +return count; + + pbuf =alloc_buf (portdev->vdev, count,0 ,GFP_ATOMIC );if (!pbuf)return -ENOMEM ;
@@ -1503,11 +1523,16 @@ static void unplug_port(struct port *port)remove_port_data (port);/* -* We should just assume the device itself has gone off -- -* else a close on an open port later will try to send out a -* control message. +* Null out portdev under outvq_lock so that __send_to_port() +* cannot race: it checks port->portdev inside the same lock +* and bails out if NULL, preventing any buffer from being +* enqueued to an already torn-down virtqueue. Also prevents +* a close on an open port later from sending a stale control +* message. */ +spin_lock_irq (&port->outvq_lock); port->portdev =NULL ; +spin_unlock_irq (&port->outvq_lock);sysfs_remove_group (&port->dev->kobj, &port_attribute_group);device_destroy (&port_class, port->dev->devt);
Signature · 1 line
2.34.1
PATCH v3 3/4 virtio_console: fix control queue race during restore
In reply to Sungho BaeFrom: Sungho Bae <[email protected]>
In virtcons_restore(), after virtio_device_ready() sets DRIVER_OK, the device becomes active. If the control receive queue (c_ivq) is populated immediately, the host can instantly deliver pending control messages (e.g., VIRTIO_CONSOLE_PORT_REMOVE).
This triggers the control_work_handler(), which can modify the portdev->ports list concurrently with the unprotected list_for_each_entry loop in virtcons_restore(), leading to list corruption or Use-After-Free.
Fix this by deferring the population of the control receive queue (fill_queue for c_ivq) until after the list iteration is complete. This ensures the host cannot inject control messages during the vulnerable window.
--- drivers/char/virtio_console.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
drivers/char/virtio_console.c +12 -3
@@ -2164,9 +2164,6 @@ static int virtcons_restore(struct virtio_device *vdev)virtio_device_ready (portdev->vdev); -if (use_multiport (portdev)) -fill_queue (portdev->c_ivq, &portdev->c_ivq_lock); -list_for_each_entry (port, &portdev->ports, list) { port->in_vq = portdev->in_vqs[port->id]; port->out_vq = portdev->out_vqs[port->id];
@@ -2183,6 +2180,18 @@ static int virtcons_restore(struct virtio_device *vdev)if (port->guest_connected) send_control_msg(port, VIRTIO_CONSOLE_PORT_OPEN, 1); } + +/* +* Populate the control receive queue only after the list iteration +* is complete. If we fill this queue before iterating, the host could +* immediately deliver a VIRTIO_CONSOLE_PORT_REMOVE message. +* This would trigger the control workqueue, which modifies the +* portdev->ports list concurrently with the unprotected loop above, +* leading to a Use-After-Free and list corruption. +*/ +if (use_multiport (portdev)) +fill_queue (portdev->c_ivq, &portdev->c_ivq_lock); +return 0 ; }#endif
Signature · 1 line
2.34.1
PATCH v3 4/4 virtio_console: fix race between hvc put_chars and virtqueue teardown on freeze
In reply to Sungho BaeFrom: Sungho Bae <[email protected]>
With no_console_suspend enabled, hvc console output can continue while virtio_console is freezing. In that window, put_chars can still enqueue buffers to the output virtqueue while virtcons_freeze is tearing queues down, triggering a BUG_ON in virtqueue_detach_unused_buf_split:
BUG_ON(vq->vq.num_free != vq->split.vring.num)
Add a pm_freezing flag to ports_device. Set it via smp_store_release() at the start of virtcons_freeze(); put_chars() and __send_to_port() drop output while the flag is set, checked via smp_load_acquire().
The check in __send_to_port() is placed under outvq_lock, making it atomic with remove_port_data() which also acquires outvq_lock. Once remove_port_data() returns for a given port, no concurrent __send_to_port() can add buffers before remove_vqs() tears down the vq.
After setting pm_freezing, acquire and release outvq_lock for each port (protected by ports_lock to prevent list manipulation races) before calling virtio_reset_device(). A TX thread that already passed the pm_freezing check may still hold outvq_lock while spinning for host acknowledgment; the drain loop ensures all such threads have completed before the device is reset.
Clear pm_freezing in virtcons_restore() only after all port->out_vq pointers have been reassigned to the newly allocated virtqueues, preventing TX paths from dereferencing freed vqs during restore.
--- drivers/char/virtio_console.c | 70 ++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 6 deletions(-)
drivers/char/virtio_console.c +64 -6
@@ -157,6 +157,12 @@ struct ports_device {/* Major number for this device. Ports will be created as minors. */ int chr_major; + +/* +* Set to true during PM freeze to block TX paths that may race +* with virtqueue teardown (e.g. hvc put_chars with no_console_suspend). +*/ +bool pm_freezing; };struct port_stats {
@@ -611,6 +617,17 @@ static ssize_t __send_to_port(struct port *port, struct scatterlist *sg, goto free_and_done; } +/* +* Check freeze flag under the lock so that the flag check and +* virtqueue_add_outbuf() are atomic with respect to +* remove_port_data() which also takes outvq_lock. This +* guarantees that once remove_port_data() returns, no new +* buffers can be added before remove_vqs() tears down the vq. +* Pairs with smp_store_release() in virtcons_freeze/restore. +*/ +if (smp_load_acquire (&portdev->pm_freezing))/* pairs with freeze/restore */ + goto free_and_done; + out_vq = port->out_vq;reclaim_consumed_buffers (port);
@@ -1125,14 +1142,27 @@ static ssize_t put_chars(u32 vtermno, const u8 *buf, size_t count)return -EPIPE ;/* -* Silently drop output if device hot-unplug is in progress. -* portdev was NULLed by unplug_port() after hvc_remove() was -* already called, so the hvc layer will stop invoking put_chars() -* very soon. Returning count avoids a pointless retry loop in the -* interim. +* Silently drop output in two cases, both by returning count so +* that the hvc layer does not spin-retry: +* +* 1. Device hot-unplug (!portdev): portdev was NULLed by +* unplug_port() after hvc_remove() was already called, so +* the hvc layer will stop invoking put_chars() very soon. +* Returning count avoids a pointless retry loop in the +* interim. +* +* 2. PM freeze (pm_freezing): the hvc console stays active +* under no_console_suspend but virtqueues are being torn +* down. Drop the output silently so the hvc layer does not +* stall suspend. +* +* This early check avoids a pointless GFP_ATOMIC allocation; +* __send_to_port() rechecks under outvq_lock for correctness. +* Pairs with smp_store_release() in virtcons_freeze/restore. */ portdev =READ_ONCE (port->portdev); -if (!portdev) +if (!portdev || +smp_load_acquire (&portdev->pm_freezing))/* pairs with freeze/restore */ return count; pbuf =alloc_buf (portdev->vdev, count,0 ,GFP_ATOMIC );
@@ -2002,6 +2032,7 @@ static int virtcons_probe(struct virtio_device *vdev)/* Attach this portdev to this virtio_device, and vice-versa. */ portdev->vdev = vdev; vdev->priv = portdev; + portdev->pm_freezing = false; portdev->chr_major =register_chrdev (0 ,"virtio-portsdev" , &portdev_fops);
@@ -2119,9 +2150,30 @@ static int virtcons_freeze(struct virtio_device *vdev) {struct ports_device *portdev;struct port *port; +unsigned long flags; portdev = vdev->priv; +/* +* Block TX paths (put_chars, __send_to_port) before resetting the +* device and tearing down virtqueues. This prevents races with +* hvc console writes that remain active under no_console_suspend. +*/ +smp_store_release (&portdev->pm_freezing, true); + +/* +* Synchronize with any concurrent __send_to_port() that may have +* passed the pm_freezing check. By acquiring and releasing the +* outvq_lock for each port, we ensure all active TX paths have +* completed before we reset the device. +*/ +spin_lock_irqsave (&portdev->ports_lock, flags); +list_for_each_entry (port, &portdev->ports, list) { +spin_lock (&port->outvq_lock); +spin_unlock (&port->outvq_lock); + } +spin_unlock_irqrestore (&portdev->ports_lock, flags); +virtio_reset_device (vdev);if (use_multiport (portdev))
@@ -2192,6 +2244,12 @@ static int virtcons_restore(struct virtio_device *vdev)if (use_multiport (portdev))fill_queue (portdev->c_ivq, &portdev->c_ivq_lock); +/* +* Allow TX paths only after all port->out_vq pointers have +* been reassigned to the newly allocated virtqueues. +*/ +smp_store_release (&portdev->pm_freezing, false); + return 0; }#endif
Signature · 1 line
2.34.1