vhost: invalidate vring access on IOTLB transitions
Subsystems (6)
PATCH v8 1/1 vhost: invalidate vring access on IOTLB transitions
When VIRTIO_F_ACCESS_PLATFORM changes, the cached desc, avail, and used addresses change meaning with the address space. Invalidate the cached vring access state when switching between direct userspace addresses and device IOTLB addresses, and discard the device IOTLB when userspace clears ACCESS_PLATFORM.
Introduce a common device-IOTLB teardown helper and use it from vhost-net and vhost-vsock. The helper drops the device-wide view, switches each VQ under its mutex, clears the IOTLB message queues, and frees the old table after the VQ handoff.
Preserve the existing IOTLB replacement semantics when ACCESS_PLATFORM is set again. On the first direct-to-IOTLB transition, invalidate the cached vring addresses. When replacing an existing IOTLB, keep the GIOVA ring addresses and reset only the metadata cache, so stale metadata pointers cannot be used after the old table is freed.
A successful live transition leaves the backend attached. Userspace must configure the vring addresses for the new address mode after ACCESS_PLATFORM is cleared.
vhost_vq_invalidate_access() clears all three vring addresses together. Treat the VQ as invalidated only when all three are zero, since an individual vring address may legitimately be GIOVA 0 in IOTLB mode.
---
Changes since v7:
- Squash the three patches so the common helper is introduced together with the vhost-net and vhost-vsock callers.
- Preserve the existing device-IOTLB replacement semantics.
- Treat a VQ as invalidated only when all three vring addresses are zero, since an individual vring address may legitimately be GIOVA 0.
- Keep the original vhost-vsock feature-test formatting.
drivers/vhost/vhost.c | 57 ++++++++++++++++++++++++++++++++++++++++++- drivers/vhost/vhost.h | 1 + drivers/vhost/vsock.c | 2 ++ drivers/vhost/net.c | 2 ++ 4 files changed, 61 insertions(+), 1 deletion(-)
drivers/vhost/vhost.c +56 -1
@@ -344,6 +344,17 @@ static void __vhost_vq_meta_reset(struct vhost_virtqueue *vq) vq->meta_iotlb[j] =NULL ; } +/* Caller must hold the virtqueue mutex. */ +static void vhost_vq_invalidate_access (struct vhost_virtqueue *vq) +{ + vq->desc =NULL ; + vq->avail =NULL ; + vq->used =NULL ; + vq->log_used = false; + vq->log_addr =-1ull ; +__vhost_vq_meta_reset (vq); +} +static void vhost_vq_meta_reset (struct vhost_dev *d) {int i;
@@ -1918,6 +1929,13 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq) {unsigned int num = vq->num; +/* +* vhost_vq_invalidate_access() clears all three addresses together. +* A single zero address may be a valid GIOVA in IOTLB mode. +*/ +if (!vq->desc && !vq->avail && !vq->used) +return 0 ; +if (!vq->iotlb)return 1 ;
@@ -2287,6 +2305,40 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg }EXPORT_SYMBOL_GPL (vhost_vring_ioctl); +/* Caller must hold the device mutex. */ +void vhost_clear_device_iotlb (struct vhost_dev *d) +{ +struct vhost_iotlb *iotlb; +int i; + + iotlb = d->iotlb; +if (!iotlb) +return ; + +/* +* Drop the device-wide view first. Each VQ then drops its +* per-VQ view and its cached ring access under its own mutex. +* Keep the old table alive until every VQ has completed this +* handoff, since a worker may still be using it while waiting +* for its VQ mutex. +*/ + d->iotlb =NULL ; + +for (i =0 ; i < d->nvqs; ++i) { +struct vhost_virtqueue *vq = d->vqs[i]; + +mutex_lock (&vq->mutex); + vq->iotlb =NULL ; +vhost_vq_invalidate_access (vq); +mutex_unlock (&vq->mutex); + } + +vhost_clear_msg (d); +vhost_iotlb_free (iotlb); +wake_up_interruptible_poll (&d->wait,EPOLLIN |EPOLLRDNORM ); +} +EXPORT_SYMBOL_GPL (vhost_clear_device_iotlb); +int vhost_init_device_iotlb (struct vhost_dev *d) {struct vhost_iotlb *niotlb, *oiotlb;
@@ -2307,7 +2359,10 @@ int vhost_init_device_iotlb(struct vhost_dev *d)mutex_lock (&vq->mutex); vq->iotlb = niotlb; -__vhost_vq_meta_reset (vq); +if (oiotlb) +__vhost_vq_meta_reset (vq); +else +vhost_vq_invalidate_access (vq);mutex_unlock (&vq->mutex); }
drivers/vhost/vhost.h +1 -0
@@ -277,6 +277,7 @@ ssize_t vhost_chr_read_iter(struct vhost_dev *dev, struct iov_iter *to, int noblock);ssize_t vhost_chr_write_iter (struct vhost_dev *dev,struct iov_iter *from); +void vhost_clear_device_iotlb (struct vhost_dev *d);int vhost_init_device_iotlb (struct vhost_dev *d); void vhost_iotlb_map_free(struct vhost_iotlb *iotlb,
drivers/vhost/vsock.c +2 -0
@@ -868,6 +868,8 @@ static int vhost_vsock_set_features(struct vhost_vsock *vsock, u64 features)if ((features & (1ULL <<VIRTIO_F_ACCESS_PLATFORM ))) {if (vhost_init_device_iotlb (&vsock->dev)) goto err; + }else { +vhost_clear_device_iotlb (&vsock->dev); } vsock->seqpacket_allow = features & (1ULL <<VIRTIO_VSOCK_F_SEQPACKET );
drivers/vhost/net.c +2 -0
@@ -1696,6 +1696,8 @@ static int vhost_net_set_features(struct vhost_net *n, const u64 *features)if (virtio_features_test_bit (features,VIRTIO_F_ACCESS_PLATFORM )) {if (vhost_init_device_iotlb (&n->dev)) goto out_unlock; + }else { +vhost_clear_device_iotlb (&n->dev); }for (i =0 ; i <VHOST_NET_VQ_MAX ; ++i) {
base-commit: b282418bc366194677eafd1dad180d92254586ac
Signature · 1 line
2.34.1
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Concurrent execution of `vhost_clear_device_iotlb()` and a vhost worker thread calling…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Concurrent execution of vhost_clear_device_iotlb() and a vhost worker thread calling translate_desc() results in a NULL pointer dereference or arbitrary host memory access due to lockless clearing of dev->iotlb. --
--- Patch [1]: [PATCH v8] vhost: invalidate vring access on IOTLB transitions --- commit 601cfcb65f9457e49ce60311ed18a7ca3496a8ba
Author: Jia Jia <[email protected]>
vhost: invalidate vring access on IOTLB transitions
This commit introduces a helper to clear the device IOTLB when the VIRTIO_F_ACCESS_PLATFORM feature is cleared. It drops the device-wide view first and then drops the per-VQ views under their respective mutexes.
Quoted 4 lines
drivers/vhost/vhost.c +0
-0
[ ... ]
Quoted 9 lines
drivers/vhost/vhost.c +8
-0+/* Caller must hold the device mutex. */ +void vhost_clear_device_iotlb (struct vhost_dev *d) +{ +struct vhost_iotlb *iotlb; +int i; + + iotlb = d->iotlb; +if (!iotlb)
drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg · line 2325
Quoted 10 lines
drivers/vhost/vhost.c +10
-0+return ; + +/* +* Drop the device-wide view first. Each VQ then drops its +* per-VQ view and its cached ring access under its own mutex. +* Keep the old table alive until every VQ has completed this +* handoff, since a worker may still be using it while waiting +* for its VQ mutex. +*/ + d->iotlb =NULL ;
[Severity: Critical]
Could setting d->iotlb = NULL here race with a running vhost worker thread?
While this update is protected by dev->mutex, it does not hold vq->mutex. If a worker thread is concurrently calling translate_desc(), it evaluates the device IOTLB locklessly:
drivers/vhost/vhost.c:translate_desc() {
...
struct vhost_iotlb *umem = dev->iotlb ? dev->iotlb : dev->umem;
...
}Since dev->iotlb is cleared without vq->mutex, could the worker erroneously fall back to dev->umem?
If dev->umem is NULL, this looks like it could trigger a NULL pointer dereference when accessing umem->root. If dev->umem is populated, could it map GIOVAs using the GPA table and lead to arbitrary host memory access?
Quoted 15 lines
drivers/vhost/vhost.c +15
-0+ +for (i =0 ; i < d->nvqs; ++i) { +struct vhost_virtqueue *vq = d->vqs[i]; + +mutex_lock (&vq->mutex); + vq->iotlb =NULL ; +vhost_vq_invalidate_access (vq); +mutex_unlock (&vq->mutex); + } + +vhost_clear_msg (d); +vhost_iotlb_free (iotlb); +wake_up_interruptible_poll (&d->wait,EPOLLIN |EPOLLRDNORM ); +} +EXPORT_SYMBOL_GPL (vhost_clear_device_iotlb);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 47 lines
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Concurrent execution ofvhost_clear_device_iotlb()and a vhost worker thread callingtranslate_desc()results in a NULL pointer dereference or arbitrary host memory access due to lockless clearing ofdev->iotlb. ----- Patch [1]: [PATCH v8] vhost: invalidate vring access on IOTLB transitions --- commit
601cfcb65f9457e49ce60311ed18a7ca3496a8ba
Author: Jia Jia <[email protected]>vhost: invalidate vring access on IOTLB transitions
This commit introduces a helper to clear the device IOTLB when the
VIRTIO_F_ACCESS_PLATFORMfeature is cleared. It drops the device-wide view first and then drops the per-VQ views under their respective mutexes.Quoted 4 lines
drivers/vhost/vhost.c +0
-0[ ... ]
Quoted 9 lines
drivers/vhost/vhost.c +8
-0+/* Caller must hold the device mutex. */ +void vhost_clear_device_iotlb (struct vhost_dev *d) +{ +struct vhost_iotlb *iotlb; +int i; + + iotlb = d->iotlb; +if (!iotlb)drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg · line 2325
Quoted 10 lines
drivers/vhost/vhost.c +10
-0+return ; + +/* +* Drop the device-wide view first. Each VQ then drops its +* per-VQ view and its cached ring access under its own mutex. +* Keep the old table alive until every VQ has completed this +* handoff, since a worker may still be using it while waiting +* for its VQ mutex. +*/ + d->iotlb =NULL ;[Severity: Critical]
Could settingd->iotlb= NULL here race with a running vhost worker thread?While this update is protected by
dev->mutex, it does not holdvq->mutex. If a worker thread is concurrently callingtranslate_desc(), it evaluates the device IOTLB locklessly:
Quoted 8 lines
drivers/vhost/vhost.c:translate_desc() { ... struct vhost_iotlb *umem = dev->iotlb ? dev->iotlb : dev->umem; ... }Since
dev->iotlbis cleared withoutvq->mutex, could the worker erroneously fall back todev->umem?
I was able to reproduce the issue. Although I did not observe the more severe symptoms described, repeated resets of ACCESS_PLATFORM showed thattranslate_desc() can select dev->umem while the device IOTLB is active. This confirms that the race is real.
Quoted 5 lines
If
dev->umemis NULL, this looks like it could trigger a NULL pointer dereference when accessingumem->root. Ifdev->umemis populated, could it map GIOVAs using the GPA table and lead to arbitrary host memory access?
The reported consequences require several additional preconditions. I could not reproduce them; at this point, they remain theoretical.
Quoted 18 lines
Quoted 5 lines
drivers/vhost/vhost.c +5
-0+ +for (i =0 ; i < d->nvqs; ++i) { +struct vhost_virtqueue *vq = d->vqs[i]; + +mutex_lock (&vq->mutex);drivers/vhost/vhost.c long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg · line 2340
Quoted 10 lines
drivers/vhost/vhost.c +10
-0+ vq->iotlb =NULL ; +vhost_vq_invalidate_access (vq); +mutex_unlock (&vq->mutex); + } + +vhost_clear_msg (d); +vhost_iotlb_free (iotlb); +wake_up_interruptible_poll (&d->wait,EPOLLIN |EPOLLRDNORM ); +} +EXPORT_SYMBOL_GPL (vhost_clear_device_iotlb);--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1