Skip to contentSkip to conversation

vhost: fix caching attributes of MMIO regions by setting them explicitly

PATCH In review Kommula Shiva Shankar 3 participants 5 messages Jan 2 – Jan 13

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vhost: fix caching attributes of MMIO regions by setting them explicitly

Explicitly set non-cached caching attributes for MMIO regions. Default write-back mode can cause CPU to cache device memory, causing invalid reads and unpredictable behavior.

Invalid read and write issues were observed on ARM64 when mapping the notification area to userspace via mmap.

---
Originally sent to net-next, now redirected to vhost tree per Jason Wang's suggestion.

 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +1 -0
index 05a481e4c385..b0179e8567ab 100644
@@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
 	if (vma->vm_end - vma->vm_start != notify.size)
 		return -ENOTSUPP;
 
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
 	vma->vm_ops = &vhost_vdpa_vm_ops;
 	return 0;
Signature · 1 line
2.48.1
#1 lore raw

Hi Michael,
Just a ping on this patch. Would appreciate your review when you get a chance.

Thanks

Quoted 39 lines · Kommula Shiva Shankar

Prioritize security for external emails:
Confirm sender and content safety before clicking links or opening attachments Report Suspicious
Explicitly set non-cached caching attributes for MMIO regions. Default write-back mode can cause CPU to cache device memory, causing invalid reads and unpredictable behavior.

Invalid read and write issues were observed on ARM64 when mapping the notification area to userspace via mmap.

---
Originally sent to net-next, now redirected to vhost tree per Jason Wang's suggestion.

drivers/vhost/vdpa.c | 1 +

1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +1 -0
index 05a481e4c385..b0179e8567ab 100644
@@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
 if (vma->vm_end - vma->vm_start != notify.size)
 return -ENOTSUPP;
 
+ vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
 vma->vm_ops = &vhost_vdpa_vm_ops;
 return 0;
Signature · 1 line
2.48.1
#2 lore raw
In reply to Kommula Shiva Shankar
Quoted 6 lines · Kommula Shiva Shankar

Explicitly set non-cached caching attributes for MMIO regions. Default write-back mode can cause CPU to cache device memory, causing invalid reads and unpredictable behavior.

Invalid read and write issues were observed on ARM64 when mapping the notification area to userspace via mmap.

device memory in question is the VQ kick, yes? So if it is cached, the kick can get delayed, but how is this causing "invalid read and write issues"? What is read/written exactly?

Quoted 3 lines

I also worry a bit about regressing on other hardware. Cc nvidia guys.

Quoted 21 lines

---
Originally sent to net-next, now redirected to vhost tree per Jason Wang's suggestion.

 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +1 -0
index 05a481e4c385..b0179e8567ab 100644
@@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
 	if (vma->vm_end - vma->vm_start != notify.size)
 		return -ENOTSUPP;
 
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
 	vma->vm_ops = &vhost_vdpa_vm_ops;
 	return 0;
Signature · 1 line
2.48.1
#3 lore raw

________________________________________
From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 13, 2026 13:00
To: Shiva Shankar Kommula
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jerin Jacob; Nithin Kumar Dabilpuram; Srujana Challa; [email protected]; [email protected]
Subject: [EXTERNAL] Re: [PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

On Fri, Jan 02, 2026 at 12: 27: 03PM +0530, Kommula Shiva Shankar wrote: > Explicitly set non-cached caching attributes for MMIO regions. > Default write-back mode can cause CPU to cache device memory, > causing invalid reads and unpredictable ZjQcmQRYFpfptBannerStart
Prioritize security for external emails:
Confirm sender and content safety before clicking links or opening attachments
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/CRVmXkqW!tc3Z1f8UYnX61G-8-Z36D0TwOyXGKE_ybqkwtAsNGONThDzce1-BzQFJIDlDn_iWLJpe2FHKyeofP3u77neSt706ScpI5Ec$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd

Quoted 11 lines · Kommula Shiva Shankar

Explicitly set non-cached caching attributes for MMIO regions. Default write-back mode can cause CPU to cache device memory, causing invalid reads and unpredictable behavior.

Invalid read and write issues were observed on ARM64 when mapping the notification area to userspace via mmap.

device memory in question is the VQ kick, yes? So if it is cached, the kick can get delayed, but how is this causing "invalid read and write issues"? What is read/written exactly?

This is the VQ notification address.
on ARM64, when memory mapped as write back, notification writes never reached the device. Reads on the device side sees stale values.

Quoted 29 lines

I also worry a bit about regressing on other hardware. Cc nvidia guys.

---
Originally sent to net-next, now redirected to vhost tree per Jason Wang's suggestion.

 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vdpa.c +1 -1
index 05a481e4c385..b0179e8567ab 100644
@@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
      if (vma->vm_end - vma->vm_start != notify.size)
              return -ENOTSUPP;
 
+     vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
      vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
      vma->vm_ops = &vhost_vdpa_vm_ops;
      return 0;
--

2.48.1

Reviewed-by
#4 lore raw
In reply to Michael S. Tsirkin
Quoted 8 lines · Michael S. Tsirkin
Quoted 2 lines

I also worry a bit about regressing on other hardware. Cc nvidia guys.

drivers/vhost/vdpa.c static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma) · line 1531

Quoted 2 lines
drivers/vhost/vdpa.c +1 -0
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);

This is definitely required and correct if notify.addr comes from a PCI BAR address.

You need to trace the origin of that memory in all the drivers to determine if it is OK or not.

For instance mlx5 is:

	kick_addr = mdev->bar_addr + offset;
	res->phys_kick_addr = kick_addr;
[..]
	addr = (phys_addr_t)ndev->mvdev.res.phys_kick_addr;

"bar_addr" is PCI memory so this patch is correct and required for mlx5.

ifcvf:

                        hw->notify_base_pa = pci_resource_start(pdev, cap.bar) +
                                        le32_to_cpu(cap.offset);
[..]
                hw->vring[i].notify_pa = hw->notify_base_pa +
                        notify_off * hw->notify_off_multiplier;
[..]
	area.addr = vf->vring[idx].notify_pa;

octep:

                        oct_hw->notify_base_pa = pci_resource_start(pdev, cap.bar) +
                                                 le32_to_cpu(cap.offset);
[..]
                oct_hw->vqs[i].notify_pa = oct_hw->notify_base_pa +
                        notify_off * oct_hw->notify_off_multiplier;
[..]
	area.addr = oct_hw->vqs[idx].notify_pa;

pds:
No idea, it is messed up though:

	area.addr = pdsv->vqs[qid].notify_pa;
    struct pds_vdpa_vq_info {
	dma_addr_t notify_pa;

Can't cast dma_addr_t to phys_addr_t!

virtio_pci:
Also messed up:

  	notify.addr = vp_vdpa->vring[qid].notify_pa;
    struct vp_vring {
	resource_size_t notify_pa;

phys_addr is not a resource_size_t

Guessing pds and virtio_pci are also both fine, even if I gave up trying to figure out where notify_pa gets set from in the end.

So the patch is OK

Jason