Skip to contentSkip to conversation

PCI passthru on Hyper-V (Part I)

PATCH In review Mukesh R 7 participants 72 messages Jan 20 – Apr 18 Page 2 of 2

Subsystems (8)
lore
Share & tools
lore
#63 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

In reply to Mukesh R
Quoted 90 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
   drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
   1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
   	};
   } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
   struct mshv_root mshv_root;
   enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
   }
   #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 652

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 654

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Quoted 9 lines

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 706

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
   static struct mshv_mem_region *
   mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
   {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
   	return ret;
   }
+
   #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
   static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
   #endif /* CONFIG_X86_64 */
   static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
   {
   	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
   	case HVMSG_GPA_INTERCEPT:
   		return mshv_handle_gpa_intercept(vp);
   	}
Signature · 1 line
2.51.2.vfs.0.1
author
#64 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

Quoted 97 lines · Stanislav Kinsburskii
Quoted 96 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
    drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
    1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
    	};
    } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
    struct mshv_root mshv_root;
    enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
    }
    #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 652

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 654

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

Quoted 8 lines
Quoted 3 lines

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

There are no ram pages for mmio regions. Also, we don't do much with mmio regions other than tell the hyp about it.

Thanks,
-Mukesh

Quoted 142 lines

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 706

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
    static struct mshv_mem_region *
    mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
    {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
    	return ret;
    }
+
    #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
    static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
    #endif /* CONFIG_X86_64 */
    static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
    {
    	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
    	case HVMSG_GPA_INTERCEPT:
    		return mshv_handle_gpa_intercept(vp);
    	}
Signature · 1 line
2.51.2.vfs.0.1
#65 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

Quoted 102 lines · Mukesh R
Quoted 101 lines · Stanislav Kinsburskii
Quoted 99 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
    drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
    1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
    	};
    } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
    struct mshv_root mshv_root;
    enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
    }
    #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 651

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 653

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

Quoted 8 lines
Quoted 4 lines

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

There are no ram pages for mmio regions. Also, we don't do much with mmio regions other than tell the hyp about it.

So, are you saying that the hypervisor does not use these pages and only tracks them? That would make things easier.
However, if we later try to map a GPA that is already mapped, will the hypervisor return an error?

Thanks,
Stanislav

Quoted 146 lines

Thanks,
-Mukesh

Quoted 142 lines

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 705

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
    static struct mshv_mem_region *
    mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
    {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
    	return ret;
    }
+
    #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
    static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
    #endif /* CONFIG_X86_64 */
    static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
    {
    	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
    	case HVMSG_GPA_INTERCEPT:
    		return mshv_handle_gpa_intercept(vp);
    	}
Signature · 1 line
2.51.2.vfs.0.1
author
#66 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

Quoted 108 lines · Stanislav Kinsburskii
Quoted 107 lines · Mukesh R
Quoted 105 lines · Stanislav Kinsburskii
Quoted 90 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
     drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
     1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
     	};
     } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
     struct mshv_root mshv_root;
     enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
     }
     #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 651

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 653

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Quoted 9 lines

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

Quoted 8 lines
Quoted 3 lines

There are no ram pages for mmio regions. Also, we don't do much with mmio regions other than tell the hyp about it.

So, are you saying that the hypervisor does not use these pages and only tracks them? That would make things easier.
However, if we later try to map a GPA that is already mapped, will the hypervisor return an error?

Hypervisor does not return an error.

Quoted 149 lines

Thanks,
Stanislav

Quoted 146 lines

Thanks,
-Mukesh

Quoted 142 lines

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 705

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
     static struct mshv_mem_region *
     mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
     {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
     	return ret;
     }
+
     #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
     static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
     #endif /* CONFIG_X86_64 */
     static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
     {
     	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
     	case HVMSG_GPA_INTERCEPT:
     		return mshv_handle_gpa_intercept(vp);
     	}
Signature · 1 line
2.51.2.vfs.0.1
#67 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

Quoted 113 lines · Mukesh R
Quoted 112 lines · Stanislav Kinsburskii
Quoted 110 lines · Mukesh R
Quoted 97 lines · Stanislav Kinsburskii
Quoted 96 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
     drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
     1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
     	};
     } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
     struct mshv_root mshv_root;
     enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
     }
     #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 651

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 653

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

Quoted 8 lines
Quoted 3 lines

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

There are no ram pages for mmio regions. Also, we don't do much with mmio regions other than tell the hyp about it.

Quoted 7 lines
Quoted 4 lines

So, are you saying that the hypervisor does not use these pages and only tracks them? That would make things easier.
However, if we later try to map a GPA that is already mapped, will the hypervisor return an error?

Hypervisor does not return an error.

So, what happenes if we map a GPA that is already mapped? Does it just remap it to the new PFN?

Thanks,
Stanislav

Quoted 151 lines
Quoted 151 lines

Thanks,
Stanislav

Quoted 146 lines

Thanks,
-Mukesh

Quoted 142 lines

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 705

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
     static struct mshv_mem_region *
     mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
     {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
     	return ret;
     }
+
     #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
     static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
     #endif /* CONFIG_X86_64 */
     static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
     {
     	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
     	case HVMSG_GPA_INTERCEPT:
     		return mshv_handle_gpa_intercept(vp);
     	}
Signature · 1 line
2.51.2.vfs.0.1
author
#68 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

Quoted 119 lines · Stanislav Kinsburskii
Quoted 118 lines · Mukesh R
Quoted 116 lines · Stanislav Kinsburskii
Quoted 102 lines · Mukesh R
Quoted 101 lines · Stanislav Kinsburskii
Quoted 99 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
      drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
      1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
      	};
      } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
      struct mshv_root mshv_root;
      enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
      }
      #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 651

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 653

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

Quoted 8 lines
Quoted 4 lines

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

There are no ram pages for mmio regions. Also, we don't do much with mmio regions other than tell the hyp about it.

So, are you saying that the hypervisor does not use these pages and only tracks them? That would make things easier.
However, if we later try to map a GPA that is already mapped, will the hypervisor return an error?

Quoted 5 lines
Quoted 2 lines

Hypervisor does not return an error.

So, what happenes if we map a GPA that is already mapped? Does it just remap it to the new PFN?

yes, otherwise it would return error, right?

Quoted 154 lines

Thanks,
Stanislav

Quoted 151 lines
Quoted 151 lines

Thanks,
Stanislav

Quoted 146 lines

Thanks,
-Mukesh

Quoted 142 lines

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 705

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
      static struct mshv_mem_region *
      mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
      {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
      	return ret;
      }
+
      #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
      static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
      #endif /* CONFIG_X86_64 */
      static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
      {
      	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
      	case HVMSG_GPA_INTERCEPT:
      		return mshv_handle_gpa_intercept(vp);
      	}
Signature · 1 line
2.51.2.vfs.0.1
#69 lore raw

Re: [PATCH v0 15/15] mshv: Populate mmio mappings for PCI passthru

Quoted 120 lines · Mukesh R
Quoted 119 lines · Stanislav Kinsburskii
Quoted 118 lines · Mukesh R
Quoted 116 lines · Stanislav Kinsburskii
Quoted 102 lines · Mukesh R
Quoted 101 lines · Stanislav Kinsburskii
Quoted 99 lines · Mukesh R
Quoted 82 lines · Stanislav Kinsburskii
Quoted 81 lines · Mukesh R
Quoted 58 lines · Stanislav Kinsburskii
Quoted 57 lines · Mukesh R

From: Mukesh Rathor <[email protected]>

Upon guest access, in case of missing mmio mapping, the hypervisor generates an unmapped gpa intercept. In this path, lookup the PCI resource pfn for the guest gpa, and ask the hypervisor to map it via hypercall. The PCI resource pfn is maintained by the VFIO driver, and obtained via fixup_user_fault call (similar to KVM).

---
      drivers/hv/mshv_root_main.c | 115 ++++++++++++++++++++++++++++++++++++
      1 file changed, 115 insertions(+)
drivers/hv/mshv_root_main.c +32 -0
index 03f3aa9f5541..4c8bc7cd0888 100644
@@ -56,6 +56,14 @@ struct hv_stats_page {
      	};
      } __packed;
+bool hv_nofull_mmio;   /* don't map entire mmio region upon fault */
+static int __init setup_hv_full_mmio(char *str)
+{
+	hv_nofull_mmio = true;
+	return 0;
+}
+__setup("hv_nofull_mmio", setup_hv_full_mmio);
+
      struct mshv_root mshv_root;
      enum hv_scheduler_type hv_scheduler_type;
@@ -612,6 +620,109 @@ mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn)
      }
      #ifdef CONFIG_X86_64
+
+/*
+ * Check if uaddr is for mmio range. If yes, return 0 with mmio_pfn filled in
+ * else just return -errno.
+ */
+static int mshv_chk_get_mmio_start_pfn(struct mshv_partition *pt, u64 gfn,
+				       u64 *mmio_pfnp)
+{
+	struct vm_area_struct *vma;
+	bool is_mmio;
+	u64 uaddr;
+	struct mshv_mem_region *mreg;
+	struct follow_pfnmap_args pfnmap_args;
+	int rc = -EINVAL;
+
+	/*
+	 * Do not allow mem region to be deleted beneath us. VFIO uses
+	 * useraddr vma to lookup pci bar pfn.
+	 */
+	spin_lock(&pt->pt_mem_regions_lock);
+
+	/* Get the region again under the lock */
+	mreg = mshv_partition_region_by_gfn(pt, gfn);
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
Quoted 8 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 651

Quoted 6 lines
drivers/hv/mshv_root_main.c +6 -0
+		goto unlock_pt_out;
+
+	uaddr = mreg->start_uaddr +
+		((gfn - mreg->start_gfn) << HV_HYP_PAGE_SHIFT);
+
+	mmap_read_lock(current->mm);

Semaphore can't be taken under spinlock.

Yeah, something didn't feel right here and I meant to recheck, now regret rushing to submit the patch.

Rethinking, I think the pt_mem_regions_lock is not needed to protect the uaddr because unmap will properly serialize via the mm lock.

Quoted 4 lines

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 653

Quoted 2 lines
drivers/hv/mshv_root_main.c +2 -0
+	vma = vma_lookup(current->mm, uaddr);
+	is_mmio = vma ? !!(vma->vm_flags & (VM_IO | VM_PFNMAP)) : 0;

Why this check is needed again?

Quoted 6 lines
Quoted 2 lines

To make sure region did not change. This check is under lock.

How can this happen? One can't change VMA type without unmapping it first. And unmapping it leads to a kernel MMIO region state dangling around without corresponding user space mapping.

Right, and vm_flags would not be mmio expected then.

Quoted 3 lines

This is similar to dangling pinned regions and should likely be addressed the same way by utilizing MMU notifiers to destpoy memoty regions is VMA is detached.

I don't think we need that. Either it succeeds if the region did not change at all, or just fails.

Quoted 8 lines
Quoted 4 lines

I'm afraid we do, as if the driver mapped a page with the previous memory region, and then the region is unmapped, the page will stay mapped in the hypervisor, but will be considered free by kernel, which in turn will lead to GPF upn next allocation.

There are no ram pages for mmio regions. Also, we don't do much with mmio regions other than tell the hyp about it.

So, are you saying that the hypervisor does not use these pages and only tracks them? That would make things easier.
However, if we later try to map a GPA that is already mapped, will the hypervisor return an error?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Hypervisor does not return an error.

So, what happenes if we map a GPA that is already mapped? Does it just remap it to the new PFN?

yes, otherwise it would return error, right?

I see.
Please summarize and document this behaviour in the commit message.

Thanks,
Stanislav

Quoted 154 lines
Quoted 154 lines

Thanks,
Stanislav

Quoted 151 lines
Quoted 151 lines

Thanks,
Stanislav

Quoted 146 lines

Thanks,
-Mukesh

Quoted 142 lines

With pinned regions we issue is similar but less impacting: pages can't be released by user space unmapping and thus will be simply leaked, but the system stays intact.

MMIO regions are simila to movable region in this regard: they don't reference the user pages, and thus this guest region replaement is a stright wat to kernel panic.

Quoted 59 lines
Quoted 59 lines
Quoted 59 lines
Quoted 58 lines

The region type is stored on the region itself.
And the type is checked on the caller side.

Quoted 42 lines
drivers/hv/mshv_root_main.c +42 -0
+	if (!is_mmio)
+		goto unlock_mmap_out;
+
+	pfnmap_args.vma = vma;
+	pfnmap_args.address = uaddr;
+
+	rc = follow_pfnmap_start(&pfnmap_args);
+	if (rc) {
+		rc = fixup_user_fault(current->mm, uaddr, FAULT_FLAG_WRITE,
+				      NULL);
+		if (rc)
+			goto unlock_mmap_out;
+
+		rc = follow_pfnmap_start(&pfnmap_args);
+		if (rc)
+			goto unlock_mmap_out;
+	}
+
+	*mmio_pfnp = pfnmap_args.pfn;
+	follow_pfnmap_end(&pfnmap_args);
+d
+unlock_mmap_out:
+	mmap_read_unlock(current->mm);
+unlock_pt_out:
+	spin_unlock(&pt->pt_mem_regions_lock);
+	return rc;
+}
+
+/*
+ * At present, the only unmapped gpa is mmio space. Verify if it's mmio
+ * and resolve if possible.
+ * Returns: True if valid mmio intercept and it was handled, else false
+ */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp)
+{
+	struct hv_message *hvmsg = vp->vp_intercept_msg_page;
+	struct hv_x64_memory_intercept_message *msg;
+	union hv_x64_memory_access_info accinfo;
+	u64 gfn, mmio_spa, numpgs;
+	struct mshv_mem_region *mreg;
+	int rc;
+	struct mshv_partition *pt = vp->vp_partition;

drivers/hv/mshv_root_main.c mshv_partition_region_by_gfn(struct mshv_partition *partition, u64 gfn) · line 705

Quoted 10 lines
drivers/hv/mshv_root_main.c +10 -0
+
+	msg = (struct hv_x64_memory_intercept_message *)hvmsg->u.payload;
+	accinfo = msg->memory_access_info;
+
+	if (!accinfo.gva_gpa_valid)
+		return false;
+
+	/* Do a fast check and bail if non mmio intercept */
+	gfn = msg->guest_physical_address >> HV_HYP_PAGE_SHIFT;
+	mreg = mshv_partition_region_by_gfn(pt, gfn);

This call needs to be protected by the spinlock.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This is sorta fast path to bail. We recheck under partition lock above.

Accessing the list of regions without lock is unsafe.

I am not sure why? This check is done by a vcpu thread, so regions will not have just gone away.

This is shared resources. Multiple VP thread get into this function simultaneously, so there is a race already. But this one we can live with without locking as they don't mutate the list of the regions.

The issue happens when VMM adds or removed another region as it mutates the list and races with VP threads doing this lookup.

Thanks,
Stanislav

Quoted 55 lines

Thanks,
-Mukesh

Quoted 51 lines

Thanks,
Stanislav

Quoted 48 lines

Thanks,
-Mukesh

Quoted 44 lines

Thanks,
Stanislav

Quoted 41 lines
drivers/hv/mshv_root_main.c +23 -0
+	if (mreg == NULL || mreg->type != MSHV_REGION_TYPE_MMIO)
+		return false;
+
+	rc = mshv_chk_get_mmio_start_pfn(pt, gfn, &mmio_spa);
+	if (rc)
+		return false;
+
+	if (!hv_nofull_mmio) {		/* default case */
+		gfn = mreg->start_gfn;
+		mmio_spa = mmio_spa - (gfn - mreg->start_gfn);
+		numpgs = mreg->nr_pages;
+	} else
+		numpgs = 1;
+
+	rc = hv_call_map_mmio_pages(pt->pt_id, gfn, mmio_spa, numpgs);
+
+	return rc == 0;
+}
+
      static struct mshv_mem_region *
      mshv_partition_region_by_gfn_get(struct mshv_partition *p, u64 gfn)
      {
@@ -666,13 +777,17 @@ static bool mshv_handle_gpa_intercept(struct mshv_vp *vp)
      	return ret;
      }
+
      #else  /* CONFIG_X86_64 */
+static bool mshv_handle_unmapped_gpa(struct mshv_vp *vp) { return false; }
      static bool mshv_handle_gpa_intercept(struct mshv_vp *vp) { return false; }
      #endif /* CONFIG_X86_64 */
      static bool mshv_vp_handle_intercept(struct mshv_vp *vp)
      {
      	switch (vp->vp_intercept_msg_page->header.message_type) {
+	case HVMSG_UNMAPPED_GPA:
+		return mshv_handle_unmapped_gpa(vp);
      	case HVMSG_GPA_INTERCEPT:
      		return mshv_handle_gpa_intercept(vp);
      	}
Signature · 1 line
2.51.2.vfs.0.1
#70 lore raw
In reply to Mukesh R

Hi Mukesh,

Quoted 11 lines · On Mon, 19 Jan 2026 22:42:15 -0800 Mukesh R <[email protected]> wrote:

From: Mukesh Rathor <[email protected]>

Implement passthru of PCI devices to unprivileged virtual machines (VMs) when Linux is running as a privileged VM on Microsoft Hyper-V hypervisor. This support is made to fit within the workings of VFIO framework, and any VMM needing to use it must use the VFIO subsystem. This supports both full device passthru and SR-IOV based VFs.

There are 3 cases where Linux can run as a privileged VM (aka MSHV):
Baremetal root (meaning Hyper-V+Linux), L1VH, and Nested.

I think some introduction/background to L1VH would help.

Quoted 3 lines

At a high level, the hypervisor supports traditional mapped iommu domains that use explicit map and unmap hypercalls for mapping and unmapping guest RAM into the iommu subsystem.

It may be clearer to state that the hypervisor supports Linux IOMMU paging domains through map/unmap hypercalls, mapping GPAs to HPAs using stage‑2 I/O page tables.

Quoted 5 lines

Hyper-V also has a
concept of direct attach devices whereby the iommu subsystem simply uses the guest HW page table (ept/npt/..). This series adds support for both, and both are made to work in VFIO type1 subsystem.

This may warrant introducing a new IOMMU domain feature flag, as it performs mappings but does not support map/unmap semantics in the same way as a paging domain.

Quoted 53 lines

While this Part I focuses on memory mappings, upcoming Part II will focus on irq bypass along with some minor irq remapping updates.

This patch series was tested using Cloud Hypervisor verion 48. Qemu support of MSHV is in the works, and that will be extended to include PCI passthru and SR-IOV support also in near future.

Based on: 8f0b4cce4481 (origin/hyperv-next)

Thanks,
-Mukesh

Mukesh Rathor (15):
iommu/hyperv: rename hyperv-iommu.c to hyperv-irq.c x86/hyperv: cosmetic changes in irqdomain.c for readability x86/hyperv: add insufficient memory support in irqdomain.c
mshv: Provide a way to get partition id if running in a VMM process
mshv: Declarations and definitions for VFIO-MSHV bridge device
mshv: Implement mshv bridge device for VFIO
mshv: Add ioctl support for MSHV-VFIO bridge device
PCI: hv: rename hv_compose_msi_msg to hv_vmbus_compose_msi_msg
mshv: Import data structs around device domains and irq remapping
PCI: hv: Build device id for a VMBus device
x86/hyperv: Build logical device ids for PCI passthru hcalls x86/hyperv: Implement hyperv virtual iommu
x86/hyperv: Basic interrupt support for direct attached devices
mshv: Remove mapping of mmio space during map user ioctl
mshv: Populate mmio mappings for PCI passthru

 MAINTAINERS                         |    1 +
 arch/arm64/include/asm/mshyperv.h   |   15 +
 arch/x86/hyperv/irqdomain.c         |  314 ++++++---
 arch/x86/include/asm/mshyperv.h     |   21 +
 arch/x86/kernel/pci-dma.c           |    2 +
 drivers/hv/Makefile                 |    3 +-
 drivers/hv/mshv_root.h              |   24 +
 drivers/hv/mshv_root_main.c         |  296 +++++++-
 drivers/hv/mshv_vfio.c              |  210 ++++++
 drivers/iommu/Kconfig               |    1 +
 drivers/iommu/Makefile              |    2 +-
 drivers/iommu/hyperv-iommu.c        | 1004
+++++++++++++++++++++------ drivers/iommu/hyperv-irq.c          |
330 +++++++++ drivers/pci/controller/pci-hyperv.c |  207 ++++--
 include/asm-generic/mshyperv.h      |    1 +
 include/hyperv/hvgdk_mini.h         |   11 +
 include/hyperv/hvhdk_mini.h         |  112 +++
 include/linux/hyperv.h              |    6 +
 include/uapi/linux/mshv.h           |   31 +
 19 files changed, 2182 insertions(+), 409 deletions(-)
 create mode 100644 drivers/hv/mshv_vfio.c
 create mode 100644 drivers/iommu/hyperv-irq.c
author
#71 lore raw
Quoted 13 lines · Jacob Pan

Hi Mukesh,

Quoted 8 lines · On Mon, 19 Jan 2026 22:42:15 -0800 Mukesh R <[email protected]> wrote:

From: Mukesh Rathor <[email protected]>

Implement passthru of PCI devices to unprivileged virtual machines (VMs) when Linux is running as a privileged VM on Microsoft Hyper-V hypervisor. This support is made to fit within the workings of VFIO framework, and any VMM needing to use it must use the VFIO subsystem. This supports both full device passthru and SR-IOV based VFs.

Quoted 4 lines
Quoted 3 lines

There are 3 cases where Linux can run as a privileged VM (aka MSHV):
Baremetal root (meaning Hyper-V+Linux), L1VH, and Nested.

I think some introduction/background to L1VH would help.

Ok, i can add something, but l1vh was very well introduced if you search the mshv commits for "l1vh".

Quoted 6 lines
Quoted 3 lines

At a high level, the hypervisor supports traditional mapped iommu domains that use explicit map and unmap hypercalls for mapping and unmapping guest RAM into the iommu subsystem.

It may be clearer to state that the hypervisor supports Linux IOMMU paging domains through map/unmap hypercalls, mapping GPAs to HPAs using stage?2 I/O page tables.

sure.

Quoted 8 lines
Quoted 5 lines

Hyper-V also has a
concept of direct attach devices whereby the iommu subsystem simply uses the guest HW page table (ept/npt/..). This series adds support for both, and both are made to work in VFIO type1 subsystem.

This may warrant introducing a new IOMMU domain feature flag, as it performs mappings but does not support map/unmap semantics in the same way as a paging domain.

Yeah, I was hoping we can get by for now without it. At least in case of the cloud hypervisor, entire guest ram is mapped anyways. We can document it and work on enhancements which are much easier once we have a baseline. For now, it's a paging domain will all pages pinned.. :).

Quoted 53 lines
Quoted 53 lines

While this Part I focuses on memory mappings, upcoming Part II will focus on irq bypass along with some minor irq remapping updates.

This patch series was tested using Cloud Hypervisor verion 48. Qemu support of MSHV is in the works, and that will be extended to include PCI passthru and SR-IOV support also in near future.

Based on: 8f0b4cce4481 (origin/hyperv-next)

Thanks,
-Mukesh

Mukesh Rathor (15):
iommu/hyperv: rename hyperv-iommu.c to hyperv-irq.c x86/hyperv: cosmetic changes in irqdomain.c for readability x86/hyperv: add insufficient memory support in irqdomain.c
mshv: Provide a way to get partition id if running in a VMM process
mshv: Declarations and definitions for VFIO-MSHV bridge device
mshv: Implement mshv bridge device for VFIO
mshv: Add ioctl support for MSHV-VFIO bridge device
PCI: hv: rename hv_compose_msi_msg to hv_vmbus_compose_msi_msg
mshv: Import data structs around device domains and irq remapping
PCI: hv: Build device id for a VMBus device
x86/hyperv: Build logical device ids for PCI passthru hcalls x86/hyperv: Implement hyperv virtual iommu
x86/hyperv: Basic interrupt support for direct attached devices
mshv: Remove mapping of mmio space during map user ioctl
mshv: Populate mmio mappings for PCI passthru

  MAINTAINERS                         |    1 +
  arch/arm64/include/asm/mshyperv.h   |   15 +
  arch/x86/hyperv/irqdomain.c         |  314 ++++++---
  arch/x86/include/asm/mshyperv.h     |   21 +
  arch/x86/kernel/pci-dma.c           |    2 +
  drivers/hv/Makefile                 |    3 +-
  drivers/hv/mshv_root.h              |   24 +
  drivers/hv/mshv_root_main.c         |  296 +++++++-
  drivers/hv/mshv_vfio.c              |  210 ++++++
  drivers/iommu/Kconfig               |    1 +
  drivers/iommu/Makefile              |    2 +-
  drivers/iommu/hyperv-iommu.c        | 1004
+++++++++++++++++++++------ drivers/iommu/hyperv-irq.c          |
330 +++++++++ drivers/pci/controller/pci-hyperv.c |  207 ++++--
  include/asm-generic/mshyperv.h      |    1 +
  include/hyperv/hvgdk_mini.h         |   11 +
  include/hyperv/hvhdk_mini.h         |  112 +++
  include/linux/hyperv.h              |    6 +
  include/uapi/linux/mshv.h           |   31 +
  19 files changed, 2182 insertions(+), 409 deletions(-)

create mode 100644 drivers/hv/mshv_vfio.c
create mode 100644 drivers/iommu/hyperv-irq.c