Skip to contentSkip to conversation

iommu/vt-d: Fix dmar pte read access not set error

PATCH In review Lu Baolu 3 participants 12 messages Dec 11, 2019 – Dec 17, 2019

Subsystems (2)
lore
Share & tools
lore

Lu Baolu

author
#0 lore raw

PATCH 1/1 iommu/vt-d: Fix dmar pte read access not set error

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
---
 drivers/iommu/intel-iommu.c | 8 --------
 1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -8
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct iommu_domain *domain,
 	int prot = 0;
 	int ret;
 
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return -EINVAL;
-
 	if (iommu_prot & IOMMU_READ)
 		prot |= DMA_PTE_READ;
 	if (iommu_prot & IOMMU_WRITE)
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain,
 	/* Cope with horrid API which requires us to unmap more than the
 	   size argument if it happens to be a large-page mapping. */
 	BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level));
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return 0;
 
 	if (size < VTD_PAGE_SIZE << level_to_offset_bits(level))
 		size = VTD_PAGE_SIZE << level_to_offset_bits(level);
@@ -5556,9 +5551,6 @@ static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain,
 	int level = 0;
 	u64 phys = 0;
 
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return 0;
-
 	pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
 	if (pte)
 		phys = dma_pte_addr(pte);
Signature · 1 line
2.17.1
#1 lore raw
Quoted 8 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Does this pose any potential issues with the reverse case where the group has a default identity domain, and the first device fits that, but a later device in the group needs dma and gets a private dma domain?

Quoted 46 lines
Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")

---
drivers/iommu/intel-iommu.c | 8 --------

1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -8
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct iommu_domain *domain,
 	int prot = 0;
 	int ret;
 
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return -EINVAL;
-
 	if (iommu_prot & IOMMU_READ)
 		prot |= DMA_PTE_READ;
 	if (iommu_prot & IOMMU_WRITE)
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain,
 	/* Cope with horrid API which requires us to unmap more than the
 	   size argument if it happens to be a large-page mapping. */
 	BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level));
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return 0;
 
 	if (size < VTD_PAGE_SIZE << level_to_offset_bits(level))
 		size = VTD_PAGE_SIZE << level_to_offset_bits(level);
@@ -5556,9 +5551,6 @@ static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain,
 	int level = 0;
 	u64 phys = 0;
 
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return 0;
-
 	pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
 	if (pte)
 		phys = dma_pte_addr(pte);
Signature · 1 line
2.17.1
author
#2 lore raw

Hi,

Quoted 10 lines · Jerry Snitselaar
Quoted 8 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Quoted 4 lines

Does this pose any potential issues with the reverse case where the group has a default identity domain, and the first device fits that, but a later device in the group needs dma and gets a private dma domain?

No. iommu_map/unmap() should not be called for default identity domain.

        if (unlikely(!(domain->type & __IOMMU_DOMAIN_PAGING)))
                return -EINVAL;

Best regards,
baolu

Quoted 51 lines
Quoted 50 lines
Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")

---
drivers/iommu/intel-iommu.c | 8 --------

1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -3
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct iommu_domain 
 *domain,
     int prot = 0;
     int ret;
 
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return -EINVAL;
-
     if (iommu_prot & IOMMU_READ)
         prot |= DMA_PTE_READ;

    if (iommu_prot & IOMMU_WRITE)

drivers/iommu/intel-iommu.c +0 -0
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct 
 iommu_domain *domain,
     /* Cope with horrid API which requires us to unmap more than the

       size argument if it happens to be a large-page mapping. */ BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level)); -    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;

    if (size < VTD_PAGE_SIZE << level_to_offset_bits(level)) size = VTD_PAGE_SIZE << level_to_offset_bits(level);

drivers/iommu/intel-iommu.c +0 -3
@@ -5556,9 +5551,6 @@ static phys_addr_t 
 intel_iommu_iova_to_phys(struct iommu_domain *domain,
     int level = 0;
     u64 phys = 0;
 
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;
-
     pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
     if (pte)

        phys = dma_pte_addr(pte);

Signature · 1 line
2.17.1
Reviewed-by on v1 1/1
#3 lore raw
In reply to Lu Baolu
Quoted 8 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Quoted 6 lines
Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 40 lines

---
drivers/iommu/intel-iommu.c | 8 --------

1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -8
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct iommu_domain *domain,
 	int prot = 0;
 	int ret;
 
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return -EINVAL;
-
 	if (iommu_prot & IOMMU_READ)
 		prot |= DMA_PTE_READ;
 	if (iommu_prot & IOMMU_WRITE)
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain,
 	/* Cope with horrid API which requires us to unmap more than the
 	   size argument if it happens to be a large-page mapping. */
 	BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level));
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return 0;
 
 	if (size < VTD_PAGE_SIZE << level_to_offset_bits(level))
 		size = VTD_PAGE_SIZE << level_to_offset_bits(level);
@@ -5556,9 +5551,6 @@ static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain,
 	int level = 0;
 	u64 phys = 0;
 
-	if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-		return 0;
-
 	pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
 	if (pte)
 		phys = dma_pte_addr(pte);
Signature · 1 line
2.17.1
author
#4 lore raw

Hi,

Quoted 9 lines · Jerry Snitselaar
Quoted 8 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Quoted 8 lines
Quoted 6 lines
Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

Quoted 45 lines
Quoted 44 lines

---
drivers/iommu/intel-iommu.c | 8 --------

1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -3
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct iommu_domain 
 *domain,
     int prot = 0;
     int ret;
 
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return -EINVAL;
-
     if (iommu_prot & IOMMU_READ)
         prot |= DMA_PTE_READ;

    if (iommu_prot & IOMMU_WRITE)

drivers/iommu/intel-iommu.c +0 -0
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct 
 iommu_domain *domain,
     /* Cope with horrid API which requires us to unmap more than the

       size argument if it happens to be a large-page mapping. */ BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level)); -    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;

    if (size < VTD_PAGE_SIZE << level_to_offset_bits(level)) size = VTD_PAGE_SIZE << level_to_offset_bits(level);

drivers/iommu/intel-iommu.c +0 -3
@@ -5556,9 +5551,6 @@ static phys_addr_t 
 intel_iommu_iova_to_phys(struct iommu_domain *domain,
     int level = 0;
     u64 phys = 0;
 
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;
-
     pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
     if (pte)

        phys = dma_pte_addr(pte);

Signature · 1 line
2.17.1
#5 lore raw
Quoted 19 lines · Lu Baolu

Hi,

Quoted 16 lines · Jerry Snitselaar
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 8 lines
Quoted 1 line

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

I'm testing with this patch, my patch that moves the direct mapping call, and Alex's patch for the ISA bridge. It solved the 2 iommu mapping errors I was seeing with default passthrough, I no longer see all the dmar pte read access errors, and the system boots allowing me to login. I'm tracking down 2 issues at the moment. With passthrough I see a problem with 01:00.4 that I mentioned in the earlier email:

[   78.978573] uhci_hcd: USB Universal Host Controller Interface driver
[   78.980842] uhci_hcd 0000:01:00.4: UHCI Host Controller
[   78.982738] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3
[   78.985222] uhci_hcd 0000:01:00.4: detected 8 ports
[   78.986907] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports
[   78.989316] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[   78.994634] uhci_hcd 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping
[   7 0000:01:00.4: unable to allocate consistent memory for frame list
[   79.499891] uhci_hcd 0000:01:00.4: startup error -16
[   79.501588] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[   79.503494] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[   79.505497] uhci_hcd: probe of 0000:01:00.4 failed with error -16

If I boot the system with iommu=nopt I see an iommu map failure due to the prot check in __domain_mapping:

[   40.940589] pci 0000:00:1f.0: iommu_group_add_device: calling iommu_group_create_direct_mappings
[   40.943558] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iterating through mappings
[   40.946402] pci 0000:00:1f.0: iommu_group_create_direct_mappings: calling apply_resv_region
[   40.949184] pci 0000:00:1f.0: iommu_group_create_direct_mappings: entry type is direct
[   40.951819] DMAR: intel_iommu_map: enter
[   40.953128] DMAR: __domain_mapping: prot & (DMA_PTE_READ|DMA_PTE_WRITE) == 0
[   40.955486] DMAR: domain_mapping: __domain_mapping failed
[   40.957348] DMAR: intel_iommu_map: domain_pfn_mapping returned -22
[   40.959466] DMAR: intel_iommu_map: leave
[   40.959468] iommu: iommu_map: ops->map failed iova 0x0 pa 0x0000000000000000 pgsize 0x1000
[   40.963511] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iommu_map failed
[   40.966026] pci 0000:00:1f.0: iommu_group_create_direct_mappings: leaving func
[   40.968487] pci 0000:00:1f.0: iommu_group_add_device: calling __iommu_attach_device
[   40.971016] pci 0000:00:1f.0: Adding to iommu group 19
[   40.972731] pci 0000:00:1f.0: DMAR: domain->type is dma

/sys/kernel/iommu_groups/19
[root@hp-dl388g8-07 19]# cat reserved_regions 0x0000000000000000 0x0000000000ffffff direct 0x00000000bdf6e000 0x00000000bdf84fff direct
0x00000000fee00000 0x00000000feefffff msi

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller

Quoted 49 lines
Quoted 35 lines
Quoted 35 lines

---
drivers/iommu/intel-iommu.c | 8 --------

1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -3
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct 
 iommu_domain *domain,
     int prot = 0;
     int ret;
 
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return -EINVAL;
-
     if (iommu_prot & IOMMU_READ)
         prot |= DMA_PTE_READ;

    if (iommu_prot & IOMMU_WRITE)

drivers/iommu/intel-iommu.c +0 -0
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct 
 iommu_domain *domain,
     /* Cope with horrid API which requires us to unmap more than the

       size argument if it happens to be a large-page mapping. */ BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level)); -    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;

    if (size < VTD_PAGE_SIZE << level_to_offset_bits(level)) size = VTD_PAGE_SIZE << level_to_offset_bits(level);

drivers/iommu/intel-iommu.c +0 -0
@@ -5556,9 +5551,6 @@ static phys_addr_t 
 intel_iommu_iova_to_phys(struct iommu_domain *domain,
     int level = 0;
     u64 phys = 0;
 
Quoted 10 lines
Quoted 9 lines
drivers/iommu/intel-iommu.c +0 -3
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;
-
     pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
     if (pte)

        phys = dma_pte_addr(pte);

Signature · 1 line
2.17.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#6 lore raw

Hi,

Quoted 68 lines · Jerry Snitselaar
Quoted 19 lines · Lu Baolu

Hi,

Quoted 16 lines · Jerry Snitselaar
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 8 lines
Quoted 1 line

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

I'm testing with this patch, my patch that moves the direct mapping call, and Alex's patch for the ISA bridge. It solved the 2 iommu mapping errors I was seeing with default passthrough, I no longer see all the dmar pte read access errors, and the system boots allowing me to login. I'm tracking down 2 issues at the moment. With passthrough I see a problem with 01:00.4 that I mentioned in the earlier email:

[   78.978573] uhci_hcd: USB Universal Host Controller Interface driver [   78.980842] uhci_hcd 0000:01:00.4: UHCI Host Controller
[   78.982738] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3 [   78.985222] uhci_hcd 0000:01:00.4: detected 8 ports
[   78.986907] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports [   78.989316] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[   78.994634] uhci_hcd 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping [   7 0000:01:00.4: unable to allocate consistent memory for frame list [   79.499891] uhci_hcd 0000:01:00.4: startup error -16
[   79.501588] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[   79.503494] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[   79.505497] uhci_hcd: probe of 0000:01:00.4 failed with error -16

If I boot the system with iommu=nopt I see an iommu map failure due to the prot check in __domain_mapping:

[   40.940589] pci 0000:00:1f.0: iommu_group_add_device: calling iommu_group_create_direct_mappings [   40.943558] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iterating through mappings [   40.946402] pci 0000:00:1f.0: iommu_group_create_direct_mappings: calling apply_resv_region [   40.949184] pci 0000:00:1f.0: iommu_group_create_direct_mappings: entry type is direct [   40.951819] DMAR: intel_iommu_map: enter
[   40.953128] DMAR: __domain_mapping: prot & (DMA_PTE_READ|DMA_PTE_WRITE) == 0
[   40.955486] DMAR: domain_mapping: __domain_mapping failed
[   40.957348] DMAR: intel_iommu_map: domain_pfn_mapping returned -22
[   40.959466] DMAR: intel_iommu_map: leave
[   40.959468] iommu: iommu_map: ops->map failed iova 0x0 pa 0x0000000000000000 pgsize 0x1000 [   40.963511] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iommu_map failed [   40.966026] pci 0000:00:1f.0: iommu_group_create_direct_mappings: leaving func
[   40.968487] pci 0000:00:1f.0: iommu_group_add_device: calling __iommu_attach_device [   40.971016] pci 0000:00:1f.0: Adding to iommu group 19
[   40.972731] pci 0000:00:1f.0: DMAR: domain->type is dma

Quoted 6 lines

/sys/kernel/iommu_groups/19
[root@hp-dl388g8-07 19]# cat reserved_regions 0x0000000000000000 0x0000000000ffffff direct 0x00000000bdf6e000 0x00000000bdf84fff direct
0x00000000fee00000 0x00000000feefffff msi

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller

This seems to be another issue?

Best regards,
baolu

#7 lore raw
Quoted 76 lines · Lu Baolu

Hi,

Quoted 73 lines · Jerry Snitselaar
Quoted 19 lines · Lu Baolu

Hi,

Quoted 16 lines · Jerry Snitselaar
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 8 lines
Quoted 1 line

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

I'm testing with this patch, my patch that moves the direct mapping call, and Alex's patch for the ISA bridge. It solved the 2 iommu mapping errors I was seeing with default passthrough, I no longer see all the dmar pte read access errors, and the system boots allowing me to login. I'm tracking down 2 issues at the moment. With passthrough I see a problem with 01:00.4 that I mentioned in the earlier email:

[   78.978573] uhci_hcd: USB Universal Host Controller Interface driver [   78.980842] uhci_hcd 0000:01:00.4: UHCI Host Controller
[   78.982738] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3 [   78.985222] uhci_hcd 0000:01:00.4: detected 8 ports
[   78.986907] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports [   78.989316] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[   78.994634] uhci_hcd 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping [   7 0000:01:00.4: unable to allocate consistent memory for frame list [   79.499891] uhci_hcd 0000:01:00.4: startup error -16
[   79.501588] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[   79.503494] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[   79.505497] uhci_hcd: probe of 0000:01:00.4 failed with error -16

If I boot the system with iommu=nopt I see an iommu map failure due to the prot check in __domain_mapping:

[   40.940589] pci 0000:00:1f.0: iommu_group_add_device: calling iommu_group_create_direct_mappings [   40.943558] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iterating through mappings [   40.946402] pci 0000:00:1f.0: iommu_group_create_direct_mappings: calling apply_resv_region [   40.949184] pci 0000:00:1f.0: iommu_group_create_direct_mappings: entry type is direct [   40.951819] DMAR: intel_iommu_map: enter
[   40.953128] DMAR: __domain_mapping: prot & (DMA_PTE_READ|DMA_PTE_WRITE) == 0
[   40.955486] DMAR: domain_mapping: __domain_mapping failed
[   40.957348] DMAR: intel_iommu_map: domain_pfn_mapping returned -22
[   40.959466] DMAR: intel_iommu_map: leave
[   40.959468] iommu: iommu_map: ops->map failed iova 0x0 pa 0x0000000000000000 pgsize 0x1000 [   40.963511] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iommu_map failed [   40.966026] pci 0000:00:1f.0: iommu_group_create_direct_mappings: leaving func
[   40.968487] pci 0000:00:1f.0: iommu_group_add_device: calling __iommu_attach_device [   40.971016] pci 0000:00:1f.0: Adding to iommu group 19
[   40.972731] pci 0000:00:1f.0: DMAR: domain->type is dma

/sys/kernel/iommu_groups/19
[root@hp-dl388g8-07 19]# cat reserved_regions 0x0000000000000000 0x0000000000ffffff direct 0x00000000bdf6e000 0x00000000bdf84fff direct
0x00000000fee00000 0x00000000feefffff msi

Quoted 6 lines
Quoted 1 line

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller

This seems to be another issue?

Best regards,
baolu

In intel_iommu_get_resv_regions this iommu_alloc_resv_region is called with prot set to 0:

                if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {
                        reg = iommu_alloc_resv_region(0, 1UL << 24, 0,
                                                      IOMMU_RESV_DIRECT_RELAXABLE);
                        if (reg)

I wonder if this is an issue with the region starting at 0x0 and this bit in iommu_group_create_mappings:

			phys_addr = iommu_iova_to_phys(domain, addr);
			if (phys_addr)
				continue;

Off to stick in some more debugging statements.

Regards,
Jerry

Quoted 4 lines

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#8 lore raw
Quoted 84 lines · Jerry Snitselaar
Quoted 76 lines · Lu Baolu

Hi,

Quoted 73 lines · Jerry Snitselaar
Quoted 19 lines · Lu Baolu

Hi,

Quoted 16 lines · Jerry Snitselaar
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 8 lines
Quoted 1 line

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

I'm testing with this patch, my patch that moves the direct mapping call, and Alex's patch for the ISA bridge. It solved the 2 iommu mapping errors I was seeing with default passthrough, I no longer see all the dmar pte read access errors, and the system boots allowing me to login. I'm tracking down 2 issues at the moment. With passthrough I see a problem with 01:00.4 that I mentioned in the earlier email:

[   78.978573] uhci_hcd: USB Universal Host Controller Interface driver [   78.980842] uhci_hcd 0000:01:00.4: UHCI Host Controller
[   78.982738] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3 [   78.985222] uhci_hcd 0000:01:00.4: detected 8 ports
[   78.986907] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports [   78.989316] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[   78.994634] uhci_hcd 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping [   7 0000:01:00.4: unable to allocate consistent memory for frame list [   79.499891] uhci_hcd 0000:01:00.4: startup error -16
[   79.501588] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[   79.503494] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[   79.505497] uhci_hcd: probe of 0000:01:00.4 failed with error -16

If I boot the system with iommu=nopt I see an iommu map failure due to the prot check in __domain_mapping:

[   40.940589] pci 0000:00:1f.0: iommu_group_add_device: calling iommu_group_create_direct_mappings [   40.943558] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iterating through mappings [   40.946402] pci 0000:00:1f.0: iommu_group_create_direct_mappings: calling apply_resv_region [   40.949184] pci 0000:00:1f.0: iommu_group_create_direct_mappings: entry type is direct [   40.951819] DMAR: intel_iommu_map: enter
[   40.953128] DMAR: __domain_mapping: prot & (DMA_PTE_READ|DMA_PTE_WRITE) == 0
[   40.955486] DMAR: domain_mapping: __domain_mapping failed
[   40.957348] DMAR: intel_iommu_map: domain_pfn_mapping returned -22
[   40.959466] DMAR: intel_iommu_map: leave
[   40.959468] iommu: iommu_map: ops->map failed iova 0x0 pa 0x0000000000000000 pgsize 0x1000 [   40.963511] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iommu_map failed [   40.966026] pci 0000:00:1f.0: iommu_group_create_direct_mappings: leaving func
[   40.968487] pci 0000:00:1f.0: iommu_group_add_device: calling __iommu_attach_device [   40.971016] pci 0000:00:1f.0: Adding to iommu group 19
[   40.972731] pci 0000:00:1f.0: DMAR: domain->type is dma

/sys/kernel/iommu_groups/19
[root@hp-dl388g8-07 19]# cat reserved_regions 0x0000000000000000 0x0000000000ffffff direct 0x00000000bdf6e000 0x00000000bdf84fff direct
0x00000000fee00000 0x00000000feefffff msi

Quoted 6 lines
Quoted 1 line

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller

This seems to be another issue?

Best regards,
baolu

Quoted 8 lines

In intel_iommu_get_resv_regions this iommu_alloc_resv_region is called with prot set to 0:

               if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) {
                       reg = iommu_alloc_resv_region(0, 1UL << 24, 0,
                                                     IOMMU_RESV_DIRECT_RELAXABLE);
                       if (reg)

Looking at the older code for the ISA bridge it looks like it called iommu_prepare_identity_map -> domain_prepare_identity_map -> iommu_domain_identity_map -> and finally __domain_mapping with DMA_PTE_READ|DMA_PTE_WRITE?

Quoted 6 lines

I wonder if this is an issue with the region starting at 0x0 and this bit in iommu_group_create_mappings:

			phys_addr = iommu_iova_to_phys(domain, addr);
			if (phys_addr)
				continue;

Disregard this

Quoted 10 lines

Off to stick in some more debugging statements.

Regards,
Jerry

Quoted 4 lines

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

author
#9 lore raw

Hi,

Quoted 91 lines · Jerry Snitselaar
Quoted 90 lines · Jerry Snitselaar
Quoted 76 lines · Lu Baolu

Hi,

Quoted 73 lines · Jerry Snitselaar
Quoted 19 lines · Lu Baolu

Hi,

Quoted 16 lines · Jerry Snitselaar
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 8 lines
Quoted 1 line

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

I'm testing with this patch, my patch that moves the direct mapping call, and Alex's patch for the ISA bridge. It solved the 2 iommu mapping errors I was seeing with default passthrough, I no longer see all the dmar pte read access errors, and the system boots allowing me to login. I'm tracking down 2 issues at the moment. With passthrough I see a problem with 01:00.4 that I mentioned in the earlier email:

[   78.978573] uhci_hcd: USB Universal Host Controller Interface driver [   78.980842] uhci_hcd 0000:01:00.4: UHCI Host Controller
[   78.982738] uhci_hcd 0000:01:00.4: new USB bus registered, assigned bus number 3 [   78.985222] uhci_hcd 0000:01:00.4: detected 8 ports
[   78.986907] uhci_hcd 0000:01:00.4: port count misdetected? forcing to 2 ports [   78.989316] uhci_hcd 0000:01:00.4: irq 16, io base 0x00003c00
[   78.994634] uhci_hcd 0000:01:00.4: DMAR: 32bit DMA uses non-identity mapping [   7 0000:01:00.4: unable to allocate consistent memory for frame list [   79.499891] uhci_hcd 0000:01:00.4: startup error -16
[   79.501588] uhci_hcd 0000:01:00.4: USB bus 3 deregistered
[   79.503494] uhci_hcd 0000:01:00.4: init 0000:01:00.4 fail, -16
[   79.505497] uhci_hcd: probe of 0000:01:00.4 failed with error -16

If I boot the system with iommu=nopt I see an iommu map failure due to the prot check in __domain_mapping:

[   40.940589] pci 0000:00:1f.0: iommu_group_add_device: calling iommu_group_create_direct_mappings [   40.943558] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iterating through mappings [   40.946402] pci 0000:00:1f.0: iommu_group_create_direct_mappings: calling apply_resv_region [   40.949184] pci 0000:00:1f.0: iommu_group_create_direct_mappings: entry type is direct [   40.951819] DMAR: intel_iommu_map: enter
[   40.953128] DMAR: __domain_mapping: prot & (DMA_PTE_READ|DMA_PTE_WRITE) == 0
[   40.955486] DMAR: domain_mapping: __domain_mapping failed
[   40.957348] DMAR: intel_iommu_map: domain_pfn_mapping returned -22
[   40.959466] DMAR: intel_iommu_map: leave
[   40.959468] iommu: iommu_map: ops->map failed iova 0x0 pa 0x0000000000000000 pgsize 0x1000 [   40.963511] pci 0000:00:1f.0: iommu_group_create_direct_mappings: iommu_map failed [   40.966026] pci 0000:00:1f.0: iommu_group_create_direct_mappings: leaving func
[   40.968487] pci 0000:00:1f.0: iommu_group_add_device: calling __iommu_attach_device [   40.971016] pci 0000:00:1f.0: Adding to iommu group 19
[   40.972731] pci 0000:00:1f.0: DMAR: domain->type is dma

/sys/kernel/iommu_groups/19
[root@hp-dl388g8-07 19]# cat reserved_regions 0x0000000000000000 0x0000000000ffffff direct 0x00000000bdf6e000 0x00000000bdf84fff direct
0x00000000fee00000 0x00000000feefffff msi

Quoted 6 lines
Quoted 1 line

00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller

This seems to be another issue?

Best regards,
baolu

In intel_iommu_get_resv_regions this iommu_alloc_resv_region is called with prot set to 0:

               if ((pdev->class >> 8) == PCI_CLASS_BRIDGE_ISA) { reg = iommu_alloc_resv_region(0, 1UL << 24, 0,

Quoted 7 lines
Quoted 3 lines

IOMMU_RESV_DIRECT_RELAXABLE);
                       if (reg)

Looking at the older code for the ISA bridge it looks like it called iommu_prepare_identity_map -> domain_prepare_identity_map -> iommu_domain_identity_map -> and finally __domain_mapping with DMA_PTE_READ|DMA_PTE_WRITE?

Yes. Returning a reserved region without any access permission makes no sense. Can you please post a fix for this?

Best regards,
baolu

#10 lore raw
In reply to Lu Baolu
Quoted 19 lines · Lu Baolu

Hi,

Quoted 16 lines · Jerry Snitselaar
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 8 lines
Quoted 1 line

Can you please try this fix and check whether it can fix your problem? If it helps, do you mind adding a Tested-by?

Best regards,
baolu

Quoted 49 lines
Quoted 35 lines
Quoted 35 lines

---
drivers/iommu/intel-iommu.c | 8 --------

1 file changed, 8 deletions(-)
drivers/iommu/intel-iommu.c +0 -3
index 0c8d81f56a30..b73bebea9148 100644
@@ -5478,9 +5478,6 @@ static int intel_iommu_map(struct 
 iommu_domain *domain,
     int prot = 0;
     int ret;
 
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return -EINVAL;
-
     if (iommu_prot & IOMMU_READ)
         prot |= DMA_PTE_READ;

    if (iommu_prot & IOMMU_WRITE)

drivers/iommu/intel-iommu.c +0 -0
@@ -5523,8 +5520,6 @@ static size_t intel_iommu_unmap(struct 
 iommu_domain *domain,
     /* Cope with horrid API which requires us to unmap more than the

       size argument if it happens to be a large-page mapping. */ BUG_ON(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level)); -    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;

    if (size < VTD_PAGE_SIZE << level_to_offset_bits(level)) size = VTD_PAGE_SIZE << level_to_offset_bits(level);

drivers/iommu/intel-iommu.c +0 -0
@@ -5556,9 +5551,6 @@ static phys_addr_t 
 intel_iommu_iova_to_phys(struct iommu_domain *domain,
     int level = 0;
     u64 phys = 0;
 
Quoted 10 lines
Quoted 9 lines
drivers/iommu/intel-iommu.c +0 -3
-    if (dmar_domain->flags & DOMAIN_FLAG_LOSE_CHILDREN)
-        return 0;
-
     pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level);
     if (pte)

        phys = dma_pte_addr(pte);

Signature · 1 line
2.17.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

maintainer
#11 lore raw
In reply to Lu Baolu
Quoted 14 lines · Lu Baolu

If the default DMA domain of a group doesn't fit a device, it will still sit in the group but use a private identity domain. When map/unmap/iova_to_phys come through iommu API, the driver should still serve them, otherwise, other devices in the same group will be impacted. Since identity domain has been mapped with the whole available memory space and RMRRs, we don't need to worry about the impact on it.

Cc Jerry Snitselaar <[email protected]>
Fixes 942067f1b6b97 ("iommu/vt-d: Identify default domains replaced with private")
Quoted 3 lines
---
 drivers/iommu/intel-iommu.c | 8 --------
 1 file changed, 8 deletions(-)

Applied for v5.5, thanks.