Skip to contentSkip to conversation

eal: add option to force IOVA as PA mode

PATCH In review Chas Williams 3 participants 7 messages Nov 30, 2017 – Jan 12, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 eal: add option to force IOVA as PA mode

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

---
 config/common_base                | 5 +++++
 lib/librte_eal/bsdapp/eal/eal.c   | 4 ++++
 lib/librte_eal/linuxapp/eal/eal.c | 4 ++++
 3 files changed, 13 insertions(+)
config/common_base +5 -0
index e74febe..06b3ded 100644
@@ -820,3 +820,8 @@ CONFIG_RTE_APP_CRYPTO_PERF=y
 # Compile the eventdev application
 #
 CONFIG_RTE_APP_EVENTDEV=y
+
+#
+# Force IOVA physical addresses in VFIO
+#
+CONFIG_RTE_LIBRTE_USE_PHYS_IOVA=n
lib/librte_eal/bsdapp/eal/eal.c +4 -0
index 369a682..b396285 100644
@@ -569,8 +569,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	if (internal_config.no_hugetlbfs == 0 &&
 			internal_config.process_type != RTE_PROC_SECONDARY &&
lib/librte_eal/linuxapp/eal/eal.c +4 -0
index 229eec9..13b02fd 100644
@@ -804,8 +804,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+#ifdef RTE_LIBRTE_USE_PHYS_IOVA
+	rte_eal_get_configuration()->iova_mode = RTE_IOVA_PA;
+#else
 	/* autodetect the iova mapping mode (default is iova_pa) */
 	rte_eal_get_configuration()->iova_mode = rte_bus_get_iommu_class();
+#endif
 
 	/* Workaround for KNI which requires physical address to work */
 	if (rte_eal_get_configuration()->iova_mode == RTE_IOVA_VA &&
Signature · 1 line
2.9.5
#1 lore raw

30/11/2017 03:57, Chas Williams:

Quoted 7 lines

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

Which machines are you talking about?
A run-time option may be a better solution.

#2 lore raw

The particular machine in my case is a Dell Optiplex 790 which is fairly similar to another system that has basically the same problem. The IOMMU advertises that is can only handle 39 bits of addressing. The DMAR address tables have a width of 36 bits, so all is well when using IOVA physical addresses. With IOVA virtual addresses, they sometimes go beyond the 39 bit boundary. I was under the impression that IOMMU widths are more typically 48 bits, but I appear to have some low end systems with a simpler IOMMU.

I could make it a runtime option.

Quoted 13 lines · Thomas Monjalon

30/11/2017 03:57, Chas Williams:

Quoted 7 lines

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

Which machines are you talking about?
A run-time option may be a better solution.

#3 lore raw

03/12/2017 01:23, Chas Williams:

Quoted 9 lines

The particular machine in my case is a Dell Optiplex 790 which is fairly similar to another system that has basically the same problem. The IOMMU advertises that is can only handle 39 bits of addressing. The DMAR address tables have a width of 36 bits, so all is well when using IOVA physical addresses. With IOVA virtual addresses, they sometimes go beyond the 39 bit boundary. I was under the impression that IOMMU widths are more typically 48 bits, but I appear to have some low end systems with a simpler IOMMU.

Quoted 16 lines

I could make it a runtime option.

Quoted 11 lines · Thomas Monjalon

30/11/2017 03:57, Chas Williams:

Quoted 7 lines

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

Which machines are you talking about?
A run-time option may be a better solution.

Please check the patch from Maxime:

	https://dpdk.org/dev/patchwork/patch/33192/
#4 lore raw

Thanks, and yes I already saw that. That seems like a much better solution. I will give it a spin in my box with this problem.

Quoted 34 lines · Thomas Monjalon

03/12/2017 01:23, Chas Williams:

Quoted 3 lines

The particular machine in my case is a Dell Optiplex 790 which is fairly similar to another system that has basically the same problem. The IOMMU advertises that is can only handle 39 bits of addressing. The DMAR

address

Quoted 4 lines

tables have a width of 36 bits, so all is well when using IOVA physical addresses. With IOVA virtual addresses, they sometimes go beyond the 39 bit boundary. I was under the impression that IOMMU widths are more typically 48 bits, but I appear to have some low end systems with a

simpler

Quoted 18 lines

IOMMU.

I could make it a runtime option.

Quoted 11 lines · Thomas Monjalon

30/11/2017 03:57, Chas Williams:

Quoted 7 lines

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

Which machines are you talking about?
A run-time option may be a better solution.

Please check the patch from Maxime:

        https://dpdk.org/dev/patchwork/patch/33192/
#5 lore raw
In reply to Chas Williams
Quoted 8 lines · Chas Williams

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

Is it possible to dynamically detect the machine limitation in bus iova code instead of compile time option?

#6 lore raw
Quoted 11 lines · Hemant Agrawal
Quoted 9 lines · Chas Williams

From: Chas Williams <[email protected]>

The IOMMU in some machines report that they can only support limited widths. IOVA virtual addresses may exceed this width making the use of IOVA virtual addresses difficult. The option CONFIG_RTE_LIBRTE_USE_PHYS_IOVA can used to force IOVA physical address usage.

Quoted 4 lines

Is it possible to dynamically detect the machine limitation in bus iova code instead of compile time option?

Yes, I think you can examine the "cap" of the IOMMU group in /sys and determine the bit width. I will have to check on that.