Skip to contentSkip to conversation

eal/bus: use RTE_IOVA_PA only if phys addresses are available

PATCH Superseded Darek Stojaczyk 4 participants 7 messages Sep 7, 2018 – Oct 30, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 eal/bus: use RTE_IOVA_PA only if phys addresses are available

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
 lib/librte_eal/common/eal_common_bus.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +14 -4
index 0943851cc..8b56979d7 100644
@@ -236,9 +236,19 @@ rte_bus_get_iommu_class(void)
 			mode |= bus->get_iommu_class();
 	}
 
-	if (mode != RTE_IOVA_VA) {
-		/* Use default IOVA mode */
-		mode = RTE_IOVA_PA;
+	if (mode == RTE_IOVA_VA)
+		return RTE_IOVA_VA;
+
+	if (mode & RTE_IOVA_PA) {
+		/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */
+		return RTE_IOVA_PA;
+	}
+
+	if (rte_eal_using_phys_addrs()) {
+		/* Default to RTE_IOVA_PA only if it's supported */
+		return RTE_IOVA_PA;
 	}
-	return mode;
+
+	/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+	return RTE_IOVA_VA;
 }
Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/1 eal/bus: use RTE_IOVA_PA only if phys addresses are available

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
Changes since v1:
* added a missing rte_memory.h include

 lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +15 -4
index 0943851cc..68c581b8a 100644
@@ -37,6 +37,7 @@
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
+#include <rte_memory.h>
 
 #include "eal_private.h"
 
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void)
 			mode |= bus->get_iommu_class();
 	}
 
-	if (mode != RTE_IOVA_VA) {
-		/* Use default IOVA mode */
-		mode = RTE_IOVA_PA;
+	if (mode == RTE_IOVA_VA)
+		return RTE_IOVA_VA;
+
+	if (mode & RTE_IOVA_PA) {
+		/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */
+		return RTE_IOVA_PA;
+	}
+
+	if (rte_eal_using_phys_addrs()) {
+		/* Default to RTE_IOVA_PA only if it's supported */
+		return RTE_IOVA_PA;
 	}
-	return mode;
+
+	/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+	return RTE_IOVA_VA;
 }
Signature · 1 line
2.17.1
#2 lore raw
Quoted 42 lines · Darek Stojaczyk

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
Changes since v1:
* added a missing rte_memory.h include

  lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++----
  1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +8 -3
index 0943851cc..68c581b8a 100644
@@ -37,6 +37,7 @@
  #include <rte_bus.h>
  #include <rte_debug.h>
  #include <rte_string_fns.h>
+#include <rte_memory.h>
 
  #include "eal_private.h"
 
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void)
  			mode |= bus->get_iommu_class();
  	}
 
-	if (mode != RTE_IOVA_VA) {
-		/* Use default IOVA mode */
-		mode = RTE_IOVA_PA;
+	if (mode == RTE_IOVA_VA)
+		return RTE_IOVA_VA;
+
+	if (mode & RTE_IOVA_PA) {
+		/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */
+		return RTE_IOVA_PA;
+	}

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +7 -1
+
+	if (rte_eal_using_phys_addrs()) {
+		/* Default to RTE_IOVA_PA only if it's supported */
+		return RTE_IOVA_PA;
  	}
-	return mode;
+
+	/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+	return RTE_IOVA_VA;
  }

This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require IOVA_PA, i should be running in IOVA_VA mode.

This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.

For example, at startup, i might have devices bound to VFIO, so IOVA_VA mode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requires IOVA_PA (leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to use IOVA_VA mode unless i happen to have a IOVA_PA device available at startup.

Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use IOVA_PA if i don't *also* have a few devices bound exclusively to VFIO.

[*] Do we have vdev devices that require IOVA_PA? I can't think of any...

Signature · 2 lines
Thanks,
Anatoly
#3 lore raw
Quoted 64 lines

-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, September 17, 2018 12:34 PM
To: Stojaczyk, Dariusz <[email protected]>; [email protected]; Santosh Shukla <[email protected]>; Hemant Agrawal <[email protected]>; Jerin Jacob <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Chas Williams <[email protected]>
Subject: Re: [PATCH v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available

Quoted 29 lines · Darek Stojaczyk

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
Changes since v1:
* added a missing rte_memory.h include

  lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++----
  1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +1 -0
index 0943851cc..68c581b8a 100644
@@ -37,6 +37,7 @@
  #include <rte_bus.h>
  #include <rte_debug.h>
  #include <rte_string_fns.h>
+#include <rte_memory.h>
 
  #include "eal_private.h"
 
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void)
  			mode |= bus->get_iommu_class();

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 244

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +5 -3
  	}
 
-	if (mode != RTE_IOVA_VA) {
-		/* Use default IOVA mode */
-		mode = RTE_IOVA_PA;
+	if (mode == RTE_IOVA_VA)
+		return RTE_IOVA_VA;
+
+	if (mode & RTE_IOVA_PA) {
+		/* Not all buses support RTE_IOVA_VA, fallback to

RTE_IOVA_PA */

Quoted 9 lines
lib/librte_eal/common/eal_common_bus.c +7 -1
+		return RTE_IOVA_PA;
+	}
+
+	if (rte_eal_using_phys_addrs()) {
+		/* Default to RTE_IOVA_PA only if it's supported */
+		return RTE_IOVA_PA;
  	}
-	return mode;
+
Quoted 8 lines

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255

Quoted 4 lines
lib/librte_eal/common/eal_common_bus.c +2 -0
+	/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+	return RTE_IOVA_VA;
  }

This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require IOVA_PA, i should be running in IOVA_VA mode.

Another problem may occur when trying to hotplug devices that support only 39bit DMA. You may not be able to map any memory with vfio when in RTE_IOVA_VA mode, as virtual addresses likely occupy more than 39 bits.

The rte_pci bus enforces RTE_IOVA_PA whenever it finds such devices on init.

I have no doubt the logic can be improved here, but for now RTE_IOVA_PA is the only safe default.

D.

Quoted 22 lines

This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.

For example, at startup, i might have devices bound to VFIO, so IOVA_VA mode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requires IOVA_PA (leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to use IOVA_VA mode unless i happen to have a IOVA_PA device available at startup.

Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use IOVA_PA if i don't *also* have a few devices bound exclusively to VFIO.

[*] Do we have vdev devices that require IOVA_PA? I can't think of any...

--
Thanks,
Anatoly

#4 lore raw
Quoted 70 lines · Dariusz
Quoted 55 lines

-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, September 17, 2018 12:34 PM
To: Stojaczyk, Dariusz <[email protected]>; [email protected]; Santosh Shukla <[email protected]>; Hemant Agrawal <[email protected]>; Jerin Jacob <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Chas Williams <[email protected]>
Subject: Re: [PATCH v2] eal/bus: use RTE_IOVA_PA only if phys addresses are available

Quoted 29 lines · Darek Stojaczyk

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
Changes since v1:
* added a missing rte_memory.h include

  lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++----
  1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +1 -0
index 0943851cc..68c581b8a 100644
@@ -37,6 +37,7 @@
  #include <rte_bus.h>
  #include <rte_debug.h>
  #include <rte_string_fns.h>
+#include <rte_memory.h>
 
  #include "eal_private.h"
 
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void)
                    mode |= bus->get_iommu_class();

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 244

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +5 -3
    }
 
-   if (mode != RTE_IOVA_VA) {
-           /* Use default IOVA mode */
-           mode = RTE_IOVA_PA;
+   if (mode == RTE_IOVA_VA)
+           return RTE_IOVA_VA;
+
+   if (mode & RTE_IOVA_PA) {
+           /* Not all buses support RTE_IOVA_VA, fallback to

RTE_IOVA_PA */

Quoted 15 lines

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255

Quoted 13 lines
lib/librte_eal/common/eal_common_bus.c +9 -1
+           return RTE_IOVA_PA;
+   }
+
+   if (rte_eal_using_phys_addrs()) {
+           /* Default to RTE_IOVA_PA only if it's supported */
+           return RTE_IOVA_PA;
    }
-   return mode;
+
+   /* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+   return RTE_IOVA_VA;
  }

This is a good change, however I think that this is too pessimistic. If

Quoted 9 lines

i don't

Quoted 2 lines

have any devices that explictly require IOVA_PA, i should be running in IOVA_VA mode.

Another problem may occur when trying to hotplug devices that support only 39bit DMA. You may not be able to map any memory with vfio when in RTE_IOVA_VA mode, as virtual addresses likely occupy more than 39 bits.

There is now a hint for trying to map memory as low as possible instead of using default Linux mmap base address. This makes devices with addressing limitations being usable as long as the physical memory to map is not more than what those devices allow.

Quoted 36 lines

The rte_pci bus enforces RTE_IOVA_PA whenever it finds such devices on init.

I have no doubt the logic can be improved here, but for now RTE_IOVA_PA is the only safe default.

D.

Quoted 2 lines

This of course doesn't take hotplug into account, so a command-line

switch

Quoted 5 lines

to force one or the other should also be available.

For example, at startup, i might have devices bound to VFIO, so IOVA_VA mode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later

hotplug a

Quoted 2 lines

device that requires IOVA_PA (leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be

forced to

Quoted 4 lines

use IOVA_VA mode unless i happen to have a IOVA_PA device available at startup.

Similarly, if i'm running DPDK as root but am only using virtual devices

like

Quoted 2 lines

pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use IOVA_PA if i don't *also* have a few

devices

Quoted 7 lines

bound exclusively to VFIO.

[*] Do we have vdev devices that require IOVA_PA? I can't think of any...

--
Thanks,
Anatoly

#5 lore raw
In reply to Burakov, Anatoly

17/09/2018 12:33, Burakov, Anatoly:

Quoted 69 lines · Darek Stojaczyk
Quoted 42 lines

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
Changes since v1:
* added a missing rte_memory.h include

  lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++----
  1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +8 -3
index 0943851cc..68c581b8a 100644
@@ -37,6 +37,7 @@
  #include <rte_bus.h>
  #include <rte_debug.h>
  #include <rte_string_fns.h>
+#include <rte_memory.h>
 
  #include "eal_private.h"
 
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void)
  			mode |= bus->get_iommu_class();
  	}
 
-	if (mode != RTE_IOVA_VA) {
-		/* Use default IOVA mode */
-		mode = RTE_IOVA_PA;
+	if (mode == RTE_IOVA_VA)
+		return RTE_IOVA_VA;
+
+	if (mode & RTE_IOVA_PA) {
+		/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */
+		return RTE_IOVA_PA;
+	}

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +7 -1
+
+	if (rte_eal_using_phys_addrs()) {
+		/* Default to RTE_IOVA_PA only if it's supported */
+		return RTE_IOVA_PA;
  	}
-	return mode;
+
+	/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+	return RTE_IOVA_VA;
  }

This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require IOVA_PA, i should be running in IOVA_VA mode.

This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.

For example, at startup, i might have devices bound to VFIO, so IOVA_VA mode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requires IOVA_PA (leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to use IOVA_VA mode unless i happen to have a IOVA_PA device available at startup.

Quoted 6 lines

Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use IOVA_PA if i don't *also* have a few devices bound exclusively to VFIO.

[*] Do we have vdev devices that require IOVA_PA? I can't think of any...

If running as root, what is the benefit of using virtual addresses?

#6 lore raw
Quoted 70 lines · Thomas Monjalon

17/09/2018 12:33, Burakov, Anatoly:

Quoted 69 lines · Darek Stojaczyk
Quoted 42 lines

When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.

---
Changes since v1:
* added a missing rte_memory.h include

   lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++----
   1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +8 -3
index 0943851cc..68c581b8a 100644
@@ -37,6 +37,7 @@
   #include <rte_bus.h>
   #include <rte_debug.h>
   #include <rte_string_fns.h>
+#include <rte_memory.h>
 
   #include "eal_private.h"
 
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void)
   			mode |= bus->get_iommu_class();
   	}
 
-	if (mode != RTE_IOVA_VA) {
-		/* Use default IOVA mode */
-		mode = RTE_IOVA_PA;
+	if (mode == RTE_IOVA_VA)
+		return RTE_IOVA_VA;
+
+	if (mode & RTE_IOVA_PA) {
+		/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */
+		return RTE_IOVA_PA;
+	}

lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +7 -1
+
+	if (rte_eal_using_phys_addrs()) {
+		/* Default to RTE_IOVA_PA only if it's supported */
+		return RTE_IOVA_PA;
   	}
-	return mode;
+
+	/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */
+	return RTE_IOVA_VA;
   }

This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require IOVA_PA, i should be running in IOVA_VA mode.

This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.

For example, at startup, i might have devices bound to VFIO, so IOVA_VA mode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requires IOVA_PA (leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to use IOVA_VA mode unless i happen to have a IOVA_PA device available at startup.

Quoted 9 lines
Quoted 6 lines

Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use IOVA_PA if i don't *also* have a few devices bound exclusively to VFIO.

[*] Do we have vdev devices that require IOVA_PA? I can't think of any...

If running as root, what is the benefit of using virtual addresses?

Contiguous memory addresses is one that comes to mind.

Signature · 2 lines
Thanks,
Anatoly