eal/bus: use RTE_IOVA_PA only if phys addresses are available
PATCH 1/1 eal/bus: use RTE_IOVA_PA only if phys addresses are available
When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.
--- lib/librte_eal/common/eal_common_bus.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +14 -4
@@ -236,9 +236,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class (); } -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */ +return RTE_IOVA_PA ; + } + +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; + +/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }
Signature · 1 line
2.17.1
PATCH v2 1/1 eal/bus: use RTE_IOVA_PA only if phys addresses are available
When neither RTE_IOVA_VA nor RTE_IOVA_PA was explicitly requested, DPDK would currently fallback to the default RTE_IOVA_PA mode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to use RTE_IOVA_VA if no phys addresses are available may help in this case.
---
Changes since v1:
* added a missing rte_memory.h include
lib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
lib/librte_eal/common/eal_common_bus.c +15 -4
@@ -37,6 +37,7 @@#include <rte_bus.h> #include <rte_debug.h> #include <rte_string_fns.h> +#include <rte_memory.h> #include "eal_private.h"
@@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class (); } -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */ +return RTE_IOVA_PA ; + } + +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; + +/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }
Signature · 1 line
2.17.1
Quoted 42 lines · Darek Stojaczyk
When neither
RTE_IOVA_VAnorRTE_IOVA_PAwas explicitly requested, DPDK would currently fallback to the defaultRTE_IOVA_PAmode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to useRTE_IOVA_VAif no phys addresses are available may help in this case.Signed-off-by Darek Stojaczyk <[email protected]>---
Changes since v1:
* added a missingrte_memory.hincludelib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)lib/librte_eal/common/eal_common_bus.c +8
-3@@ -37,6 +37,7 @@#include <rte_bus.h> #include <rte_debug.h> #include <rte_string_fns.h> +#include <rte_memory.h> #include "eal_private.h" @@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class (); } -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */ +return RTE_IOVA_PA ; + }
lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255
Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +7
-1+ +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; + +/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }
This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require IOVA_PA, i should be running in IOVA_VA mode.
This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.
For example, at startup, i might have devices bound to VFIO, so IOVA_VA mode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requires IOVA_PA (leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to use IOVA_VA mode unless i happen to have a IOVA_PA device available at startup.
Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use IOVA_PA if i don't *also* have a few devices bound exclusively to VFIO.
[*] Do we have vdev devices that require IOVA_PA? I can't think of any...
Signature · 2 lines
Thanks, Anatoly
Quoted 64 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, September 17, 2018 12:34 PM
To: Stojaczyk, Dariusz <[email protected]>; [email protected]; Santosh Shukla <[email protected]>; Hemant Agrawal <[email protected]>; Jerin Jacob <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Chas Williams <[email protected]>
Subject: Re: [PATCH v2] eal/bus: useRTE_IOVA_PAonly if phys addresses are availableQuoted 29 lines · Darek Stojaczyk
When neither
RTE_IOVA_VAnorRTE_IOVA_PAwas explicitly requested, DPDK would currently fallback to the defaultRTE_IOVA_PAmode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to useRTE_IOVA_VAif no phys addresses are available may help in this case.Signed-off-by Darek Stojaczyk <[email protected]>---
Changes since v1:
* added a missingrte_memory.hincludelib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)lib/librte_eal/common/eal_common_bus.c +1
-0@@ -37,6 +37,7 @@#include <rte_bus.h> #include <rte_debug.h> #include <rte_string_fns.h> +#include <rte_memory.h> #include "eal_private.h" @@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class ();lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 244
Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +5
-3} -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to
RTE_IOVA_PA*/Quoted 9 lines
lib/librte_eal/common/eal_common_bus.c +7
-1+return RTE_IOVA_PA ; + } + +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; +
Quoted 8 lines
lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255
Quoted 4 lines
lib/librte_eal/common/eal_common_bus.c +2
-0+/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require
IOVA_PA, i should be running inIOVA_VAmode.
Another problem may occur when trying to hotplug devices that support only 39bit DMA. You may not be able to map any memory with vfio when in RTE_IOVA_VA mode, as virtual addresses likely occupy more than 39 bits.
The rte_pci bus enforces RTE_IOVA_PA whenever it finds such devices on init.
I have no doubt the logic can be improved here, but for now RTE_IOVA_PA is the only safe default.
D.
Quoted 22 lines
This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.
For example, at startup, i might have devices bound to VFIO, so
IOVA_VAmode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requiresIOVA_PA(leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to useIOVA_VAmode unless i happen to have aIOVA_PAdevice available at startup.Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use
IOVA_PAif i don't *also* have a few devices bound exclusively to VFIO.[*] Do we have vdev devices that require
IOVA_PA? I can't think of any...--
Thanks,
Anatoly
Quoted 70 lines · Dariusz
Quoted 55 lines
-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, September 17, 2018 12:34 PM
To: Stojaczyk, Dariusz <[email protected]>; [email protected]; Santosh Shukla <[email protected]>; Hemant Agrawal <[email protected]>; Jerin Jacob <[email protected]>
Cc: Maxime Coquelin <[email protected]>; Chas Williams <[email protected]>
Subject: Re: [PATCH v2] eal/bus: useRTE_IOVA_PAonly if phys addresses are availableQuoted 29 lines · Darek Stojaczyk
When neither
RTE_IOVA_VAnorRTE_IOVA_PAwas explicitly requested, DPDK would currently fallback to the defaultRTE_IOVA_PAmode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to useRTE_IOVA_VAif no phys addresses are available may help in this case.Signed-off-by Darek Stojaczyk <[email protected]>---
Changes since v1:
* added a missingrte_memory.hincludelib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)lib/librte_eal/common/eal_common_bus.c +1
-0@@ -37,6 +37,7 @@#include <rte_bus.h> #include <rte_debug.h> #include <rte_string_fns.h> +#include <rte_memory.h> #include "eal_private.h" @@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class ();lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 244
Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +5
-3} -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to
RTE_IOVA_PA*/Quoted 15 lines
lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255
Quoted 13 lines
lib/librte_eal/common/eal_common_bus.c +9
-1+return RTE_IOVA_PA ; + } + +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; + +/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }This is a good change, however I think that this is too pessimistic. If
Quoted 9 lines
i don't
Quoted 2 lines
have any devices that explictly require
IOVA_PA, i should be running inIOVA_VAmode.Another problem may occur when trying to hotplug devices that support only 39bit DMA. You may not be able to map any memory with vfio when in
RTE_IOVA_VAmode, as virtual addresses likely occupy more than 39 bits.
There is now a hint for trying to map memory as low as possible instead of using default Linux mmap base address. This makes devices with addressing limitations being usable as long as the physical memory to map is not more than what those devices allow.
Quoted 36 lines
The
rte_pcibus enforcesRTE_IOVA_PAwhenever it finds such devices on init.I have no doubt the logic can be improved here, but for now
RTE_IOVA_PAis the only safe default.D.
Quoted 2 lines
This of course doesn't take hotplug into account, so a command-line
switch
Quoted 5 lines
to force one or the other should also be available.
For example, at startup, i might have devices bound to VFIO, so
IOVA_VAmode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might laterhotplug a
Quoted 2 lines
device that requires
IOVA_PA(leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will beforced to
Quoted 4 lines
use
IOVA_VAmode unless i happen to have aIOVA_PAdevice available at startup.Similarly, if i'm running DPDK as root but am only using virtual devices
like
Quoted 2 lines
pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use
IOVA_PAif i don't *also* have a fewdevices
Quoted 7 lines
bound exclusively to VFIO.
[*] Do we have vdev devices that require
IOVA_PA? I can't think of any...--
Thanks,
Anatoly
17/09/2018 12:33, Burakov, Anatoly:
Quoted 69 lines · Darek Stojaczyk
Quoted 42 lines
When neither
RTE_IOVA_VAnorRTE_IOVA_PAwas explicitly requested, DPDK would currently fallback to the defaultRTE_IOVA_PAmode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to useRTE_IOVA_VAif no phys addresses are available may help in this case.Signed-off-by Darek Stojaczyk <[email protected]>---
Changes since v1:
* added a missingrte_memory.hincludelib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)lib/librte_eal/common/eal_common_bus.c +8
-3@@ -37,6 +37,7 @@#include <rte_bus.h> #include <rte_debug.h> #include <rte_string_fns.h> +#include <rte_memory.h> #include "eal_private.h" @@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class (); } -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */ +return RTE_IOVA_PA ; + }lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255
Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +7
-1+ +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; + +/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require
IOVA_PA, i should be running inIOVA_VAmode.This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.
For example, at startup, i might have devices bound to VFIO, so
IOVA_VAmode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requiresIOVA_PA(leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to useIOVA_VAmode unless i happen to have aIOVA_PAdevice available at startup.
Quoted 6 lines
Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use
IOVA_PAif i don't *also* have a few devices bound exclusively to VFIO.[*] Do we have vdev devices that require
IOVA_PA? I can't think of any...
If running as root, what is the benefit of using virtual addresses?
Quoted 70 lines · Thomas Monjalon
17/09/2018 12:33, Burakov, Anatoly:
Quoted 69 lines · Darek Stojaczyk
Quoted 42 lines
When neither
RTE_IOVA_VAnorRTE_IOVA_PAwas explicitly requested, DPDK would currently fallback to the defaultRTE_IOVA_PAmode and possibly encounter a failure later on if running as a non-priviledged user. Attempting to useRTE_IOVA_VAif no phys addresses are available may help in this case.Signed-off-by Darek Stojaczyk <[email protected]>---
Changes since v1:
* added a missingrte_memory.hincludelib/librte_eal/common/eal_common_bus.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)lib/librte_eal/common/eal_common_bus.c +8
-3@@ -37,6 +37,7 @@#include <rte_bus.h> #include <rte_debug.h> #include <rte_string_fns.h> +#include <rte_memory.h> #include "eal_private.h" @@ -236,9 +237,19 @@ rte_bus_get_iommu_class(void) mode |= bus->get_iommu_class (); } -if (mode !=RTE_IOVA_VA ) { -/* Use default IOVA mode */ - mode =RTE_IOVA_PA ; +if (mode ==RTE_IOVA_VA ) +return RTE_IOVA_VA ; + +if (mode &RTE_IOVA_PA ) { +/* Not all buses support RTE_IOVA_VA, fallback to RTE_IOVA_PA */ +return RTE_IOVA_PA ; + }lib/librte_eal/common/eal_common_bus.c rte_bus_get_iommu_class(void) · line 255
Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +7
-1+ +if (rte_eal_using_phys_addrs ()) { +/* Default to RTE_IOVA_PA only if it's supported */ +return RTE_IOVA_PA ; } -return mode; + +/* Since RTE_IOVA_PA is unsupported, fallback to RTE_IOVA_VA */ +return RTE_IOVA_VA ; }This is a good change, however I think that this is too pessimistic. If i don't have any devices that explictly require
IOVA_PA, i should be running inIOVA_VAmode.This of course doesn't take hotplug into account, so a command-line switch to force one or the other should also be available.
For example, at startup, i might have devices bound to VFIO, so
IOVA_VAmode is picked. However, even though at a time of startup none of the devices require physical addresses, i also know that i might later hotplug a device that requiresIOVA_PA(leaving the question of hotplug brokenness aside for now...) - currently, this scenario will not work, as i will be forced to useIOVA_VAmode unless i happen to have aIOVA_PAdevice available at startup.
Quoted 9 lines
Quoted 6 lines
Similarly, if i'm running DPDK as root but am only using virtual devices like pcap, i should be able to force DPDK into using VA addresses [*], yet currently i will be forced to use
IOVA_PAif i don't *also* have a few devices bound exclusively to VFIO.[*] Do we have vdev devices that require
IOVA_PA? I can't think of any...If running as root, what is the benefit of using virtual addresses?
Contiguous memory addresses is one that comes to mind.
Signature · 2 lines
Thanks, Anatoly