Skip to contentSkip to conversation

vfio: retry creating sPAPR DMA window

PATCH Applied Takeshi Yoshimura 4 participants 14 messages Jun 7, 2019 – Jul 11, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfio: retry creating sPAPR DMA window

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
 lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
lib/librte_eal/linux/eal/eal_vfio.c +23 -3
index 6892a2c14..f16c5c3c0 100644
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
 	/* create new DMA window */
 	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
 	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;
+
+			while (pgsizes != 0) {
+				create->page_shift = 31 - __builtin_clz(pgsizes);
+				create->levels = levels;
+				ret = ioctl(vfio_container_fd,
+					VFIO_IOMMU_SPAPR_TCE_CREATE, create);
+				if (!ret)
+					break;
+				pgsizes &= ~(1 << create->page_shift);
+			}
+			if (!ret)
+				break;
+		}
+		if (ret) {
+			RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
+					"error %i (%s)\n", errno, strerror(errno));
+			return -1;
+		}
 	}
 
 	if (create->start_addr != 0) {
Signature · 1 line
2.17.1
#1 lore raw

Please, any review or ack for this patch?

07/06/2019 04:28, Takeshi Yoshimura:

Quoted 8 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
 lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
#2 lore raw
Quoted 12 lines

Please, any review or ack for this patch?

07/06/2019 04:28, Takeshi Yoshimura:

Quoted 8 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
  lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
  1 file changed, 23 insertions(+), 3 deletions(-)

I've shared my comments with Takeshi and I don't think the patch is ready in it's current state. It appears to modify x86 functionality and I'm not comfortable with signing off on that since it's in an anrea I'm not familiar with.

Dave

#3 lore raw

08/07/2019 18:47, David Christensen:

Quoted 7 lines
Quoted 7 lines

Please, any review or ack for this patch?

07/06/2019 04:28, Takeshi Yoshimura:

Quoted 4 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

Quoted 6 lines
Quoted 1 line
Quoted 1 line

I've shared my comments with Takeshi and I don't think the patch is ready in it's current state. It appears to modify x86 functionality and I'm not comfortable with signing off on that since it's in an anrea I'm not familiar with.

It is a function specific to sPAPR. So there is no risk for x86 or Arm. As you didn't share your comments publicly, this patch has been merged in 19.08-rc1.

#4 lore raw
Quoted 15 lines · Thomas Monjalon

08/07/2019 18:47, David Christensen:

Quoted 7 lines
Quoted 7 lines

Please, any review or ack for this patch?

07/06/2019 04:28, Takeshi Yoshimura:

Quoted 4 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

Quoted 6 lines
Quoted 1 line
Quoted 1 line

I've shared my comments with Takeshi and I don't think the patch is ready in it's current state. It appears to modify x86 functionality and I'm not comfortable with signing off on that since it's in an anrea I'm not familiar with.

Quoted 5 lines

It is a function specific to sPAPR. So there is no risk for x86 or Arm. As you didn't share your comments publicly, this patch has been merged in 19.08-rc1.

I have a suspicion that David has confused this patch with another one, which did indeed touch a lot of code (but still didn't touch x86 AFAIR).

Signature · 2 lines
Thanks,
Anatoly
Acked-by on v1 1/1
#5 lore raw
In reply to Takeshi Yoshimura
Quoted 6 lines · Takeshi Yoshimura

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---

This doesn't affect any code outside of sPAPR and looks sane, so

Signature · 2 lines
Thanks,
Anatoly
#6 lore raw

05/07/2019 10:15, Burakov, Anatoly:

Quoted 11 lines · Takeshi Yoshimura
Quoted 6 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---

This doesn't affect any code outside of sPAPR and looks sane, so

Applied, thanks

#7 lore raw
In reply to Takeshi Yoshimura
Quoted 15 lines · Takeshi Yoshimura

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
  lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
  1 file changed, 23 insertions(+), 3 deletions(-)
lib/librte_eal/linux/eal/eal_vfio.c +0 -0
index 6892a2c14..f16c5c3c0 100644
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
  	/* create new DMA window */

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int vfio_container_fd, · line 1455

Quoted 10 lines
lib/librte_eal/linux/eal/eal_vfio.c +5 -3
  	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
  	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):

/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function ‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ for (levels = 1; levels <= info.ddw.levels; levels++) {
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option "-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2

Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw structures, but this isn't simply a case of a missing struct; rather it is an issue of differing definitions between what recent kernels have, and what older kernels had.

@Thomas Do we still support these older distros?

Signature · 2 lines
Thanks,
Anatoly
#8 lore raw

10/07/2019 12:32, Burakov, Anatoly:

Quoted 57 lines · Takeshi Yoshimura
Quoted 15 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
  lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
  1 file changed, 23 insertions(+), 3 deletions(-)
lib/librte_eal/linux/eal/eal_vfio.c +0 -0
index 6892a2c14..f16c5c3c0 100644
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
  	/* create new DMA window */

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int vfio_container_fd, · line 1455

Quoted 10 lines
lib/librte_eal/linux/eal/eal_vfio.c +5 -3
  	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
  	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):

/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function ‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
for (levels = 1; levels <= info.ddw.levels; levels++) {
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option "-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2

Quoted 6 lines

Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw structures, but this isn't simply a case of a missing struct; rather it is an issue of differing definitions between what recent kernels have, and what older kernels had.

@Thomas Do we still support these older distros?

If the kernel is not maintained, we don't support it. For most distros, Linux < 3.16 is not maintained.

#9 lore raw
Quoted 63 lines · Thomas Monjalon

10/07/2019 12:32, Burakov, Anatoly:

Quoted 62 lines · Takeshi Yoshimura
Quoted 15 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
   lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
   1 file changed, 23 insertions(+), 3 deletions(-)
lib/librte_eal/linux/eal/eal_vfio.c +0 -0
index 6892a2c14..f16c5c3c0 100644
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
   	/* create new DMA window */

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int vfio_container_fd, · line 1455

Quoted 10 lines
lib/librte_eal/linux/eal/eal_vfio.c +5 -3
   	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
   	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):

/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function ‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
for (levels = 1; levels <= info.ddw.levels; levels++) {
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option "-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2

Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw structures, but this isn't simply a case of a missing struct; rather it is an issue of differing definitions between what recent kernels have, and what older kernels had.

Quoted 5 lines
Quoted 1 line

@Thomas Do we still support these older distros?

If the kernel is not maintained, we don't support it. For most distros, Linux < 3.16 is not maintained.

This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's published schedule [1] has stopped receiving even maintenance updates, and is only updated for select customers). It was never an LTS kernel to begin with, i think.

[1] https://ubuntu.com/about/release-cycle

Signature · 2 lines
Thanks,
Anatoly
#10 lore raw

10/07/2019 14:35, Burakov, Anatoly:

Quoted 66 lines · Thomas Monjalon
Quoted 65 lines

10/07/2019 12:32, Burakov, Anatoly:

Quoted 63 lines · Takeshi Yoshimura
Quoted 15 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
   lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
   1 file changed, 23 insertions(+), 3 deletions(-)
lib/librte_eal/linux/eal/eal_vfio.c +0 -0
index 6892a2c14..f16c5c3c0 100644
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
   	/* create new DMA window */

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int vfio_container_fd, · line 1455

Quoted 10 lines
lib/librte_eal/linux/eal/eal_vfio.c +5 -3
   	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
   	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):

/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function ‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
for (levels = 1; levels <= info.ddw.levels; levels++) {
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option "-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2

Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw structures, but this isn't simply a case of a missing struct; rather it is an issue of differing definitions between what recent kernels have, and what older kernels had.

@Thomas Do we still support these older distros?

Quoted 7 lines
Quoted 2 lines

If the kernel is not maintained, we don't support it. For most distros, Linux < 3.16 is not maintained.

This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's published schedule [1] has stopped receiving even maintenance updates, and is only updated for select customers). It was never an LTS kernel to begin with, i think.

So we can ignore this issue.

#11 lore raw
Quoted 70 lines · Thomas Monjalon

10/07/2019 14:35, Burakov, Anatoly:

Quoted 69 lines · Thomas Monjalon
Quoted 67 lines

10/07/2019 12:32, Burakov, Anatoly:

Quoted 57 lines · Takeshi Yoshimura
Quoted 15 lines

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO ioctl. However, Linux 4.17 or before returns incorrect page_shift for Power9. I added the code for retrying creation of sPAPR DMA window.

---
    lib/librte_eal/linux/eal/eal_vfio.c | 26 +++++++++++++++++++++++---
    1 file changed, 23 insertions(+), 3 deletions(-)
lib/librte_eal/linux/eal/eal_vfio.c +0 -0
index 6892a2c14..f16c5c3c0 100644
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int vfio_container_fd,
    	/* create new DMA window */

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int vfio_container_fd, · line 1455

Quoted 10 lines
lib/librte_eal/linux/eal/eal_vfio.c +5 -3
    	ret = ioctl(vfio_container_fd, VFIO_IOMMU_SPAPR_TCE_CREATE, create);
    	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

This is failing compilation on some older distros (Fedora 20, Ubuntu 14.04):

/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function ‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’
for (levels = 1; levels <= info.ddw.levels; levels++) {
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error: ‘struct vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option "-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2

Quoted 6 lines

Looking at the eal_vfio.h we are handling case of missing sPAPR info/ddw structures, but this isn't simply a case of a missing struct; rather it is an issue of differing definitions between what recent kernels have, and what older kernels had.

@Thomas Do we still support these older distros?

If the kernel is not maintained, we don't support it. For most distros, Linux < 3.16 is not maintained.

Quoted 7 lines
Quoted 4 lines

This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's published schedule [1] has stopped receiving even maintenance updates, and is only updated for select customers). It was never an LTS kernel to begin with, i think.

So we can ignore this issue.

It seems that it's also reproducible on Ubuntu 14.04 with kernel 3.19. Which is not an upstream LTS kernel, but it's >3.16.

It also looks like it may be an issue for us specifically, so i would suggest reverting the patch for rc2 if the submitter/maintainers don't come back and fix it.

Signature · 2 lines
Thanks,
Anatoly
#12 lore raw
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Quoted 1 line

@Thomas Do we still support these older distros?

If the kernel is not maintained, we don't support it. For most distros, Linux < 3.16 is not maintained.

This is on kernel 3.13 (Ubuntu 14.04 LTS which according to Canonical's published schedule [1] has stopped receiving even maintenance updates, and is only updated for select customers). It was never an LTS kernel to begin with, i think.

Quoted 6 lines
Quoted 2 lines

So we can ignore this issue.

It seems that it's also reproducible on Ubuntu 14.04 with kernel 3.19. Which is not an upstream LTS kernel, but it's >3.16.

It also looks like it may be an issue for us specifically, so i would suggest reverting the patch for rc2 if the submitter/maintainers don't come back and fix it.

Just to confirm, these failures are on x86 builds, is that correct? Power 9 systems aren't supported by these older distros, Linux kernel 4.6 was the first kernel with Power 9 support.

Dave

#13 lore raw
In reply to Burakov, Anatoly

-----"Burakov, Anatoly" <[email protected]> wrote: -----

Quoted 106 lines

To: Thomas Monjalon <[email protected]>
From: "Burakov, Anatoly" <[email protected]>
Date: 07/10/2019 09:01AM
Cc: Takeshi Yoshimura <[email protected]>, [email protected], David Christensen <[email protected]>
Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] vfio: retry creating sPAPR DMA window

Quoted 5 lines · Thomas Monjalon

10/07/2019 14:35, Burakov, Anatoly:

Quoted 4 lines · Thomas Monjalon
Quoted 3 lines

10/07/2019 12:32, Burakov, Anatoly:

Quoted 2 lines · Takeshi Yoshimura
Quoted 1 line

sPAPR allows only page_shift from VFIO_IOMMU_SPAPR_TCE_GET_INFO

ioctl.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

However, Linux 4.17 or before returns incorrect page_shift for

Power9.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

I added the code for retrying creation of sPAPR DMA window.

---
    lib/librte_eal/linux/eal/eal_vfio.c | 26

+++++++++++++++++++++++---

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
    1 file changed, 23 insertions(+), 3 deletions(-)
(quoted diff) +0 -0

b/lib/librte_eal/linux/eal/eal_vfio.c

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int

Quoted 4 lines

index 6892a2c14..f16c5c3c0 100644

lib/librte_eal/linux/eal/eal_vfio.c +0 -0
@@ -1448,9 +1448,29 @@ vfio_spapr_create_new_dma_window(int

vfio_container_fd,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int · line 1449

Quoted 2 lines
lib/librte_eal/linux/eal/eal_vfio.c +0 -0
    	/* create new DMA window */
    	ret = ioctl(vfio_container_fd,

VFIO_IOMMU_SPAPR_TCE_CREATE, create);

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

lib/librte_eal/linux/eal/eal_vfio.c vfio_spapr_create_new_dma_window(int vfio_container_fd, · line 1455

Quoted 9 lines
lib/librte_eal/linux/eal/eal_vfio.c +5 -3
    	if (ret) {
-		RTE_LOG(ERR, EAL, "  cannot create new DMA window, "
-				"error %i (%s)\n", errno, strerror(errno));
-		return -1;
+		/* try possible page_shift and levels for workaround */
+		uint32_t levels;
+
+		for (levels = 1; levels <= info.ddw.levels; levels++) {
+			uint32_t pgsizes = info.ddw.pgsizes;

+CC POWER maintainer from MAINTAINERS file.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

This is failing compilation on some older distros (Fedora 20,

Ubuntu 14.04):

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: In function ‘vfio_spapr_create_new_dma_window’:
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1451:34: error:

‘struct

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ for (levels = 1; levels <= info.ddw.levels; levels++) { ^ /root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c:1452:27: error:

‘struct

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

vfio_iommu_spapr_tce_info’ has no member named ‘ddw’ uint32_t pgsizes = info.ddw.pgsizes;
^
/root/dpdk/lib/librte_eal/linux/eal/eal_vfio.c: At top level:
cc1: error: unrecognized command line option "-Wno-address-of-packed-member" [-Werror]
cc1: all warnings being treated as errors
make[7]: *** [eal_vfio.o] Error 1
make[7]: *** Waiting for unfinished jobs.... make[6]: *** [eal] Error 2
make[5]: *** [linux] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Looking at the eal_vfio.h we are handling case of missing sPAPR

info/ddw

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

structures, but this isn't simply a case of a missing struct;

rather it

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

is an issue of differing definitions between what recent kernels

have,

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

and what older kernels had.

@Thomas Do we still support these older distros?

If the kernel is not maintained, we don't support it. For most distros, Linux < 3.16 is not maintained.

This is on kernel 3.13 (Ubuntu 14.04 LTS which according to

Canonical's

Quoted 1 line
Quoted 1 line

published schedule [1] has stopped receiving even maintenance

updates,

Quoted 1 line
Quoted 1 line

and is only updated for select customers). It was never an LTS

kernel to

Quoted 4 lines
Quoted 1 line

begin with, i think.

So we can ignore this issue.

It seems that it's also reproducible on Ubuntu 14.04 with kernel 3.19.
Which is not an upstream LTS kernel, but it's >3.16.

Quoted 11 lines

It also looks like it may be an issue for us specifically, so i would

suggest reverting the patch for rc2 if the submitter/maintainers don't
come back and fix it.

Signature · 2 lines
Thanks,
Anatoly

Hi Anatoly,
Thank you for your report and analysis. This can be avoided by adding #ifdef VFIO_IOMMU_SPAPR_INFO_DDW. I will quickly submit the fix soon after this email.

Regards,
Takeshi