Skip to contentSkip to conversation

doc: announce change in dma mapping/unmapping

PATCH In review Xuan Ding 4 participants 7 messages Aug 25, 2021 – Aug 31, 2021

lore
Share & tools
lore

Xuan Ding

author
#0 lore raw

PATCH 1/1 doc: announce change in dma mapping/unmapping

Currently, the VFIO subsystem will compact adjacent DMA regions for the purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to be adjacent in memory. Since VFIO implementation on IA platforms also does not allow partial unmapping of memory mapped for DMA, the current DPDK VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API to also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
 doc/guides/rel_notes/deprecation.rst | 3 +++
 1 file changed, 3 insertions(+)
doc/guides/rel_notes/deprecation.rst +3 -0
index 76a4abfd6b..272ffa993e 100644
@@ -287,3 +287,6 @@ Deprecation Notices
   reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
   information from the crypto/security operation. This field will be used to
   communicate events such as soft expiry with IPsec in lookaside mode.
+
+  * vfio: the functions `rte_vfio_container_dma_map` and `rte_vfio_container_dma_unmap`
+  will be amended to include page size. This change is targeted for DPDK 21.11.
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#1 lore raw
Quoted 24 lines · Xuan Ding

Currently, the VFIO subsystem will compact adjacent DMA regions for the purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to be adjacent in memory. Since VFIO implementation on IA platforms also does not allow partial unmapping of memory mapped for DMA, the current DPDK VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API to also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
  doc/guides/rel_notes/deprecation.rst | 3 +++
  1 file changed, 3 insertions(+)
doc/guides/rel_notes/deprecation.rst +0 -0
index 76a4abfd6b..272ffa993e 100644

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 292

Quoted 6 lines
doc/guides/rel_notes/deprecation.rst +3 -0
@@ -287,3 +287,6 @@ Deprecation Notices
    reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
    information from the crypto/security operation. This field will be used to
    communicate events such as soft expiry with IPsec in lookaside mode.
+
+  * vfio: the functions `rte_vfio_container_dma_map` and `rte_vfio_container_dma_unmap`
+  will be amended to include page size. This change is targeted for DPDK 21.11.
Signature · 2 lines
Thanks,
Anatoly
#2 lore raw
Quoted 19 lines · Anatoly
Quoted 18 lines · Xuan Ding

Currently, the VFIO subsystem will compact adjacent DMA regions for the purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to be adjacent in memory. Since VFIO implementation on IA platforms also does not allow partial unmapping of memory mapped for DMA, the current DPDK VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API to also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
  doc/guides/rel_notes/deprecation.rst | 3 +++

  1 file changed, 3 insertions(+)
Quoted 17 lines
Quoted 6 lines
doc/guides/rel_notes/deprecation.rst +0 -0
index 76a4abfd6b..272ffa993e 100644
@@ -287,3 +287,6 @@ Deprecation Notices
Quoted 8 lines

    reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other information from the crypto/security operation. This field will be used to communicate events such as soft expiry with IPsec in lookaside mode. +
+  * vfio: the functions rte_vfio_container_dma_map and rte_vfio_container_dma_unmap
+  will be amended to include page size. This change is targeted for DPDK 21.11.

Techboard decision was to add a new API, instead of updating existing ones, to not break the apps using this API.

@Xuan, @Anatoly, can you please confirm if this will solve your problem?

#3 lore raw
Quoted 38 lines · Ferruh Yigit
Quoted 19 lines · Anatoly
Quoted 18 lines · Xuan Ding

Currently, the VFIO subsystem will compact adjacent DMA regions for the purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to be adjacent in memory. Since VFIO implementation on IA platforms also does not allow partial unmapping of memory mapped for DMA, the current DPDK VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API to also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
   doc/guides/rel_notes/deprecation.rst | 3 +++
   1 file changed, 3 insertions(+)
Quoted 17 lines
Quoted 11 lines
doc/guides/rel_notes/deprecation.rst +2 -0
index 76a4abfd6b..272ffa993e 100644
@@ -287,3 +287,6 @@ Deprecation Notices
     reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
     information from the crypto/security operation. This field will be used to
     communicate events such as soft expiry with IPsec in lookaside mode.
+
+  * vfio: the functions `rte_vfio_container_dma_map` and
Quoted 3 lines

rte_vfio_container_dma_unmap
+  will be amended to include page size. This change is targeted for DPDK 21.11.

Quoted 5 lines

Techboard decision was to add a new API, instead of updating existing ones, to not break the apps using this API.

@Xuan, @Anatoly, can you please confirm if this will solve your problem?

I don't think adding a new API is a particularly good solution. The "new" API will be almost exactly as the old one, but adding one parameter. I don't expect code duplication to be an issue, but having two API's that do the same thing seems like it's rife for potential confusion.

If we add a new API, we can then either remove the old API entirely in 22.11 (effectively renaming it), or we remove the new API in 22.11 and rename it back to the old function name. I don't think neither of these is a good solution, as we risk introducing more users for the API that will later change.

I think the pain of updating current software for 21.11 (while keeping compatibility with 20.11 ABI!) is going to happen regardless, and whether we decide to add a "temporary" new API or permanently rename the old one. It's (in my opinion) easier to just bite the bullet and update the function in 21.11.

However, if the tech board feels like adding a new API is a good solution, then okay, but we need to flesh out roadmap a bit better. Do we rename the old API, or do we add a temporary new API?

Signature · 2 lines
Thanks,
Anatoly
#4 lore raw
Quoted 42 lines · Anatoly
Quoted 41 lines · Ferruh Yigit
Quoted 19 lines · Anatoly
Quoted 18 lines · Xuan Ding

Currently, the VFIO subsystem will compact adjacent DMA regions for the purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to be adjacent in memory. Since VFIO implementation on IA platforms also does not allow partial unmapping of memory mapped for DMA, the current DPDK VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API to also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
   doc/guides/rel_notes/deprecation.rst | 3 +++
   1 file changed, 3 insertions(+)
Quoted 17 lines
Quoted 11 lines
doc/guides/rel_notes/deprecation.rst +2 -0
index 76a4abfd6b..272ffa993e 100644
@@ -287,3 +287,6 @@ Deprecation Notices
     reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
     information from the crypto/security operation. This field will be used to
     communicate events such as soft expiry with IPsec in lookaside mode.
+
+  * vfio: the functions `rte_vfio_container_dma_map` and
Quoted 3 lines

rte_vfio_container_dma_unmap
+  will be amended to include page size. This change is targeted for DPDK 21.11.

Techboard decision was to add a new API, instead of updating existing ones, to not break the apps using this API.

Quoted 8 lines
Quoted 2 lines

@Xuan, @Anatoly, can you please confirm if this will solve your problem?

I don't think adding a new API is a particularly good solution. The "new" API will be almost exactly as the old one, but adding one parameter. I don't expect code duplication to be an issue, but having two API's that do the same thing seems like it's rife for potential confusion.

Well, if one API is marked as deprecated, then there will be no confusion for users, since using the wrong one will give a warning pointing to the right one.

Quoted 5 lines

If we add a new API, we can then either remove the old API entirely in 22.11 (effectively renaming it), or we remove the new API in 22.11 and rename it back to the old function name. I don't think neither of these is a good solution, as we risk introducing more users for the API that will later change.

The new API will not be renamed to the old one, since that would break apps using it without proper deprecation process. Removing the old one alone would be the approach to be used, but it would be correctly following the deprecation process and giving users at least 1 year, if no 2, of notice about the change.

Quoted 6 lines

I think the pain of updating current software for 21.11 (while keeping compatibility with 20.11 ABI!) is going to happen regardless, and whether we decide to add a "temporary" new API or permanently rename the old one. It's (in my opinion) easier to just bite the bullet and update the function in 21.11.

I fail to see the issue with adding a new function. Whether we add a new function or add a parameter to the existing one, code will have to change either way. The advantage of the former scheme, adding the new function, is that it shows that we are serious about our ABI/API compatibility process, and are not lax about passing exceptions when other options are available.

Quoted 4 lines

However, if the tech board feels like adding a new API is a good solution, then okay, but we need to flesh out roadmap a bit better. Do we rename the old API, or do we add a temporary new API?

New API added, old API deprecated. In future old API goes away leaving new API as the only option.

/Bruce

#5 lore raw
Quoted 79 lines · Bruce Richardson
Quoted 42 lines · Anatoly
Quoted 41 lines · Ferruh Yigit
Quoted 19 lines · Anatoly
Quoted 18 lines · Xuan Ding

Currently, the VFIO subsystem will compact adjacent DMA regions for the purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to be adjacent in memory. Since VFIO implementation on IA platforms also does not allow partial unmapping of memory mapped for DMA, the current DPDK VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API to also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
    doc/guides/rel_notes/deprecation.rst | 3 +++
    1 file changed, 3 insertions(+)
Quoted 17 lines
Quoted 11 lines
doc/guides/rel_notes/deprecation.rst +2 -0
index 76a4abfd6b..272ffa993e 100644
@@ -287,3 +287,6 @@ Deprecation Notices
      reserved bytes to 2 (from 3), and use 1 byte to indicate warnings and other
      information from the crypto/security operation. This field will be used to
      communicate events such as soft expiry with IPsec in lookaside mode.
+
+  * vfio: the functions `rte_vfio_container_dma_map` and
Quoted 3 lines

rte_vfio_container_dma_unmap
+  will be amended to include page size. This change is targeted for DPDK 21.11.

Techboard decision was to add a new API, instead of updating existing ones, to not break the apps using this API.

Quoted 8 lines
Quoted 2 lines

@Xuan, @Anatoly, can you please confirm if this will solve your problem?

I don't think adding a new API is a particularly good solution. The "new" API will be almost exactly as the old one, but adding one parameter. I don't expect code duplication to be an issue, but having two API's that do the same thing seems like it's rife for potential confusion.

Well, if one API is marked as deprecated, then there will be no confusion for users, since using the wrong one will give a warning pointing to the right one.

Quoted 5 lines

If we add a new API, we can then either remove the old API entirely in 22.11 (effectively renaming it), or we remove the new API in 22.11 and rename it back to the old function name. I don't think neither of these is a good solution, as we risk introducing more users for the API that will later change.

The new API will not be renamed to the old one, since that would break apps using it without proper deprecation process. Removing the old one alone would be the approach to be used, but it would be correctly following the deprecation process and giving users at least 1 year, if no 2, of notice about the change.

Quoted 6 lines

I think the pain of updating current software for 21.11 (while keeping compatibility with 20.11 ABI!) is going to happen regardless, and whether we decide to add a "temporary" new API or permanently rename the old one. It's (in my opinion) easier to just bite the bullet and update the function in 21.11.

I fail to see the issue with adding a new function. Whether we add a new function or add a parameter to the existing one, code will have to change either way. The advantage of the former scheme, adding the new function, is that it shows that we are serious about our ABI/API compatibility process, and are not lax about passing exceptions when other options are available.

Quoted 1 line
Quoted 9 lines
Quoted 3 lines

However, if the tech board feels like adding a new API is a good solution, then okay, but we need to flesh out roadmap a bit better. Do we rename the old API, or do we add a temporary new API?

New API added, old API deprecated. In future old API goes away leaving new API as the only option.

/Bruce

Okay, so it's settled then. I revoke my ack for this patch, and we need a new deprecation notice.

Signature · 2 lines
Thanks,
Anatoly
author
#6 lore raw

Hi,

Quoted 111 lines

-----Original Message-----
From: Burakov, Anatoly <[email protected]>
Sent: Thursday, August 26, 2021 6:15 PM
To: Richardson, Bruce <[email protected]>
Cc: Yigit, Ferruh <[email protected]>; Ding, Xuan <[email protected]>; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Hu, Jiayu <[email protected]>; [email protected]; David Marchand <[email protected]>
Subject: Re: [PATCH] doc: announce change in dma mapping/unmapping

Quoted 5 lines · Bruce Richardson
Quoted 4 lines · Anatoly
Quoted 3 lines · Ferruh Yigit
Quoted 2 lines · Anatoly
Quoted 1 line · Xuan Ding

Currently, the VFIO subsystem will compact adjacent DMA regions for

the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

purposes of saving space in the internal list of mappings. This has a side effect of compacting two separate mappings that just happen to

be

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

adjacent in memory. Since VFIO implementation on IA platforms also

does

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

not allow partial unmapping of memory mapped for DMA, the current

DPDK

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

VFIO implementation will prevent unmapping of accidentally adjacent maps even though it could have been unmapped [1].

The proper fix for this issue is to change the VFIO DMA mapping API

to

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

also include page size, and always map memory page-by-page.

[1] https://mails.dpdk.org/archives/dev/2021-July/213493.html

---
    doc/guides/rel_notes/deprecation.rst | 3 +++
    1 file changed, 3 insertions(+)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 287

Quoted 7 lines
doc/guides/rel_notes/deprecation.rst +0 -0
index 76a4abfd6b..272ffa993e 100644
@@ -287,3 +287,6 @@ Deprecation Notices
      reserved bytes to 2 (from 3), and use 1 byte to indicate warnings

and other

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 288

Quoted 1 line
doc/guides/rel_notes/deprecation.rst +0 -0
      information from the crypto/security operation. This field will be

used to

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

doc/guides/rel_notes/deprecation.rst Deprecation Notices · line 289

Quoted 1 line
doc/guides/rel_notes/deprecation.rst +0 -0
      communicate events such as soft expiry with IPsec in lookaside

mode.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
doc/guides/rel_notes/deprecation.rst +2 -0
+
+  * vfio: the functions `rte_vfio_container_dma_map` and

rte_vfio_container_dma_unmap
+  will be amended to include page size. This change is targeted for

DPDK 21.11.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Techboard decision was to add a new API, instead of updating existing

ones, to

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

not break the apps using this API.

@Xuan, @Anatoly, can you please confirm if this will solve your problem?

I don't think adding a new API is a particularly good solution. The "new" API will be almost exactly as the old one, but adding one parameter. I

don't

Quoted 7 lines
Quoted 3 lines

expect code duplication to be an issue, but having two API's that do the same thing seems like it's rife for potential confusion.

Well, if one API is marked as deprecated, then there will be no confusion for users, since using the wrong one will give a warning pointing to the right one.

Quoted 6 lines
Quoted 5 lines

If we add a new API, we can then either remove the old API entirely in 22.11 (effectively renaming it), or we remove the new API in 22.11 and rename it back to the old function name. I don't think neither of these is a good solution, as we risk introducing more users for the API that will later change.

The new API will not be renamed to the old one, since that would break

apps

Quoted 8 lines

using it without proper deprecation process. Removing the old one alone would be the approach to be used, but it would be correctly following the deprecation process and giving users at least 1 year, if no 2, of notice about the change.

Quoted 3 lines

I think the pain of updating current software for 21.11 (while keeping compatibility with 20.11 ABI!) is going to happen regardless, and whether

we

Quoted 1 line
Quoted 1 line

decide to add a "temporary" new API or permanently rename the old one.

It's

Quoted 5 lines
Quoted 2 lines

(in my opinion) easier to just bite the bullet and update the function in 21.11.

I fail to see the issue with adding a new function. Whether we add a new function or add a parameter to the existing one, code will have to change either way. The advantage of the former scheme, adding the new function,

is

Quoted 6 lines

that it shows that we are serious about our ABI/API compatibility process, and are not lax about passing exceptions when other options are available.

Quoted 3 lines

However, if the tech board feels like adding a new API is a good solution, then okay, but we need to flesh out roadmap a bit better. Do we rename

the

Quoted 3 lines
Quoted 1 line

old API, or do we add a temporary new API?

New API added, old API deprecated. In future old API goes away leaving

Quoted 8 lines

new

Quoted 4 lines

API as the only option.

/Bruce

Okay, so it's settled then. I revoke my ack for this patch, and we need a new deprecation notice.

A new depreciation notice was sent [1], targeting for API change in DPDK-22.02. For the unmapping issue mentioned before, we developed a compromised solution to optimize the partial unmap logic in DPDK-21.11, and it is compatible with current API.

[1] https://mails.dpdk.org/archives/dev/2021-August/217802.html

Thanks for your suggestion and support!

Regards,
Xuan

Quoted 4 lines

--
Thanks,
Anatoly