Skip to contentSkip to conversation

avoid break due to failure to cleanup temporary directory

PATCH Superseded Andre Muezerie 3 participants 14 messages May 5, 2025 – Jun 4, 2025 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover avoid break due to failure to cleanup temporary directory

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows.

The proposed fix adds a mitigation for this problem, and also sets a flag to ignore cleanup errors.

For completeness, the error message is included below.

--- stderr ---
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe

    os.rmdir(path)
    ~~~~~~~~^^^^^^

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc

    _os.unlink(path)
    ~~~~~~~~~~^^^^^^

PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__

    self.cleanup()
    ~~~~~~~~~~~~^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
    cls._rmtree(path, ignore_errors=ignore_errors,
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                repeated=(path == name))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.

Andre Muezerie (1):
buildtools: avoid break due to failure to cleanup temporary directory

 buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

--
2.49.0.vfs.0.2

#1 lore raw

PATCH 1/1 buildtools: avoid break due to failure to cleanup temporary directory

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".

---
 buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
buildtools/gen-pmdinfo-cfile.py +9 -1
index 5fbd51658a..2ddbdb9502 100644
@@ -6,9 +6,10 @@
 import subprocess
 import sys
 import tempfile
+import time
 
 _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
+with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
     paths = []
     for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
                                check=True).stdout.decode().splitlines():
@@ -19,3 +20,10 @@
                            check=True, cwd=temp)
             paths.append(os.path.join(temp, name))
     subprocess.run(pmdinfogen + paths + [output], check=True)
+
+    if os.name == "nt":
+        # Instances have been seen on Windows where the temporary directory fails to get cleaned
+        # up due to ERROR_SHARING_VIOLATION (32).
+        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
+        # failures when the issue is hit despite the mitigation.
+        time.sleep(1)
Signature · 1 line
2.49.0.vfs.0.2
#2 lore raw

Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

Quoted 20 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".

---
 buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
buildtools/gen-pmdinfo-cfile.py +0 -0
index 5fbd51658a..2ddbdb9502 100644

buildtools/gen-pmdinfo-cfile.py line 12

Quoted 8 lines
buildtools/gen-pmdinfo-cfile.py +2 -1
@@ -6,9 +6,10 @@
 import subprocess
 import sys
 import tempfile
+import time
 
 _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
+with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:

This built fine on my machine but failed on the CI.

According to doc\guides\linux_gsg\sys_reqs.rst, currently the minimum required Python version is 3.6.
However, parameter ignore_cleanup_errors parameter was added in 3.10
(https://docs.python.org/3/library/tempfile.html)

Is there something that blocks us from requiring 3.10 at minimum? Is this something that could be done for the next release?

Quoted 16 lines
buildtools/gen-pmdinfo-cfile.py +7 -0
     paths = []
     for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
                                check=True).stdout.decode().splitlines():
@@ -19,3 +20,10 @@
                            check=True, cwd=temp)
             paths.append(os.path.join(temp, name))
     subprocess.run(pmdinfogen + paths + [output], check=True)
+
+    if os.name == "nt":
+        # Instances have been seen on Windows where the temporary directory fails to get cleaned
+        # up due to ERROR_SHARING_VIOLATION (32).
+        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
+        # failures when the issue is hit despite the mitigation.
+        time.sleep(1)
Signature · 1 line
2.49.0.vfs.0.2
#3 lore raw

Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

In reply to Andre Muezerie

Hi Andre,

Quoted 32 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".

---
  buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
buildtools/gen-pmdinfo-cfile.py +2 -1
index 5fbd51658a..2ddbdb9502 100644
@@ -6,9 +6,10 @@
  import subprocess
  import sys
  import tempfile
+import time
 
  _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
+with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
      paths = []
      for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
                                 check=True).stdout.decode().splitlines():

buildtools/gen-pmdinfo-cfile.py line 29

Quoted 10 lines
buildtools/gen-pmdinfo-cfile.py +7 -0
@@ -19,3 +20,10 @@
                             check=True, cwd=temp)
              paths.append(os.path.join(temp, name))
      subprocess.run(pmdinfogen + paths + [output], check=True)
+
+    if os.name == "nt":
+        # Instances have been seen on Windows where the temporary directory fails to get cleaned
+        # up due to ERROR_SHARING_VIOLATION (32).
+        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
+        # failures when the issue is hit despite the mitigation.
+        time.sleep(1)

Not fond of this timing hack at all.
Meson 0.51 (minimum requirement is 0.57) introduced |@PRIVATE_DIR@ for custom_target() [1]. Can we use it to get rid of tempfile.TemporaryDirectory?

drivers/meson.build:
    custom_target(command: [pmdinfo, '@PRIVATE_DIR@', ...], ...)

buildtools/gen-pmdinfo-cfile.py:
    _, temp, ..., *pmdinfogen = sys.argv
    for name in subprocess.run([ar, "t", ...):
       ...
       subprocess.run([ar, "x", ...], cwd=temp)
       ...
|

|[1]: https://mesonbuild.com/Reference-manual_functions.html#custom_target
|

#4 lore raw

Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

Quoted 61 lines · Dmitry Kozlyuk

Hi Andre,

Quoted 31 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".

---
 buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
buildtools/gen-pmdinfo-cfile.py +2 -1
index 5fbd51658a..2ddbdb9502 100644
@@ -6,9 +6,10 @@
 import subprocess
 import sys
 import tempfile
+import time
 _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
+with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
     paths = []
     for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
                                check=True).stdout.decode().splitlines():

buildtools/gen-pmdinfo-cfile.py line 29

Quoted 10 lines
buildtools/gen-pmdinfo-cfile.py +7 -0
@@ -19,3 +20,10 @@
                            check=True, cwd=temp)
             paths.append(os.path.join(temp, name))
     subprocess.run(pmdinfogen + paths + [output], check=True)
+
+    if os.name == "nt":
+        # Instances have been seen on Windows where the temporary directory fails to get cleaned
+        # up due to ERROR_SHARING_VIOLATION (32).
+        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
+        # failures when the issue is hit despite the mitigation.
+        time.sleep(1)

Not fond of this timing hack at all.
Meson 0.51 (minimum requirement is 0.57) introduced |@PRIVATE_DIR@ for custom_target() [1].
Can we use it to get rid of tempfile.TemporaryDirectory?

drivers/meson.build:
    custom_target(command: [pmdinfo, '@PRIVATE_DIR@', ...], ...)

buildtools/gen-pmdinfo-cfile.py:
    _, temp, ..., *pmdinfogen = sys.argv
    for name in subprocess.run([ar, "t", ...):
       ...
       subprocess.run([ar, "x", ...], cwd=temp)
       ...
|

Quoted 2 lines

|[1]: https://mesonbuild.com/Reference-manual_functions.html#custom_target
|

Yes, I like your suggestion. I'll send out a V2 with the change you suggested. Thanks for providing all the details about custom_target() and @PRIVATE_DIR@.

#5 lore raw

v2 cover avoid break due to failure to cleanup temporary directory

In reply to Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows.

The proposed fix adds a mitigation for this problem, and also sets a flag to ignore cleanup errors.

For completeness, the error message is included below.

--- stderr ---
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe

    os.rmdir(path)
    ~~~~~~~~^^^^^^

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc

    _os.unlink(path)
    ~~~~~~~~~~^^^^^^

PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__

    self.cleanup()
    ~~~~~~~~~~~~^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
    cls._rmtree(path, ignore_errors=ignore_errors,
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                repeated=(path == name))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.

v2:
- removed use of Python's tempfile.TemporaryDirectory and used '@PRIVATE_DIR@' in meson's custom_target command instead.

Andre Muezerie (1):
buildtools: avoid break due to failure to cleanup temporary directory

 buildtools/gen-pmdinfo-cfile.py | 24 +++++++++++-------------
 drivers/meson.build             |  2 +-
 2 files changed, 12 insertions(+), 14 deletions(-)

--
2.49.0.vfs.0.2

#6 lore raw

PATCH v2 1/1 buildtools: avoid break due to failure to cleanup temporary directory

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".

---
 buildtools/gen-pmdinfo-cfile.py | 24 +++++++++++-------------
 drivers/meson.build             |  2 +-
 2 files changed, 12 insertions(+), 14 deletions(-)
buildtools/gen-pmdinfo-cfile.py +11 -13
index 5fbd51658a..cf8881297f 100644
@@ -5,17 +5,15 @@
 import os
 import subprocess
 import sys
-import tempfile
 
-_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
-    paths = []
-    for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
-                               check=True).stdout.decode().splitlines():
-        if os.path.exists(name):
-            paths.append(name)
-        else:
-            subprocess.run([ar, "x", os.path.abspath(archive), name],
-                           check=True, cwd=temp)
-            paths.append(os.path.join(temp, name))
-    subprocess.run(pmdinfogen + paths + [output], check=True)
+_, tmp_root, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv
+paths = []
+for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
+                            check=True).stdout.decode().splitlines():
+    if os.path.exists(name):
+        paths.append(name)
+    else:
+        subprocess.run([ar, "x", os.path.abspath(archive), name],
+                        check=True, cwd=tmp_dir)
+        paths.append(os.path.join(tmp_dir, name))
+subprocess.run(pmdinfogen + paths + [output], check=True)
drivers/meson.build +1 -1
index caee1e3b79..8aad099c40 100644
@@ -274,7 +274,7 @@ foreach subpath:subdirs
                 c_args: cflags)
         objs += tmp_lib.extract_all_objects(recursive: true)
         sources_pmd_info = custom_target(out_filename,
-                command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
+                command: [pmdinfo, '@PRIVATE_DIR@', tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
                 output: out_filename,
                 depends: [tmp_lib])
 
Signature · 1 line
2.49.0.vfs.0.2
#7 lore raw

Re: [PATCH v2 1/1] buildtools: avoid break due to failure to cleanup temporary directory

Quoted 9 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".

This paragraph needs an update.

Quoted 11 lines
buildtools/gen-pmdinfo-cfile.py +0 -4
@@ -5,17 +5,15 @@
  import os
  import subprocess
  import sys
-import tempfile
 
-_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
-    paths = []

buildtools/gen-pmdinfo-cfile.py line 9

Quoted 10 lines
buildtools/gen-pmdinfo-cfile.py +1 -9
-    for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
-                               check=True).stdout.decode().splitlines():
-        if os.path.exists(name):
-            paths.append(name)
-        else:
-            subprocess.run([ar, "x", os.path.abspath(archive), name],
-                           check=True, cwd=temp)
-            paths.append(os.path.join(temp, name))
-    subprocess.run(pmdinfogen + paths + [output], check=True)
+_, tmp_root, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv

No need for tmp_root anymore: it was required to put temporary files inside the build directory, but @PRIVATE_DIR@ is already there.

#8 lore raw

v3 cover avoid break due to failure to cleanup temporary directory

In reply to Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.

For completeness, the error message is included below.

--- stderr ---
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe

    os.rmdir(path)
    ~~~~~~~~^^^^^^

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc

    _os.unlink(path)
    ~~~~~~~~~~^^^^^^

PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__

    self.cleanup()
    ~~~~~~~~~~~~^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
    cls._rmtree(path, ignore_errors=ignore_errors,
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                repeated=(path == name))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.

v3:
- removed parameter tmp_root from gen-pmdinfo-cfile.py
- modified gen-pmdinfo-cfile.py to create '@PRIVATE_DIR@' when it does not exist.

v2:
- removed use of Python's tempfile.TemporaryDirectory and used '@PRIVATE_DIR@' in meson's custom_target command instead.

Andre Muezerie (1):
buildtools: avoid break due to failure to cleanup temporary directory

 buildtools/gen-pmdinfo-cfile.py | 26 +++++++++++++-------------
 buildtools/meson.build          |  2 +-
 drivers/meson.build             |  2 +-
 3 files changed, 15 insertions(+), 15 deletions(-)

--
2.49.0.vfs.0.2

#9 lore raw

PATCH v3 1/1 buildtools: avoid break due to failure to cleanup temporary directory

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.

---
 buildtools/gen-pmdinfo-cfile.py | 26 +++++++++++++-------------
 buildtools/meson.build          |  2 +-
 drivers/meson.build             |  2 +-
 3 files changed, 15 insertions(+), 15 deletions(-)
buildtools/gen-pmdinfo-cfile.py +13 -13
index 5fbd51658a..8ffc192c40 100644
@@ -5,17 +5,17 @@
 import os
 import subprocess
 import sys
-import tempfile
 
-_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
-    paths = []
-    for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
-                               check=True).stdout.decode().splitlines():
-        if os.path.exists(name):
-            paths.append(name)
-        else:
-            subprocess.run([ar, "x", os.path.abspath(archive), name],
-                           check=True, cwd=temp)
-            paths.append(os.path.join(temp, name))
-    subprocess.run(pmdinfogen + paths + [output], check=True)
+_, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv
+paths = []
+for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
+                            check=True).stdout.decode().splitlines():
+    if os.path.exists(name):
+        paths.append(name)
+    else:
+        if not os.path.exists(tmp_dir):
+            os.makedirs(tmp_dir)
+        subprocess.run([ar, "x", os.path.abspath(archive), name],
+                        check=True, cwd=tmp_dir)
+        paths.append(os.path.join(tmp_dir, name))
+subprocess.run(pmdinfogen + paths + [output], check=True)
buildtools/meson.build +1 -1
index 1cd1ce02fd..e7c53c9a84 100644
@@ -34,7 +34,7 @@ install_data([
         install_dir: 'bin')
 
 # select library and object file format
-pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
+pmdinfo = py3 + files('gen-pmdinfo-cfile.py')
 pmdinfogen = py3 + files('pmdinfogen.py')
 if host_machine.system() == 'windows'
     if cc.get_id() == 'gcc'
drivers/meson.build +1 -1
index caee1e3b79..8aad099c40 100644
@@ -274,7 +274,7 @@ foreach subpath:subdirs
                 c_args: cflags)
         objs += tmp_lib.extract_all_objects(recursive: true)
         sources_pmd_info = custom_target(out_filename,
-                command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
+                command: [pmdinfo, '@PRIVATE_DIR@', tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
                 output: out_filename,
                 depends: [tmp_lib])
 
Signature · 1 line
2.49.0.vfs.0.2
#11 lore raw

Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory

In reply to Andre Muezerie

Hello Andre,

Quoted 11 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.

How does this change interact (except a merge conflict) with the other series that adds pmdinfo for MSVC?
https://patchwork.dpdk.org/project/dpdk/list/?series=35065&state=%2A&archive=both

Signature · 1 line
David Marchand
#12 lore raw

Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory

Quoted 11 lines · David Marchand

Hello Andre,

Quoted 7 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

Quoted 9 lines
Quoted 4 lines

This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.

How does this change interact (except a merge conflict) with the other series that adds pmdinfo for MSVC?
https://patchwork.dpdk.org/project/dpdk/list/?series=35065&state=%2A&archive=both

I suggest applying this patch as early as possible. I can rebase the other patch which will merge conflict with it after if needed.

On my dev machine I'm building the drivers folder, and that doesn't succeed without this patch. I want to enable that on the official branch as well, but can't until this patch is merged.

Quoted 3 lines
Signature · 2 lines
-- 
David Marchand
#13 lore raw

Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory

In reply to Andre Muezerie
Quoted 11 lines · Andre Muezerie

When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.

Applied, thanks.

Signature · 1 line
David Marchand