avoid break due to failure to cleanup temporary directory
cover avoid break due to failure to cleanup temporary directory
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).
Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows.
The proposed fix adds a mitigation for this problem, and also sets a flag to ignore cleanup errors.
For completeness, the error message is included below.
--- stderr ---
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
os.rmdir(path)
~~~~~~~~^^^^^^PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc
_os.unlink(path)
~~~~~~~~~~^^^^^^PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__
self.cleanup()
~~~~~~~~~~~~^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
_shutil.rmtree(name, onexc=onexc)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
return _rmtree_unsafe(path, onexc)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
onexc(os.rmdir, path, err)
~~~~~^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
cls._rmtree(path, ignore_errors=ignore_errors,
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
repeated=(path == name))
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
_shutil.rmtree(name, onexc=onexc)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
return _rmtree_unsafe(path, onexc)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
onexc(os.rmdir, path, err)
~~~~~^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
os.rmdir(path)
~~~~~~~~^^^^^^PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.
Andre Muezerie (1):
buildtools: avoid break due to failure to cleanup temporary directory
buildtools/gen-pmdinfo-cfile.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
--
2.49.0.vfs.0.2
PATCH 1/1 buildtools: avoid break due to failure to cleanup temporary directory
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).
Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".
--- buildtools/gen-pmdinfo-cfile.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
buildtools/gen-pmdinfo-cfile.py +9 -1
@@ -6,9 +6,10 @@ import subprocess import sys import tempfile +import time _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: +with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp: paths = [] for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, check=True).stdout.decode().splitlines():
@@ -19,3 +20,10 @@ check=True, cwd=temp) paths.append(os.path.join(temp, name)) subprocess.run(pmdinfogen + paths + [output], check=True) + + if os.name == "nt": + # Instances have been seen on Windows where the temporary directory fails to get cleaned + # up due to ERROR_SHARING_VIOLATION (32). + # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids + # failures when the issue is hit despite the mitigation. + time.sleep(1)
Signature · 1 line
2.49.0.vfs.0.2
Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory
Quoted 20 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "
ignore_cleanup_errors" is set to "True".Signed-off-by Andre Muezerie <[email protected]>--- buildtools/gen-pmdinfo-cfile.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)buildtools/gen-pmdinfo-cfile.py +0
-0
buildtools/gen-pmdinfo-cfile.py line 12
Quoted 8 lines
buildtools/gen-pmdinfo-cfile.py +2
-1@@ -6,9 +6,10 @@ import subprocess import sys import tempfile +import time _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: +with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
This built fine on my machine but failed on the CI.
According to doc\guides\linux_gsg\sys_reqs.rst, currently the minimum required Python version is 3.6.
However, parameter ignore_cleanup_errors parameter was added in 3.10
(https://docs.python.org/3/library/tempfile.html)
Is there something that blocks us from requiring 3.10 at minimum? Is this something that could be done for the next release?
Quoted 16 lines
buildtools/gen-pmdinfo-cfile.py +7
-0paths = [] for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, check=True).stdout.decode().splitlines():@@ -19,3 +20,10 @@ check=True, cwd=temp) paths.append(os.path.join(temp, name)) subprocess.run(pmdinfogen + paths + [output], check=True) + + if os.name == "nt": + # Instances have been seen on Windows where the temporary directory fails to get cleaned + # up due to ERROR_SHARING_VIOLATION (32). + # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids + # failures when the issue is hit despite the mitigation. + time.sleep(1)Signature · 1 line
2.49.0.vfs.0.2
Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory
In reply to Andre MuezerieHi Andre,
Quoted 32 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "
ignore_cleanup_errors" is set to "True".Signed-off-by Andre Muezerie <[email protected]>--- buildtools/gen-pmdinfo-cfile.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)buildtools/gen-pmdinfo-cfile.py +2
-1@@ -6,9 +6,10 @@ import subprocess import sys import tempfile +import time _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: +with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp: paths = [] for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, check=True).stdout.decode().splitlines():
buildtools/gen-pmdinfo-cfile.py line 29
Quoted 10 lines
buildtools/gen-pmdinfo-cfile.py +7
-0@@ -19,3 +20,10 @@ check=True, cwd=temp) paths.append(os.path.join(temp, name)) subprocess.run(pmdinfogen + paths + [output], check=True) + + if os.name == "nt": + # Instances have been seen on Windows where the temporary directory fails to get cleaned + # up due to ERROR_SHARING_VIOLATION (32). + # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids + # failures when the issue is hit despite the mitigation. + time.sleep(1)
Not fond of this timing hack at all.
Meson 0.51 (minimum requirement is 0.57) introduced |@PRIVATE_DIR@ for custom_target() [1]. Can we use it to get rid of tempfile.TemporaryDirectory?
drivers/meson.build:
custom_target(command: [pmdinfo, '@PRIVATE_DIR@', ...], ...)
buildtools/gen-pmdinfo-cfile.py:
_, temp, ..., *pmdinfogen = sys.argv
for name in subprocess.run([ar, "t", ...):
...
subprocess.run([ar, "x", ...], cwd=temp)
...
|
|[1]: https://mesonbuild.com/Reference-manual_functions.html#custom_target
|
Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory
Quoted 61 lines · Dmitry Kozlyuk
Hi Andre,
Quoted 31 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "
ignore_cleanup_errors" is set to "True".Signed-off-by Andre Muezerie <[email protected]>--- buildtools/gen-pmdinfo-cfile.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)buildtools/gen-pmdinfo-cfile.py +2
-1@@ -6,9 +6,10 @@ import subprocess import sys import tempfile +import time _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: +with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp: paths = [] for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, check=True).stdout.decode().splitlines():buildtools/gen-pmdinfo-cfile.py line 29
Quoted 10 lines
buildtools/gen-pmdinfo-cfile.py +7
-0@@ -19,3 +20,10 @@ check=True, cwd=temp) paths.append(os.path.join(temp, name)) subprocess.run(pmdinfogen + paths + [output], check=True) + + if os.name == "nt": + # Instances have been seen on Windows where the temporary directory fails to get cleaned + # up due to ERROR_SHARING_VIOLATION (32). + # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids + # failures when the issue is hit despite the mitigation. + time.sleep(1)Not fond of this timing hack at all.
Meson 0.51 (minimum requirement is 0.57) introduced |@PRIVATE_DIR@ forcustom_target()[1].
Can we use it to get rid of tempfile.TemporaryDirectory?
drivers/meson.build:
custom_target(command: [pmdinfo, '@PRIVATE_DIR@', ...], ...)
buildtools/gen-pmdinfo-cfile.py:
_, temp, ..., *pmdinfogen = sys.argv
for name insubprocess.run([ar, "t", ...):
...
subprocess.run([ar, "x", ...], cwd=temp)
...
|
Quoted 2 lines
|[1]: https://mesonbuild.com/Reference-manual_functions.html#custom_target
|
Yes, I like your suggestion. I'll send out a V2 with the change you suggested. Thanks for providing all the details about custom_target() and @PRIVATE_DIR@.
v2 cover avoid break due to failure to cleanup temporary directory
In reply to Andre MuezerieWhen compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).
Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows.
The proposed fix adds a mitigation for this problem, and also sets a flag to ignore cleanup errors.
For completeness, the error message is included below.
--- stderr ---
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
os.rmdir(path)
~~~~~~~~^^^^^^PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc
_os.unlink(path)
~~~~~~~~~~^^^^^^PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__
self.cleanup()
~~~~~~~~~~~~^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
_shutil.rmtree(name, onexc=onexc)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
return _rmtree_unsafe(path, onexc)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
onexc(os.rmdir, path, err)
~~~~~^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
cls._rmtree(path, ignore_errors=ignore_errors,
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
repeated=(path == name))
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
_shutil.rmtree(name, onexc=onexc)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
return _rmtree_unsafe(path, onexc)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
onexc(os.rmdir, path, err)
~~~~~^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
os.rmdir(path)
~~~~~~~~^^^^^^PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.
v2:
- removed use of Python's tempfile.TemporaryDirectory and used '@PRIVATE_DIR@' in meson's custom_target command instead.
Andre Muezerie (1):
buildtools: avoid break due to failure to cleanup temporary directory
buildtools/gen-pmdinfo-cfile.py | 24 +++++++++++------------- drivers/meson.build | 2 +- 2 files changed, 12 insertions(+), 14 deletions(-)
--
2.49.0.vfs.0.2
PATCH v2 1/1 buildtools: avoid break due to failure to cleanup temporary directory
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).
Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "ignore_cleanup_errors" is set to "True".
--- buildtools/gen-pmdinfo-cfile.py | 24 +++++++++++------------- drivers/meson.build | 2 +- 2 files changed, 12 insertions(+), 14 deletions(-)
buildtools/gen-pmdinfo-cfile.py +11 -13
@@ -5,17 +5,15 @@ import os import subprocess import sys -import tempfile -_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: - paths = [] - for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, - check=True).stdout.decode().splitlines(): - if os.path.exists(name): - paths.append(name) - else: - subprocess.run([ar, "x", os.path.abspath(archive), name], - check=True, cwd=temp) - paths.append(os.path.join(temp, name)) - subprocess.run(pmdinfogen + paths + [output], check=True) +_, tmp_root, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv +paths = [] +for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, + check=True).stdout.decode().splitlines(): + if os.path.exists(name): + paths.append(name) + else: + subprocess.run([ar, "x", os.path.abspath(archive), name], + check=True, cwd=tmp_dir) + paths.append(os.path.join(tmp_dir, name)) +subprocess.run(pmdinfogen + paths + [output], check=True)
drivers/meson.build +1 -1
@@ -274,7 +274,7 @@ foreach subpath:subdirs c_args: cflags) objs += tmp_lib.extract_all_objects(recursive: true) sources_pmd_info = custom_target(out_filename, - command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], + command: [pmdinfo, '@PRIVATE_DIR@', tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, depends: [tmp_lib])
Signature · 1 line
2.49.0.vfs.0.2
Re: [PATCH v2 1/1] buildtools: avoid break due to failure to cleanup temporary directory
Quoted 9 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).Code inspection did not reveal problems with the DPDK code and scripts, and this issue was only seen on Windows. Adding a 1 second sleep before cleaning up the temporary directory seems to be effective, but to guarantee that this break does not happen anymore, flag "
ignore_cleanup_errors" is set to "True".
This paragraph needs an update.
Quoted 11 lines
buildtools/gen-pmdinfo-cfile.py +0
-4@@ -5,17 +5,15 @@ import os import subprocess import sys -import tempfile -_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: - paths = []
buildtools/gen-pmdinfo-cfile.py line 9
Quoted 10 lines
buildtools/gen-pmdinfo-cfile.py +1
-9- for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, - check=True).stdout.decode().splitlines(): - if os.path.exists(name): - paths.append(name) - else: - subprocess.run([ar, "x", os.path.abspath(archive), name], - check=True, cwd=temp) - paths.append(os.path.join(temp, name)) - subprocess.run(pmdinfogen + paths + [output], check=True) +_, tmp_root, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv
No need for tmp_root anymore: it was required to put temporary files inside the build directory, but @PRIVATE_DIR@ is already there.
v3 cover avoid break due to failure to cleanup temporary directory
In reply to Andre MuezerieWhen compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).
This issue was not seen on operating systems other than Windows.
This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.
For completeness, the error message is included below.
--- stderr ---
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
os.rmdir(path)
~~~~~~~~^^^^^^PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc
_os.unlink(path)
~~~~~~~~~~^^^^^^PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__
self.cleanup()
~~~~~~~~~~~~^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
_shutil.rmtree(name, onexc=onexc)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
return _rmtree_unsafe(path, onexc)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
onexc(os.rmdir, path, err)
~~~~~^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
cls._rmtree(path, ignore_errors=ignore_errors,
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
repeated=(path == name))
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
_shutil.rmtree(name, onexc=onexc)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
return _rmtree_unsafe(path, onexc)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
onexc(os.rmdir, path, err)
~~~~~^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
os.rmdir(path)
~~~~~~~~^^^^^^PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'
[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.
v3:
- removed parameter tmp_root from gen-pmdinfo-cfile.py
- modified gen-pmdinfo-cfile.py to create '@PRIVATE_DIR@' when it does not exist.
v2:
- removed use of Python's tempfile.TemporaryDirectory and used '@PRIVATE_DIR@' in meson's custom_target command instead.
Andre Muezerie (1):
buildtools: avoid break due to failure to cleanup temporary directory
buildtools/gen-pmdinfo-cfile.py | 26 +++++++++++++------------- buildtools/meson.build | 2 +- drivers/meson.build | 2 +- 3 files changed, 15 insertions(+), 15 deletions(-)
--
2.49.0.vfs.0.2
PATCH v3 1/1 buildtools: avoid break due to failure to cleanup temporary directory
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to ERROR_SHARING_VIOLATION (32).
This issue was not seen on operating systems other than Windows.
This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.
--- buildtools/gen-pmdinfo-cfile.py | 26 +++++++++++++------------- buildtools/meson.build | 2 +- drivers/meson.build | 2 +- 3 files changed, 15 insertions(+), 15 deletions(-)
buildtools/gen-pmdinfo-cfile.py +13 -13
@@ -5,17 +5,17 @@ import os import subprocess import sys -import tempfile -_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: - paths = [] - for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, - check=True).stdout.decode().splitlines(): - if os.path.exists(name): - paths.append(name) - else: - subprocess.run([ar, "x", os.path.abspath(archive), name], - check=True, cwd=temp) - paths.append(os.path.join(temp, name)) - subprocess.run(pmdinfogen + paths + [output], check=True) +_, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv +paths = [] +for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, + check=True).stdout.decode().splitlines(): + if os.path.exists(name): + paths.append(name) + else: + if not os.path.exists(tmp_dir): + os.makedirs(tmp_dir) + subprocess.run([ar, "x", os.path.abspath(archive), name], + check=True, cwd=tmp_dir) + paths.append(os.path.join(tmp_dir, name)) +subprocess.run(pmdinfogen + paths + [output], check=True)
buildtools/meson.build +1 -1
@@ -34,7 +34,7 @@ install_data([ install_dir: 'bin') # select library and object file format -pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()] +pmdinfo = py3 + files('gen-pmdinfo-cfile.py') pmdinfogen = py3 + files('pmdinfogen.py') if host_machine.system() == 'windows' if cc.get_id() == 'gcc'
drivers/meson.build +1 -1
@@ -274,7 +274,7 @@ foreach subpath:subdirs c_args: cflags) objs += tmp_lib.extract_all_objects(recursive: true) sources_pmd_info = custom_target(out_filename, - command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], + command: [pmdinfo, '@PRIVATE_DIR@', tmp_lib.full_path(), '@OUTPUT@', pmdinfogen], output: out_filename, depends: [tmp_lib])
Signature · 1 line
2.49.0.vfs.0.2
Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory
Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory
In reply to Andre MuezerieHello Andre,
Quoted 11 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).This issue was not seen on operating systems other than Windows.
This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.
Signed-off-by Andre Muezerie <[email protected]>
How does this change interact (except a merge conflict) with the other series that adds pmdinfo for MSVC?
https://patchwork.dpdk.org/project/dpdk/list/?series=35065&state=%2A&archive=both
Signature · 1 line
David Marchand
Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory
Quoted 11 lines · David Marchand
Hello Andre,
Quoted 7 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).This issue was not seen on operating systems other than Windows.
Quoted 9 lines
Quoted 4 lines
This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.
Signed-off-by Andre Muezerie <[email protected]>How does this change interact (except a merge conflict) with the other series that adds pmdinfo for MSVC?
https://patchwork.dpdk.org/project/dpdk/list/?series=35065&state=%2A&archive=both
I suggest applying this patch as early as possible. I can rebase the other patch which will merge conflict with it after if needed.
On my dev machine I'm building the drivers folder, and that doesn't succeed without this patch. I want to enable that on the official branch as well, but can't until this patch is merged.
Quoted 3 lines
Signature · 2 lines
-- David Marchand
David Marchand
AppliedRe: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory
In reply to Andre MuezerieQuoted 11 lines · Andre Muezerie
When compiling drivers on Windows, instances have been seen where a temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION(32).This issue was not seen on operating systems other than Windows.
This patch eliminates the use of tempfile.TemporaryDirectory, which was triggering the issue.
Signed-off-by Andre Muezerie <[email protected]>
Applied, thanks.
Signature · 1 line
David Marchand