tests: CUDA DMA-BUF resources leave exported descriptors open
Hello,
At rdma-core master a70151dfec390eb5f7fc5b266f51d601da2a7778, the CUDA DMA-BUF test resources obtain an FD from cuMemGetHandleForAddressRange but do not close it after registration. This affects DmabufCudaRes, MRExDmabufCudaRes, MRExDmabufDmaHCudaRes, and Mlx5DmabufCudaRes. It occurs on successful registration and on failures or skips after the FD is obtained. The integer FD passed to the pyverbs MR constructor is caller-owned.
The following uses a real process FD and a mocked constructor to exercise the first resource method without requiring a CUDA GPU or RDMA device. Run it from a checkout with an in-place pyverbs build:
```sh
PYTHONPATH=build/python:tests python3 - <<'PY'
import errno
import os
import sys
from types import SimpleNamespace
from unittest.mock import patch
sys.path.append(sys.path.pop(0)) # Prefer the built pyverbs over the source directory.
import tests.test_cuda_dmabuf as test
fd = os.open('/dev/null', os.O_RDONLY)
res = SimpleNamespace(pd=object(), msg_size=64, mr_access=1,
dmabuf_cuda_init=lambda: fd)
try:
with patch.object(test, 'DmaBufMR', return_value=object()):
test.DmabufCudaRes.create_mr(res)
try:
os.fstat(fd)
except OSError as exc:
if exc.errno != errno.EBADF:
raise
print('descriptor closed')
else:
print('descriptor still open')
finally:
try:
os.close(fd)
except OSError as exc:
if exc.errno != errno.EBADF:
raise
PY
```Observed: descriptor still open. Expected: descriptor closed after registration returns, including on error. A proposed fix closes it in a finally block around registration and, for the DMAHandle path, around handle creation as well. On Linux the RDMA DMA-BUF registration path acquires its own buffer reference.
I reproduced the behavior on Linux aarch64 with Python 3.12 and a local rdma-core/pyverbs build. Mock-based FD tests cover the four registration methods, unsupported and error paths, and DMAHandle creation failure; they fail on the base and pass with the fix. No CUDA or RDMA traffic was run for this comparison.
Quoted 48 lines · Dezhen Lu
Hello,
At rdma-core master
a70151dfec390eb5f7fc5b266f51d601da2a7778, the CUDA DMA-BUF test resources obtain an FD from cuMemGetHandleForAddressRange but do not close it after registration. This affects DmabufCudaRes, MRExDmabufCudaRes, MRExDmabufDmaHCudaRes, and Mlx5DmabufCudaRes. It occurs on successful registration and on failures or skips after the FD is obtained. The integer FD passed to the pyverbs MR constructor is caller-owned.The following uses a real process FD and a mocked constructor to exercise the first resource method without requiring a CUDA GPU or RDMA device. Run it from a checkout with an in-place pyverbs build:
```sh PYTHONPATH=build/python:tests python3 - <<'PY' import errno import os import sys from types import SimpleNamespace from unittest.mock import patch sys.path.append(sys.path.pop(0)) # Prefer the built pyverbs over the source directory. import tests.test_cuda_dmabuf as test fd = os.open('/dev/null', os.O_RDONLY) res = SimpleNamespace(pd=object(), msg_size=64, mr_access=1, dmabuf_cuda_init=lambda: fd) try: with patch.object(test, 'DmaBufMR', return_value=object()): test.DmabufCudaRes.create_mr(res) try: os.fstat(fd) except OSError as exc: if exc.errno != errno.EBADF: raise print('descriptor closed') else: print('descriptor still open') finally: try: os.close(fd) except OSError as exc: if exc.errno != errno.EBADF: raise PY ```
Quoted 5 lines
Observed: descriptor still open. Expected: descriptor closed after registration returns, including on error. A proposed fix closes it in a finally block around registration and, for the DMAHandle path, around handle creation as well. On Linux the RDMA DMA-BUF registration path acquires its own buffer reference.
Please open PR against rdma-core with your fix for both test reports.
Thanks