Skip to contentSkip to conversation

tests: CUDA DMA-BUF resources leave exported descriptors open

DISCUSSION Dezhen Lu 2 participants 2 messages Nov 4 – Nov 9 InfiniBand

lore
Share & tools
lore

Dezhen Lu

author
#0 lore raw

Hello,

At rdma-core master a70151dfec390eb5f7fc5b266f51d601da2a7778, the CUDA DMA-BUF test resources obtain an FD from cuMemGetHandleForAddressRange but do not close it after registration. This affects DmabufCudaRes, MRExDmabufCudaRes, MRExDmabufDmaHCudaRes, and Mlx5DmabufCudaRes. It occurs on successful registration and on failures or skips after the FD is obtained. The integer FD passed to the pyverbs MR constructor is caller-owned.

The following uses a real process FD and a mocked constructor to exercise the first resource method without requiring a CUDA GPU or RDMA device. Run it from a checkout with an in-place pyverbs build:

```sh
PYTHONPATH=build/python:tests python3 - <<'PY'
import errno
import os
import sys
from types import SimpleNamespace
from unittest.mock import patch

sys.path.append(sys.path.pop(0))  # Prefer the built pyverbs over the source directory.
import tests.test_cuda_dmabuf as test

fd = os.open('/dev/null', os.O_RDONLY)
res = SimpleNamespace(pd=object(), msg_size=64, mr_access=1,
                      dmabuf_cuda_init=lambda: fd)
try:
    with patch.object(test, 'DmaBufMR', return_value=object()):
        test.DmabufCudaRes.create_mr(res)
    try:
        os.fstat(fd)
    except OSError as exc:
        if exc.errno != errno.EBADF:
            raise
        print('descriptor closed')
    else:
        print('descriptor still open')
finally:
    try:
        os.close(fd)
    except OSError as exc:
        if exc.errno != errno.EBADF:
            raise
PY
```

Observed: descriptor still open. Expected: descriptor closed after registration returns, including on error. A proposed fix closes it in a finally block around registration and, for the DMAHandle path, around handle creation as well. On Linux the RDMA DMA-BUF registration path acquires its own buffer reference.

I reproduced the behavior on Linux aarch64 with Python 3.12 and a local rdma-core/pyverbs build. Mock-based FD tests cover the four registration methods, unsupported and error paths, and DMAHandle creation failure; they fail on the base and pass with the fix. No CUDA or RDMA traffic was run for this comparison.

#1 lore raw
Quoted 48 lines · Dezhen Lu

Hello,

At rdma-core master a70151dfec390eb5f7fc5b266f51d601da2a7778, the CUDA DMA-BUF test resources obtain an FD from cuMemGetHandleForAddressRange but do not close it after registration. This affects DmabufCudaRes, MRExDmabufCudaRes, MRExDmabufDmaHCudaRes, and Mlx5DmabufCudaRes. It occurs on successful registration and on failures or skips after the FD is obtained. The integer FD passed to the pyverbs MR constructor is caller-owned.

The following uses a real process FD and a mocked constructor to exercise the first resource method without requiring a CUDA GPU or RDMA device. Run it from a checkout with an in-place pyverbs build:

```sh
PYTHONPATH=build/python:tests python3 - <<'PY'
import errno
import os
import sys
from types import SimpleNamespace
from unittest.mock import patch

sys.path.append(sys.path.pop(0))  # Prefer the built pyverbs over the source directory.
import tests.test_cuda_dmabuf as test

fd = os.open('/dev/null', os.O_RDONLY)
res = SimpleNamespace(pd=object(), msg_size=64, mr_access=1,
                      dmabuf_cuda_init=lambda: fd)
try:
    with patch.object(test, 'DmaBufMR', return_value=object()):
        test.DmabufCudaRes.create_mr(res)
    try:
        os.fstat(fd)
    except OSError as exc:
        if exc.errno != errno.EBADF:
            raise
        print('descriptor closed')
    else:
        print('descriptor still open')
finally:
    try:
        os.close(fd)
    except OSError as exc:
        if exc.errno != errno.EBADF:
            raise
PY
```
Quoted 5 lines

Observed: descriptor still open. Expected: descriptor closed after registration returns, including on error. A proposed fix closes it in a finally block around registration and, for the DMAHandle path, around handle creation as well. On Linux the RDMA DMA-BUF registration path acquires its own buffer reference.

Please open PR against rdma-core with your fix for both test reports.

Thanks