Skip to contentSkip to conversation

CUDA not working with ib_write_bw

DISCUSSION Steve Wise 1 participant 2 messages Feb 2, 2015 – Feb 4, 2015 2 messages not in the archive InfiniBand

lore
Share & tools
lore

Steve Wise

author
#0 lore raw

Hey Gil,

I'm trying to test iWARP RDMA <-> GPU memory and I compiled CUDA into the top-o-tree perftest repo. My Nvidia setup is working because I have verified it with another gpu rdma package (donard from pmc). But when using ib_write_bw the server gets an error registering the gpu memory with the device. Below is the output from ib_write_bw. I instrumented the kernel registration path and I find that get_user_pages() is returning -14 (-EFAULT) when called by ib_umem_get().

Q: Is this supposed to work with the upstream RDMA drivers? I'm using a 3.16.3 kernel.org kernel.

Thanks,

Steve
---

[root@stevo1 perftest]# ./ib_write_bw -R --use_cuda

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test

 Dual-port       : OFF          Device         : cxgb4_1
 Number of qps   : 1            Transport type : IW
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 Gid index       : 0

Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
Waiting for client rdma_cm QP to connect
Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
initializing CUDA
There is 1 device supporting CUDA
[pid = 14124, dev = 0] device name = [Tesla K20Xm] creating CUDA Ctx
making it the current CUDA Ctx
cuMemAlloc() of a 131072 bytes GPU buffer
allocated GPU buffer address at 0000001304260000 pointer=0x1304260000 Couldn't allocate MR
Unable to create the resources needed by comm struct
Unable to perform rdma_client function
[root@stevo1 perftest]#

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): E51087C8852F7244A3C5126F7988F0D36A0DAF90@nkgeml511-mbx.china.huawei.com

A message that is not in the archive (sent off-list or not yet received): E51087C8852F7244A3C5126F7988F0D36A0DAF90-gH8YcUvTLkOF0yysJWCP+gK1hpo4iccwjNknBlVQO8k@public.gmane.org

author
#3 lore raw

I want to RDMA to/from gpu memory from a remote peer. So the data arrives at the RDMA device from the wire, and is DMA'd directly to GPU memory of the peer adapter in the system.

Quoted 18 lines

-----Original Message-----
From: Zhangfengwei [mailto:[email protected]]
Sent: Wednesday, February 04, 2015 12:42 AM
To: Steve Wise; [email protected]
Cc: [email protected]; Stephen Bates
Subject: 答复: CUDA not working with ib_write_bw

Hi Steve，

Do you want to transfer the data to the gpu buffer directly ? I guess the DMA seems not to do this all by itself.

-----邮件原件-----
发件人: [email protected] [mailto:[email protected]] 代表 Steve Wise 发送时间: 2015年2月2日 23:59
收件人: [email protected]
抄送: [email protected]; Stephen Bates
主题: CUDA not working with ib_write_bw

Quoted 5 lines

Hey Gil,

I'm trying to test iWARP RDMA <-> GPU memory and I compiled CUDA into the top-o-tree perftest repo. My Nvidia setup is working because I have verified it with another gpu rdma package (donard from pmc). But when using ib_write_bw the server gets an error registering the gpu memory with the device. Below is the output from ib_write_bw. I instrumented the kernel registration path

and I find

Quoted 36 lines

that get_user_pages() is returning -14 (-EFAULT) when called by ib_umem_get().

Q: Is this supposed to work with the upstream RDMA drivers? I'm using a 3.16.3 kernel.org kernel.

Thanks,

Steve
---

[root@stevo1 perftest]# ./ib_write_bw -R --use_cuda

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test

 Dual-port       : OFF          Device         : cxgb4_1
 Number of qps   : 1            Transport type : IW
 Connection type : RC           Using SRQ      : OFF
 CQ Moderation   : 100
 Mtu             : 1024[B]
 Link type       : Ethernet
 Gid index       : 0

Max inline data : 0[B]
rdma_cm QPs : ON
Data ex. method : rdma_cm
---------------------------------------------------------------------------------------
Waiting for client rdma_cm QP to connect Please run the same command with the IB/RoCE interface IP
---------------------------------------------------------------------------------------
initializing CUDA
There is 1 device supporting CUDA
[pid = 14124, dev = 0] device name = [Tesla K20Xm] creating CUDA Ctx making it the current CUDA Ctx cuMemAlloc() of a 131072 bytes GPU buffer allocated GPU buffer address at 0000001304260000 pointer=0x1304260000 Couldn't allocate MR Unable to create the resources needed by comm struct Unable to perform rdma_client function
[root@stevo1 perftest]#

Quoted 3 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html