Skip to contentSkip to conversation

ibv_reg_mr fails to register memory

DISCUSSION Mohammed Rafi K C 2 participants 3 messages Nov 11, 2014 – Nov 12, 2014 2 messages not in the archive InfiniBand

lore
Share & tools
lore

Mohammed Rafi K C

author
#0 lore raw

Hi All,

I was trying to implement an application that support rdma using librdmacm. When the process is trying to register the buffer using ibv_reg_mr, it failes after registering some number buffer.

We are using Mellanox Technologies MT27500 Family [ConnectX-3] cards.

Reconfigured options are :

1) cat /sys/module/mlx4_core/parameters/log_num_mtt

    24
2) cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
    7

Can anyone point out reason behind ibv_reg_mr fails ?

Thanks & Regards
Rafi KC

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw

Hi,

Likely, it's caused by limits in /etc/security/limits.conf. See p.30 in Mellanox OFED for Linux User Manual Rev 1.5.3-4.0.42.

Best regards,
Eugene

Quoted 17 lines · Mohammed Rafi K C

Hi All,

I was trying to implement an application that support rdma using librdmacm. When the process is trying to register the buffer using ibv_reg_mr, it failes after registering some number buffer.

We are using Mellanox Technologies MT27500 Family [ConnectX-3] cards.

Reconfigured options are :

1) cat /sys/module/mlx4_core/parameters/log_num_mtt

    24
2) cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
    7

Can anyone point out reason behind ibv_reg_mr fails ?

Quoted 7 lines

Thanks & Regards
Rafi KC

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAMtRa4DW2_Aj13enG2RNoLhCro5VPb+3Ak8im4b0GWAW3TTyEw@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CAMtRa4DW2_Aj13enG2RNoLhCro5VPb+3Ak8im4b0GWAW3TTyEw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

author
#4 lore raw
Quoted 9 lines · kushagra misra

yes plz check your ulimits value under /etc/security/limits.conf that may be causing the issue.

On Wed, Nov 12, 2014 at 4:14 PM, Evgenii Smirnov <[email protected]> wrote:
Hi,

Quoted 7 lines

Likely, it's caused by limits in /etc/security/limits.conf. See p.30
in Mellanox OFED for Linux User Manual Rev 1.5.3-4.0.42.

Best regards,
Eugene

Eugene/kushagra

Thank you very much for your reply. AS you guys suggested , I checked /etc/security/limits.conf file and the value was set to unlimited for both soft memlock and hard memlock.

* soft memlock unlimited
* hard memlock unlimited

So I hope that will allow us to pin unlimited memory with kernel. I did some experiment to know how much memory is able to register successfully , and the result was approximately 128MB (127.40625 MB). Further allocation is failed by returning NULL from ibv_reg_mr, by setting errno=11 (Resource temporarily unavailable).

Thanks & Regrads
Rafi KC

Quoted 29 lines

On Tue, Nov 11, 2014 at 11:12 AM, Mohammed Rafi K C <[email protected]> wrote:
> Hi All,
>
> I was trying to implement an application that support rdma using
> librdmacm. When the process is trying to register the buffer using
> ibv_reg_mr, it failes after registering some number buffer. >
> We are using Mellanox Technologies MT27500 Family [ConnectX-3] cards.
>
> Reconfigured options are :
>
> 1) cat /sys/module/mlx4_core/parameters/log_num_mtt
> 24
> 2) cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
> 7
>
> Can anyone point out reason behind ibv_reg_mr fails ?
>
> Thanks & Regards
> Rafi KC
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to [email protected]

Quoted 19 lines

> More majordomo info at
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Signature · 4 lines
Real leaders are ordinary people with extraordinary determination.!!!!

Thanks & Regard's
Kushagra Misra

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html