Skip to contentSkip to conversation

librdmacm: lazy initialization for ib devices]

PATCH Shamir Rabinovith 3 participants 4 messages Mar 27, 2014 – Mar 31, 2014 1 message not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

----- Forwarded message from Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]> -----

So what exact capability is under the spot here? and how it's related to a certain HCA driver and not to the kernel uverbs layer?

----- End forwarded message -----

ibv_open_device fail when opening the hca many times. ucma_init call this api a-priori on all the devices for any process that use the library. The patch remove this a-priori open and open the device only when verbs are really needed. I do not think its issue at the verbs but more issue in how the librdmacm use the verbs.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw
Quoted 6 lines · Shamir Rabinovith

So what exact capability is under the spot here? and how it's related to a certain HCA driver and not to the kernel uverbs layer?

----- End forwarded message -----

ibv_open_device fail when opening the hca many times.

sorry, but you didn't provide an answer to my question - why is ibv_open_device failing if called from N > M (what's N and what's M) processes/threads.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 5333E37B.5080202-VPRAkNaXOzVWk0Htik3J/[email protected]

#3 lore raw
Quoted 5 lines · Or Gerlitz

sorry, but you didn't provide an answer to my question - why is ibv_open_device failing if called from N > M (what's N and what's M) processes/threads.

Or.

The limit is due to firmware UAR BAR size. The issue happen here:
ib_uverbs_get_context -> ibdev->alloc_ucontext -> mlx4_ib_alloc_ucontext -> mlx4_uar_alloc -> mlx4_bitmap_alloc . mlx4_QUERY_DEV_CAP return value of 4194304 when reading the UAR from the firmware which turn to UAR table size of 1024 in mlx4_init_uar_table. This is the result of dividing the UAR BAR size by page size. We can try to extend the size of the UAR BAR for Oracle servers as we know the HW and we can verify it will boot with this change but anyhow we will still hit the limit of number of processes that can open the library. More then this - UAR is set to value that will allow less stronger machines to boot properly so if you do not know which machine will use the firmware you cannot change this limit as all. So the overall is that I think we need this change anyhow, with or without the UAR BAR size change.

BR, Shamir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
Quoted 10 lines · Shamir Rabinovith
Quoted 5 lines · Or Gerlitz

sorry, but you didn't provide an answer to my question - why is ibv_open_device failing if called from N > M (what's N and what's M) processes/threads.

Or.

The limit is due to firmware UAR BAR size. The issue happen here:
ib_uverbs_get_context -> ibdev->alloc_ucontext -> mlx4_ib_alloc_ucontext -> mlx4_uar_alloc -> mlx4_bitmap_alloc .

[...]

So what **exact** operation in librdmacm fails after N processes loaded the library on a node with M HCAs and what are N and M?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html