librdmacm: lazy initialization for ib devices]
----- Forwarded message from Or Gerlitz <ogerlitz-VPRAkNaXOzVWk0Htik3J/[email protected]> -----
So what exact capability is under the spot here? and how it's related to a certain HCA driver and not to the kernel uverbs layer?
----- End forwarded message -----
ibv_open_device fail when opening the hca many times. ucma_init call this api a-priori on all the devices for any process that use the library. The patch remove this a-priori open and open the device only when verbs are really needed. I do not think its issue at the verbs but more issue in how the librdmacm use the verbs.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Shamir Rabinovith
So what exact capability is under the spot here? and how it's related to a certain HCA driver and not to the kernel uverbs layer?
----- End forwarded message -----
ibv_open_devicefail when opening the hca many times.
sorry, but you didn't provide an answer to my question - why is ibv_open_device failing if called from N > M (what's N and what's M) processes/threads.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 5333E37B.5080202-VPRAkNaXOzVWk0Htik3J/[email protected]
Quoted 5 lines · Or Gerlitz
sorry, but you didn't provide an answer to my question - why is
ibv_open_devicefailing if called from N > M (what's N and what's M) processes/threads.Or.
The limit is due to firmware UAR BAR size. The issue happen here:ib_uverbs_get_context -> ibdev->alloc_ucontext -> mlx4_ib_alloc_ucontext -> mlx4_uar_alloc -> mlx4_bitmap_alloc . mlx4_QUERY_DEV_CAP return value of 4194304 when reading the UAR from the firmware which turn to UAR table size of 1024 in mlx4_init_uar_table. This is the result of dividing the UAR BAR size by page size. We can try to extend the size of the UAR BAR for Oracle servers as we know the HW and we can verify it will boot with this change but anyhow we will still hit the limit of number of processes that can open the library. More then this - UAR is set to value that will allow less stronger machines to boot properly so if you do not know which machine will use the firmware you cannot change this limit as all. So the overall is that I think we need this change anyhow, with or without the UAR BAR size change.
BR, Shamir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Shamir Rabinovith
Quoted 5 lines · Or Gerlitz
sorry, but you didn't provide an answer to my question - why is
ibv_open_devicefailing if called from N > M (what's N and what's M) processes/threads.Or.
The limit is due to firmware UAR BAR size. The issue happen here:
ib_uverbs_get_context->ibdev->alloc_ucontext->mlx4_ib_alloc_ucontext->mlx4_uar_alloc->mlx4_bitmap_alloc.
[...]
So what **exact** operation in librdmacm fails after N processes loaded the library on a node with M HCAs and what are N and M?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html