Maximum size for memory registration (ibv_reg_mr)

Is there any limit on the maximum size of memory that can be registered using ibv_reg_mr. I am using OFED-1.4.2 stack, and when I try to pass a size greater than 16GB ibv_reg_mr returns NULL. The total amount of physical memory on the node is 48GB, max locked memory(ulimit -l) is set to unlimited.

Regards,

Bharath

The limit depends on the HCA.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

max_mr_size returned by calling ibv_devinfo is 0xffffffffffffffff. Does this mean the size is unlimited?

Regards,

Bharath

It means that the query function might be returning bad data. What adapter is it?

- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I don't know the exact adapter it is a QDR ConnectX adapter with board_id:
MT_0C40110009. Is there any documentations where one can find such information if the query function returns bad data.

Regards,

Bharath

All Mellanox adapters have some limit on the amount of memory they can register. You can mess around with module parameters to adjust it. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

You can use mlx4_core module parameter (log_mtts_per_seg) to enlarge number of MTTs per segment. This enable to register more memory with the same number of segments.
In OFED 1.5.2 the max value was enlarged to 7

Tziporet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yep, mlx4_ib_query_device does props->max_mr_size = ~0ull ...

Jack, what would it take to fix that? I took a look and other settings are taken from dev->dev->caps.xxx where caps is of type mlx4_caps which is in turn filled @ drivers/net/mlx4/main.c :: mlx4_dev_cap I wasn't sure what field/computation would exactly account for later setting max_mr_size

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

It is not that trivial to calculate this value.
If you create a MR in kernel, it covers the entire address space and the HCA does not pose any limit since you do not consume MTTs. And if you use MTTs then the page size is a parameter in this calculation - huge page, regular page etc.

Do leaving it as is seems to be the most accurate thing...
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I agree that the kernel case is not of large interest, even though what you wrote only applies for dma mr, when some FMR scheme is used, MTTs are consumed, ofcourse. But, typically, kernel code will not go to the order of giga-bytes, and in other words will not hit the HCA limit.

I would implement it for regular pages and drop a note in the libibverbs man page that if huge pages are used (well, the huge pages patch set isn't fully merged, maybe its about time to make this happen...) then the actual limit is bigger, e.g follows the proportion between the regular to the huge pages used.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

What you're proposing is to change something that is not accurate in another not accurate thing. Even when you don't use huge pages, a calculation based on regular pages would be inaccurate too. That's because the allocator allocates MTTs in 2^n quantities while the registered memory need not be sized like that. Moreover, such calculation is less accurate then what we have now.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html