Infinihost HCA ->MTT -are they per HCA?

We have the following HCAs in one of the machines:

InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20)

There are two adapters in the machine and from various experiments we find that we are limited to being able to register (ibv_reg_mr()) 28 GB. My calculations show one should be able to register up to 56 GB. I would like to understand the difference.

In /sys/module/ib_mthca/parameters I see the following values:

fmr_reserved_mtts = 262144
log_mtts_per_seg = 3
num_mpt = 131072
num_mtt = 2097152

Therefore max memory that one can register should be (2097152 -262144) * 4K * 2^(log_mtts_per_seg) i.e. 1.75 MB * 4K * 8 = 56 GB.

Since ibv_reg_mr() ties one to a pd and thus a particular device, is the 56 GB split between the 2 HCAs in the machine? How does one explain the difference?

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Your calaculations are correct; each device should be able, theoretically, to register 56 GB of memory. I assume you have enough memory in your machine for the memory to be registered and also ICM memory that is required to support all the registered memory. Also it depends on how many registered memory regions you use and and their sizes. For example, since the MTT segments allocator is a buddy allocator, it allocates 2 ^ n mtt segments each time. So if your memory region requires only [2 ^ (n -1)] + 1 entries, it will still consume 2 ^ n entries from the allocator.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html