Naïve question wrt ibv_reg_mr(), pinned pages and Linux VM

Hello everybody,
I have a question regarding behavior of the memory pinning using ibv_reg_mr() on RHEL/Centos 5.3/OFED-1.4.1. This may be more a Linux feature/issue, and I apologize if that is the case. Here is the scenario

1. An application runs which allocates a significant fraction of the system memory as RDMA buffers (e.g. 3GB of ram on a 4GB system). These are setup using ibv_reg_mr(). It is clear that the pages are pinned from the kernel perspective. With just this program running, the resident set size of the program approaches the allocation size.
2. If other memory-intensive processes are now started, the resident set size of the RDMA-using program shrinks dramatically.
3. Even if the other memory-intensive programs are stopped, and the RDMA-using program is forced to read its memory, the resident set never grows to a ³reasonable² size again.

My potentially foolish assumptions are/were that:
i) Since the memory is pinned anyway, it would be locked into the process address space, and
ii) even if that were not the case, that the process would be able to regain a large RSS when any competing processes stopped.

For ii), it almost seems that the VM doesn¹t realize that the pages it would be grabbing back are already resident and therefore won¹t actually take any more memory.

Clearly, I can use mlock() to avoid the issue, but I was wondering if I have missed something obvious here. Any clues/brickbats gratefully received!

Regards,

Tim Wright

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Tim,
the memory you register using ibv_reg_mr() is pinned and the pages cannot be used for anything else until you release them through a call to ibv_dereg_mr() (or the process terminates). There is no need to use mlock.
Did you check that the call to ibv_reg_mr() succedded?

Quoted 37 lines · Tim Wright

Hello everybody,
I have a question regarding behavior of the memory pinning using ibv_reg_mr() on RHEL/Centos 5.3/OFED-1.4.1. This may be more a Linux feature/issue, and I apologize if that is the case. Here is the scenario

1. An application runs which allocates a significant fraction of the system memory as RDMA buffers (e.g. 3GB of ram on a 4GB system). These are setup using ibv_reg_mr(). It is clear that the pages are pinned from the kernel perspective. With just this program running, the resident set size of the program approaches the allocation size.
2. If other memory-intensive processes are now started, the resident set size of the RDMA-using program shrinks dramatically.
3. Even if the other memory-intensive programs are stopped, and the RDMA-using program is forced to read its memory, the resident set never grows to a ³reasonable² size again.

My potentially foolish assumptions are/were that:
i) Since the memory is pinned anyway, it would be locked into the process address space, and
ii) even if that were not the case, that the process would be able to regain a large RSS when any competing processes stopped.

For ii), it almost seems that the VM doesn¹t realize that the pages it would be grabbing back are already resident and therefore won¹t actually take any more memory.

Clearly, I can use mlock() to avoid the issue, but I was wondering if I have missed something obvious here. Any clues/brickbats gratefully received!

Regards,

Tim Wright

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Eli,
Thank you for replying. Yes the call succeeds and the memory is clearly pinned at least from the perspective of the kernel, and the RDMA operations do happen on the correct memory. The problem is that the pages get stolen from the resident set of the program, which is inefficient, and doesn't seem to make a lot of sense since it doesn't free much of anything since the memory is locked. What does seem to happen under load is that the process page tables can get kicked out making access to the pinned memory very slow. Of course other parts of the application are also subject to paging, so mlock is really necessary anyway, but I was surprised that the memory was only pinned into physical memory and not also locked into the calling process' address space.

For my purposes, it seems mlockall() is the solution, and it appears to interact just fine with the underlying get_user_pages() operations.