InfiniBand userspace verbs implementation
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Andy Isaacson
Quoted 6 lines · Timur Tabi
I don't know if
VM_REGISTEREDis a good idea or not, but it should be absolutely impossible for the kernel to reclaim "registered" (aka pinned) memory, no matter what. For RDMA services (such as Infiniband, iWARP, etc), it's normal for non-root processes to pin hundreds of megabytes of memory, and that memory better be locked to those physical pages until the application deregisters them.
Quoted 3 lines
If you take the hardline position that "the app is the only thing that matters", your code is unlikely to get merged. Linux is a general-purpose OS.
All userspace hardware drivers with DMA will require pinned pages (and some of them will require continuous memory). Since this memory may be scheduled to be accessed by DMA, reclaiming those pages may (aka. will) result in "random" memory corruption unless done by the driver itself.
You can't even set a time limit, the driver may have allocated all DMA memory to queued transfers, and some media needs to get plugged in by the lazy robot. As soon as the robot arrives - boom. (For the same reason, this memory MUST NOT be freed if the application terminates abnormally, e.g. killed by OOM).
In other words, you need to make this memory as unaccessible as the framebuffer on a graphic card. If that causes a lockup, you better had prevented that while allocating.
Quoted 11 lines
In a Linux context, I doubt that fullblown SA is necessary or appropriate. Rather, I'd suggest two new signals, SIGMEMLOW and SIGMEMCRIT. The userland comms library registers handlers for both. When the kernel decides that it needs to reclaim some memory from the app, it sends SIGMEMLOW. The comms library then has the responsibility to un-reserve some memory in an orderly fashion. If a reasonable [1] time has expired since SIGMEMLOW and the kernel is still hungry, the kernel sends SIGMEMCRIT. At this point, the comms lib *must* unregister some memory [2] even if it has to drop state to do so; if it returns from the signal handler without having unregistered the memory, the kernel will SIGKILL.
Choosing Data loss vs. finitely stalled system may sometimes be a bad decision.
If I designes an application that might get a "gimme memory or die", I'd reserve an extra bunch of memory with the only purpose of being released in this situation. If the kernel had done that instead, this part of memory could have been used e.g. as a read-only disk cache in the meantime (off cause provided somebody cared to implement that).
Quoted 2 lines
[2] Is there a way for the kernel to pass down to userspace how many pages it wants, maybe in the sigcontext?
Then you'd need only one signal.
I think this interface is usefull, it would e.g. allow a picture viewer to cache as many decoded and scaled pictures as the RAM permits, freeing them if the RAM gets full and the swap would have to be used.
Signature · 2 lines
"When the pin is pulled, Mr. Grenade is not our friend. -U.S. Marine Corps
RE: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation
Quoted 7 lines
From: Bodo Eggert <[email protected]>
Sent: Friday, April 22, 2005 6:10 AMAll userspace hardware drivers with DMA will require pinned pages (and some of them will require continuous memory). Since this memory may be scheduled to be accessed by DMA, reclaiming those pages may (aka. will) result in "random" memory corruption unless done by the driver itself.
Any reclaim must involve the driver. That doesn't mean that it must involve the application. That said this isn't trivial to implement.
Quoted 6 lines
You can't even set a time limit, the driver may have allocated all DMA memory to queued transfers, and some media needs to get plugged in by the lazy robot. As soon as the robot arrives - boom. (For the same reason, this memory MUST NOT be freed if the application terminates abnormally, e.g. killed by OOM).
InfiniBand provides support for deregistering memory that might be referenced at some future time by an RDMA operation. The only side effect this has is that the QP on both sides of the connection transition to an error state.
Upon abnormal termination, all registrations must be undone and the memory unpinned. This must be synchronized with the hardware so that there are no races. The IB deregistration semantics provide such synchronization. I'd venture that any HW design that does not do this is broken.
Requiring the memory to never be freed upon abnormal termination equates to a serious memory leak, in that physical memory is leaked, not virtual.
- Fab
RE: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbsimplementation
Quoted 9 lines · Fab Tillier
Quoted 8 lines
From: Bodo Eggert <[email protected]>
Sent: Friday, April 22, 2005 6:10 AMYou can't even set a time limit, the driver may have allocated all DMA memory to queued transfers, and some media needs to get plugged in by the lazy robot. As soon as the robot arrives - boom. (For the same reason, this memory MUST NOT be freed if the application terminates abnormally, e.g. killed by OOM).
Quoted 8 lines
InfiniBand provides support for deregistering memory that might be referenced at some future time by an RDMA operation. The only side effect this has is that the QP on both sides of the connection transition to an error state.
Upon abnormal termination, all registrations must be undone and the memory unpinned. This must be synchronized with the hardware so that there are no races.
If you know the hardware. If you have userspace drivers, this will be impossible, and even if you have kernel drivers, you'll need to know which of them is responsible for each part of the pinned memory.
This doesn't imply the affected memory to be lost. The same application that created the pinned memory can reset the hardware (provided nobody changed the configuration), then reconnect to the shared memory segment you'll use for that purpose and use or free it.
Signature · 1 line
To iterate is human; to recurse, divine.