mm: Add f_ops->populate()

Sometimes you might want to use MAP_POPULATE to ask a device driver to initialize the device memory in some specific manner. SGX driver can use this to request more memory by issuing ENCLS[EAUG] x86 opcode for each page in the address range.

Add f_ops->populate() with the same parameters as f_ops->mmap() and make it conditionally called inside call_mmap(). Update call sites accodingly.

Quoted 4 lines · On Sun, 6 Mar 2022 05:26:55 +0200 Jarkko Sakkinen <[email protected]> wrote:

Sometimes you might want to use MAP_POPULATE to ask a device driver to initialize the device memory in some specific manner. SGX driver can use this to request more memory by issuing ENCLS[EAUG] x86 opcode for each page in the address range.

Why is this useful? Please fully describe the benefit to kernel users. Convince us that the benefit justifies the code churn, maintenance cost and larger kernel footprint.

Quoted 9 lines · Andrew Morton

Quoted 4 lines · On Sun, 6 Mar 2022 05:26:55 +0200 Jarkko Sakkinen <[email protected]> wrote:

Sometimes you might want to use MAP_POPULATE to ask a device driver to initialize the device memory in some specific manner. SGX driver can use this to request more memory by issuing ENCLS[EAUG] x86 opcode for each page in the address range.

Why is this useful? Please fully describe the benefit to kernel users. Convince us that the benefit justifies the code churn, maintenance cost and larger kernel footprint.

In short: page faults stink. The core kernel has lots of ways of avoiding page faults like madvise(MADV_WILLNEED) or mmap(MAP_POPULATE).
But, those only work on normal RAM that the core mm manages.

SGX is weird. SGX memory is managed outside the core mm. It doesn't have a 'struct page' and get_user_pages() doesn't work on it. Its VMAs are marked with VM_IO. So, none of the existing methods for avoiding page faults work on SGX memory.

This essentially helps extend existing "normal RAM" kernel ABIs to work for avoiding faults for SGX too. SGX users want to enjoy all of the benefits of a delayed allocation policy (better resource use, overcommit, NUMA affinity) but without the cost of millions of faults.

That said, this isn't how I would have implemented it. I probably would have hooked in to populate_vma_page_range() or its callers.

But e.g. in __mm_populate() anything with (VM_IO | VM_PFNMAP) gets filtered out and never reach that function.

I don't know unorthodox that'd be but could we perhaps have a VM flag for SGX?

Quoted 5 lines · Jarkko Sakkinen

But e.g. in __mm_populate() anything with (VM_IO | VM_PFNMAP) gets filtered out and never reach that function.

I don't know unorthodox that'd be but could we perhaps have a VM flag for SGX?

SGX only works on a subset of the chips from one vendor on one architecture. That doesn't seem worth burning a VM flag.

In short: page faults stink. The core kernel has lots of ways of avoiding page faults like madvise(MADV_WILLNEED) or mmap(MAP_POPULATE).
But, those only work on normal RAM that the core mm manages.

SGX is weird. SGX memory is managed outside the core mm. It doesn't have a 'struct page' and get_user_pages() doesn't work on it. Its VMAs are marked with VM_IO. So, none of the existing methods for avoiding page faults work on SGX memory.

This essentially helps extend existing "normal RAM" kernel ABIs to work for avoiding faults for SGX too. SGX users want to enjoy all of the benefits of a delayed allocation policy (better resource use, overcommit, NUMA affinity) but without the cost of millions of faults.

Quoted 9 lines · Dave Hansen

In short: page faults stink. The core kernel has lots of ways of avoiding page faults like madvise(MADV_WILLNEED) or mmap(MAP_POPULATE).
But, those only work on normal RAM that the core mm manages.

SGX is weird. SGX memory is managed outside the core mm. It doesn't have a 'struct page' and get_user_pages() doesn't work on it. Its VMAs are marked with VM_IO. So, none of the existing methods for avoiding page faults work on SGX memory.

Quoted 4 lines

This essentially helps extend existing "normal RAM" kernel ABIs to work for avoiding faults for SGX too. SGX users want to enjoy all of the benefits of a delayed allocation policy (better resource use, overcommit, NUMA affinity) but without the cost of millions of faults.

We have a mechanism for dynamically reducing the number of page faults already; it's just buried in the page cache code. You have vma->vm_file, which contains a file_ra_state. You can use this to track where recent faults have been and grow the size of the region you fault in per page fault. You don't have to (indeed probably don't want to) use the same algorithm as the page cache, but the _principle_ is the same -- were recent speculative faults actually used; should we grow the number of pages actually faulted in, or is this a random sparse workload where we want to allocate individual pages.

Don't rely on the user to ask. They don't know.

Quoted 4 lines · On Sun, 6 Mar 2022 05:26:55 +0200 Jarkko Sakkinen <[email protected]> wrote:

Sometimes you might want to use MAP_POPULATE to ask a device driver to initialize the device memory in some specific manner. SGX driver can use this to request more memory by issuing ENCLS[EAUG] x86 opcode for each page in the address range.

Why is this useful? Please fully describe the benefit to kernel users. Convince us that the benefit justifies the code churn, maintenance cost and larger kernel footprint.

Do we know of any other drivers which might use this?

Quoted 3 lines

Add f_ops->populate() with the same parameters as f_ops->mmap() and make it conditionally called inside call_mmap(). Update call sites accodingly.

spello

include/linux/fs.h static inline ssize_t call_write_iter(struct file *file, struct kiocb *kio, · line 2086

Quoted 12 lines

include/linux/fs.h +7 -2

-static inline int call_mmap(struct file *file, struct vm_area_struct *vma)
+static inline int call_mmap(struct file *file, struct vm_area_struct *vma, bool do_populate)
 {
-	return file->f_op->mmap(file, vma);
+	int ret = file->f_op->mmap(file, vma);
+
+	if (!ret && do_populate && file->f_op->populate)
+		ret = file->f_op->populate(file, vma);
+
+	return ret;
 }

Should this still be inlined?