How to use huge pages in drivers?

Hi ; not quite sure where to ask so will start here...

Some context first. I'm inquiring in the context of mckernel[1], a lightweight kernel that works next to linux (basically offlines a few/most cores, reserve some memory and have boot a second OS on that to run HPC applications).
Being brutally honest here, this is mostly research and anyone here looking into it will probably scream, but I might as well try not to add too many more reasons to do so....

One of the mecanisms here is that sometimes we want to access the mckernel memory from linux (either from the process that spawned the mckernel side process or from a driver in linux), and to do that we have mapped the mckernel side virtual memory range to that process so it can page fault.
The (horrible) function doing that can be found here[2], rus_vm_fault - sends a message to the other side to identify the physical address corresponding from what we had reserved earlier and map it quite manually.

We could know at this point if it had been a huge page (very likely) or not; I'm observing a huge difference of performance with some interconnect if I add a huge kludge emulating huge pages here (directly manipulating the process' page table) so I'd very much like to use huge pages when we know a huge page has been mapped on the other side.

What I'd like to know is:
- we know (assuming the other side isn't too bugged, but if it is we're
fucked up anyway) exactly what huge-page-sized physical memory range has been mapped on the other side, is there a way to manually gather the pages corresponding and merge them into a huge page?

- from what I understand that does not seem possible/recommended, the
way to go being to have a userland process get huge pages and pass these to a device (ioctl or something); but I assume that means said process needs to keep on running all the time that memory is required? If the page fault needs to split the page (because the other side handed a "small" page so we can only map a regular page here), can it be merged back into a huge page for the next time this physical region is used?

[1] https://github.com/RIKEN-SysSoft/mckernel
[2] https://github.com/RIKEN-SysSoft/mckernel/blob/development/executer/kernel/mcctrl/syscall.c#L538

Any input will be appreciated,

Dominique Martinet

You're using the word 'page' here, but I suspect what you really mean is "pfn" or "pte". As you've described it, it doesn't matter what data structure Linux is using for the memory, since Linux doesn't know about the memory.

We have vmf_insert_pfn_pmd() which is designed to be called from your ->huge_fault handler. See dev_dax_huge_fault() -> __dev_dax_pmd_fault() for an example. It's a fairly new mechanism, so I don't think it's popular with device drivers yet.

All you really need is the physical address of the memory to make this work.

Matthew Wilcox wrote on Tue, Sep 03, 2019:

Correct, we're already using vmf_insert_pfn

Great; I'm not sure how I had missed the pmd variant here. It's even been around for long enough to be available on our "old" el7 kernels so I'll be able to test this quickly.

Thanks!

Dominique

Dominique Martinet wrote on Tue, Sep 03, 2019:

Actually let me take that back, vmf_insert_pfn is only used if pfn_valid() is false, probably as a safeguard of sort(?). The normal case went with pfn_to_page(pfn) + vm_insert_page() so, as things stands.
I do have a few more questions if you could humor me a bit more...

- the vma was created with a vm_flags including VM_MIXEDMAP for some
reason, I don't know why.
If I change it to VM_PFNMAP (which sounds better here from the little I understand of this as we do not need cow and looks a bit simpler?), I can remove the vm_insert_page() path and use the vmf_insert_pfn one instead, which appears to work fine for simple programs... But the kernel thread for my network adapter (bxi... which is not upstream either I guess.. sigh..) no longer tries to fault via my custom .fault vm operation... Which means I probably did need MIXEDMAP ?

I'm honestly not sure where to read up on what these two flags imply, looking at the page fault handler code I do not see why the request from a kernel thread would care what kind of vma it is...

- ignoring that for now (it's not like I need to switch to PFNMAP);
adding vmf_insert_pfn_pmd() for when the remote side uses large pages, it complains that the vmf->pmd is not a pmd_none nor huge nor a devmap (this check appears specific to rhel7 kernel, I could temporarily test with an upstream kernel but the network adapter won't work there so I'll need this to work on this ultimately)

It looks like handle_mm_fault() will always try to allocate a pmd so it should never be empty in my fault handler, and I don't see anything else than vmf_insert_pfn_pmd() setting the mkdevmap flag, and it's not huge either...
(on a dump, the the pmd content is 175cb18067, so these flags according to crash for x86_64 are (PRESENT|RW|USER|ACCESSED|DIRTY))

I tried adding a huge_fault vm op thinking it might be called with a more appropriate pmd but it doesn't seem to be called at all in my case..? I would have assumed from the code that it would try every page

and if I try to somehow force it by using pmd_mkdevmap on the vmf->pmd, things appear to work until the process exits and zap_page does a null deref on pgtable_trans_huge_withdraw because the pgtable was never deposited - this looks gone on newer kernels, but once again I do not see where these should come from; I'm just assuming I reap what I sew messing with the flags.

Long story short, I think I have some deeper undestanding problem about the whole thing. Do I also need to use some specific flags when that special file is mmap'd to allow huge_fault to be called ?
I think transparent_hugepage_enabled(vma) is fine, but the vmf.pmd found in __handle_mm_fault is probably already not none at this point...?

Thanks again, feel free to ignore me for a bit longer I'll keep digging my own grave, writing to a rubber duck that might have an idea of how far the wrong way I've gone already helps... :D

Dominique

Strange ... PFNMAP absolutely should try to fault via the ->fault vm operation (although see below)

You shouldn't be calling vmf_insert_pfn_pmd() from a regular ->fault handler, as by then the fault handler has already inserted a PMD. The ->huge_fault handler is the place to call it from.

You may need to force PMD-alignment for your call to mmap().

Hope these pointers are slightly more useful than a rubber duck ;-)

Matthew Wilcox wrote on Wed, Sep 04, 2019:

It does fault in some context, just not in another.. A bit weird but I'll stick to MIXEDMAP for now - I'm really curious as to what the difference is, "normal" applications seem to work fine with either mode, it's only the bxi driver that

I was missing setting the VM_HUGE_FAULT vm_flags2 bit in the vma - the huge_fault handler is now called, and I no longer have the pre-existing pmd problem; that's a much better solution than manually fiddling with flags :)

Question though - is it ok to insert small pages if the huge_fault handler is called with PE_SIZE_PMD ?
(I think the pte insertion will automatically create the pmd, but would be good to confirm)

Now I've got this I'm back to where I stood with my kludge though, programs work until they exit, and the zap_huge_pmd() function tries to withdraw the pagetable from some magic field that was never set in my case... I realize this is old code no longer upstream, but my new workaround for this (looking at the zap_huge_pmd function) was to pretend my file is dax.
Now that I've set it as dax I think it actually makes sense as in "there's memory here that points to something linux no longer manages directly, just let it be" and we might benefit from the other exceptions dax have, I'll need to look at what this implies in more details...

Much appreciated, thank you for taking the time! :)

Off to debug my network driver for the PFNMAP behaviour next, and then some more testing... I'm sure I broke something seemingly unrelated on the other side of the project!

Dominique

No, you need to return VM_FAULT_FALLBACK, at which point the generic code will create a PMD for you and then call your ->fault handler which can insert PTEs.

It works the same way from PUDs to PMDs by the way, in case you ever have a 1GB mapping ;-)

I think that should be fine, but I don't really know RHEL 7.3 all that well ;-)

No problem ... these APIs are relatively new and not necessarily all that intuitive.

Matthew Wilcox wrote on Thu, Sep 05, 2019:

Hmm, that's a shame actually.
There is a rather costly round-trip between linux and mckernel to determine what page size is used for this virtual address on the remote side and to get the corresponding physical address, so basically when we get the fault we do know know if this will be a PMD or PTE.

I'd rather avoid having to do one round-trip at the PMD stage, get told this is a PTE, temporarily give up and wait to be called again with PE_SIZE_PTE and do a second round-trip in this case.
I didn't see anywhere in the vm_fault struct that I could piggy-back to remember something from the previous call, and I'm pretty sure it would be a bad idea to use the vma's vm_private_data here because there could be multiple faults in parallel on other threads.

Looking at vmf_insert_pfn(), it will allocate a pmd because of insert_pfn's get_locked_pte, so it does end up working (I never return a page - we always return VM_FAULT_NOPAGE on success, so I do not see the harm in doing it early if we can)

Following the code in __handle_vm_fault assuming the pmd fault would have returned fallback I do not see any harm here - the pmd actually already has been allocated here (at pmd level fault), it's just set to none.

Not exactly pretty, though, and very definitely no guarantee it'll keep working... I'll stick a comment saying what we should do at least :P

Yes, already returning fallback in this case - but I'm just assuming that won't happen so no round-trip here :)

Good enough for me, tests will tell me what I broke :)

Looking at a recent vanilla linux on evening and rhel's kernel at work didn't help on my side (some fun differences like the VM_HUGE_FAULT flag in the vma, but now I understand it was added for abi compatibility it does make sense after I found about it - on an older module the function could just have been left uninitialized and thus non-null yet not valid)

Definitely did help to point at huge_fault() again.

Thanks,

Dominique