mmap_sem in ->fault and ->page_mkwrite

Hi,

I'm not sure if this is such a great topic but it's a question which
I came across a few times already and LSF/MM is a good place for brainstorming somewhat crazy ideas ;).

So currently ->fault() and ->page_mkwrite() are called under mmap_sem held for reading. Now this creates sometimes unpleasant locking dependencies for filesystems (modern filesystems have to do an equivalent of ->write_begin in ->page_mkwrite and that is a non-trivial operation). Just to mention my last itch, I had to split reader side of filesystem freezing lock into two locks - one which ranks above mmap_sem and one which ranks below it. Then writer side has to wait for both locks. It works but ...

So I was wondering: Would it be somehow possible we could drop mmap_sem in these two callbacks (especially ->page_mkwrite())? I understand process' mapping can change under us once we drop the semaphore so we'd have to somehow recheck we have still the right page after re-taking mmap_sem. Like if we protected VMAs with SRCU so that they don't disappear under us once we drop mmap_sem and after retaking mmap_sem we would recheck whether VMA still applies to our fault.

And I know there's VM_FAULT_RETRY but that really seems like a special hack for x86 architecture page fault code. Making it work for all architectures and callers such as get_user_pages() didn't really seem plausible to me.

Quoted 22 lines · Jan Kara

Hi,

I'm not sure if this is such a great topic but it's a question which
I came across a few times already and LSF/MM is a good place for brainstorming somewhat crazy ideas ;).

So currently ->fault() and ->page_mkwrite() are called under mmap_sem held for reading. Now this creates sometimes unpleasant locking dependencies for filesystems (modern filesystems have to do an equivalent of ->write_begin in ->page_mkwrite and that is a non-trivial operation). Just to mention my last itch, I had to split reader side of filesystem freezing lock into two locks - one which ranks above mmap_sem and one which ranks below it. Then writer side has to wait for both locks. It works but ...

So I was wondering: Would it be somehow possible we could drop mmap_sem in these two callbacks (especially ->page_mkwrite())? I understand process' mapping can change under us once we drop the semaphore so we'd have to somehow recheck we have still the right page after re-taking mmap_sem. Like if we protected VMAs with SRCU so that they don't disappear under us once we drop mmap_sem and after retaking mmap_sem we would recheck whether VMA still applies to our fault.

Quoted 3 lines

And I know there's VM_FAULT_RETRY but that really seems like a special hack for x86 architecture page fault code. Making it work for all architectures and callers such as get_user_pages() didn't really seem plausible to me.

Please, *please*, don't. VMA locking is complete horror without SRCU mess thrown in. It's a bloody bad idea, at least without a very massive cleanup prior to that thing.

Start with drawing the call graph for vma-related code - at least the parts from relevant locks grabbed to accesses of fields protected by said locks.

Quoted 15 lines · Al Viro

Quoted 14 lines · Jan Kara

Hi,

I'm not sure if this is such a great topic but it's a question which
I came across a few times already and LSF/MM is a good place for brainstorming somewhat crazy ideas ;).

So currently ->fault() and ->page_mkwrite() are called under mmap_sem held for reading. Now this creates sometimes unpleasant locking dependencies for filesystems (modern filesystems have to do an equivalent of ->write_begin in ->page_mkwrite and that is a non-trivial operation). Just to mention my last itch, I had to split reader side of filesystem freezing lock into two locks - one which ranks above mmap_sem and one which ranks below it. Then writer side has to wait for both locks. It works but ...

Quoted 7 lines

Quoted 7 lines

So I was wondering: Would it be somehow possible we could drop mmap_sem in these two callbacks (especially ->page_mkwrite())? I understand process' mapping can change under us once we drop the semaphore so we'd have to somehow recheck we have still the right page after re-taking mmap_sem. Like if we protected VMAs with SRCU so that they don't disappear under us once we drop mmap_sem and after retaking mmap_sem we would recheck whether VMA still applies to our fault.

Quoted 3 lines

Quoted 3 lines

And I know there's VM_FAULT_RETRY but that really seems like a special hack for x86 architecture page fault code. Making it work for all architectures and callers such as get_user_pages() didn't really seem plausible to me.

Quoted 7 lines

Please, *please*, don't. VMA locking is complete horror without SRCU mess thrown in. It's a bloody bad idea, at least without a very massive cleanup prior to that thing.

Start with drawing the call graph for vma-related code - at least the parts from relevant locks grabbed to accesses of fields protected by said locks.

VMAs are protected by mmap_sem AFAIK so that doesn't look all that
complex. But I guess you are pointing at the fact that sometimes mmap_sem is acquired rather far (sometimes even in arch code) from the places which use the protection of mmap_sem and so it would be difficult (if possible at all) to verify that once we drop mmap_sem, all these places will happily handle that fact. I agree it would be a mess unless we somehow simplify things first...