page lock ordering and OCFS2
Subsystems (4)
RFC 1/1 page lock ordering and OCFS2
I sent an ealier version of this patch to linux-fsdevel and was met with deafening silence. I'm resending the commentary from that first mail and am including a new version of the patch. This time it has much clearer naming and some kerneldoc blurbs. Here goes...
--
In recent weeks we've been reworking the locking in OCFS2 to simplify things and make it behave more like a "local" Linux file system. We've run into an ordering inversion between a page's PG_locked and OCFS2's DLM locks that protect page cache contents. I'm including a patch at the end of this mail that I think is a clean way to give the file system a chance to get the ordering right, but we're open to any and all suggestions. We want to do the cleanest thing.
OCFS2 maintains page cache coherence between nodes by requiring that a node hold a valid lock while there are active pages in the page cache. The page cache is invalidated before a node releases a lock so that another node can acquire it. While this invalidation is happening new locks can not be acquired on that node. This is equivalent to a DLM processing thread acquiring PG_locked during truncation while holding a DLM lock. Normal file system user tasks come to the a_ops with PG_locked acquired by their callers before they have a chance to get DLM locks.
We talked a little about modifying the invalidation path to avoid waiting for pages that are held by an OCFS2 path that is waiting for a DLM lock. It seems awfully hard to get right without some very nasty code. The truncation on lock removal has to write back dirty pages so that other nodes can read it -- it can't simply skip these pages if they happened to be locked.
So we aimed right for the lock ordering inversion problem with the appended patch. It gives file systems a callback that is tried before page locks are acquired that are going to be passed in to the file system's a_ops methods.
So, what do people think about this? Is it reasonable to patch the core to help OCFS2 with this ordering inversion?
- z
drivers/block/loop.c +7 -1
@@ -229,9 +229,13 @@ size =PAGE_CACHE_SIZE - offset;if (size > len) size = len; +if (mapping_guard_page_locks (mapping,1 )) + goto fail; page =grab_cache_page (mapping, index); -if (unlikely (!page)) +if (unlikely (!page)) { +mapping_page_locks_done (mapping,1 ); goto fail; + }if (unlikely (aops->prepare_write (file, page, offset, offset + size))) goto unlock;
@@ -263,6 +267,7 @@ pos += size;unlock_page (page);page_cache_release (page); +mapping_page_locks_done (mapping,1 ); } out:up (&mapping->host->i_sem);
@@ -270,6 +275,7 @@ unlock:unlock_page (page);page_cache_release (page); +mapping_page_locks_done (mapping,1 ); fail: ret =-1 ; goto out;
fs/buffer.c +19 -4
@@ -2174,16 +2174,24 @@if ((offset & (inode->i_sb->s_blocksize -1 )) ==0 ) { offset++; } + + err =mapping_guard_page_locks (mapping,1 ); +if (err) + goto out; + index = size >>PAGE_CACHE_SHIFT ; - err = -ENOMEM ; page =grab_cache_page (mapping, index); -if (!page) +if (!page) { + err = -ENOMEM ; +mapping_page_locks_done (mapping,1 ); goto out; + } err = mapping->a_ops->prepare_write (NULL , page, offset, offset);if (!err) { err = mapping->a_ops->commit_write (NULL , page, offset, offset); }unlock_page (page); +mapping_page_locks_done (mapping,1 );page_cache_release (page);if (err >0 ) err =0 ;
@@ -2572,10 +2580,16 @@if ((offset & (blocksize -1 )) ==0 ) goto out; - ret = -ENOMEM ; + ret =mapping_guard_page_locks (mapping,1 ); +if (ret) + goto out; + page =grab_cache_page (mapping, index); -if (!page) +if (!page) { +mapping_page_locks_done (mapping,1 ); + ret = -ENOMEM ; goto out; + } to = (offset + blocksize) & ~(blocksize -1 ); ret = a_ops->prepare_write (NULL , page, offset, to);
@@ -2588,6 +2602,7 @@ }unlock_page (page);page_cache_release (page); +mapping_page_locks_done (mapping,1 ); out:return ret; }
include/linux/fs.h +4 -0
@@ -325,6 +325,10 @@ loff_t offset, unsigned long nr_segs);struct page * (*get_xip_page)(struct address_space *,sector_t ,int ); + +/* see mapping_guard_page_locks() */ +int (*guard_page_locks)(struct address_space *,int write); +void (*page_locks_done)(struct address_space *,int write); };struct backing_dev_info ;
include/linux/pagemap.h +48 -0
54 lines not on this page · raw message
mm/filemap.c +89 -15
292 lines not on this page · raw message
mm/readahead.c +5 -0
@@ -269,6 +269,10 @@ end_index = ((isize -1 ) >>PAGE_CACHE_SHIFT ); + ret =mapping_guard_page_locks (mapping,0 ); +if (ret) + goto out; +/* * Preallocate as many pages as we will need. */
@@ -302,6 +306,7 @@if (ret)read_pages (mapping, filp, &page_pool, ret);BUG_ON (!list_empty (&page_pool)); +mapping_page_locks_done (mapping,0 ); out:return ret; }
Quoted 4 lines · Zach Brown
I sent an ealier version of this patch to linux-fsdevel and was met with deafening silence.
Maybe because nobody understood your description ;)
Quoted 6 lines
I'm resending the commentary from that first mail and am
including a new version of the patch. This time it has much clearer naming and some kerneldoc blurbs. Here goes...--
Quoted 7 lines
In recent weeks we've been reworking the locking in OCFS2 to simplify things and make it behave more like a "local" Linux file system. We've run into an ordering inversion between a page's
PG_lockedand OCFS2's DLM locks that protect page cache contents. I'm including a patch at the end of this mail that I think is a clean way to give the file system a chance to get the ordering right, but we're open to any and all suggestions. We want to do the cleanest thing.
The patch is of course pretty unwelcome: lots of weird stuff in the core VFS which everyone has to maintain but probably will not test.
So I think we need a better understanding of what the locking inversion problem is, so we can perhaps find a better solution. Bear in mind that ext3 has (rare, unsolved) lock inversion problems in this area as well, so commonality will be sought for.
Quoted 2 lines
OCFS2 maintains page cache coherence between nodes by requiring that a node hold a valid lock while there are active pages in the page cache.
"active pages in the page cache" means present pagecache pages in the node which holds pages in its pagecache, yes?
Quoted 7 lines
The page
cache is invalidated before a node releases a lock so that another node can acquire it. While this invalidation is happening new locks can not be acquired on that node. This is equivalent to a DLM processing thread acquiringPG_lockedduring truncation while holding a DLM lock. Normal file system user tasks come to thea_opswithPG_lockedacquired by their callers before they have a chance to get DLM locks.
So where is the lock inversion?
Perhaps if you were to cook up one of those little threadA/threadB ascii diagrams we could see where the inversion occurs?
Quoted 8 lines · Andrew Morton
Quoted 3 lines · Zach Brown
I sent an ealier version of this patch to linux-fsdevel and was met with deafening silence.
Maybe because nobody understood your description ;)
I have no trouble believing that :) Adding the DLM to the already complicated FS universe isn't exactly a barrel of laughs.
Quoted 2 lines
The patch is of course pretty unwelcome: lots of weird stuff in the core VFS which everyone has to maintain but probably will not test.
Yeah, believe me, we want minimal impact. With this patch we were considering the whole kernel once OCFS2 was merged. We think we *could* probably handle this internally in OCFS2, it would just be a bit of a mess. It could conceivably involve ocfs2-local truncation loops that check special lists of page pointers, that kind of thing. It's unclear to me which is the lesser evil.
Quoted 4 lines
So I think we need a better understanding of what the locking inversion problem is, so we can perhaps find a better solution. Bear in mind that ext3 has (rare, unsolved) lock inversion problems in this area as well, so commonality will be sought for.
This hits almost immediately with one node spinning with appending writes (>> from the shell, say) and another node sitting in tail -f. We don't have the luxury of leaving it unsolved :(
I'd love to learn more about the ext3 case. Maybe I'll bug Andreas :)
Quoted 5 lines
Quoted 2 lines
OCFS2 maintains page cache coherence between nodes by requiring that a node hold a valid lock while there are active pages in the page cache.
"active pages in the page cache" means present pagecache pages in the node which holds pages in its pagecache, yes?
Yeah, a node associates a held DLM lock with the pages it has in its page cache. Other nodes might be doing the same thing, say, if they all have read locks and all have cached reads in their page caches. We can illustrate the mechanics a little more by starting with that scenario and adding another node who wants to write to this file (these associations and locks are per-inode). This new node will try to acquire a write lock on the inode. Before its write lock is granted it will ask the other nodes to all drop their read locks. As part of dropping their locks they will invalidate their page caches. In the future those nodes will have to read the pages back from shared disk, after having gotten the right to do so from the DLM..
I guess it's important to realize that DLM locks can remain valid between system calls. They're pinned by references *during* the system calls, but they remain valid and held on the node between system calls as long as another node doesn't send network messages asking them to be dropped.
Quoted 9 lines
Quoted 7 lines
The page
cache is invalidated before a node releases a lock so that another node can acquire it. While this invalidation is happening new locks can not be acquired on that node. This is equivalent to a DLM processing thread acquiringPG_lockedduring truncation while holding a DLM lock. Normal file system user tasks come to thea_opswithPG_lockedacquired by their callers before they have a chance to get DLM locks.
Quoted 4 lines
So where is the lock inversion?
Perhaps if you were to cook up one of those little threadA/threadB ascii diagrams we could see where the inversion occurs?
Yeah, let me give that a try. I'll try to trim it down to the relevant bits. First let's start with a totally fresh node and have a read get a read DLM lock and populate the page cache on this node:
sys_read
generic_file_aio_read
ocfs2_readpage
ocfs2_data_lock
block_read_full_page
ocfs2_data_unlock
So it was only allowed to proceed past ocfs2_data_lock() towards block_read_full_page() once the DLM granted it a read lock. As it calls ocfs2_data_unlock() it only is dropping this caller's local reference on the lock. The lock still exists on that node and is still valid and holding data in the page cache until it gets a network message saying that another node, who is probably going to be writing, would like the lock dropped.
DLM kernel threads respond to the network messages and truncate the page cache. While the thread is busy with this inode's lock other paths on that node won't be able get locks. Say one of those messages arrives. While a local DLM thread is invalidating the page cache another user thread tries to read:
user thread dlm thread
kthread ... ocfs2_data_convert_worker truncate_inode_pages
sys_read
generic_file_aio_read
* gets page lock
ocfs2_readpage
ocfs2_data_lock
(stuck waiting for dlm)
lock_page
(stuck waiting for page)
The user task holds a page lock while waiting for the DLM to allow it to proceed. The DLM thread is preventing lock granting progress while waiting for the page lock that the user task holds.
I don't know how far to go in explaining what leads up to laying out the locking like this. It is typical (and OCFS2 used to do this) to wait for the DLM locks up in file->{read,write} and pin them for the duration of the IO. This avoids the page lock and DLM lock inversion problem, but it suffers from a host of other problems -- most fatally needing that vma walking to govern holding multiple DLM locks during an IO.
Hmm, I managed to dig up a console log of a sysrq-T of one of the dlm threads stuck in this state.. maybe it clarifies a little. I can't seem to find the matching read state, but it should be more comfortable.. ocfs2_readpage really is the first significant ocfs2 call in that path and the first thing it does is ask the DLM for a read lock.
[<c035a46e>] __wait_on_bit_lock+0x5e/0x70 [<c0147934>] __lock_page+0x84/0x90 [<c0154e47>] truncate_inode_pages_range+0x207/0x320 [<c0154f91>] truncate_inode_pages+0x31/0x40 [<d0f55978>] ocfs2_data_convert_worker+0x148/0x260 [ocfs2] [<d0f55540>] ocfs2_generic_unblock_lock+0x90/0x380 [ocfs2] [<d0f55b11>] ocfs2_unblock_data+0x81/0x360 [ocfs2] [<d0f5674d>] ocfs2_process_blocked_lock+0x9d/0x3f0 [ocfs2] [<d0f8b089>] ocfs2_vote_thread_do_work+0xa9/0x270 [ocfs2] [<d0f8b36f>] ocfs2_vote_thread+0x5f/0x170 [ocfs2] [<c0137c9a>] kthread+0xba/0xc0
I guess all that really illustrates its lots of funky DLM internals and then truncate_inode_pages(). :)
Does this help?
- z
Quoted 31 lines · Zach Brown
Quoted 5 lines
So where is the lock inversion?
Perhaps if you were to cook up one of those little threadA/threadB ascii diagrams we could see where the inversion occurs?
Yeah, let me give that a try. I'll try to trim it down to the relevant bits. First let's start with a totally fresh node and have a read get a read DLM lock and populate the page cache on this node:
sys_read
generic_file_aio_read
ocfs2_readpage
ocfs2_data_lock
block_read_full_page
ocfs2_data_unlockSo it was only allowed to proceed past
ocfs2_data_lock()towardsblock_read_full_page()once the DLM granted it a read lock. As it callsocfs2_data_unlock()it only is dropping this caller's local reference on the lock. The lock still exists on that node and is still valid and holding data in the page cache until it gets a network message saying that another node, who is probably going to be writing, would like the lock dropped.DLM kernel threads respond to the network messages and truncate the page cache. While the thread is busy with this inode's lock other paths on that node won't be able get locks. Say one of those messages arrives. While a local DLM thread is invalidating the page cache another user thread tries to read:
Quoted 6 lines
user thread dlm thread
kthread ... ocfs2_data_convert_worker
I assume there's an ocfs2_data_lock hereabouts?
Quoted 10 lines
truncate_inode_pagessys_read
generic_file_aio_read
* gets page lock
ocfs2_readpage
ocfs2_data_lock
(stuck waiting for dlm)
lock_page
(stuck waiting for page)
Why does ocfs2_readpage() need to take ocfs2_data_lock? (Is ocfs2_data_lock a range-based read-lock thing, or what?)
Quoted 10 lines
The user task holds a page lock while waiting for the DLM to allow it to proceed. The DLM thread is preventing lock granting progress while waiting for the page lock that the user task holds.
I don't know how far to go in explaining what leads up to laying out the locking like this. It is typical (and OCFS2 used to do this) to wait for the DLM locks up in file->{read,write} and pin them for the duration of the IO. This avoids the page lock and DLM lock inversion problem, but it suffers from a host of other problems -- most fatally needing that vma walking to govern holding multiple DLM locks during an IO.
Oh.
Have you considered using invalidate_inode_pages() instead of truncate_inode_pages()? If that leaves any pages behind, drop the read lock, sleep a bit, try again - something klunky like that might get you out of trouble, dunno.
Quoted 53 lines · Andrew Morton
Quoted 31 lines · Zach Brown
Quoted 5 lines
So where is the lock inversion?
Perhaps if you were to cook up one of those little threadA/threadB ascii diagrams we could see where the inversion occurs?
Yeah, let me give that a try. I'll try to trim it down to the relevant bits. First let's start with a totally fresh node and have a read get a read DLM lock and populate the page cache on this node:
sys_read
generic_file_aio_read
ocfs2_readpage
ocfs2_data_lock
block_read_full_page
ocfs2_data_unlockSo it was only allowed to proceed past
ocfs2_data_lock()towardsblock_read_full_page()once the DLM granted it a read lock. As it callsocfs2_data_unlock()it only is dropping this caller's local reference on the lock. The lock still exists on that node and is still valid and holding data in the page cache until it gets a network message saying that another node, who is probably going to be writing, would like the lock dropped.DLM kernel threads respond to the network messages and truncate the page cache. While the thread is busy with this inode's lock other paths on that node won't be able get locks. Say one of those messages arrives. While a local DLM thread is invalidating the page cache another user thread tries to read:
Quoted 6 lines
user thread dlm thread
kthread ... ocfs2_data_convert_workerI assume there's an
ocfs2_data_lockhereabouts?Quoted 10 lines
truncate_inode_pagessys_read
generic_file_aio_read
* gets page lock
ocfs2_readpage
ocfs2_data_lock
(stuck waiting for dlm)
lock_page
(stuck waiting for page)
Quoted 2 lines
Why does
ocfs2_readpage()need to takeocfs2_data_lock? (Isocfs2_data_locka range-based read-lock thing, or what?)
If I get this right, then they need it to guarantee that noone is writing to that file offset on a different node at the same time otherwise this readpage will see stale data.
What I would ask is why does the above dlm thread need to hold the data_lock duing truncate_inode_pages? Just take it _after_ truncate_inode_pages(), check that there noone instantiated any pages in between truncate_inode_pages and data_lock acquisition and there are no pages all is great and if there are some drop data_lock, cond_resched(), and repeat truncate_inode_pages, etc. Eventually it will succeed. And no need for nasty VFS patch you are proposing...
Quoted 11 lines
Quoted 10 lines
The user task holds a page lock while waiting for the DLM to allow it to proceed. The DLM thread is preventing lock granting progress while waiting for the page lock that the user task holds.
I don't know how far to go in explaining what leads up to laying out the locking like this. It is typical (and OCFS2 used to do this) to wait for the DLM locks up in file->{read,write} and pin them for the duration of the IO. This avoids the page lock and DLM lock inversion problem, but it suffers from a host of other problems -- most fatally needing that vma walking to govern holding multiple DLM locks during an IO.
Quoted 6 lines
Oh.
Have you considered using
invalidate_inode_pages()instead oftruncate_inode_pages()? If that leaves any pages behind, drop the read lock, sleep a bit, try again - something klunky like that might get you out of trouble, dunno.
Or my suggestion above, I think mine has higher chance of needing less taking/dropping of data_lock as at the end of the truncate there will be no pages left, unless there is an overeager read process at work on that mapping at the same time.
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
Quoted 2 lines · Anton Altaparmakov
What I would ask is why does the above dlm thread need to hold the
data_lockduingtruncate_inode_pages?
I hope the mail I just sent made that a little more clear.
Quoted 2 lines
and repeat
truncate_inode_pages, etc. Eventually it will succeed. And no need for nasty VFS patch you are proposing...
Yeah, this also came to me this morning in the shower :) There are some hard cases because these are actually read-write locks, but it might be doable. We're discussing it.
Quoted 2 lines
no pages left, unless there is an overeager read process at work on that mapping at the same time.
I fear that it'll be pretty easy to get bad capture effects, but maybe that's ok. We'll see.
- z
Quoted 10 lines · Andrew Morton
Quoted 6 lines
user thread dlm thread
kthread ... ocfs2_data_convert_workerI assume there's an
ocfs2_data_lockhereabouts?
No. The thread is working behind the scenes to bring the lock to a state where others can get the lock again. It doesn't explicitly hold the lock like regular fs paths do, but no one else can get the lock until it finishes its work.
A wild over-simplification of the logic goes something like this:
dlm_lock(lock) {
wait_event(, !lock->blocked);
atomic_inc(&lock->ref);
}dlm_unlock(lock) {
if (atomic_dec_and_test(&lock->ref) {
wake_worker_thread()
}
}incoming_drop_message() {
lock->blocked = 1;
wake_worker_thread();
}worker_thread() {
wait_event(, atomic_read(&lock->ref) == 0);
truncate_inode_pages();
lock->blocked = 0;
}It's much worse than that, but that's the basic idea for our problem here. A network message comes in which forbids additional acquiry of the lock until all the current holders have released it and the the worker thread has truncated the page cache.
So in our inversion ocfs2_readpage() holds a page lock while waiting in dlm_lock() for ->blocked to be clear, but worker_thread() can't clear ->blocked until it locks and truncates the page that ocfs2_readpage() holds.
Quoted 4 lines
Have you considered using
invalidate_inode_pages()instead oftruncate_inode_pages()? If that leaves any pages behind, drop the read lock, sleep a bit, try again - something klunky like that might get you out of trouble, dunno.
Yeah, that doesn't work because the locked pages aren't making forward progress. In the read case it would be fine to just skip them, they're about to be re-read once ocfs2_readpage() gets its DLM lock. But we'd have to strongly identify locked pages waiting in ocfs2_readpage(). And in the write case the DLM thread is responsible writing those pages back before releasing the lock so that other nodes can read the pages in.
We think we could do this sort of thing with a specific truncation loop, but that's the nasty code I wasn't sure would be any more acceptable than this nasty core patch. The truncation loop would need a way to identify locked pages that it can just ignore or initiate writeback on because it'd know that it's just another ocfs2 path that holds the page lock while waiting for a dlm lock. I wonder if we could just do it with PG_fs_misc. I can give it a try if that doesn't send shivers down everyone's spine.
- z
Quoted 4 lines
We think we could do this sort of thing with a specific truncation loop, but that's the nasty code I wasn't sure would be any more acceptable than this nasty core patch.
OK, I'm appending a patch which attempts to solve this problem in OCFS2 code instead of up in the VFS. It's against a branch in OCFS2 svn, but one which isn't incomprehensibly different from what is in -mm.
The idea is to stop the DLM thread from deadlocking on locked pages (which are held by other tasks waiting for the DLM) by tagging them with PG_fs_misc. The DLM thread knows that the pages are locked by callers who are waiting for the thread so they can assume ownership of the page lock, of a sort. The DLM thread writes back dirty data in the page manually so it doesn't unlock the callers page. It can basically ignore clean pages because they're going to be reread by the lock holder once they get the DLM lock. Comments in the patch above ocfs2_data_convert_worker, get_lock_or_fs_misc, ocfs2_write_dirty_fs_misc_page, and ocfs2_set_fs_misc lay this out.
It introduces block_read_full_page() and truncate_inode_pages() derivatives which understand the PG_fs_misc special case. It needs a few export patches to the core, but the real burden is on OCFS2 to keep these derivatives up to date. Maybe after they stabilize for a while, and say another clustered file system has similar problems, we could roll some core helpers out of them. But for now it illustrates what the solution would look like in OCFS2. I'm sure I missed some corner cases, but this passes the most basic functional testing.
The specific exports it needs from 2.6.14-rc4-mm1 are:
$ grep '+EXPORT' patches/*.patch patches/add-wake_up_page_all.patch:+EXPORT_SYMBOL(__wake_up_bit_all); patches/add-wake_up_page_all.patch:+EXPORT_SYMBOL(wake_up_page_all); patches/export-pagevec-helpers.patch:+EXPORT_SYMBOL_GPL(pagevec_lookup); patches/export-page_waitqueue.patch:+EXPORT_SYMBOL_GPL(page_waitqueue); patches/export-truncate_complete_pate.patch:+EXPORT_SYMBOL(truncate_complete_page); patches/export-wake_up_page.patch:+EXPORT_SYMBOL(wake_up_page);
that wake_up_page_all() is just a variant that provides 0 nr_exclusive to __wake_up_bit():
-void fastcall __wake_up_bit(wait_queue_head_t *wq, void *word, int bit) +static inline int __wake_up_bit_nr(wait_queue_head_t *wq, void *word, int bit,
+ int nr_exclusive)
{
struct wait_bit_key key = __WAIT_BIT_KEY_INITIALIZER(word, bit);
if (waitqueue_active(wq))
- __wake_up(wq, TASK_INTERRUPTIBLE|TASK_UNINTERRUPTIBLE, 1, &key);
+ __wake_up(wq, TASK_INTERRUPTIBLE|TASK_UNINTERRUPTIBLE,+ nr_exclusive, &key);
+}
+
+void fastcall __wake_up_bit(wait_queue_head_t *wq, void *word, int bit) +{
+ __wake_up_bit_nr(wq, word, bit, 1);
}
EXPORT_SYMBOL(__wake_up_bit);
+void fastcall __wake_up_bit_all(wait_queue_head_t *wq, void *word, int bit) +{
+ __wake_up_bit_nr(wq, word, bit, 0);
+}
+EXPORT_SYMBOL(__wake_up_bit_all);
Is this preferable to the core changes and is it something that's mergeable? We'd love to come to a solution that won't be a barrier to merging so we can get on with it. I can send that exporting series if we decide this is the right thing.
- z
ocfs2/dlmglue.h +7 -0
@@ -54,6 +54,8 @@ int ocfs2_create_new_inode_locks(structint ocfs2_drop_inode_locks (struct inode *inode);int ocfs2_data_lock (struct inode *inode,int write); +int ocfs2_data_lock_holding_page (struct inode *inode,int write, +struct page *page);void ocfs2_data_unlock (struct inode *inode,int write);int ocfs2_rw_lock (struct inode *inode,int write);
@@ -70,6 +72,11 @@ int ocfs2_meta_lock_full(struct inode *i/* 99% of the time we don't want to supply any additional flags -- * those are for very specific cases only. */ #define ocfs2_meta_lock (i, h, b, e) ocfs2_meta_lock_full(i, h, b, e, 0) +int ocfs2_meta_lock_holding_page (struct inode *inode, +ocfs2_journal_handle *handle, +struct buffer_head **ret_bh, +int ex, +struct page *page);void ocfs2_meta_unlock (struct inode *inode,int ex); int ocfs2_super_lock(ocfs2_super *osb,
ocfs2/aops.c +8 -7
@@ -130,8 +130,8 @@ bail:return err; } -static int ocfs2_get_block (struct inode *inode,sector_t iblock, -struct buffer_head *bh_result,int create) +int ocfs2_get_block (struct inode *inode,sector_t iblock, +struct buffer_head *bh_result,int create) {int err =0 ;u64 p_blkno, past_eof;
@@ -198,7 +198,7 @@ static int ocfs2_readpage(struct file *fmlog_entry ("(0x%p, %lu)\n" , file, (page ? page->index :0 )); - ret =ocfs2_meta_lock (inode,NULL ,NULL ,0 ); + ret =ocfs2_meta_lock_holding_page (inode,NULL ,NULL ,0 , page);if (ret <0 ) {mlog_errno (ret); goto out;
@@ -226,7 +226,7 @@ static int ocfs2_readpage(struct file *f goto out_alloc; } - ret =ocfs2_data_lock (inode,0 ); + ret =ocfs2_data_lock_holding_page (inode,0 , page);if (ret <0 ) {mlog_errno (ret); goto out_alloc;
@@ -283,7 +283,7 @@ int ocfs2_prepare_write(struct file *filmlog_entry ("(0x%p, 0x%p, %u, %u)\n" , file, page, from, to); - ret =ocfs2_meta_lock (inode,NULL ,NULL ,0 ); + ret =ocfs2_meta_lock_holding_page (inode,NULL ,NULL ,0 , page);if (ret <0 ) {mlog_errno (ret); goto out;
@@ -397,13 +397,14 @@ static int ocfs2_commit_write(struct fil locklevel =1 ; } - ret =ocfs2_meta_lock (inode,NULL , &di_bh, locklevel); + ret =ocfs2_meta_lock_holding_page (inode,NULL , &di_bh, locklevel, + page);if (ret <0 ) {mlog_errno (ret); goto out; } - ret =ocfs2_data_lock (inode,1 ); + ret =ocfs2_data_lock_holding_page (inode,1 , page);if (ret <0 ) {mlog_errno (ret); goto out_unlock_meta;
ocfs2/aops.h +2 -0
@@ -22,6 +22,8 @@#ifndef OCFS2_AOPS_H #define OCFS2_AOPS_H +int ocfs2_get_block (struct inode *inode,sector_t iblock, +struct buffer_head *bh_result,int create);int ocfs2_prepare_write (struct file *file,struct page *page,unsigned from,unsigned to);
ocfs2/dlmglue.c +249 -20
314 lines not on this page · raw message
Quoted 3 lines · Zach Brown
It introduces
block_read_full_page()andtruncate_inode_pages()derivatives which understand thePG_fs_miscspecial case. It needs a few export patches to the core, but the real burden is on OCFS2 to keep these derivatives up to date.
The way you do it looks nice, but the exports aren't a big no-way. That stuff is far too internal to be exported. Either we can get Andrew to agree on moving those bits into the codepath for all filesystems or we need to do some hackery where every functions gets renamed to __function with an argument int cluster_aware and we have to functions inling them, one normal and one for cluster filesystems.
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Zach Brown
It introduces
block_read_full_page()andtruncate_inode_pages()derivatives which understand thePG_fs_miscspecial case. It needs a few export patches to the core, but the real burden is on OCFS2 to keep these derivatives up to date.
Quoted 6 lines
The way you do it looks nice, but the exports aren't a big no-way. That stuff is far too internal to be exported. Either we can get Andrew to agree on moving those bits into the codepath for all filesystems or we need to do some hackery where every functions gets renamed to
__functionwith an argument intcluster_awareand we have to functions inling them, one normal and one for cluster filesystems.
Yeah, I can certainly appreciate that line of reasoning. I'm happy to do that work, but it'd be nice to get some assurance that it won't be wasted effort. Andrew, is this a reasonable direction to take things in? We'd avoid the exports by introducing some wrappers and helpers to the core that OCFS2 would call..
- z
Quoted 15 lines · Zach Brown
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines · Zach Brown
It introduces
block_read_full_page()andtruncate_inode_pages()derivatives which understand thePG_fs_miscspecial case. It needs a few export patches to the core, but the real burden is on OCFS2 to keep these derivatives up to date.Quoted 6 lines
The way you do it looks nice, but the exports aren't a big no-way. That stuff is far too internal to be exported. Either we can get Andrew to agree on moving those bits into the codepath for all filesystems or we need to do some hackery where every functions gets renamed to
__functionwith an argument intcluster_awareand we have to functions inling them, one normal and one for cluster filesystems.
Quoted 5 lines
Yeah, I can certainly appreciate that line of reasoning. I'm happy to do that work, but it'd be nice to get some assurance that it won't be wasted effort. Andrew, is this a reasonable direction to take things in? We'd avoid the exports by introducing some wrappers and helpers to the core that OCFS2 would call..
It depends on what the patch ends up looking like I guess.
All those games with PG_fs_misc look awfully similar to lock_page() - I'd have thought there's some room for rationalising code in there.
The overall approach would be to avoid adding overhead and complexity for other filesystems and to only export symbols which constitute a sensible
API: avoid exporting weirdo internal helpers which we might change in the future (but we don't care about external modules, so why does this matter? Because some of them are GPL?)
Quoted 3 lines
All those games with
PG_fs_misclook awfully similar tolock_page()- I'd have thought there's some room for rationalising code in there.
Yeah, it's very much like a secondary restricted page lock. Instead of sleeping when you can't get the page lock, you can also try acquiring this sort of secondary page lock bit which lets you do *very* specific things. Maybe it's not *that* weird, I just didn't think that the core would be interested in supporting that notion. I can try rolling a patch to see what a more sensible API would look like.
Quoted 3 lines
The overall approach would be to avoid adding overhead and complexity for other filesystems and to only export symbols which constitute a sensible API:
*nod*
- z
Quoted 3 lines · Christoph Hellwig
The way you do it looks nice, but the exports aren't a big no-way. That stuff is far too internal to be exported.
Yeah, understood.
So here's an entirely different approach. We add a return value to readpage, prepare_write, and commit_write in the style of WRITEPAGE_ACTIVATE. It tells the callers that the locked page they handed to the op has been unlocked and that they should back off and try again. This lets the ocfs2 methods notice when they're going to sleep on the DLM and can unlock the page before sleeping.
This compiles but hasn't been tested.
include/linux/fs.h +24 -0
@@ -292,6 +292,30 @@ struct iattr {*/ #include <linux/quota.h> +/** +* enum positive_aop_returns - aop return codes with specific semantics +* +* @WRITEPAGE_ACTIVATE: IO was not started: activate page. Returned by +* writepage(); +* +* @AOP_TRUNCATED_PAGE: The AOP method that was handed a locked page has +* unlocked it and the page might have been truncated. +* The caller should back up to acquiring a new page and +* trying again. The aop will be taking reasonable +* precautions not to livelock. If the caller held a page +* reference, it should drop it before retrying. Returned +* by readpage(), prepare_write(), and commit_write(). +* +* address_space_operation functions return these large constants to indicate +* special semantics to the caller. These are much larger than the bytes in a +* page to allow for functions that return the number of bytes operated on in a +* given page. +*/ +enum positive_aop_returns { +WRITEPAGE_ACTIVATE =0x80000 , +AOP_TRUNCATED_PAGE =0x80001 , +}; +/* * oh the beauties of C type declarations. */
include/linux/writeback.h +0 -6
@@ -60,12 +60,6 @@ struct writeback_control { };/* -* ->writepage() return values (make these much larger than a pagesize, in -* case some fs is returning number-of-bytes-written from writepage) -*/ -#define WRITEPAGE_ACTIVATE 0x80000/* IO was not started: activate page */ - -/* * fs/fs-writeback.c */ void writeback_inodes (struct writeback_control *wbc);
drivers/block/loop.c +18 -5
@@ -213,7 +213,7 @@ static int do_lo_send_aops(struct loop_dstruct address_space_operations *aops = mapping->a_ops;pgoff_t index;unsigned offset, bv_offs; -int len, ret =0 ; +int len, ret;down (&mapping->host->i_sem); index = pos >>PAGE_CACHE_SHIFT ;
@@ -232,9 +232,15 @@ static int do_lo_send_aops(struct loop_d page =grab_cache_page (mapping, index);if (unlikely (!page)) goto fail; -if (unlikely (aops->prepare_write (file, page, offset, - offset + size))) + ret = aops->prepare_write (file, page, offset, + offset + size); +if (unlikely (ret)) { +if (ret ==AOP_TRUNCTED_PAGE ) { +page_cache_release (page); +continue ; + } goto unlock; + } transfer_result =lo_do_transfer (lo,WRITE , page, offset, bvec->bv_page, bv_offs, size,IV );if (unlikely (transfer_result)) {
@@ -251,9 +257,15 @@ static int do_lo_send_aops(struct loop_dkunmap_atomic (kaddr,KM_USER0 ); }flush_dcache_page (page); -if (unlikely (aops->commit_write (file, page, offset, - offset + size))) + ret = aops->commit_write (file, page, offset, + offset + size); +if (unlikely (ret)) { +if (ret ==AOP_TRUNCATED_PAGE ) { +page_cache_release (page); +continue ; + } goto unlock; + }if (unlikely (transfer_result)) goto unlock; bv_offs += size;
@@ -264,6 +276,7 @@ static int do_lo_send_aops(struct loop_dunlock_page (page);page_cache_release (page); } + ret =0 ; out:up (&mapping->host->i_sem);return ret;
mm/filemap.c +50 -23
142 lines not on this page · raw message
mm/readahead.c +8 -5
@@ -159,7 +159,7 @@ static int read_pages(struct address_spa {unsigned page_idx;struct pagevec lru_pvec; -int ret =0 ; +int ret;if (mapping->a_ops->readpages) { ret = mapping->a_ops->readpages (filp, mapping, pages, nr_pages);
@@ -172,14 +172,17 @@ static int read_pages(struct address_spalist_del (&page->lru);if (!add_to_page_cache (page, mapping, page->index,GFP_KERNEL )) { - mapping->a_ops->readpage (filp, page); -if (!pagevec_add (&lru_pvec, page)) + ret = mapping->a_ops->readpage (filp, page); +if (ret !=AOP_TRUNCATED_PAGE && + !pagevec_add (&lru_pvec, page)) {__pagevec_lru_add (&lru_pvec); - }else { -page_cache_release (page); +continue ; + } } +page_cache_release (page); }pagevec_lru_add (&lru_pvec); + ret =0 ; out:return ret; }
Hmm, I forgot to cc: -fsdevel on this mail yesterday. My mistake. This is part of the thread that is trying to address an inversion between page locks and dlm locks in OCFS2:
http://lkml.org/lkml/2005/10/17/275
- z
---------------------
Quoted 3 lines · Christoph Hellwig
The way you do it looks nice, but the exports aren't a big no-way. That stuff is far too internal to be exported.
Yeah, understood.
So here's an entirely different approach. We add a return value to readpage, prepare_write, and commit_write in the style of WRITEPAGE_ACTIVATE. It tells the callers that the locked page they handed to the op has been unlocked and that they should back off and try again. This lets the ocfs2 methods notice when they're going to sleep on the DLM and can unlock the page before sleeping.
This compiles but hasn't been tested.
include/linux/fs.h +24 -0
@@ -292,6 +292,30 @@ struct iattr {*/ #include <linux/quota.h> +/** +* enum positive_aop_returns - aop return codes with specific semantics +* +* @WRITEPAGE_ACTIVATE: IO was not started: activate page. Returned by +* writepage(); +* +* @AOP_TRUNCATED_PAGE: The AOP method that was handed a locked page has +* unlocked it and the page might have been truncated. +* The caller should back up to acquiring a new page and +* trying again. The aop will be taking reasonable +* precautions not to livelock. If the caller held a page +* reference, it should drop it before retrying. Returned +* by readpage(), prepare_write(), and commit_write(). +* +* address_space_operation functions return these large constants to indicate +* special semantics to the caller. These are much larger than the bytes in a +* page to allow for functions that return the number of bytes operated on in a +* given page. +*/ +enum positive_aop_returns { +WRITEPAGE_ACTIVATE =0x80000 , +AOP_TRUNCATED_PAGE =0x80001 , +}; +/* * oh the beauties of C type declarations. */
include/linux/writeback.h +0 -6
@@ -60,12 +60,6 @@ struct writeback_control { };/* -* ->writepage() return values (make these much larger than a pagesize, in -* case some fs is returning number-of-bytes-written from writepage) -*/ -#define WRITEPAGE_ACTIVATE 0x80000/* IO was not started: activate page */ - -/* * fs/fs-writeback.c */ void writeback_inodes (struct writeback_control *wbc);
drivers/block/loop.c +18 -5
@@ -213,7 +213,7 @@ static int do_lo_send_aops(struct loop_dstruct address_space_operations *aops = mapping->a_ops;pgoff_t index;unsigned offset, bv_offs; -int len, ret =0 ; +int len, ret;down (&mapping->host->i_sem); index = pos >>PAGE_CACHE_SHIFT ;
@@ -232,9 +232,15 @@ static int do_lo_send_aops(struct loop_d page =grab_cache_page (mapping, index);if (unlikely (!page)) goto fail; -if (unlikely (aops->prepare_write (file, page, offset, - offset + size))) + ret = aops->prepare_write (file, page, offset, + offset + size); +if (unlikely (ret)) { +if (ret ==AOP_TRUNCTED_PAGE ) { +page_cache_release (page); +continue ; + } goto unlock; + } transfer_result =lo_do_transfer (lo,WRITE , page, offset, bvec->bv_page, bv_offs, size,IV );if (unlikely (transfer_result)) {
@@ -251,9 +257,15 @@ static int do_lo_send_aops(struct loop_dkunmap_atomic (kaddr,KM_USER0 ); }flush_dcache_page (page); -if (unlikely (aops->commit_write (file, page, offset, - offset + size))) + ret = aops->commit_write (file, page, offset, + offset + size); +if (unlikely (ret)) { +if (ret ==AOP_TRUNCATED_PAGE ) { +page_cache_release (page); +continue ; + } goto unlock; + }if (unlikely (transfer_result)) goto unlock; bv_offs += size;
@@ -264,6 +276,7 @@ static int do_lo_send_aops(struct loop_dunlock_page (page);page_cache_release (page); } + ret =0 ; out:up (&mapping->host->i_sem);return ret;
mm/filemap.c +50 -23
142 lines not on this page · raw message
mm/readahead.c +8 -5
@@ -159,7 +159,7 @@ static int read_pages(struct address_spa {unsigned page_idx;struct pagevec lru_pvec; -int ret =0 ; +int ret;if (mapping->a_ops->readpages) { ret = mapping->a_ops->readpages (filp, mapping, pages, nr_pages);
@@ -172,14 +172,17 @@ static int read_pages(struct address_spalist_del (&page->lru);if (!add_to_page_cache (page, mapping, page->index,GFP_KERNEL )) { - mapping->a_ops->readpage (filp, page); -if (!pagevec_add (&lru_pvec, page)) + ret = mapping->a_ops->readpage (filp, page); +if (ret !=AOP_TRUNCATED_PAGE && + !pagevec_add (&lru_pvec, page)) {__pagevec_lru_add (&lru_pvec); - }else { -page_cache_release (page); +continue ; + } } +page_cache_release (page); }pagevec_lru_add (&lru_pvec); + ret =0 ; out:return ret; }
Quoted 10 lines · Zach Brown
The specific exports it needs from 2.6.14-rc4-mm1 are:
$ grep '+EXPORT' patches/*.patch patches/add-wake_up_page_all.patch:+EXPORT_SYMBOL(__wake_up_bit_all); patches/add-wake_up_page_all.patch:+EXPORT_SYMBOL(wake_up_page_all); patches/export-pagevec-helpers.patch:+EXPORT_SYMBOL_GPL(pagevec_lookup); patches/export-page_waitqueue.patch:+EXPORT_SYMBOL_GPL(page_waitqueue); patches/export-truncate_complete_pate.patch:+EXPORT_SYMBOL(truncate_complete_page); patches/export-wake_up_page.patch:+EXPORT_SYMBOL(wake_up_page);
Exporting page_waitqueue seems wrong. Might be better to add a core function to do the wait_event(*page_waitqueue(page), PageFsMisc(page)); and export that.
How did you come up with this mix of GPL and non-GPL?
Quoted 26 lines
that
wake_up_page_all()is just a variant that provides 0nr_exclusiveto__wake_up_bit():-void fastcall
__wake_up_bit(wait_queue_head_t *wq, void *word, int bit)+static inline int __wake_up_bit_nr(wait_queue_head_t *wq, void *word, int bit,
+ intnr_exclusive){ struct wait_bit_key key = __WAIT_BIT_KEY_INITIALIZER(word, bit); if (waitqueue_active(wq)) - __wake_up(wq, TASK_INTERRUPTIBLE|TASK_UNINTERRUPTIBLE, 1, &key); + __wake_up(wq, TASK_INTERRUPTIBLE|TASK_UNINTERRUPTIBLE,+
nr_exclusive, &key);
+}
+
+void fastcall__wake_up_bit(wait_queue_head_t *wq, void *word, int bit)+{
+__wake_up_bit_nr(wq, word, bit, 1);
}
EXPORT_SYMBOL(__wake_up_bit);+void fastcall
__wake_up_bit_all(wait_queue_head_t *wq, void *word, int bit)+{
+__wake_up_bit_nr(wq, word, bit, 0);
+}
+EXPORT_SYMBOL(__wake_up_bit_all);
Quoted 4 lines
Is this preferable to the core changes and is it something that's mergeable? We'd love to come to a solution that won't be a barrier to merging so we can get on with it. I can send that exporting series if we decide this is the right thing.
The above looks sane enough. Please run it by Bill?
Quoted 9 lines
Quoted 7 lines
patches/add-wake_up_page_all.patch:+EXPORT_SYMBOL(__wake_up_bit_all); patches/add-wake_up_page_all.patch:+EXPORT_SYMBOL(wake_up_page_all); patches/export-pagevec-helpers.patch:+EXPORT_SYMBOL_GPL(pagevec_lookup); patches/export-page_waitqueue.patch:+EXPORT_SYMBOL_GPL(page_waitqueue); patches/export-truncate_complete_pate.patch:+EXPORT_SYMBOL(truncate_complete_page); patches/export-wake_up_page.patch:+EXPORT_SYMBOL(wake_up_page);
Quoted 3 lines
Exporting
page_waitqueueseems wrong. Might be better to add a core function to do thewait_event(*page_waitqueue(page), PageFsMisc(page)); and export that.
Sure thing.
Quoted 1 line
How did you come up with this mix of GPL and non-GPL?
Carelessness. I'll aim for _GPL for anything that still needs to be exported.
Quoted 1 line
The above looks sane enough. Please run it by Bill?
OK.
- z
Quoted 30 lines · Zach Brown
I sent an ealier version of this patch to linux-fsdevel and was met with deafening silence. I'm resending the commentary from that first mail and am including a new version of the patch. This time it has much clearer naming and some kerneldoc blurbs. Here goes...
--
In recent weeks we've been reworking the locking in OCFS2 to simplify things and make it behave more like a "local" Linux file system. We've run into an ordering inversion between a page's
PG_lockedand OCFS2's DLM locks that protect page cache contents. I'm including a patch at the end of this mail that I think is a clean way to give the file system a chance to get the ordering right, but we're open to any and all suggestions. We want to do the cleanest thing.OCFS2 maintains page cache coherence between nodes by requiring that a node hold a valid lock while there are active pages in the page cache. The page cache is invalidated before a node releases a lock so that another node can acquire it. While this invalidation is happening new locks can not be acquired on that node. This is equivalent to a DLM processing thread acquiring
PG_lockedduring truncation while holding a DLM lock. Normal file system user tasks come to thea_opswithPG_lockedacquired by their callers before they have a chance to get DLM locks.We talked a little about modifying the invalidation path to avoid waiting for pages that are held by an OCFS2 path that is waiting for a DLM lock. It seems awfully hard to get right without some very nasty code. The truncation on lock removal has to write back dirty pages so that other nodes can read it -- it can't simply skip these pages if they happened to be locked.
Quoted 6 lines
So we aimed right for the lock ordering inversion problem with the appended patch. It gives file systems a callback that is tried before page locks are acquired that are going to be passed in to the file system's
a_opsmethods.So, what do people think about this? Is it reasonable to patch the core to help OCFS2 with this ordering inversion?
Sorry for the "deafening silence" on fs-devel. I was trying to see what exactly I need and how to combine with what you are trying to do, before I reply. Unfortunately, I don't understand your lock inversion problem well enough :(
I was looking at ext3 pagelock, trasaction ordering. I wanted a way to reverse the ordering to support writepages() cleanly. I guess what you are proposing could be used (in a weird way) to that, except that I need the support in different places (callers of writepage, writepages). BTW, I wasn't comfortable touching VFS locking just of ext3 purpose :(
Thanks,
Badari