ocfs2: fix slow deleting
cover ocfs2: fix slow deleting
There is a use case that the app deletes huge number(XX kilo) of files in every 5 minutes. The deletions of some specific files are extreamly slow(costing xx~xxx seconds). That is unacceptable.
Reading out the dir entries and the relavent inodes cost time. And we are doing that with i_mutex held, it causes unlink path waiting on the mutex for long time.
fix:
We drops and retake the mutex in the duration giving change to unlink to go on. Also, for live nodes, one node only scan and recover this slot where the node resides(helps performance). And always do it at each scan time. For those dead (not mounted), we do it when we "should". And for dead slots, no dropping-retaking mutex is needed.
cover ocfs2: fix slow deleting
Quoted 7 lines · Wengang Wang
There is a use case that the app deletes huge number(XX kilo) of files in every 5 minutes. The deletions of some specific files are extreamly slow(costing xx~xxx seconds). That is unacceptable.
Reading out the dir entries and the relavent inodes cost time. And we are doing that with
i_mutexheld, it causes unlink path waiting on the mutex for long time.
Quoted 6 lines
fix:
We drops and retake the mutex in the duration giving change to unlink to go on. Also, for live nodes, one node only scan and recover this slot where the node resides(helps performance). And always do it at each scan time. For those dead (not mounted), we do it when we "should". And for dead slots, no dropping-retaking mutex is needed.
Yes, this is a good issue to tackle. I will read the patch in greater detail later. But offhand, I have two comments.
1. "should" is not descriptive. I am assuming you mean do it only during actual recovery. If so, that would be incorrect. Say node 0 unlinks a file that was being used by node 1. Node 0 dies. Recovery will notice that that inode is active and not delete it. If node 1 dies, or is unable to delete
the file for any other reason, then our only hope is orphan scan.
2. All nodes have to scan all slots. Even live slots. I remember we did for a reason. And that reason should be in the comment in the patch written by Srini.
cover ocfs2: fix slow deleting
Quoted 18 lines · Sunil Mushran
Quoted 7 lines · Wengang Wang
There is a use case that the app deletes huge number(XX kilo) of files in every 5 minutes. The deletions of some specific files are extreamly slow(costing xx~xxx seconds). That is unacceptable.
Reading out the dir entries and the relavent inodes cost time. And we are doing that with
i_mutexheld, it causes unlink path waiting on the mutex for long time.Quoted 6 lines
fix:
We drops and retake the mutex in the duration giving change to unlink to go on. Also, for live nodes, one node only scan and recover this slot where the node resides(helps performance). And always do it at each scan time. For those dead (not mounted), we do it when we "should". And for dead slots, no dropping-retaking mutex is needed.Yes, this is a good issue to tackle. I will read the patch in greater detail later. But offhand, I have two comments.
Quoted 6 lines
1. "should" is not descriptive. I am assuming you mean do it only during actual recovery. If so, that would be incorrect. Say node 0 unlinks a file that was being used by node 1. Node 0 dies. Recovery will notice that that inode is active and not delete it. If node 1 dies, or is unable to delete
the file for any other reason, then our only hope is orphan scan.
Sorry. the "should" doesn't mean a actual recovery. I meant when "os->os_seqno == seqno", the orginal condition determining whether we do queue the scans.
Quoted 4 lines
2. All nodes have to scan all slots. Even live slots. I remember we did for a reason. And that reason should be in the comment in the patch written by Srini.
Oh... I will check Srini's patch.
thanks,
wengang.
cover ocfs2: fix slow deleting
Quoted 31 lines · Wengang Wang
Quoted 18 lines · Sunil Mushran
Quoted 7 lines · Wengang Wang
There is a use case that the app deletes huge number(XX kilo) of files in every 5 minutes. The deletions of some specific files are extreamly slow(costing xx~xxx seconds). That is unacceptable.
Reading out the dir entries and the relavent inodes cost time. And we are doing that with
i_mutexheld, it causes unlink path waiting on the mutex for long time.Quoted 6 lines
fix:
We drops and retake the mutex in the duration giving change to unlink to go on. Also, for live nodes, one node only scan and recover this slot where the node resides(helps performance). And always do it at each scan time. For those dead (not mounted), we do it when we "should". And for dead slots, no dropping-retaking mutex is needed.Yes, this is a good issue to tackle. I will read the patch in greater detail later. But offhand, I have two comments.
Quoted 6 lines
1. "should" is not descriptive. I am assuming you mean do it only during actual recovery. If so, that would be incorrect. Say node 0 unlinks a file that was being used by node 1. Node 0 dies. Recovery will notice that that inode is active and not delete it. If node 1 dies, or is unable to delete
the file for any other reason, then our only hope is orphan scan.Sorry. the "should" doesn't mean a actual recovery. I meant when "
os->os_seqno== seqno", the orginal condition determining whether we do queue the scans.Quoted 1 line
Quoted 5 lines
Quoted 3 lines
2. All nodes have to scan all slots. Even live slots. I remember we did for a reason. And that reason should be in the comment in the patch written by Srini.
Oh... I will check Srini's patch.
The whole description is the following:
-------------
When a dentry is unlinked, the unlinking node takes an EX on the dentry lock
before moving the dentry to the orphan directory. Other nodes that have this dentry in cache have a PR on the same dentry lock. When the EX is requested, the other nodes flag the corresponding inode as MAYBE_ORPHANED
during downconvert. The inode is finally deleted when the last node to iput
the inode sees that i_nlink==0 and the MAYBE_ORPHANED flag is set.
A problem arises if a node is forced to free dentry locks because of memory
pressure. If this happens, the node will no longer get downconvert notifications for the dentries that have been unlinked on another node. If it also happens that node is actively using the corresponding inode and
happens to be the one performing the last iput on that inode, it will fail
to delete the inode as it will not have the MAYBE_ORPHANED flag set.
This patch fixes this shortcoming by introducing a periodic scan of the orphan directories to delete such inodes. Care has been taken to distribute
the workload across the cluster so that no one node has to perform the task
all the time.
---------------
I didn't see the reason All nodes have to scan all slots. If there is one, the "load balance"? Or the real reason is not in the description?
thanks,
wengang.
cover ocfs2: fix slow deleting
In reply to Sunil MushranQuoted 4 lines · Sunil Mushran
2. All nodes have to scan all slots. Even live slots. I remember we did for
a reason. And that reason should be in the comment in the patch written by Srini.
When a node unlinks a file it inserts an entry into it's own orphan slot. If another node is the last one to close the file and dentry got flushed then it will not do the cleanup as it doesn't know the file was orphaned. The file will remain in the orphan slot till the node umounts and the same slot is reused again. To overcome this problem a node has to rescan all slots(including live slots) and try to do the cleanup.
cover ocfs2: fix slow deleting
Quoted 11 lines · Srinivas Eeda
Quoted 4 lines · Sunil Mushran
2. All nodes have to scan all slots. Even live slots. I remember we did for
a reason. And that reason should be in the comment in the patch written by Srini.When a node unlinks a file it inserts an entry into it's own orphan slot. If another node is the last one to close the file and dentry got flushed then it will not do the cleanup as it doesn't know the file was orphaned. The file will remain in the orphan slot till the node umounts and the same slot is reused again. To overcome this problem a node has to rescan all slots(including live slots) and try to do the cleanup.
The qs is not why are we scanning all live slots on all nodes. As in, why not just recover the local slot. There was a reason for that. Yes, we have to recover unused slots for the reason listed previously.
cover ocfs2: fix slow deleting
Quoted 12 lines · Sunil Mushran
Quoted 11 lines · Srinivas Eeda
Quoted 4 lines · Sunil Mushran
2. All nodes have to scan all slots. Even live slots. I remember we did for
a reason. And that reason should be in the comment in the patch written by Srini.When a node unlinks a file it inserts an entry into it's own orphan slot. If another node is the last one to close the file and dentry got flushed then it will not do the cleanup as it doesn't know the file was orphaned. The file will remain in the orphan slot till the node umounts and the same slot is reused again. To overcome this problem a node has to rescan all slots(including live slots) and try to do the cleanup.
Quoted 3 lines
The qs is not why are we scanning all live slots on all nodes. As in, why not just recover the local slot. There was a reason for that. Yes, we have to recover unused slots for the reason listed previously.
bleh.... let me rephrase.
The qs is why are we scanning all live slots on all nodes. Wengangs patch limits the scanning to the local (live) slot only. And I remember we had a reason for it.
cover ocfs2: fix slow deleting
Below is excerpts from Joel's email for the same question :)
Quoted 4 lines
Currently, orphan scan just iterate all the slots and call
ocfs2_queue_recovery_completion, but I don't think it is proper for a node to query another mounted one since that node will query it by itself.
Node 1 has an inode it was using. The dentry went away due to
memory pressure. Node 1 closes the inode, but it's on the free list. The node has the open lock.
Node 2 unlinks the inode. It grabs the dentry lock to notify
others, but node 1 has no dentry and doesn't get the message. It trylocks the open lock, sees that another node has a PR, and does nothing.
Later node 2 runs its orphan dir. It igets the inode, trylocks
the open lock, sees the PR still, and does nothing.
Basically, we have to trigger an orphan iput on node 1. The
only way for this to happen is if node 1 runs node 2's orphan dir. This patch exists because that wasn't happening.
Quoted 24 lines · Sunil Mushran
Quoted 14 lines · Sunil Mushran
Quoted 6 lines · Srinivas Eeda
Quoted 5 lines · Sunil Mushran
2. All nodes have to scan all slots. Even live slots. I remember we did for
a reason. And that reason should be in the comment in the patch written
by Srini.Quoted 7 lines
When a node unlinks a file it inserts an entry into it's own orphan slot. If another node is the last one to close the file and dentry got flushed then it will not do the cleanup as it doesn't know the file was orphaned. The file will remain in the orphan slot till the node umounts
and the same slot is reused again. To overcome this problem a node has to rescan all slots(including live slots) and try to do the cleanup.Quoted 3 lines
The qs is not why are we scanning all live slots on all nodes. As in, why not just recover the local slot. There was a reason for that. Yes, we have to recover unused slots for the reason listed previously.
bleh.... let me rephrase.
The qs is why are we scanning all live slots on all nodes. Wengangs patch limits the scanning to the local (live) slot only. And I remember we had a reason for it.
cover ocfs2: fix slow deleting
Quoted 7 lines · Srinivas Eeda
Below is excerpts from Joel's email for the same question :)
Quoted 4 lines
Currently, orphan scan just iterate all the slots and call
ocfs2_queue_recovery_completion, but I don't think it is proper for a node to query another mounted one since that node will query it by itself.
Quoted 12 lines
Node 1 has an inode it was using. The dentry went away due to
memory pressure. Node 1 closes the inode, but it's on the free list. The node has the open lock.
Node 2 unlinks the inode. It grabs the dentry lock to notify
others, but node 1 has no dentry and doesn't get the message. It trylocks the open lock, sees that another node has a PR, and does nothing.
Later node 2 runs its orphan dir. It igets the inode, trylocks
the open lock, sees the PR still, and does nothing.
Basically, we have to trigger an orphan iput on node 1. The
only way for this to happen is if node 1 runs node 2's orphan dir. This patch exists because that wasn't happening.
Thanks for the reminder.
Joel Becker
maintainercover ocfs2: fix slow deleting
In reply to Srinivas EedaQuoted 7 lines · Srinivas Eeda
Below is excerpts from Joel's email for the same question :)
Quoted 4 lines
Currently, orphan scan just iterate all the slots and call
ocfs2_queue_recovery_completion, but I don't think it is proper for a node to query another mounted one since that node will query it by itself.
Quoted 12 lines
Node 1 has an inode it was using. The dentry went away due to
memory pressure. Node 1 closes the inode, but it's on the free list. The node has the open lock.
Node 2 unlinks the inode. It grabs the dentry lock to notify
others, but node 1 has no dentry and doesn't get the message. It trylocks the open lock, sees that another node has a PR, and does nothing.
Later node 2 runs its orphan dir. It igets the inode, trylocks
the open lock, sees the PR still, and does nothing.
Basically, we have to trigger an orphan iput on node 1. The
only way for this to happen is if node 1 runs node 2's orphan dir. This patch exists because that wasn't happening.
What he said. And yes, I know I'm late to the party.
Joel
Signature · 7 lines
"Lately I've been talking in my sleep. Can't imagine what I'd have to say. Except my world will be right When love comes back my way." http://www.jlbec.org/ jlbec at evilplan.org