configfs: Fix deadlock rmdir() vs rename()
cover configfs: Fix deadlock rmdir() vs rename()
Hi,
This patchset fixes the deadlock described below. I did not exactly implement what I described earlier, since I found this more elegant solution meanwhile. Please see the third patch header for a detailed explanation of the fix.
The following procedure can trigger a deadlock in configfs (see
http://www.ussg.iu.edu/hypermail/linux/kernel/0806.1/0380.html for a patch that makes it easier to trigger):
# mkdir /config/cluster/foo # cd /config/cluster/foo # mv heartbeat/dead_threshold node/bar
and in another shell, right after having launched test_deadlock:
# rmdir /config/cluster/foo
First, lockdep warns as usual (see below), and after two minutes (standard task deadlock parameters), we get the dead lock alerts:
<log>
=============================================
[ INFO: possible recursive locking detected ]
2.6.26-rc5 #13
---------------------------------------------
rmdir/3997 is trying to acquire lock:
(&sb->s_type->i_mutex_key#11){--..}, at: [<ffffffff802d2131>] configfs_detach_prep+0x58/0xaa
but task is already holding lock:
(&sb->s_type->i_mutex_key#11){--..}, at: [<ffffffff80296070>] vfs_rmdir+0x49/0xac
other info that might help us debug this:
2 locks held by rmdir/3997:
#0: (&sb->s_type->i_mutex_key#3/1){--..}, at: [<ffffffff80297c77>] do_rmdir+0x82/0x108 #1: (&sb->s_type->i_mutex_key#11){--..}, at: [<ffffffff80296070>] vfs_rmdir+0x49/0xac
stack backtrace:
Pid: 3997, comm: rmdir Not tainted 2.6.26-rc5 #13
Call Trace: [<ffffffff8024aa65>] __lock_acquire+0x8d2/0xc78 [<ffffffff802495ec>] find_usage_backwards+0x9d/0xbe [<ffffffff802d2131>] configfs_detach_prep+0x58/0xaa [<ffffffff8024b1de>] lock_acquire+0x51/0x6c [<ffffffff802d2131>] configfs_detach_prep+0x58/0xaa [<ffffffff80247dad>] debug_mutex_lock_common+0x16/0x23 [<ffffffff805d63a4>] mutex_lock_nested+0xcd/0x23b [<ffffffff802d2131>] configfs_detach_prep+0x58/0xaa [<ffffffff802d327b>] configfs_rmdir+0xb8/0x1c3 [<ffffffff80296092>] vfs_rmdir+0x6b/0xac [<ffffffff80297cac>] do_rmdir+0xb7/0x108 [<ffffffff80249d1e>] trace_hardirqs_on+0xef/0x113 [<ffffffff805d74c4>] trace_hardirqs_on_thunk+0x35/0x3a [<ffffffff8020b0cb>] system_call_after_swapgs+0x7b/0x80 INFO: task test_deadlock:3996 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.test_deadlock D 0000000000000001 0 3996 3980
ffff81007cc93d78 0000000000000046 ffff81007cc93d40 ffffffff808ed280 ffffffff808ed280 ffff81007cc93d28 ffffffff808ed280 ffffffff808ed280 ffffffff808ed280 ffffffff808ea120 ffffffff808ed280 ffff81007cdcaa10
Call Trace: [<ffffffff802955e3>] lock_rename+0x11e/0x126 [<ffffffff805d641e>] mutex_lock_nested+0x147/0x23b [<ffffffff802955e3>] lock_rename+0x11e/0x126 [<ffffffff80297838>] sys_renameat+0xd7/0x21c [<ffffffff805d74c4>] trace_hardirqs_on_thunk+0x35/0x3a [<ffffffff80249d1e>] trace_hardirqs_on+0xef/0x113 [<ffffffff805d74c4>] trace_hardirqs_on_thunk+0x35/0x3a [<ffffffff8020b0cb>] system_call_after_swapgs+0x7b/0x80
INFO: lockdep is turned off.
INFO: task rmdir:3997 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rmdir D 0000000000000000 0 3997 3986
ffff81007cdb9dd8 0000000000000046 0000000000000000 ffffffff808ed280 ffffffff808ed280 ffff81007cdb9d88 ffffffff808ed280 ffffffff808ed280 ffffffff808ed280 ffffffff808ea120 ffffffff808ed280 ffff81007cde0a50
Call Trace: [<ffffffff802d2131>] configfs_detach_prep+0x58/0xaa [<ffffffff805d641e>] mutex_lock_nested+0x147/0x23b [<ffffffff802d2131>] configfs_detach_prep+0x58/0xaa [<ffffffff802d327b>] configfs_rmdir+0xb8/0x1c3 [<ffffffff80296092>] vfs_rmdir+0x6b/0xac [<ffffffff80297cac>] do_rmdir+0xb7/0x108 [<ffffffff80249d1e>] trace_hardirqs_on+0xef/0x113 [<ffffffff805d74c4>] trace_hardirqs_on_thunk+0x35/0x3a [<ffffffff8020b0cb>] system_call_after_swapgs+0x7b/0x80
INFO: lockdep is turned off.
</log>
The issue here is that the VFS locks the i_mutex of the source and target directories of the rename in source -> target order (because none is ascendent of the other one), while configfs_detach_prep() takes them in default group order (or reverse order, I'm not sure), following the order specified by the groups' creator.
Louis
Signature · 4 lines
Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes
PATCH 1/3 configfs: Introduce configfs_dirent_lock
An embedded and charset-unspecified text was scrubbed...
Name: configfs-introduce-configfs_dirent_lock.patch
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20080612/dd1c0153/attachment.pl
Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 3 lines · Louis Rilling
This patch introduces
configfs_dirent_lockspinlock to protectconfigfs_direnttraversals against linkage mutations (add/del/move). This will allowconfigfs_detach_prep()to avoid lockingi_mutexes.
I like that you expanded the scope to cover all configfs_dirent
linkages. These are all protected by i_mutex in the current code, but your rename fix removes that protection.
Quoted 3 lines
Locking rules for
configfs_direntlinkage mutations are the same plus the requirement of takingconfigfs_dirent_lock. Forconfigfs_direntwalking, one can either take appropriatei_mutexas before, or takeconfigfs_dirent_lock.
Nope, you *must* take configfs_dirent_lock now. You've removedi_mutex protection in the last patch.
Quoted 2 lines
The spinlock could actually be a mutex, but the critical sections are either O(1) or should not be too long (default groups walking in last patch).
I'm wary of someone's reasonably deep groups. Discussing it
yesterday I'd been convinced that a mutex was good to start with. But given your increased scope of the lock, let's try the spinlock and see what happens.
Quoted 1 line
+extern
spinlock_tconfigfs_dirent_lock;
Boy I wish this could be static to one file :-(
Review static struct configfs_dirent *configfs_ · line 90
Quoted 8 lines
(quoted diff) +2
-0@@ -79,7 +84,9 @@ static struct configfs_dirent *configfs_ atomic_set(&sd->s_count, 1); INIT_LIST_HEAD(&sd->s_links); INIT_LIST_HEAD(&sd->s_children); + spin_lock(&configfs_dirent_lock); list_add(&sd->s_sibling, &parent_sd->s_children); + spin_unlock(&configfs_dirent_lock); sd->s_element = element;
You need to set s_element either under the lock or before taking
the lock. Once you've dropped the lock, someone can reach this dirent from the parent, and should see s_element.
Quoted 11 lines
(quoted diff) +2
-0@@ -173,7 +180,9 @@ static int create_dir(struct config_item } else { struct configfs_dirent *sd = d->d_fsdata; if (sd) { + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_put(sd); } }
Review int configfs_create_link(struct configfs · line 241
Quoted 9 lines
(quoted diff) +2
-0@@ -224,7 +233,9 @@ int configfs_create_link(struct configfs else { struct configfs_dirent *sd = dentry->d_fsdata; if (sd) { + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_put(sd); } }
These strike me as wrong - I would think that one should either
see the old configfs_dirent tree or the new one. That is, one would take the dirent lock at the beginning of configfs_mkdir() and release it at the end - so any other code that looks at the dirent tree will see an atomic change. Here, some other path could see the new dirent after configfs_make_dirent() but then it disappears when configfs_create() fails.
If you did that, though, it'd have to be a mutex. Now, the only thing that sees this intermediate condition is
configfs itself. Everyone else is protected by i_mutex. I guess it's OK - but can you comment that fact? i_mutex does *not* protect traversal of the configfs_dirent tree, but it does prevent the outside world from seeing the intermediate states.
Quoted 69 lines
(quoted diff) +10
-0@@ -238,7 +249,9 @@ static void remove_dir(struct dentry * d struct configfs_dirent * sd; sd = d->d_fsdata; + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_put(sd); if (d->d_inode) simple_rmdir(parent->d_inode,d);@@ -410,7 +423,9 @@ static void detach_attrs(struct config_i list_for_each_entry_safe(sd, tmp, &parent_sd->s_children, s_sibling) { if (!sd->s_element || !(sd->s_type & CONFIGFS_NOT_PINNED)) continue; + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_drop_dentry(sd, dentry); configfs_put(sd); }@@ -1268,7 +1283,9 @@ static int configfs_dir_close(struct ino struct configfs_dirent * cursor = file->private_data; mutex_lock(&dentry->d_inode->i_mutex); + spin_lock(&configfs_dirent_lock); list_del_init(&cursor->s_sibling); + spin_unlock(&configfs_dirent_lock); mutex_unlock(&dentry->d_inode->i_mutex); release_configfs_dirent(cursor);@@ -1362,7 +1383,9 @@ static loff_t configfs_dir_lseek(struct struct list_head *p; loff_t n = file->f_pos - 2; + spin_lock(&configfs_dirent_lock); list_del(&cursor->s_sibling); + spin_unlock(&configfs_dirent_lock); p = sd->s_children.next; while (n && p != &sd->s_children) { struct configfs_dirent *next;@@ -1372,7 +1395,9 @@ static loff_t configfs_dir_lseek(struct n--; p = p->next; } + spin_lock(&configfs_dirent_lock); list_add_tail(&cursor->s_sibling, p); + spin_unlock(&configfs_dirent_lock); } } mutex_unlock(&dentry->d_inode->i_mutex);Index:
b/fs/configfs/inode.c
===================================================================fs/configfs/inode.c +2
-0@@ -247,7 +247,9 @@ void configfs_hash_and_remove(struct denif (!sd->s_element)continue ;if (!strcmp (configfs_get_name (sd), name)) { +spin_lock (&configfs_dirent_lock);list_del_init (&sd->s_sibling); +spin_unlock (&configfs_dirent_lock);configfs_drop_dentry (sd, dir);configfs_put (sd);break ;Index:
b/fs/configfs/symlink.c
===================================================================
fs/configfs/symlink.c int configfs_unlink(struct inode *dir, s · line 177
Quoted 9 lines
fs/configfs/symlink.c +2
-0@@ -169,7 +169,9 @@ int configfs_unlink(struct inode *dir, s parent_item =configfs_get_config_item (dentry->d_parent); type = parent_item->ci_type; +spin_lock (&configfs_dirent_lock);list_del_init (&sd->s_sibling); +spin_unlock (&configfs_dirent_lock);configfs_drop_dentry (sd, dentry->d_parent);dput (dentry);configfs_put (sd);
You do the lock,del(sibling),unlock so often, maybe it should be
a helper. Then you can make configfs_dirent_lock static to dir.c. Well, you use dirent_lock in your s_links patch, so maybe not static.
Joel
Signature · 8 lines
"Copy from one, it's plagiarism; copy from two, it's research."
- Wilson Mizner
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 9 lines · Joel Becker
Quoted 3 lines · Louis Rilling
This patch introduces
configfs_dirent_lockspinlock to protectconfigfs_direnttraversals against linkage mutations (add/del/move). This will allowconfigfs_detach_prep()to avoid lockingi_mutexes.I like that you expanded the scope to cover all
configfs_dirent
linkages. These are all protected byi_mutexin the current code, but your rename fix removes that protection.
Quoted 6 lines
Quoted 3 lines
Locking rules for
configfs_direntlinkage mutations are the same plus the requirement of takingconfigfs_dirent_lock. Forconfigfs_direntwalking, one can either take appropriatei_mutexas before, or takeconfigfs_dirent_lock.Nope, you *must* take
configfs_dirent_locknow. You've removedi_mutexprotection in the last patch.
Oh well. Do you mean because of CONFIGFS_USET_DROPPING being set without i_mutex locked? This is the only mutation (except in the s_links patch) done without i_mutex locked. I thought that actually either other configfs_dirent traversals like readdir() and dir_lseek() would prevent detach_prep() from succeeding because they add dirents before, or are done in places where detach_prep() cannot do harm because new_dirent() fails whenever it sees CONFIGFS_USET_DROPPING: detach_attrs() and detach_groups() must ignore CONFIGFS_USET_DROPPING, depend_prep() is protected since the whole path is locked from configfs root, lookup() can succeed since at worst its result will be invalidated when actually detaching the default groups. The only function for which I can not figure out is configfs_hash_and_remove(), but it is not used.
I admit that the case of symlink() needs an extra check to ensure
that the target is not about to be removed. The bug was already there though, right?
Anyway, if it looks conceptually simpler to useconfigfs_dirent_lock (probably better a mutex in that case) wherever i_mutex are supposed to protect configfs_dirent traversals, I'm ok with it.
Quoted 14 lines
Quoted 4 lines
The spinlock could actually be a mutex, but the critical sections are either O(1) or should not be too long (default groups walking in last patch).
I'm wary of someone's reasonably deep groups. Discussing it
yesterday I'd been convinced that a mutex was good to start with. But given your increased scope of the lock, let's try the spinlock and see what happens.Quoted 1 line
+extern
spinlock_tconfigfs_dirent_lock;Boy I wish this could be static to one file :-(
Quoted 12 lines
Review static struct configfs_dirent *configfs_ · line 90
Quoted 8 lines
(quoted diff) +2
-0@@ -79,7 +84,9 @@ static struct configfs_dirent *configfs_ atomic_set(&sd->s_count, 1); INIT_LIST_HEAD(&sd->s_links); INIT_LIST_HEAD(&sd->s_children); + spin_lock(&configfs_dirent_lock); list_add(&sd->s_sibling, &parent_sd->s_children); + spin_unlock(&configfs_dirent_lock); sd->s_element = element;You need to set
s_elementeither under the lock or before taking
the lock. Once you've dropped the lock, someone can reach this dirent from the parent, and should sees_element.
Will do.
Quoted 22 lines
Quoted 12 lines
(quoted diff) +2
-0@@ -173,7 +180,9 @@ static int create_dir(struct config_item } else { struct configfs_dirent *sd = d->d_fsdata; if (sd) { + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_put(sd); } }Review int configfs_create_link(struct configfs · line 241
Quoted 9 lines
(quoted diff) +2
-0@@ -224,7 +233,9 @@ int configfs_create_link(struct configfs else { struct configfs_dirent *sd = dentry->d_fsdata; if (sd) { + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_put(sd); } }
Quoted 8 lines
These strike me as wrong - I would think that one should either
see the oldconfigfs_direnttree or the new one. That is, one would take the dirent lock at the beginning ofconfigfs_mkdir()and release it at the end - so any other code that looks at the dirent tree will see an atomic change. Here, some other path could see the new dirent afterconfigfs_make_dirent()but then it disappears whenconfigfs_create()fails.
If you did that, though, it'd have to be a mutex.
And we should not take other i_mutex in populate_groups() and populate_attrs(), otherwise deadlocks could happen.
Quoted 5 lines
Now, the only thing that sees this intermediate condition is
configfs itself. Everyone else is protected byi_mutex. I guess it's OK - but can you comment that fact?i_mutexdoes *not* protect traversal of theconfigfs_direnttree, but it does prevent the outside world from seeing the intermediate states.
The only intermediate conditions that may hurt one's mind is that an mkdir() (resp. symlink()) racing with an rmdir() can successfuly call make_item()/make_group() (resp. allow_link()) and immediately fail when finalizing with attach_item()/attach_group() (resp. create_link()). So, from userspace and the VFS this seems like "mkdir foo/bar/baz" simply failed because of "rmdir foo", while at the same time from the subsystem point of view this seems like userspace did "mkdir foo/bar/baz; rmdir foo/bar/baz; rmdir foo". As I pointed out in the rename fix, this however can already happen when attach_item()/attach_group() (resp. create_link()) fails because of memory pressure for instance.
For the other intermediate conditions, see above about locking rules.
Quoted 73 lines
Quoted 73 lines
(quoted diff) +10
-0@@ -238,7 +249,9 @@ static void remove_dir(struct dentry * d struct configfs_dirent * sd; sd = d->d_fsdata; + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_put(sd); if (d->d_inode) simple_rmdir(parent->d_inode,d);@@ -410,7 +423,9 @@ static void detach_attrs(struct config_i list_for_each_entry_safe(sd, tmp, &parent_sd->s_children, s_sibling) { if (!sd->s_element || !(sd->s_type & CONFIGFS_NOT_PINNED)) continue; + spin_lock(&configfs_dirent_lock); list_del_init(&sd->s_sibling); + spin_unlock(&configfs_dirent_lock); configfs_drop_dentry(sd, dentry); configfs_put(sd); }@@ -1268,7 +1283,9 @@ static int configfs_dir_close(struct ino struct configfs_dirent * cursor = file->private_data; mutex_lock(&dentry->d_inode->i_mutex); + spin_lock(&configfs_dirent_lock); list_del_init(&cursor->s_sibling); + spin_unlock(&configfs_dirent_lock); mutex_unlock(&dentry->d_inode->i_mutex); release_configfs_dirent(cursor);@@ -1362,7 +1383,9 @@ static loff_t configfs_dir_lseek(struct struct list_head *p; loff_t n = file->f_pos - 2; + spin_lock(&configfs_dirent_lock); list_del(&cursor->s_sibling); + spin_unlock(&configfs_dirent_lock); p = sd->s_children.next; while (n && p != &sd->s_children) { struct configfs_dirent *next;@@ -1372,7 +1395,9 @@ static loff_t configfs_dir_lseek(struct n--; p = p->next; } + spin_lock(&configfs_dirent_lock); list_add_tail(&cursor->s_sibling, p); + spin_unlock(&configfs_dirent_lock); } } mutex_unlock(&dentry->d_inode->i_mutex);Index:
b/fs/configfs/inode.c
===================================================================fs/configfs/inode.c +2
-0@@ -247,7 +247,9 @@ void configfs_hash_and_remove(struct denif (!sd->s_element)continue ;if (!strcmp (configfs_get_name (sd), name)) { +spin_lock (&configfs_dirent_lock);list_del_init (&sd->s_sibling); +spin_unlock (&configfs_dirent_lock);configfs_drop_dentry (sd, dir);configfs_put (sd);break ;Index:
b/fs/configfs/symlink.c
===================================================================fs/configfs/symlink.c +0
-0@@ -169,7 +169,9 @@ int configfs_unlink(struct inode *dir, s parent_item =configfs_get_config_item (dentry->d_parent); type = parent_item->ci_type;
Quoted 10 lines
fs/configfs/symlink.c int configfs_unlink(struct inode *dir, s · line 177
Quoted 6 lines
fs/configfs/symlink.c +2
-0+spin_lock (&configfs_dirent_lock);list_del_init (&sd->s_sibling); +spin_unlock (&configfs_dirent_lock);configfs_drop_dentry (sd, dentry->d_parent);dput (dentry);configfs_put (sd);You do the lock,del(sibling),unlock so often, maybe it should be
a helper. Then you can makeconfigfs_dirent_lockstatic todir.c. Well, you usedirent_lockin yours_linkspatch, so maybe not static.
Yes I need it not static, or implement helpers for s_children/s_sibling and for s_links/sl_list. And if the scope should be extended to all cases of configfs_dirent traversals plus whatever would fix symlink()'s target not disappearing, it should definitely not be static.
Louis
Signature · 5 lines
Dr Louis Rilling Kerlabs - IRISA Skype: louis.rilling Campus Universitaire de Beaulieu Phone: (+33|0) 2 99 84 71 52 Avenue du General Leclerc Fax: (+33|0) 2 99 84 71 71 35042 Rennes CEDEX - France http://www.kerlabs.com/
Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 9 lines · Louis Rilling
Quoted 7 lines · Joel Becker
Quoted 3 lines · Louis Rilling
Locking rules for
configfs_direntlinkage mutations are the same plus the requirement of takingconfigfs_dirent_lock. Forconfigfs_direntwalking, one can either take appropriatei_mutexas before, or takeconfigfs_dirent_lock.Nope, you *must* take
configfs_dirent_locknow. You've removedi_mutexprotection in the last patch.
Quoted 12 lines
Oh well. Do you mean because of
CONFIGFS_USET_DROPPINGbeing set withouti_mutexlocked? This is the only mutation (except in thes_linkspatch) done withouti_mutexlocked. I thought that actually either otherconfigfs_direnttraversals likereaddir()anddir_lseek()would preventdetach_prep()from succeeding because they add dirents before, or are done in places wheredetach_prep()cannot do harm becausenew_dirent()fails whenever it seesCONFIGFS_USET_DROPPING:detach_attrs()anddetach_groups()must ignoreCONFIGFS_USET_DROPPING,depend_prep()is protected since the whole path is locked from configfs root,lookup()can succeed since at worst its result will be invalidated when actually detaching the default groups. The only function for which I can not figure out isconfigfs_hash_and_remove(), but it is not used.
I don't mean that your code is wrong, I mean that the comment is
unclear. The locking rules aren't "you can use i_mutex or dirent_lock, take your pick". I think you are right that configfs_detach_prep() is safe to set dropping as it does without i_mutex.
This is related to the discussion below about VFS visible
changes (i_mutex protection) vs subsystem internal changes (dirent_lock protection). The protections have different scope, but your comment made them sound interchangable.
Quoted 6 lines
I admit that the case of
symlink()needs an extra check to ensure
that the target is not about to be removed. The bug was already there though, right?
Anyway, if it looks conceptually simpler to useconfigfs_dirent_lock(probably better a mutex in that case) whereveri_mutexare supposed to protectconfigfs_direnttraversals, I'm ok with it.
Leave it as a spinlock.
Going over the changes, I was pretty convinced your detach_prep
was safe vis-a-vis mkdir. You're under i_mutex for the immediate directory, and both attach_* and detach_* are under the immediate i_mutex when they make the change. Also, you have your readdir and lookup walking s_children without a lock. I *think* that's safe, because it's also against the immediate directory, and thus the vfs is holding i_mutex for you.
Quoted 2 lines
And we should not take other
i_mutexinpopulate_groups()andpopulate_attrs(), otherwise deadlocks could happen.
Huh, we certainly should. perhaps you are speaking as if we
were turning dirent_lock into a mutex. We're not turning dirent_lock into a mutex yet.
Quoted 6 lines
Quoted 5 lines
Now, the only thing that sees this intermediate condition is
configfs itself. Everyone else is protected byi_mutex. I guess it's OK - but can you comment that fact?i_mutexdoes *not* protect traversal of theconfigfs_direnttree, but it does prevent the outside world from seeing the intermediate states.
Quoted 10 lines
The only intermediate conditions that may hurt one's mind is that an
mkdir()(resp.symlink()) racing with anrmdir()can successfuly call make_item()/make_group() (resp.allow_link()) and immediately fail when finalizing with attach_item()/attach_group() (resp.create_link()). So, from userspace and the VFS this seems like "mkdir foo/bar/baz" simply failed because of "rmdir foo", while at the same time from the subsystem point of view this seems like userspace did "mkdir foo/bar/baz; rmdir foo/bar/baz; rmdir foo". As I pointed out in the rename fix, this however can already happen when attach_item()/attach_group() (resp.create_link()) fails because of memory pressure for instance.
I'm not even sure what you said here :-)
Joel
Signature · 8 lines
"Egotist: a person more interested in himself than in me."
- Ambrose Bierce
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 23 lines · Joel Becker
Quoted 9 lines · Louis Rilling
Quoted 7 lines · Joel Becker
Quoted 3 lines · Louis Rilling
Locking rules for
configfs_direntlinkage mutations are the same plus the requirement of takingconfigfs_dirent_lock. Forconfigfs_direntwalking, one can either take appropriatei_mutexas before, or takeconfigfs_dirent_lock.Nope, you *must* take
configfs_dirent_locknow. You've removedi_mutexprotection in the last patch.Quoted 12 lines
Oh well. Do you mean because of
CONFIGFS_USET_DROPPINGbeing set withouti_mutexlocked? This is the only mutation (except in thes_linkspatch) done withouti_mutexlocked. I thought that actually either otherconfigfs_direnttraversals likereaddir()anddir_lseek()would preventdetach_prep()from succeeding because they add dirents before, or are done in places wheredetach_prep()cannot do harm becausenew_dirent()fails whenever it seesCONFIGFS_USET_DROPPING:detach_attrs()anddetach_groups()must ignoreCONFIGFS_USET_DROPPING,depend_prep()is protected since the whole path is locked from configfs root,lookup()can succeed since at worst its result will be invalidated when actually detaching the default groups. The only function for which I can not figure out isconfigfs_hash_and_remove(), but it is not used.
Quoted 8 lines
I don't mean that your code is wrong, I mean that the comment is
unclear. The locking rules aren't "you can usei_mutexordirent_lock, take your pick". I think you are right thatconfigfs_detach_prep()is safe to set dropping as it does withouti_mutex.
This is related to the discussion below about VFS visible
changes (i_mutexprotection) vs subsystem internal changes (dirent_lockprotection). The protections have different scope, but your comment made them sound interchangable.
Agreed.
Quoted 8 lines
Quoted 7 lines
I admit that the case of
symlink()needs an extra check to ensure
that the target is not about to be removed. The bug was already there though, right?
Anyway, if it looks conceptually simpler to useconfigfs_dirent_lock(probably better a mutex in that case) whereveri_mutexare supposed to protectconfigfs_direnttraversals, I'm ok with it.
Quoted 8 lines
Leave it as a spinlock.
Going over the changes, I was pretty convinced yourdetach_prep
was safe vis-a-vis mkdir. You're underi_mutexfor the immediate directory, and both attach_* and detach_* are under the immediatei_mutexwhen they make the change. Also, you have your readdir and lookup walkings_childrenwithout a lock. I *think* that's safe, because it's also against the immediate directory, and thus the vfs is holdingi_mutexfor you.
Unfortunately, thinking a bit more about it I found some issues with i_mutex lock free detach_prep(), but nothing that can't be fixed ;)
Between detach_prep() in A and mkdir() in a default group A/B:detach_prep() can be called in the middle of attach_group(), for instance after having attached A/B/C, but attach_group() may then fail (because of memory pressure for instance) while attaching C's default group A/B/C/D. This would lead to both mkdir(A/B/C) and rmdir(A) failing, the reason for rmdir failure being at best obscure: the user would have expected to either see mkdir succeed and rmdir fail because of the new A/B/C group, or see mkdir fail and rmdir succeed because no user-created group lived under A. Solution: tag A/B with USET_IN_MKDIR on mkdir entrance, remove that tag on mkdir exit, and retry detach_prep() as long as USET_IN_MKDIR is found under A/*.
Between rmdir() and readdir(): dir_open() might add a configfs_dirent
to a default group A/B that detach_prep() already marked with USET_DROPPING. This could result in detach_groups() dropping the dirent and make readdir() in A/B crash afterwards. Solution: check USET_DROPPING in dir_open() and fail if it is set.
Between rmdir() and lookup(): several lookup() called under A/* while
rmdir(A) in the middle of detach_groups() could return inconsistent results (for instance some default groups being there and some other ones not). Solution:
lock dirent_lock for the whole lookup() duration, check USET_DROPPING of current dir, and fail with ENOENT if it is set.
I'll send a corrected patch later.
Quoted 7 lines
Quoted 3 lines
And we should not take other
i_mutexinpopulate_groups()andpopulate_attrs(), otherwise deadlocks could happen.Huh, we certainly should. perhaps you are speaking as if we
were turningdirent_lockinto a mutex. We're not turningdirent_lockinto a mutex yet.
I was speaking as if we replaced i_mutex protection with dirent_lock protection for a whole mkdir(), that is taking the lock before attach_* and releasing it after.
Quoted 18 lines
Quoted 7 lines
Quoted 6 lines
Now, the only thing that sees this intermediate condition is
configfs itself. Everyone else is protected byi_mutex. I guess it's OK - but can you comment that fact?i_mutexdoes *not* protect traversal of theconfigfs_direnttree, but it does prevent the outside world from seeing the intermediate states.Quoted 10 lines
The only intermediate conditions that may hurt one's mind is that an
mkdir()(resp.symlink()) racing with anrmdir()can successfuly call make_item()/make_group() (resp.allow_link()) and immediately fail when finalizing with attach_item()/attach_group() (resp.create_link()). So, from userspace and the VFS this seems like "mkdir foo/bar/baz" simply failed because of "rmdir foo", while at the same time from the subsystem point of view this seems like userspace did "mkdir foo/bar/baz; rmdir foo/bar/baz; rmdir foo". As I pointed out in the rename fix, this however can already happen when attach_item()/attach_group() (resp.create_link()) fails because of memory pressure for instance.
Quoted 1 line
I'm not even sure what you said here :-)
I was just saying that with i_mutex lock free detach_prep(), we have kind of optimistic mkdir(), with conflicts resolved as error cases of attach_*.
The intermediate conditions that really matter are:
1/ the existence of partial default groups trees (I mean configfs_dirent trees)
in the middle of attach_group() and detach_group(),
2/ the existence of default group trees that are tagged as USET_DROPPING and
should be treated as not existing anymore.
These intermediate conditions lead to the issues pointed out above, and
will be correctly handled (I hope) in the coming corrected patch.
Louis
Signature · 11 lines
Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20080613/7fb62cf0/attachment.bin
Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 15 lines · Louis Rilling
Unfortunately, thinking a bit more about it I found some issues with
i_mutexlock freedetach_prep(), but nothing that can't be fixed ;)
Betweendetach_prep()in A andmkdir()in a default group A/B:detach_prep()can be called in the middle ofattach_group(), for instance after having attached A/B/C, butattach_group()may then fail (because of memory pressure for instance) while attaching C's default group A/B/C/D. This would lead to both mkdir(A/B/C) and rmdir(A) failing, the reason for rmdir failure being at best obscure: the user would have expected to either see mkdir succeed and rmdir fail because of the new A/B/C group, or see mkdir fail and rmdir succeed because no user-created group lived under A. Solution: tag A/B withUSET_IN_MKDIRon mkdir entrance, remove that tag on mkdir exit, and retrydetach_prep()as long asUSET_IN_MKDIRis found under A/*.
Betweenrmdir()andreaddir():dir_open()might add aconfigfs_dirent
to a default group A/B thatdetach_prep()already marked withUSET_DROPPING.
Quoted 8 lines
This could result in
detach_groups()dropping the dirent and makereaddir()in A/B crash afterwards. Solution: checkUSET_DROPPINGindir_open()and fail if it is set.
Betweenrmdir()andlookup(): severallookup()called under A/* while
rmdir(A) in the middle ofdetach_groups()could return inconsistent results (for instance some default groups being there and some other ones not). Solution:
lockdirent_lockfor the wholelookup()duration, checkUSET_DROPPINGof current dir, and fail with ENOENT if it is set.
Oh, should probably provide some d_revalidate() also, which would return -ENOENT for a dentry under a directory flagged with USET_DROPPING. But I'm realizing that such "inconsistencies" (some default groups being valid in the d_cache and some other ones not) already happen between the time detach_prep() has flagged a default group with USET_DROPPING and the default group is actually detached. Am I wrong?
Louis
Signature · 11 lines
Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20080613/4f400cd4/attachment-0001.bin
Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 6 lines · Louis Rilling
Oh, should probably provide some
d_revalidate()also, which would return-ENOENTfor a dentry under a directory flagged withUSET_DROPPING. But I'm realizing that such "inconsistencies" (some default groups being valid in thed_cacheand some other ones not) already happen between the timedetach_prep()has flagged a default group withUSET_DROPPINGand the default group is actually detached. Am I wrong?
We don't need d_revalidate(). As I stated at the end of my last
email, USET_DROPPING does not mean 'It already went away'. It just means we're safe to do so, because we prevent new children. We actually make it go away underneath i_mutex.
The VFS handles inconsistencies between lookup and action. It's
part of normal operation. Otherwise, they'd have to hold all the i_mutexes around lookup and action.
Joel
Signature · 9 lines
"When choosing between two evils, I always like to try the one
I've never tried before."
- Mae West
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
In reply to Louis RillingLouis,
Can I just say, you're the first person to do serious review
other than myself, and I really appreciate it :-)
Quoted 5 lines · Louis Rilling
On Thu, Jun 12, 2008 at 07:41:31PM -0700, Joel Becker wrote:
Unfortunately, thinking a bit more about it I found some issues withi_mutexlock freedetach_prep(), but nothing that can't be fixed ;)
Betweendetach_prep()in A andmkdir()in a default group A/B:detach_prep()can be called in the middle ofattach_group(), for instance after
Quoted 8 lines
having attached A/B/C, but
attach_group()may then fail (because of memory pressure for instance) while attaching C's default group A/B/C/D. This would lead to both mkdir(A/B/C) and rmdir(A) failing, the reason for rmdir failure being at best obscure: the user would have expected to either see mkdir succeed and rmdir fail because of the new A/B/C group, or see mkdir fail and rmdir succeed because no user-created group lived under A. Solution: tag A/B withUSET_IN_MKDIRon mkdir entrance, remove that tag on mkdir exit, and retrydetach_prep()as long asUSET_IN_MKDIRis found under A/*.
I see what you are saying here. I'm not sure if that is worth
the complexity - we can say "it was kind of there". No one will ever hit it :-) But let me think about it more.
Quoted 5 lines
Between
rmdir()andreaddir():dir_open()might add aconfigfs_dirent
to a default group A/B thatdetach_prep()already marked withUSET_DROPPING. This could result indetach_groups()dropping the dirent and makereaddir()in A/B crash afterwards. Solution: checkUSET_DROPPINGindir_open()and fail if it is set.
I was trying to see why this could happen, given that we can
come to this from other places - the dir could have been open before we set USET_DROPPING. Oh! We actually fail rmdir with ENOTEMPTY when the dir is open? That's wrong. Ignore it though - we'll fix it later.
But back to your concern. configfs_readdir() can't crash for
two reasons. First, detach_groups() won't remove this dirent. A readdir placeholder has s_element==NULL. Note the check in detach_groups():
if (!sd->s_element || !(sd->s_type & CONFIGFS_USET_DEFAULT)) continue;
It skips our readdir placeholder, allowing us to free it in dir_close().
There's another reason this can't be a problem. If we get intodetach_groups(), we take i_mutex, locking out readdir(). Then we delete the directory, setting S_DEAD. In vfs_readdir(), they check IS_DEADDIR() after getting i_mutex. So they will see S_DEAD and not call our ->readdir(). S_DEAD is important. Someone could actually have our default_group as their cwd. S_DEAD prevents them from doing anything :-)
Quoted 5 lines
Between
rmdir()andlookup(): severallookup()called under A/* while
rmdir(A) in the middle ofdetach_groups()could return inconsistent results (for instance some default groups being there and some other ones not). Solution:
lockdirent_lockfor the wholelookup()duration, checkUSET_DROPPINGof current dir, and fail with ENOENT if it is set.
Nah, we don't care about the spurious lookups. This is a normal
race of i_mutex. USET_DROPPING is not a way to prevent VFS views from changing - it's only a way to prevent new children.
Remember, ->lookup() comes with i_mutex locking. We hold
i_mutex during the entire delete, so they can't call ->lookup() until we're done with a directory. Conversely, if they win i_mutex and ->lookup() a default group, then try to use it after we've removed it, they'll just ENOENT. This is evident back in do_rename(). They call lookup, which takes and drops locks, then call lock_rename() to get the locks back. And they can handle ENOENT at that point.
Quoted 3 lines
I was speaking as if we replaced
i_mutexprotection withdirent_lockprotection for a wholemkdir(), that is taking the lock before attach_* and releasing it after.
Ok. I think that's not the way to go, what you currently have
is better.
Quoted 4 lines
Quoted 1 line
I'm not even sure what you said here :-)
I was just saying that with
i_mutexlock freedetach_prep(), we have kind of optimisticmkdir(), with conflicts resolved as error cases of attach_*.
Basically, the concerns you had above.
Quoted 3 lines
The intermediate conditions that really matter are:
1/ the existence of partial default groups trees (I meanconfigfs_direnttrees)
in the middle ofattach_group()anddetach_group(),
This is your first case, the mkdir ENOMEM vs rmdir ENOTEMPTY.
Quoted 2 lines
2/ the existence of default group trees that are tagged as
USET_DROPPINGand
should be treated as not existing anymore.
This is not an issue. USET_DROPPING does *not* mean it went
away. It means we're safe to make it go away. We protect the actual going-away with i_mutex. And that's normal VFS behavior.
Joel
Signature · 9 lines
"I don't know anything about music. In my line you don't have
to."
- Elvis Presley
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 3 lines · Joel Becker
Louis,
Can I just say, you're the first person to do serious review
other than myself, and I really appreciate it :-)
It's just that I use configfs in my own work, and I'm playing hard with it, especially with modules. So I need to understand exactly what it does, and what is possible with it.
Quoted 16 lines · Louis Rilling
Quoted 5 lines · Joel Becker
Unfortunately, thinking a bit more about it I found some issues with
i_mutexlock freedetach_prep(), but nothing that can't be fixed ;)
Betweendetach_prep()in A andmkdir()in a default group A/B:detach_prep()can be called in the middle ofattach_group(), for instance afterQuoted 8 lines
having attached A/B/C, but
attach_group()may then fail (because of memory pressure for instance) while attaching C's default group A/B/C/D. This would lead to both mkdir(A/B/C) and rmdir(A) failing, the reason for rmdir failure being at best obscure: the user would have expected to either see mkdir succeed and rmdir fail because of the new A/B/C group, or see mkdir fail and rmdir succeed because no user-created group lived under A. Solution: tag A/B withUSET_IN_MKDIRon mkdir entrance, remove that tag on mkdir exit, and retrydetach_prep()as long asUSET_IN_MKDIRis found under A/*.
Quoted 3 lines
I see what you are saying here. I'm not sure if that is worth
the complexity - we can say "it was kind of there". No one will ever hit it :-) But let me think about it more.
To me it's an issue only if we want to provide some atomic view to
userspace: either userspace sees a group with all of its default groups, or it sees none. So the question is: does userspace need such atomicity? Currently configfs provides it, so this would be a userspace visible change if we break it.
Quoted 16 lines
Quoted 6 lines
Between
rmdir()andreaddir():dir_open()might add aconfigfs_dirent
to a default group A/B thatdetach_prep()already marked withUSET_DROPPING. This could result indetach_groups()dropping the dirent and makereaddir()in A/B crash afterwards. Solution: checkUSET_DROPPINGindir_open()and fail if it is set.I was trying to see why this could happen, given that we can
come to this from other places - the dir could have been open before we setUSET_DROPPING. Oh! We actually fail rmdir with ENOTEMPTY when the dir is open? That's wrong. Ignore it though - we'll fix it later.
But back to your concern.configfs_readdir()can't crash for
two reasons. First,detach_groups()won't remove this dirent. A readdir placeholder has s_element==NULL. Note the check indetach_groups():
Quoted 5 lines
if (!sd->s_element || !(sd->s_type & CONFIGFS_USET_DEFAULT)) continue;It skips our readdir placeholder, allowing us to free it in
dir_close().
I had not noticed this. Thanks for pointed it out.
Quoted 7 lines
There's another reason this can't be a problem. If we get into
detach_groups(), we takei_mutex, locking outreaddir(). Then we delete the directory, settingS_DEAD. Invfs_readdir(), they checkIS_DEADDIR()after gettingi_mutex. So they will seeS_DEADand not call our ->readdir().S_DEADis important. Someone could actually have ourdefault_groupas their cwd.S_DEADprevents them from doing anything :-)
As I told you in a previous email, I'm missing some VFS skills, so thanks again for the explanation.
Quoted 10 lines
Quoted 6 lines
Between
rmdir()andlookup(): severallookup()called under A/* while
rmdir(A) in the middle ofdetach_groups()could return inconsistent results (for instance some default groups being there and some other ones not). Solution:
lockdirent_lockfor the wholelookup()duration, checkUSET_DROPPINGof current dir, and fail with ENOENT if it is set.Nah, we don't care about the spurious lookups. This is a normal
race ofi_mutex.USET_DROPPINGis not a way to prevent VFS views from changing - it's only a way to prevent new children.
Quoted 7 lines
Remember, ->lookup() comes with i_mutex locking. We hold
i_mutexduring the entire delete, so they can't call ->lookup() until we're done with a directory. Conversely, if they wini_mutexand ->lookup() a default group, then try to use it after we've removed it, they'll just ENOENT. This is evident back indo_rename(). They call lookup, which takes and drops locks, then calllock_rename()to get the locks back. And they can handle ENOENT at that point.
Sure, my only concern is the atomic view of userspace: can userspace tolerate that (pwd=A/B, with B a default group of A, B having default groups C and D, and A being removed) 'ls C' returns error because default group C is already removed and 'ls D' is ok because default group D is not removed yet?
Quoted 7 lines
Quoted 4 lines
I was speaking as if we replaced
i_mutexprotection withdirent_lockprotection for a wholemkdir(), that is taking the lock before attach_* and releasing it after.Ok. I think that's not the way to go, what you currently have
is better.
Agreed.
Quoted 6 lines
Quoted 4 lines
The intermediate conditions that really matter are:
1/ the existence of partial default groups trees (I meanconfigfs_direnttrees)
in the middle ofattach_group()anddetach_group(),This is your first case, the mkdir ENOMEM vs rmdir ENOTEMPTY.
Exactly.
Quoted 7 lines
Quoted 3 lines
2/ the existence of default group trees that are tagged as
USET_DROPPINGand
should be treated as not existing anymore.This is not an issue.
USET_DROPPINGdoes *not* mean it went
away. It means we're safe to make it go away. We protect the actual going-away withi_mutex. And that's normal VFS behavior.
Again this is the concern of atomicity from userspace point of view: to provide such atomic view, mkdir(), lookup(), readdir(), and probably attributes open() should just fail when done in a default group flagged with USET_DROPPING.
Anyway, if atomicity from userspace point of view is not a concern, this just makes things simpler, and I'm ok with it.
Louis
Signature · 5 lines
Dr Louis Rilling Kerlabs - IRISA Skype: louis.rilling Campus Universitaire de Beaulieu Phone: (+33|0) 2 99 84 71 52 Avenue du General Leclerc Fax: (+33|0) 2 99 84 71 71 35042 Rennes CEDEX - France http://www.kerlabs.com/
Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 8 lines · Louis Rilling
Quoted 3 lines · Joel Becker
Louis,
Can I just say, you're the first person to do serious review
other than myself, and I really appreciate it :-)It's just that I use configfs in my own work, and I'm playing hard with it, especially with modules. So I need to understand exactly what it does, and what is possible with it.
I'm all for it. And see my other email responding to Luis about
the current concerns with configfs.
Quoted 9 lines
Quoted 3 lines
I see what you are saying here. I'm not sure if that is worth
the complexity - we can say "it was kind of there". No one will ever hit it :-) But let me think about it more.To me it's an issue only if we want to provide some atomic view to
userspace: either userspace sees a group with all of its default groups, or it sees none. So the question is: does userspace need such atomicity? Currently configfs provides it, so this would be a userspace visible change if we break it.
People *won't* see that. default groups are populated and
cleaned under i_mutex. The race of mkdir vs rmdir isn't about seeing partial default groups, it's about the ENOMEM racing the ENOTEMPTY. It doesn't impact lookup or other operations. We can fix it. I'm just not sure it's worth the complexity (and this is an open question).
Quoted 10 lines
Quoted 7 lines
There's another reason this can't be a problem. If we get into
detach_groups(), we takei_mutex, locking outreaddir(). Then we delete the directory, settingS_DEAD. Invfs_readdir(), they checkIS_DEADDIR()after gettingi_mutex. So they will seeS_DEADand not call our ->readdir().S_DEADis important. Someone could actually have ourdefault_groupas their cwd.S_DEADprevents them from doing anything :-)As I told you in a previous email, I'm missing some VFS skills, so thanks again for the explanation.
Oh, don't worry about that. There's nothing wrong with not
knowing - that's why you asked me. Now you do :-)
Quoted 4 lines
Sure, my only concern is the atomic view of userspace: can userspace tolerate that (pwd=A/B, with B a default group of A, B having default groups C and D, and A being removed) 'ls C' returns error because default group C is already removed and 'ls D' is ok because default group D is not removed yet?
They can't see that. We take i_mutex in detach_group. This
locks out lookup and readdir. When we're done with detach_group, all default groups are gone.
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
2/ the existence of default group trees that are tagged as
USET_DROPPINGand
should be treated as not existing anymore.This is not an issue.
USET_DROPPINGdoes *not* mean it went
away. It means we're safe to make it go away. We protect the actual going-away withi_mutex. And that's normal VFS behavior.Again this is the concern of atomicity from userspace point of view: to provide such atomic view,
mkdir(),lookup(),readdir(), and probably attributesopen()should just fail when done in a default group flagged withUSET_DROPPING.
It's not atomic, though, and never has been. I'm not quite sure
what you are unsure of here. Let me try to clarify a little.
Are you worried about two separate runs of the ls(1) command?
# ls A/B/C # ls A/B/D
These can't be atomic, because someone else could rmdir(1) in the middle:
# ls A/B/C # rmdir A/B # ls A/B/D ls: No such file or directory
This is perfectly normal, and there is no way to prevent it - it is separate entrances to the system call.
Do you mean inside one call? That is "ls A/B" would print "C"
but not "D"? That cannot happen, because we hold B's i_mutex during detach_group. So, if readdir beat us to i_mutex, it lists "C D". If we win, we remove both before releasing B's i_Mutex, and readdir errors with ENOENT - we removed B.
I'm not quite sure what inconsistency you are asking about here.
Joel
Signature · 10 lines
"The one important thing i have learned over the years is the difference between taking one's work seriously and taking one's self seriously. The first is imperative and the second is disastrous." -Margot Fonteyn Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
Re: [PATCH 1/3][BUGFIX] configfs: Introduce configfs_dirent_lock
Quoted 11 lines · Joel Becker
Quoted 5 lines
To me it's an issue only if we want to provide some atomic view to
userspace: either userspace sees a group with all of its default groups, or it sees none. So the question is: does userspace need such atomicity? Currently configfs provides it, so this would be a userspace visible change if we break it.People *won't* see that. default groups are populated and
cleaned underi_mutex. The race of mkdir vs rmdir isn't about seeing partial default groups, it's about the ENOMEM racing the ENOTEMPTY. It doesn't impact lookup or other operations. We can fix it. I'm just not sure it's worth the complexity (and this is an open question).
It's not that difficult to implement, you may just find it a bit ugly... I hope to send you a corrected "rename fix" today.
Quoted 9 lines
Quoted 5 lines
Sure, my only concern is the atomic view of userspace: can userspace tolerate that (pwd=A/B, with B a default group of A, B having default groups C and D, and A being removed) 'ls C' returns error because default group C is already removed and 'ls D' is ok because default group D is not removed yet?
They can't see that. We take
i_mutexindetach_group. This
locks out lookup and readdir. When we're done withdetach_group, all default groups are gone.
If I understand correctly, lookup() is not called each time userspace does ls, and in configfs case, it is never called for existing items since the d_cache is populated as soon as the user creates items. So lookup() does not block 'ls' during rmdir() (unless it is a lookup for a never accessed attribute). I think that this is the point that invalidates all my theory about atomicity :)
Quoted 28 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
2/ the existence of default group trees that are tagged as
USET_DROPPINGand
should be treated as not existing anymore.Quoted 8 lines
Quoted 3 lines
This is not an issue.
USET_DROPPINGdoes *not* mean it went
away. It means we're safe to make it go away. We protect the actual going-away withi_mutex. And that's normal VFS behavior.Again this is the concern of atomicity from userspace point of view: to provide such atomic view,
mkdir(),lookup(),readdir(), and probably attributesopen()should just fail when done in a default group flagged withUSET_DROPPING.It's not atomic, though, and never has been. I'm not quite sure
what you are unsure of here. Let me try to clarify a little.
Are you worried about two separate runs of the ls(1) command?# ls A/B/C # ls A/B/DThese can't be atomic, because someone else could rmdir(1) in the middle:
# ls A/B/C # rmdir A/B # ls A/B/D ls: No such file or directory
Quoted 8 lines
This is perfectly normal, and there is no way to prevent it - it is separate entrances to the system call.
Do you mean inside one call? That is "ls A/B" would print "C"
but not "D"? That cannot happen, because we hold B'si_mutexduringdetach_group. So, if readdir beat us toi_mutex, it lists "C D". If we win, we remove both before releasing B'si_Mutex, and readdir errors with ENOENT - we removed B.
I'm not quite sure what inconsistency you are asking about here.
The scenario that made me worry was more:
process 1:
/* PWD=A/B */
# ls C ls: No such file or directory /* some sync between process 1 and 2 */ process 2: /* PWD=A/D */ # ls E /* ok */ # ls E ls: No such file or directory
From a user's point of view, this looks as if somebody did 'rmdir A; mkdir A; rmdir A', while there actually were only 'rmdir A'.
If there were no d_cache, this would be impossible with the current implementation of detach_prep() locking all default groups. But with the d_cache this has always been possible.
Anyway, I give up with this (wrong) atomicity concern.
Louis
Signature · 11 lines
Dr Louis Rilling Kerlabs Skype: louis.rilling Batiment Germanium Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes http://www.kerlabs.com/ 35700 Rennes -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20080616/b3829e80/attachment.bin
PATCH 2/3 configfs: Make configfs_new_dirent() return error code instead of NULL
In reply to Louis RillingAn embedded and charset-unspecified text was scrubbed...
Name: configfs-make-configfs_new_dirent-return-errno.patch
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20080612/3cdbf679/attachment.pl
PATCH 3/3 configfs: Fix deadlock with racing rmdir() and rename()
In reply to Louis RillingAn embedded and charset-unspecified text was scrubbed...
Name: configfs-fix-rmdir-vs-rename-deadlock.patch
Url: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20080612/4c8f4ea8/attachment.pl