Skip to contentSkip to conversation

i_mutex locking in generic_file_splice_write()

DISCUSSION Mark Fasheh 4 participants 13 messages Oct 12, 2006 – Oct 16, 2006 VFS

lore
Share & tools
lore
#0 lore raw

Hi,
generic_file_splice_write() will call into a file systems
->prepare_write() and ->commit_write() via the the pipe_to_file() actor. pipe_to_file() is careful to take the pipe inode i_mutex, but nowhere in the call path do I see i_mutex on the inode being written to taken.

Shouldn't we be taking this before calling into ->prepare_write() and ->commit_write(). What's preventing generic_file_splice_write() from racing a truncate? Or maybe even another write?

A quick look through other callers reveals that generic_file_aio_write() and do_lo_send_aops() both are careful to take i_mutex.

Thanks,

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

#1 lore raw
Quoted 5 lines · On Thu, 12 Oct 2006 12:01:52 -0700 Mark Fasheh <[email protected]> wrote:

Hi,
generic_file_splice_write() will call into a file systems
->prepare_write() and ->commit_write() via the the pipe_to_file() actor. pipe_to_file() is careful to take the pipe inode i_mutex, but nowhere in the call path do I see i_mutex on the inode being written to taken.

I'm not sure that ecryptfs has full i_mutex coverage either.

Quoted 3 lines

Shouldn't we be taking this before calling into ->prepare_write() and ->commit_write(). What's preventing generic_file_splice_write() from racing a truncate? Or maybe even another write?

The lock_page() will block truncate and will block write()s to this particular page.

Quoted 2 lines

A quick look through other callers reveals that generic_file_aio_write() and do_lo_send_aops() both are careful to take i_mutex.

I'm trying to remember what i_mutex actually protects in this context. i_size, certainly - if we go changing the file size without locks then other places might get surprised. For example, a concurrent write() at a larger file offset might try to increase i_size but if it loses the race against the unlocked i_size-changing thread, the inode ends up with the smaller i_size.

So yup, we need i_mutex if only for that reason.

#2 lore raw
Quoted 6 lines · Andrew Morton
Quoted 3 lines

Shouldn't we be taking this before calling into ->prepare_write() and ->commit_write(). What's preventing generic_file_splice_write() from racing a truncate? Or maybe even another write?

The lock_page() will block truncate and will block write()s to this particular page.

Ok.

Quoted 9 lines
Quoted 2 lines

A quick look through other callers reveals that generic_file_aio_write() and do_lo_send_aops() both are careful to take i_mutex.

I'm trying to remember what i_mutex actually protects in this context. i_size, certainly - if we go changing the file size without locks then other places might get surprised. For example, a concurrent write() at a larger file offset might try to increase i_size but if it loses the race against the unlocked i_size-changing thread, the inode ends up with the smaller i_size.

I'm also worried about concurrent allocation tree changes. Perhaps I'm mistaken and all file systems we care about can handle them happening concurrently, but otherwise couldn't two processes writing to different sparse regions in a file cause problems? One process via file write and the other via a splice write.

Quoted 1 line

So yup, we need i_mutex if only for that reason.

Ok. Here's a first pass. The double lock is ugly, but as far as I can tell we need it. Unless there's a rule about ordering between pipe inodes and "other" inodes that I don't know about.

Compile tested only. I probably won't get a chance to actually run it until late this weekend at the earliest :/

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

From: Mark Fasheh <[email protected]>

[PATCH] Take i_mutex in splice_from_pipe()

The splice_actor may be calling ->prepare_write() and ->commit_write(). We want i_mutex on the inode being written to before calling those so that we don't race i_size changes.

---

 fs/splice.c |   21 +++++++++++++++++++--
 1 files changed, 19 insertions(+), 2 deletions(-)

96e880f22bd1c0e2809ebbfe5bf122ed67019e33

fs/splice.c +19 -2
index 13e92dd..e1ecb9e 100644
@@ -713,6 +713,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 {
 	int ret, do_wakeup, err;
 	struct splice_desc sd;
+	struct inode *inode = out->f_mapping->host;
 
 	ret = 0;
 	do_wakeup = 0;
@@ -722,8 +723,23 @@ ssize_t splice_from_pipe(struct pipe_ino
 	sd.file = out;
 	sd.pos = *ppos;
 
-	if (pipe->inode)
-		mutex_lock(&pipe->inode->i_mutex);
+	/*
+	 * The actor worker might be calling ->prepare_write and
+	 * ->commit_write. Most of the time, these expect i_mutex to
+	 * be held. Since this may result in an ABBA deadlock with
+	 * pipe->inode, we have to order lock acquiry here.
+	 */
+	if (pipe->inode) {
+		if (pipe->inode < inode) {
+			mutex_lock_nested(&pipe->inode->i_mutex, I_MUTEX_PARENT);
+			mutex_lock_nested(&inode->i_mutex, I_MUTEX_CHILD);
+		} else {
+			mutex_lock_nested(&inode->i_mutex, I_MUTEX_PARENT);
+			mutex_lock_nested(&pipe->inode->i_mutex, I_MUTEX_CHILD);
+		}
+	} else {
+		mutex_lock(&inode->i_mutex);
+	}
 
 	for (;;) {
 		if (pipe->nrbufs) {
@@ -799,6 +815,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 
 	if (pipe->inode)
 		mutex_unlock(&pipe->inode->i_mutex);
+	mutex_unlock(&inode->i_mutex);
 
 	if (do_wakeup) {
 		smp_mb();
Signature · 1 line
1.3.3
#3 lore raw
Quoted 26 lines · Mark Fasheh
Quoted 6 lines · Andrew Morton
Quoted 3 lines

Shouldn't we be taking this before calling into ->prepare_write() and ->commit_write(). What's preventing generic_file_splice_write() from racing a truncate? Or maybe even another write?

The lock_page() will block truncate and will block write()s to this particular page.

Ok.

Quoted 9 lines
Quoted 2 lines

A quick look through other callers reveals that generic_file_aio_write() and do_lo_send_aops() both are careful to take i_mutex.

I'm trying to remember what i_mutex actually protects in this context. i_size, certainly - if we go changing the file size without locks then other places might get surprised. For example, a concurrent write() at a larger file offset might try to increase i_size but if it loses the race against the unlocked i_size-changing thread, the inode ends up with the smaller i_size.

I'm also worried about concurrent allocation tree changes. Perhaps I'm mistaken and all file systems we care about can handle them happening concurrently, but otherwise couldn't two processes writing to different sparse regions in a file cause problems? One process via file write and the other via a splice write.

Quoted 7 lines
Quoted 1 line

So yup, we need i_mutex if only for that reason.

Ok. Here's a first pass. The double lock is ugly, but as far as I can tell we need it. Unless there's a rule about ordering between pipe inodes and "other" inodes that I don't know about.

Compile tested only. I probably won't get a chance to actually run it until late this weekend at the earliest :/

Patch looks ok to me. The double lock test only works, as long as splice is the only one ever locking both mutexes. Or if others follow the same ordering rules. I'm not very well versed in vfs matters, is that guarenteed?

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 7 lines · On Fri, 13 Oct 2006 09:45:17 +0200 Jens Axboe <[email protected]> wrote:
Quoted 2 lines

Compile tested only. I probably won't get a chance to actually run it until late this weekend at the earliest :/

Patch looks ok to me. The double lock test only works, as long as splice is the only one ever locking both mutexes. Or if others follow the same ordering rules. I'm not very well versed in vfs matters, is that guarenteed?

Taking the lowest-addressed lock first is the usual convention, if we really have to do that. I'm not aware of anywhere else where we pull this trick with i_mutex though.

#5 lore raw
Quoted 6 lines · Andrew Morton
Quoted 3 lines · On Fri, 13 Oct 2006 09:45:17 +0200 Jens Axboe <[email protected]> wrote:
Quoted 2 lines

Compile tested only. I probably won't get a chance to actually run it until late this weekend at the earliest :/

Quoted 8 lines
Quoted 4 lines

Patch looks ok to me. The double lock test only works, as long as splice is the only one ever locking both mutexes. Or if others follow the same ordering rules. I'm not very well versed in vfs matters, is that guarenteed?

Taking the lowest-addressed lock first is the usual convention, if we really have to do that. I'm not aware of anywhere else where we pull this trick with i_mutex though.

It is, but people had concerns with that approach when it was originally done for pipe tee'ing. Things like lock_rename() look scary.

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 6 lines · Jens Axboe
Quoted 3 lines

Taking the lowest-addressed lock first is the usual convention, if we really have to do that. I'm not aware of anywhere else where we pull this trick with i_mutex though.

It is, but people had concerns with that approach when it was originally done for pipe tee'ing. Things like lock_rename() look scary.

Yeah, I'm not too thrilled with the double locking either. Eventually I think we'll need some _nolock variant of generic_file_splice_write() so that OCFS2 can order cluster locks within i_mutex (much like you see with ocfs2_file_aio_write(), etc).

At the very least, I'd rather not have this stuff open coded everywhere. If we consolidate the behavior within a set of functions, we'll be able to avoid questions later about locking order, etc.

I felt dirty just typing up this patch. Sigh. Let me know what you think.

	--Mark

From: Mark Fasheh <[email protected]>

[PATCH] Take i_mutex in splice_from_pipe()

The splice_actor may be calling ->prepare_write() and ->commit_write(). We want i_mutex on the inode being written to before calling those so that we don't race i_size changes.

The double locking behavior is done elsewhere in splice.c, and if we eventually want _nolock variants of generic_file_splice_write(), fs modules might have to replicate the nasty locking code. We introduce inode_double_lock() and inode_double_unlock() to consolidate the locking rules into one set of functions.

---
 fs/splice.c        |   24 +++++++++++-------------
 include/linux/fs.h |   29 +++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 13 deletions(-)
fs/splice.c +11 -13
index 13e92dd..4d8a774 100644
@@ -713,6 +713,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 {
 	int ret, do_wakeup, err;
 	struct splice_desc sd;
+	struct inode *inode = out->f_mapping->host;
 
 	ret = 0;
 	do_wakeup = 0;
@@ -722,8 +723,13 @@ ssize_t splice_from_pipe(struct pipe_ino
 	sd.file = out;
 	sd.pos = *ppos;
 
-	if (pipe->inode)
-		mutex_lock(&pipe->inode->i_mutex);
+	/*
+	 * The actor worker might be calling ->prepare_write and
+	 * ->commit_write. Most of the time, these expect i_mutex to
+	 * be held. Since this may result in an ABBA deadlock with
+	 * pipe->inode, we have to order lock acquiry here.
+	 */
+	inode_double_lock(inode, pipe->inode);
 
 	for (;;) {
 		if (pipe->nrbufs) {
@@ -797,8 +803,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 		pipe_wait(pipe);
 	}
 
-	if (pipe->inode)
-		mutex_unlock(&pipe->inode->i_mutex);
+	inode_double_unlock(inode, pipe->inode);
 
 	if (do_wakeup) {
 		smp_mb();
@@ -1400,13 +1405,7 @@ static int link_pipe(struct pipe_inode_i
 	 * grabbing by inode address. Otherwise two different processes
 	 * could deadlock (one doing tee from A -> B, the other from B -> A).
 	 */
-	if (ipipe->inode < opipe->inode) {
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_CHILD);
-	} else {
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_CHILD);
-	}
+	inode_double_lock(ipipe->inode, opipe->inode);
 
 	do {
 		if (!opipe->readers) {
@@ -1450,8 +1449,7 @@ static int link_pipe(struct pipe_inode_i
 		i++;
 	} while (len);
 
-	mutex_unlock(&ipipe->inode->i_mutex);
-	mutex_unlock(&opipe->inode->i_mutex);
+	inode_double_unlock(ipipe->inode, opipe->inode);
 
 	/*
 	 * If we put data in the output pipe, wakeup any potential readers.
include/linux/fs.h +29 -0
index 34406ed..31d8548 100644
@@ -624,6 +624,35 @@ enum inode_i_mutex_lock_class
 };
 
 /*
+ * We rarely want to lock two inodes that do not have a parent/child
+ * relationship (such as directory, child inode) simultaneously. The
+ * vast majority of file systems should be able to get along fine
+ * without this. Do not use these functions except as a last resort.
+ */
+static inline void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (!inode2) {
+		mutex_lock(&inode1->i_mutex);
+		return;
+	}
+
+	if (inode1 < inode2) {
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
+	} else {
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
+	}
+}
+
+static inline void inode_double_unlock(struct inode *inode1, struct inode *inode2)
+{
+	mutex_unlock(&inode1->i_mutex);
+	if (inode2)
+		mutex_unlock(&inode2->i_mutex);
+}
+
+/*
  * NOTE: in a 32bit arch with a preemptable kernel and
  * an UP compile the i_size_read/write must be atomic
  * with respect to the local cpu (unlike with preempt disabled),
Signature · 1 line
1.4.2.3
#7 lore raw
Quoted 11 lines · Mark Fasheh

/*
+ * We rarely want to lock two inodes that do not have a parent/child
+ * relationship (such as directory, child inode) simultaneously. The
+ * vast majority of file systems should be able to get along fine
+ * without this. Do not use these functions except as a last resort.
+ */

include/linux/fs.h +5 -0
+static inline void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (!inode2) {
+		mutex_lock(&inode1->i_mutex);
+		return;

include/linux/fs.h enum inode_i_mutex_lock_class · line 646

Quoted 10 lines
include/linux/fs.h +10 -0
+	}
+
+	if (inode1 < inode2) {
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
+	} else {
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
+	}
+}

This wont fly, it's completely tailored to splice in that one one inode can be NULL. It also requires the 2nd inode to be that potentially NULL pointer.

The function should work for any of the inodes to be NULL, if exported as a helper.

Signature · 1 line
Jens Axboe
#8 lore raw
Quoted 3 lines · Jens Axboe

This wont fly, it's completely tailored to splice in that one one inode can be NULL. It also requires the 2nd inode to be that potentially NULL pointer.

*shrug* from my memory, 1st pointer always valid, 2nd possibly NULL seemed to be a common paradigm, though I can't really point to any code, now that I think of it.

Quoted 2 lines

The function should work for any of the inodes to be NULL, if exported as a helper.

Here's a version that also handles both inodes being the same. We're quickly approaching an elevated level of scariness in inode_double_lock :) Also, things have been un-inlined.

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

From: Mark Fasheh <[email protected]>

[PATCH] Take i_mutex in splice_from_pipe()

The splice_actor may be calling ->prepare_write() and ->commit_write(). We want i_mutex on the inode being written to before calling those so that we don't race i_size changes.

The double locking behavior is done elsewhere in splice.c, and if we eventually want _nolock variants of generic_file_splice_write(), fs modules might have to replicate the nasty locking code. We introduce inode_double_lock() and inode_double_unlock() to consolidate the locking rules into one set of functions.

---
 fs/inode.c         |   31 +++++++++++++++++++++++++++++++
 fs/splice.c        |   24 +++++++++++-------------
 include/linux/fs.h |    3 +++
 3 files changed, 45 insertions(+), 13 deletions(-)
fs/inode.c +31 -0
index bf6bec4..f42920d 100644
@@ -1306,6 +1306,37 @@ void wake_up_inode(struct inode *inode)
 	wake_up_bit(&inode->i_state, __I_LOCK);
 }
 
+/*
+ * We rarely want to lock two inodes that do not have a parent/child
+ * relationship (such as directory, child inode) simultaneously. The
+ * vast majority of file systems should be able to get along fine
+ * without this. Do not use these functions except as a last resort.
+ */
+void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (!inode2) {
+		mutex_lock(&inode1->i_mutex);
+		return;
+	}
+
+	if (inode1 < inode2) {
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
+	} else {
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
+	}
+}
+EXPORT_SYMBOL(inode_double_lock);
+
+void inode_double_unlock(struct inode *inode1, struct inode *inode2)
+{
+	mutex_unlock(&inode1->i_mutex);
+	if (inode2)
+		mutex_unlock(&inode2->i_mutex);
+}
+EXPORT_SYMBOL(inode_double_unlock);
+
 static __initdata unsigned long ihash_entries;
 static int __init set_ihash_entries(char *str)
 {
fs/splice.c +11 -13
index 13e92dd..4d8a774 100644
@@ -713,6 +713,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 {
 	int ret, do_wakeup, err;
 	struct splice_desc sd;
+	struct inode *inode = out->f_mapping->host;
 
 	ret = 0;
 	do_wakeup = 0;
@@ -722,8 +723,13 @@ ssize_t splice_from_pipe(struct pipe_ino
 	sd.file = out;
 	sd.pos = *ppos;
 
-	if (pipe->inode)
-		mutex_lock(&pipe->inode->i_mutex);
+	/*
+	 * The actor worker might be calling ->prepare_write and
+	 * ->commit_write. Most of the time, these expect i_mutex to
+	 * be held. Since this may result in an ABBA deadlock with
+	 * pipe->inode, we have to order lock acquiry here.
+	 */
+	inode_double_lock(inode, pipe->inode);
 
 	for (;;) {
 		if (pipe->nrbufs) {
@@ -797,8 +803,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 		pipe_wait(pipe);
 	}
 
-	if (pipe->inode)
-		mutex_unlock(&pipe->inode->i_mutex);
+	inode_double_unlock(inode, pipe->inode);
 
 	if (do_wakeup) {
 		smp_mb();
@@ -1400,13 +1405,7 @@ static int link_pipe(struct pipe_inode_i
 	 * grabbing by inode address. Otherwise two different processes
 	 * could deadlock (one doing tee from A -> B, the other from B -> A).
 	 */
-	if (ipipe->inode < opipe->inode) {
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_CHILD);
-	} else {
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_CHILD);
-	}
+	inode_double_lock(ipipe->inode, opipe->inode);
 
 	do {
 		if (!opipe->readers) {
@@ -1450,8 +1449,7 @@ static int link_pipe(struct pipe_inode_i
 		i++;
 	} while (len);
 
-	mutex_unlock(&ipipe->inode->i_mutex);
-	mutex_unlock(&opipe->inode->i_mutex);
+	inode_double_unlock(ipipe->inode, opipe->inode);
 
 	/*
 	 * If we put data in the output pipe, wakeup any potential readers.
include/linux/fs.h +3 -0
index 34406ed..ddbb833 100644
@@ -623,6 +623,9 @@ enum inode_i_mutex_lock_class
 	I_MUTEX_QUOTA
 };
 
+extern void inode_double_lock(struct inode *inode1, struct inode *inode2);
+extern void inode_double_unlock(struct inode *inode1, struct inode *inode2);
+
 /*
  * NOTE: in a 32bit arch with a preemptable kernel and
  * an UP compile the i_size_read/write must be atomic
Signature · 1 line
1.4.2.3
#9 lore raw
Quoted 9 lines · Mark Fasheh
Quoted 3 lines · Jens Axboe

This wont fly, it's completely tailored to splice in that one one inode can be NULL. It also requires the 2nd inode to be that potentially NULL pointer.

*shrug* from my memory, 1st pointer always valid, 2nd possibly NULL seemed to be a common paradigm, though I can't really point to any code, now that I think of it.

Quoted 5 lines
Quoted 2 lines

The function should work for any of the inodes to be NULL, if exported as a helper.

Here's a version that also handles both inodes being the same. We're quickly approaching an elevated level of scariness in inode_double_lock :) Also, things have been un-inlined.

Hmm, it doesn't seem any different :-). Bad diff attached?

Quoted 15 lines
fs/inode.c +15 -0
+void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (!inode2) {
+		mutex_lock(&inode1->i_mutex);
+		return;
+	}
+
+	if (inode1 < inode2) {
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
+	} else {
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
+	}
+}
Signature · 1 line
Jens Axboe
#10 lore raw
Quoted 1 line · Jens Axboe

Hmm, it doesn't seem any different :-). Bad diff attached?

Ahh, silly me - I applied the wrong version of the patch :) Here's the correct one.

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

From: Mark Fasheh <[email protected]>

[PATCH] Take i_mutex in splice_from_pipe()

The splice_actor may be calling ->prepare_write() and ->commit_write(). We want i_mutex on the inode being written to before calling those so that we don't race i_size changes.

The double locking behavior is done elsewhere in splice.c, and if we eventually want _nolock variants of generic_file_splice_write(), fs modules might have to replicate the nasty locking code. We introduce inode_double_lock() and inode_double_unlock() to consolidate the locking rules into one set of functions.

---
 fs/inode.c         |   38 ++++++++++++++++++++++++++++++++++++++
 fs/splice.c        |   24 +++++++++++-------------
 include/linux/fs.h |    3 +++
 3 files changed, 52 insertions(+), 13 deletions(-)
fs/inode.c +38 -0
index d9a21d1..f94b02b 100644
@@ -1306,6 +1306,44 @@ void wake_up_inode(struct inode *inode)
 	wake_up_bit(&inode->i_state, __I_LOCK);
 }
 
+/*
+ * We rarely want to lock two inodes that do not have a parent/child
+ * relationship (such as directory, child inode) simultaneously. The
+ * vast majority of file systems should be able to get along fine
+ * without this. Do not use these functions except as a last resort.
+ */
+void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (!inode1 || !inode2) {
+		if (inode1)
+			mutex_lock(&inode1->i_mutex);
+		else if (inode2)
+			mutex_lock(&inode2->i_mutex);
+		return;
+	}
+
+	if (inode1 == inode2) {
+		mutex_lock(&inode1->i_mutex);
+	} else if (inode1 < inode2) {
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
+	} else {
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
+	}
+}
+EXPORT_SYMBOL(inode_double_lock);
+
+void inode_double_unlock(struct inode *inode1, struct inode *inode2)
+{
+	if (inode1)
+		mutex_unlock(&inode1->i_mutex);
+
+	if (inode2 && inode2 != inode1)
+		mutex_unlock(&inode2->i_mutex);
+}
+EXPORT_SYMBOL(inode_double_unlock);
+
 static __initdata unsigned long ihash_entries;
 static int __init set_ihash_entries(char *str)
 {
fs/splice.c +11 -13
index a567010..c1072b6 100644
@@ -713,6 +713,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 {
 	int ret, do_wakeup, err;
 	struct splice_desc sd;
+	struct inode *inode = out->f_mapping->host;
 
 	ret = 0;
 	do_wakeup = 0;
@@ -722,8 +723,13 @@ ssize_t splice_from_pipe(struct pipe_ino
 	sd.file = out;
 	sd.pos = *ppos;
 
-	if (pipe->inode)
-		mutex_lock(&pipe->inode->i_mutex);
+	/*
+	 * The actor worker might be calling ->prepare_write and
+	 * ->commit_write. Most of the time, these expect i_mutex to
+	 * be held. Since this may result in an ABBA deadlock with
+	 * pipe->inode, we have to order lock acquiry here.
+	 */
+	inode_double_lock(inode, pipe->inode);
 
 	for (;;) {
 		if (pipe->nrbufs) {
@@ -797,8 +803,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 		pipe_wait(pipe);
 	}
 
-	if (pipe->inode)
-		mutex_unlock(&pipe->inode->i_mutex);
+	inode_double_unlock(inode, pipe->inode);
 
 	if (do_wakeup) {
 		smp_mb();
@@ -1400,13 +1405,7 @@ static int link_pipe(struct pipe_inode_i
 	 * grabbing by inode address. Otherwise two different processes
 	 * could deadlock (one doing tee from A -> B, the other from B -> A).
 	 */
-	if (ipipe->inode < opipe->inode) {
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_CHILD);
-	} else {
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_CHILD);
-	}
+	inode_double_lock(ipipe->inode, opipe->inode);
 
 	do {
 		if (!opipe->readers) {
@@ -1450,8 +1449,7 @@ static int link_pipe(struct pipe_inode_i
 		i++;
 	} while (len);
 
-	mutex_unlock(&ipipe->inode->i_mutex);
-	mutex_unlock(&opipe->inode->i_mutex);
+	inode_double_unlock(ipipe->inode, opipe->inode);
 
 	/*
 	 * If we put data in the output pipe, wakeup any potential readers.
include/linux/fs.h +3 -0
index 34406ed..ddbb833 100644
@@ -623,6 +623,9 @@ enum inode_i_mutex_lock_class
 	I_MUTEX_QUOTA
 };
 
+extern void inode_double_lock(struct inode *inode1, struct inode *inode2);
+extern void inode_double_unlock(struct inode *inode1, struct inode *inode2);
+
 /*
  * NOTE: in a 32bit arch with a preemptable kernel and
  * an UP compile the i_size_read/write must be atomic
Signature · 1 line
1.4.2.3
#11 lore raw

fs/inode.c void wake_up_inode(struct inode *inode) · line 1326

Quoted 12 lines · Mark Fasheh
fs/inode.c +12 -0
+void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (!inode1 || !inode2) {
+		if (inode1)
+			mutex_lock(&inode1->i_mutex);
+		else if (inode2)
+			mutex_lock(&inode2->i_mutex);
+		return;
+	}
+
+	if (inode1 == inode2) {
+		mutex_lock(&inode1->i_mutex);

Could be a tiny bit cleaner if you put the "inode1 == inode2" case above:

void inode_double_lock(struct inode *inode1, struct inode *inode2)

{
	if (inode1 == NULL || inode2 == NULL || inode1 == inode2) {
		if (inode1)
			mutex_lock(&inode1->i_mutex);
		else if (inode2)
			mutex_lock(&inode2->i_mutex);
		return;
	}

	if (inode1 < inode2) {
		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
	} else {
		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
	}
}

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

#12 lore raw
Quoted 1 line · Andreas Dilger

Could be a tiny bit cleaner if you put the "inode1 == inode2" case above:

Good catch - thanks for pointing it out. Here's a version with your cleanup. This version of the patch was actually tested too.

	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
[email protected]

From: Mark Fasheh <[email protected]>
[PATCH] Take i_mutex in splice_from_pipe()

The splice_actor may be calling ->prepare_write() and ->commit_write(). We want i_mutex on the inode being written to before calling those so that we don't race i_size changes.

The double locking behavior is done elsewhere in splice.c, and if we eventually want _nolock variants of generic_file_splice_write(), fs modules might have to replicate the nasty locking code. We introduce inode_double_lock() and inode_double_unlock() to consolidate the locking rules into one set of functions.

---

 fs/inode.c         |   36 ++++++++++++++++++++++++++++++++++++
 fs/splice.c        |   24 +++++++++++-------------
 include/linux/fs.h |    3 +++
 3 files changed, 50 insertions(+), 13 deletions(-)

049fb2a0358c842a6c5256a4c0d488441bfd8e11

fs/inode.c +36 -0
index d9a21d1..26cdb11 100644
@@ -1306,6 +1306,42 @@ void wake_up_inode(struct inode *inode)
 	wake_up_bit(&inode->i_state, __I_LOCK);
 }
 
+/*
+ * We rarely want to lock two inodes that do not have a parent/child
+ * relationship (such as directory, child inode) simultaneously. The
+ * vast majority of file systems should be able to get along fine
+ * without this. Do not use these functions except as a last resort.
+ */
+void inode_double_lock(struct inode *inode1, struct inode *inode2)
+{
+	if (inode1 == NULL || inode2 == NULL || inode1 == inode2) {
+		if (inode1)
+			mutex_lock(&inode1->i_mutex);
+		else if (inode2)
+			mutex_lock(&inode2->i_mutex);
+		return;
+	}
+
+	if (inode1 < inode2) {
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_CHILD);
+	} else {
+		mutex_lock_nested(&inode2->i_mutex, I_MUTEX_PARENT);
+		mutex_lock_nested(&inode1->i_mutex, I_MUTEX_CHILD);
+	}
+}
+EXPORT_SYMBOL(inode_double_lock);
+
+void inode_double_unlock(struct inode *inode1, struct inode *inode2)
+{
+	if (inode1)
+		mutex_unlock(&inode1->i_mutex);
+
+	if (inode2 && inode2 != inode1)
+		mutex_unlock(&inode2->i_mutex);
+}
+EXPORT_SYMBOL(inode_double_unlock);
+
 static __initdata unsigned long ihash_entries;
 static int __init set_ihash_entries(char *str)
 {
fs/splice.c +11 -13
index a567010..c1072b6 100644
@@ -713,6 +713,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 {
 	int ret, do_wakeup, err;
 	struct splice_desc sd;
+	struct inode *inode = out->f_mapping->host;
 
 	ret = 0;
 	do_wakeup = 0;
@@ -722,8 +723,13 @@ ssize_t splice_from_pipe(struct pipe_ino
 	sd.file = out;
 	sd.pos = *ppos;
 
-	if (pipe->inode)
-		mutex_lock(&pipe->inode->i_mutex);
+	/*
+	 * The actor worker might be calling ->prepare_write and
+	 * ->commit_write. Most of the time, these expect i_mutex to
+	 * be held. Since this may result in an ABBA deadlock with
+	 * pipe->inode, we have to order lock acquiry here.
+	 */
+	inode_double_lock(inode, pipe->inode);
 
 	for (;;) {
 		if (pipe->nrbufs) {
@@ -797,8 +803,7 @@ ssize_t splice_from_pipe(struct pipe_ino
 		pipe_wait(pipe);
 	}
 
-	if (pipe->inode)
-		mutex_unlock(&pipe->inode->i_mutex);
+	inode_double_unlock(inode, pipe->inode);
 
 	if (do_wakeup) {
 		smp_mb();
@@ -1400,13 +1405,7 @@ static int link_pipe(struct pipe_inode_i
 	 * grabbing by inode address. Otherwise two different processes
 	 * could deadlock (one doing tee from A -> B, the other from B -> A).
 	 */
-	if (ipipe->inode < opipe->inode) {
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_CHILD);
-	} else {
-		mutex_lock_nested(&opipe->inode->i_mutex, I_MUTEX_PARENT);
-		mutex_lock_nested(&ipipe->inode->i_mutex, I_MUTEX_CHILD);
-	}
+	inode_double_lock(ipipe->inode, opipe->inode);
 
 	do {
 		if (!opipe->readers) {
@@ -1450,8 +1449,7 @@ static int link_pipe(struct pipe_inode_i
 		i++;
 	} while (len);
 
-	mutex_unlock(&ipipe->inode->i_mutex);
-	mutex_unlock(&opipe->inode->i_mutex);
+	inode_double_unlock(ipipe->inode, opipe->inode);
 
 	/*
 	 * If we put data in the output pipe, wakeup any potential readers.
include/linux/fs.h +3 -0
index 34406ed..ddbb833 100644
@@ -623,6 +623,9 @@ enum inode_i_mutex_lock_class
 	I_MUTEX_QUOTA
 };
 
+extern void inode_double_lock(struct inode *inode1, struct inode *inode2);
+extern void inode_double_unlock(struct inode *inode1, struct inode *inode2);
+
 /*
  * NOTE: in a 32bit arch with a preemptable kernel and
  * an UP compile the i_size_read/write must be atomic
Signature · 1 line
1.3.3