Skip to contentSkip to conversation

ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

PATCH New akpm at linux-foundation.org 4 participants 12 messages Jan 24, 2014 – Feb 11, 2014 OCFS2

lore
Share & tools
lore
#0 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

on NodeA:
---------
1) open the test fileA, lseek the end of file. and print the position.
2) close the test fileA

on NodeB:
1) open the test fileA, append the 5k data to test FileA.
2) lseek the end of file. and print the position.
3) close file.

At first we run the test program1 on NodeA , the result is 10k. And then run the test program2 on NodeB, the result is 15k. At last, we run the test program1 on NodeA again, the result is 10k.

After applying this patch the three step result is 15k.

---

 fs/ocfs2/file.c |   11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
fs/ocfs2/file.c renamed from fs/ocfs2/file.c~ocfs2-llseek-requires-ocfs2-inode-lock-for-the-file-in-seek_end +10 -1
diff -puN fs/ocfs2/file.c~ocfs2-llseek-requires-ocfs2-inode-lock-for-the-file-in-seek_end fs/ocfs2/file.c
@@ -2626,7 +2626,16 @@ static loff_t ocfs2_file_llseek(struct f
 	case SEEK_SET:
 		break;
 	case SEEK_END:
-		offset += inode->i_size;
+		/* SEEK_END requires the OCFS2 inode lock for the file
+		 * because it references the file's size.
+		 */
+		ret = ocfs2_inode_lock(inode, NULL, 0);
+		if (ret < 0) {
+			mlog_errno(ret);
+			goto out;
+		}
+		offset += i_size_read(inode);
+		ocfs2_inode_unlock(inode, 0);
 		break;
 	case SEEK_CUR:
 		if (offset == 0) {

_

#1 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 8 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

	--Mark

--
Mark Fasheh

#2 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 4 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 3 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 8 lines
Quoted 5 lines

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Quoted 1 line

I suppose we should fix the bug it added.

fs/ocfs2/file.c renamed from fs/ocfs2/file.c~ocfs2-llseek-requires-ocfs2-inode-lock-for-the-file-in-seek_end-fix +1 -0
@@ -2631,6 +2631,7 @@ static loff_t ocfs2_file_llseek(struct f
 		 */
 		ret = ocfs2_inode_lock(inode, NULL, 0);
 		if (ret < 0) {
+			ocfs2_inode_unlock(inode, 0);
 			mlog_errno(ret);
 			goto out;
 		}

_

#3 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

fs/ocfs2/file.c static loff_t ocfs2_file_llseek(struct f · line 2637

Quoted 12 lines · On Thu, 6 Feb 2014 15:50:29 -0800 Andrew Morton <[email protected]>…

I suppose we should fix the bug it added.

fs/ocfs2/file.c renamed from fs/ocfs2/file.c~ocfs2-llseek-requires-ocfs2-inode-lock-for-the-file-in-seek_end-fix +1 -0
@@ -2631,6 +2631,7 @@ static loff_t ocfs2_file_llseek(struct f
 		 */
 		ret = ocfs2_inode_lock(inode, NULL, 0);
 		if (ret < 0) {
+			ocfs2_inode_unlock(inode, 0);
 			mlog_errno(ret);
 			goto out;
 		}

oops, scratch that.

#4 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 6 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Performance is my primary concern as well. I thought of writing it up but realized I don't really have any evidence off the top of my head one way or the other that this might slow us down.

That said, I kind of question the usefulness of this patch - we got along pretty well so far without locking in lseek and some random dd(1) test doesn't really provide a great end-user reason for why we should do this.

I will note that gfs2 locks for SEEK_END.

Testing would help to answer this, yes. Jensen is this something you can do? I'm not sure exactly what we would run yet though, I have to think about that (or maybe someone can suggest something).

Thanks,

	--Mark

--
Mark Fasheh

#5 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 27 lines · Mark Fasheh
Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 6 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Performance is my primary concern as well. I thought of writing it up but realized I don't really have any evidence off the top of my head one way or the other that this might slow us down.

That said, I kind of question the usefulness of this patch - we got along pretty well so far without locking in lseek and some random dd(1) test doesn't really provide a great end-user reason for why we should do this.

I will note that gfs2 locks for SEEK_END.

Quoted 7 lines

Testing would help to answer this, yes. Jensen is this something you can do? I'm not sure exactly what we would run yet though, I have to think about that (or maybe someone can suggest something).

Thanks,

	--Mark

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

Thanks,
--Jensen

Quoted 5 lines

--
Mark Fasheh

.

#6 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 32 lines · Jensen
Quoted 31 lines · Mark Fasheh
Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 6 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Performance is my primary concern as well. I thought of writing it up but realized I don't really have any evidence off the top of my head one way or the other that this might slow us down.

That said, I kind of question the usefulness of this patch - we got along pretty well so far without locking in lseek and some random dd(1) test doesn't really provide a great end-user reason for why we should do this.

I will note that gfs2 locks for SEEK_END.

Testing would help to answer this, yes. Jensen is this something you can do? I'm not sure exactly what we would run yet though, I have to think about that (or maybe someone can suggest something).

Quoted 5 lines
Quoted 3 lines

Thanks,

	--Mark

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

That doesn't mean we can't or shouldn't quantify the performance impact of your patch.

Please help us measure what the end-user impact of this change will be.

	--Mark

--
Mark Fasheh

#7 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 36 lines · Mark Fasheh
Quoted 35 lines · Jensen
Quoted 34 lines · Mark Fasheh
Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 6 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Performance is my primary concern as well. I thought of writing it up but realized I don't really have any evidence off the top of my head one way or the other that this might slow us down.

That said, I kind of question the usefulness of this patch - we got along pretty well so far without locking in lseek and some random dd(1) test doesn't really provide a great end-user reason for why we should do this.

I will note that gfs2 locks for SEEK_END.

Testing would help to answer this, yes. Jensen is this something you can do? I'm not sure exactly what we would run yet though, I have to think about that (or maybe someone can suggest something).

Thanks,

	--Mark
Quoted 6 lines
Quoted 2 lines

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

That doesn't mean we can't or shouldn't quantify the performance impact of your patch.

Please help us measure what the end-user impact of this change will be.

	--Mark

Ok, I see. I think that how to test the performance impact. we found the bug in web application which only concerned about the correctness. Thanks,
--Jensen

#8 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

In reply to Mark Fasheh
Quoted 36 lines · Mark Fasheh
Quoted 35 lines · Jensen
Quoted 34 lines · Mark Fasheh
Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 6 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Performance is my primary concern as well. I thought of writing it up but realized I don't really have any evidence off the top of my head one way or the other that this might slow us down.

That said, I kind of question the usefulness of this patch - we got along pretty well so far without locking in lseek and some random dd(1) test doesn't really provide a great end-user reason for why we should do this.

I will note that gfs2 locks for SEEK_END.

Testing would help to answer this, yes. Jensen is this something you can do? I'm not sure exactly what we would run yet though, I have to think about that (or maybe someone can suggest something).

Thanks,

	--Mark
Quoted 6 lines
Quoted 2 lines

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

That doesn't mean we can't or shouldn't quantify the performance impact of your patch.

Please help us measure what the end-user impact of this change will be.

	--Mark

test result: 1000000 times lseek call;
index lseek with inode lock (unit:us) lseek without inode lock (unit:us)

    1                   1168162                                    555383
    2                   1168011                                    549504
    3                   1170538                                    549396
    4                   1170375                                    551685
    5                   1170444                                    556719
    6                   1174364                                    555307
    7                   1163294                                    551552
    8                   1170080                                    549350
    9                   1162464                                    553700

10 1165441 552594

avg 1168317 552519

avg with lock - avg without lock = 615798 (avg with lock - avg without lock)/1000000=0.615798 us

Quoted 6 lines

--
Mark Fasheh

.

#9 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 17 lines · On Mon, 10 Feb 2014 16:51:32 +0800 Jensen <[email protected]> wrote:
Quoted 6 lines
Quoted 2 lines

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

That doesn't mean we can't or shouldn't quantify the performance impact of your patch.

Please help us measure what the end-user impact of this change will be.

	--Mark

test result: 1000000 times lseek call;
index lseek with inode lock (unit:us) lseek without inode lock (unit:us)

    1                   1168162                                    555383
    2                   1168011                                    549504
    3                   1170538                                    549396
    4                   1170375                                    551685
    5                   1170444                                    556719
    6                   1174364                                    555307
    7                   1163294                                    551552
    8                   1170080                                    549350
    9                   1162464                                    553700
Quoted 6 lines

10 1165441 552594

avg 1168317 552519

avg with lock - avg without lock = 615798 (avg with lock - avg without lock)/1000000=0.615798 us

hm, what does that actually mean. I guess that to get a feel for the impact on real workloads, this latency needs to be compared with the time for a small IO on fast hardware.

One could test that by creating a large file then doing lots of "lseek(random)+read" operations. Which requires forgetting about the existence of pread() ;)

#10 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

Quoted 29 lines · Andrew Morton
Quoted 17 lines · On Mon, 10 Feb 2014 16:51:32 +0800 Jensen <[email protected]> wrote:
Quoted 6 lines
Quoted 2 lines

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

That doesn't mean we can't or shouldn't quantify the performance impact of your patch.

Please help us measure what the end-user impact of this change will be.

	--Mark

test result: 1000000 times lseek call;
index lseek with inode lock (unit:us) lseek without inode lock (unit:us)

    1                   1168162                                    555383
    2                   1168011                                    549504
    3                   1170538                                    549396
    4                   1170375                                    551685
    5                   1170444                                    556719
    6                   1174364                                    555307
    7                   1163294                                    551552
    8                   1170080                                    549350
    9                   1162464                                    553700
Quoted 6 lines

10 1165441 552594

avg 1168317 552519

avg with lock - avg without lock = 615798 (avg with lock - avg without lock)/1000000=0.615798 us

hm, what does that actually mean. I guess that to get a feel for the impact on real workloads, this latency needs to be compared with the time for a small IO on fast hardware.

Quoted 5 lines

One could test that by creating a large file then doing lots of "lseek(random)+read" operations. Which requires forgetting about the existence of pread() ;)

.

because lseek only lock in SEEK_END. so your above mention can't test.

Quoted 1 line
#11 lore raw

PATCH 9/11 ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

In reply to Jensen

Hi Canquan,

Quoted 32 lines · Jensen
Quoted 31 lines · Mark Fasheh
Quoted 9 lines · Andrew Morton
Quoted 7 lines · On Thu, 6 Feb 2014 15:42:53 -0800 Mark Fasheh <[email protected]> wrote:
Quoted 6 lines · akpm at linux-foundation.org

From: Jensen <[email protected]>
Subject: ocfs2: llseek requires ocfs2 inode lock for the file in SEEK_END

llseek requires ocfs2 inode lock for updating the file size in SEEK_END. because the file size maybe update on another node.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

This bug can be reproduce the following scenario: at first, we dd a test fileA, the file size is 10k.

Basically, you want to amke SEEK_END cluster-aware. This patch would be the right way to do it.

Sunil was worried about the performance impact. Correctness beats performance, but some quantitative testing would be useful?

Performance is my primary concern as well. I thought of writing it up but realized I don't really have any evidence off the top of my head one way or the other that this might slow us down.

That said, I kind of question the usefulness of this patch - we got along pretty well so far without locking in lseek and some random dd(1) test doesn't really provide a great end-user reason for why we should do this.

I will note that gfs2 locks for SEEK_END.

Testing would help to answer this, yes. Jensen is this something you can do? I'm not sure exactly what we would run yet though, I have to think about that (or maybe someone can suggest something).

Quoted 5 lines
Quoted 3 lines

Thanks,

	--Mark

ocfs2 is a cluster file system. as like read/write/open/rmdir/unlink interface which think of cluster-aware. I think the seek interface need cluster-aware. May be it has the performance impact. but it's correctness is more important than performance.

That would be great if you can show any testing results regarding performance as you're the author of this patch and if you'd like to continue to push it, though this is not requested always.

Thanks,
-Jeff