Skip to contentSkip to conversation

On current git head webservers stopped working

DISCUSSION Gabriel C 5 participants 12 messages Jul 13, 2007 VFS

lore
Share & tools
lore

Gabriel C

author
#0 lore raw

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

This is :

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blobdiff;f=fs/read_write.c;h=47da8a4a0fbecd1bbf00aaf29fd1fdc6dc7408d8;hp=4d03008f015b9ae6d2565cdbc179d034687c3d08;hb=534f2aaa6ab07cd71164180bc958a7dcde41db11;hpb=6a14b90bb6bc7cd83e2a444bf457a2ea645cbfe7

crazy@lara:/work/crazy/linux-git/linux-2.6$ git bisect log
git-bisect start
# bad: [4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4] lots-of-architectures:
enable arbitary speed tty support
git-bisect bad 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4
# good: [7dcca30a32aadb0520417521b0c44f42d09fe05c] Linux 2.6.22
git-bisect good 7dcca30a32aadb0520417521b0c44f42d09fe05c
# good: [bf74b964775009071cf12f9d59d4dd5e388fbe0b] [MIPS] DEC: Fix
modpost warning.
git-bisect good bf74b964775009071cf12f9d59d4dd5e388fbe0b
# bad: [1b21f458ddbc8fb6fceeb68158e9e04b2571dabd] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/steve/gfs2-2.6-nmw
git-bisect bad 1b21f458ddbc8fb6fceeb68158e9e04b2571dabd
# bad: [01370f0603f8435d415a19f7e62d1bab826c3589] Merge branch
'splice-2.6.23' of git://git.kernel.dk/data/git/linux-2.6-block
git-bisect bad 01370f0603f8435d415a19f7e62d1bab826c3589
# good: [4f179f71607382888c5ce5385a4ee136e1a2002a] Input: serio_raw -
shut up errorneous warning
git-bisect good 4f179f71607382888c5ce5385a4ee136e1a2002a
# bad: [cac36bb06efe4880234524e117e0e712b10b1f16] pipe: change the
->pin() operation to ->confirm()
git-bisect bad cac36bb06efe4880234524e117e0e712b10b1f16
# bad: [cf8208d0eabd1d5d2625ec02a175a294c3f30d36] sendfile: convert nfsd
to splice_direct_to_actor()
git-bisect bad cf8208d0eabd1d5d2625ec02a175a294c3f30d36
# bad: [5ffc4ef45b3b0a57872f631b4e4ceb8ace0d7496] sendfile: remove
.sendfile from filesystems that use generic_file_sendfile()
git-bisect bad 5ffc4ef45b3b0a57872f631b4e4ceb8ace0d7496
# good: [6a14b90bb6bc7cd83e2a444bf457a2ea645cbfe7] vmsplice: add
vmsplice-to-user support
git-bisect good 6a14b90bb6bc7cd83e2a444bf457a2ea645cbfe7
# bad: [534f2aaa6ab07cd71164180bc958a7dcde41db11] sys_sendfile: switch
to using ->splice_read, if available
git-bisect bad 534f2aaa6ab07cd71164180bc958a7dcde41db11

Reverting 534f2aaa6ab07cd71164180bc958a7dcde41db11 from git2 solved that problem for me.

I've compiled current git head and the problem still exists but I cannot revert this commit while some conflicts.

Regards ,

Gabriel C

#1 lore raw
Quoted 17 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

Quoted 7 lines

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Does this work?

fs/splice.c +11 -23
index ed2ce99..92646aa 100644
@@ -491,7 +491,7 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos,
 
 	ret = 0;
 	spliced = 0;
-	while (len) {
+	while (len && !spliced) {
 		ret = __generic_file_splice_read(in, ppos, pipe, len, flags);
 
 		if (ret < 0)
@@ -1051,15 +1051,10 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
 	sd->flags &= ~SPLICE_F_NONBLOCK;
 
 	while (len) {
-		size_t read_len, max_read_len;
-
-		/*
-		 * Do at most PIPE_BUFFERS pages worth of transfer:
-		 */
-		max_read_len = min(len, (size_t)(PIPE_BUFFERS*PAGE_SIZE));
+		size_t read_len;
 
-		ret = do_splice_to(in, &sd->pos, pipe, max_read_len, flags);
-		if (unlikely(ret < 0))
+		ret = do_splice_to(in, &sd->pos, pipe, len, flags);
+		if (unlikely(ret <= 0))
 			goto out_release;
 
 		read_len = ret;
@@ -1071,26 +1066,17 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
 		 * could get stuck data in the internal pipe:
 		 */
 		ret = actor(pipe, sd);
-		if (unlikely(ret < 0))
+		if (unlikely(ret <= 0))
 			goto out_release;
 
 		bytes += ret;
 		len -= ret;
 
-		/*
-		 * In nonblocking mode, if we got back a short read then
-		 * that was due to either an IO error or due to the
-		 * pagecache entry not being there. In the IO error case
-		 * the _next_ splice attempt will produce a clean IO error
-		 * return value (not a short read), so in both cases it's
-		 * correct to break out of the loop here:
-		 */
-		if ((flags & SPLICE_F_NONBLOCK) && (read_len < max_read_len))
-			break;
+		if (ret < read_len)
+			goto out_release;
 	}
 
 	pipe->nrbufs = pipe->curbuf = 0;
-
 	return bytes;
 
 out_release:
@@ -1152,10 +1138,12 @@ long do_splice_direct(struct file *in, loff_t *ppos, struct file *out,
 		.pos		= *ppos,
 		.u.file		= out,
 	};
-	size_t ret;
+	long ret;
 
 	ret = splice_direct_to_actor(in, &sd, direct_splice_actor);
-	*ppos = sd.pos;
+	if (ret > 0)
+		*ppos += ret;
+
 	return ret;
 }
 
Signature · 1 line
Jens Axboe
author
#2 lore raw
Quoted 22 lines · Jens Axboe
Quoted 20 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200
    sys_sendfile: switch to using ->splice_read, if available

    This patch makes sendfile prefer to use ->splice_read(), if it's
    available in the file_operations structure.
Quoted 8 lines
Quoted 5 lines

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Does this work?

Yes , it does. Thx :)

Quoted 79 lines
fs/splice.c +11 -23
index ed2ce99..92646aa 100644
@@ -491,7 +491,7 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos,
 
 	ret = 0;
 	spliced = 0;
-	while (len) {
+	while (len && !spliced) {
 		ret = __generic_file_splice_read(in, ppos, pipe, len, flags);
 
 		if (ret < 0)
@@ -1051,15 +1051,10 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
 	sd->flags &= ~SPLICE_F_NONBLOCK;
 
 	while (len) {
-		size_t read_len, max_read_len;
-
-		/*
-		 * Do at most PIPE_BUFFERS pages worth of transfer:
-		 */
-		max_read_len = min(len, (size_t)(PIPE_BUFFERS*PAGE_SIZE));
+		size_t read_len;
 
-		ret = do_splice_to(in, &sd->pos, pipe, max_read_len, flags);
-		if (unlikely(ret < 0))
+		ret = do_splice_to(in, &sd->pos, pipe, len, flags);
+		if (unlikely(ret <= 0))
 			goto out_release;
 
 		read_len = ret;
@@ -1071,26 +1066,17 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd,
 		 * could get stuck data in the internal pipe:
 		 */
 		ret = actor(pipe, sd);
-		if (unlikely(ret < 0))
+		if (unlikely(ret <= 0))
 			goto out_release;
 
 		bytes += ret;
 		len -= ret;
 
-		/*
-		 * In nonblocking mode, if we got back a short read then
-		 * that was due to either an IO error or due to the
-		 * pagecache entry not being there. In the IO error case
-		 * the _next_ splice attempt will produce a clean IO error
-		 * return value (not a short read), so in both cases it's
-		 * correct to break out of the loop here:
-		 */
-		if ((flags & SPLICE_F_NONBLOCK) && (read_len < max_read_len))
-			break;
+		if (ret < read_len)
+			goto out_release;
 	}
 
 	pipe->nrbufs = pipe->curbuf = 0;
-
 	return bytes;
 
 out_release:
@@ -1152,10 +1138,12 @@ long do_splice_direct(struct file *in, loff_t *ppos, struct file *out,
 		.pos		= *ppos,
 		.u.file		= out,
 	};
-	size_t ret;
+	long ret;
 
 	ret = splice_direct_to_actor(in, &sd, direct_splice_actor);
-	*ppos = sd.pos;
+	if (ret > 0)
+		*ppos += ret;
+
 	return ret;
 }
 
#3 lore raw
Quoted 25 lines · Gabriel C
Quoted 24 lines · Jens Axboe
Quoted 22 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Does this work?

Yes , it does. Thx :)

Great, I'll polish it a bit and get it upstream asap. Thanks for the report and quick test feedback!

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 30 lines · Jens Axboe
Quoted 29 lines · Gabriel C
Quoted 28 lines · Jens Axboe
Quoted 25 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Does this work?

Yes , it does. Thx :)

Great, I'll polish it a bit and get it upstream asap. Thanks for the report and quick test feedback!

Any chance someone could hack up a ltp testcase for this?

#5 lore raw
Quoted 34 lines · Christoph Hellwig
Quoted 33 lines · Jens Axboe
Quoted 32 lines · Gabriel C
Quoted 30 lines · Jens Axboe
Quoted 17 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

Quoted 8 lines

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Does this work?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Yes , it does. Thx :)

Great, I'll polish it a bit and get it upstream asap. Thanks for the report and quick test feedback!

Any chance someone could hack up a ltp testcase for this?

Certainly, I already created a private test case for it anyway.

Signature · 1 line
Jens Axboe
author
#6 lore raw
In reply to Jens Axboe
Quoted 33 lines · Jens Axboe
Quoted 31 lines · Gabriel C
Quoted 29 lines · Jens Axboe
Quoted 26 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

Does this work?

Yes , it does. Thx :)

Great, I'll polish it a bit and get it upstream asap. Thanks for the report and quick test feedback!

You are welcome :)

If you want me to test the final version of this patch before you push it upstream just ping me.

#7 lore raw
Quoted 45 lines · Gabriel C
Quoted 33 lines · Jens Axboe
Quoted 31 lines · Gabriel C
Quoted 29 lines · Jens Axboe
Quoted 26 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200

sys_sendfile: switch to using ->splice_read, if available

This patch makes sendfile prefer to use ->splice_read(), if it's available in the file_operations structure.

:040000 040000 b19013793692eee0f632b3703dca0bd40cea753f
ed50576c0d3f0d3ce9f1a2ac9336d1164b45f63f M fs

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

Does this work?

Yes , it does. Thx :)

Great, I'll polish it a bit and get it upstream asap. Thanks for the report and quick test feedback!

Quoted 4 lines

You are welcome :)

If you want me to test the final version of this patch before you push it upstream just ping me.

Thanks, I wrote a small test case though, so hopefully I'll be ok from now though.

Signature · 1 line
Jens Axboe
#8 lore raw
In reply to Jens Axboe
Quoted 17 lines · Jens Axboe
Quoted 16 lines · Gabriel C

Hello ,

While doing some tests with 2.6.22-git2 ( at the time head 4eb6bf6bfb580afaf1e1a1d30cba17a078530cf4 ) all my webservers stopped working.
I can't get any file using wget or whatever else , everything hangs after 1% forever.

I bisected this and here the result:

534f2aaa6ab07cd71164180bc958a7dcde41db11 is first bad commit commit 534f2aaa6ab07cd71164180bc958a7dcde41db11

Author: Jens Axboe <[email protected]>
Date:   Fri Jun 1 14:52:37 2007 +0200
    sys_sendfile: switch to using ->splice_read, if available

...

Quoted 1 line

Does this work?

<patch snipped>

Hi Jens,
I was hoping this patch would fix a different problem caused by the same merge of 'splice-2.6.23' three days ago, but it didn't.

When I run nfsd on the latest git kernel, it serves up what seems to be the wrong inodes for every fetched file. I can navigate the served directory structure with no problems, but the contents of any file I fetch are completely unrelated to the real contents.

Does this suggest a possible cause or fix?

Thanks.

#9 lore raw
In reply to Jens Axboe
Quoted 2 lines · Jens Axboe

Does this work?

Ok, so it apparently works, but:

fs/splice.c ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, · line 495

Quoted 11 lines
fs/splice.c +1 -1
index ed2ce99..92646aa 100644
@@ -491,7 +491,7 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos,
 
 	ret = 0;
 	spliced = 0;
-	while (len) {
+	while (len && !spliced) {
 		ret = __generic_file_splice_read(in, ppos, pipe, len, flags);

This hunk basically make sthe whole "while" loop pointless. You'll never go through it more than once.

So at that point, maybe the loop should be removed entirely? Or maybe this part of the patch is wrong, and the fix was in the _other_ parts?

Linus

#10 lore raw
Quoted 23 lines · Linus Torvalds
Quoted 2 lines · Jens Axboe

Does this work?

Ok, so it apparently works, but:

fs/splice.c ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, · line 495

Quoted 11 lines
fs/splice.c +1 -1
index ed2ce99..92646aa 100644
@@ -491,7 +491,7 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos,
 
 	ret = 0;
 	spliced = 0;
-	while (len) {
+	while (len && !spliced) {
 		ret = __generic_file_splice_read(in, ppos, pipe, len, flags);

This hunk basically make sthe whole "while" loop pointless. You'll never go through it more than once.

Quoted 2 lines

So at that point, maybe the loop should be removed entirely? Or maybe this part of the patch is wrong, and the fix was in the _other_ parts?

It'll likely work without that hunk, and you are right - the loop should just go now. I'll update that piece in a sub sequent patch!

Signature · 1 line
Jens Axboe
#11 lore raw
Quoted 24 lines · Jens Axboe
Quoted 23 lines · Linus Torvalds
Quoted 2 lines · Jens Axboe

Does this work?

Ok, so it apparently works, but:

fs/splice.c ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, · line 495

Quoted 11 lines
fs/splice.c +1 -1
index ed2ce99..92646aa 100644
@@ -491,7 +491,7 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos,
 
 	ret = 0;
 	spliced = 0;
-	while (len) {
+	while (len && !spliced) {
 		ret = __generic_file_splice_read(in, ppos, pipe, len, flags);

This hunk basically make sthe whole "while" loop pointless. You'll never go through it more than once.

Quoted 5 lines
Quoted 2 lines

So at that point, maybe the loop should be removed entirely? Or maybe this part of the patch is wrong, and the fix was in the _other_ parts?

It'll likely work without that hunk, and you are right - the loop should just go now. I'll update that piece in a sub sequent patch!

Oh btw, forgot to mention - that hunk isn't wrong, it's just not the fix for that particular bug. In reality it should probably have been part of a different patch dropping that loop as well... Anyway, I'll do that patch and indicate as such in the changelog for that commit.

Signature · 1 line
Jens Axboe