splice blocks indefinitely when len > 64k?
Hi,
I'm experimenting with splice and have run into some unusual behaviour.
I am using the utilities in git://brick.kernel.dk/data/git/splice.git
In splice.h, when changing SPLICE_SIZE from:
#define SPLICE_SIZE (64*1024)
to
#define SPLICE_SIZE ((64*1024)+1)
splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.
Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.
Thanks!
Daniel
While it should not simply hang, the splice size needs to be an even multiple of the page size.
Quoted 22 lines · Daniel Drake
Hi,
I'm experimenting with splice and have run into some unusual behaviour.
I am using the utilities in
git://brick.kernel.dk/data/git/splice.gitIn
splice.h, when changingSPLICE_SIZEfrom:#define
SPLICE_SIZE(64*1024)to
#define
SPLICE_SIZE((64*1024)+1)splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first
splice()call.Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.
Thanks!
Daniel
Quoted 2 lines · Phillip Susi
While it should not simply hang, the splice size needs to be an even multiple of the page size.
I don't think that is true, at least on this level. Firstly, splice-cp does this:
#define BS SPLICE_SIZE
int this_len = min((off_t) BS, sb.st_size);
int ret = splice(in_fd, NULL, pfds[1], NULL, this_len, 0);
No considerations are made regarding page size.
Secondly, the problem still exists when I increase SPLICE_SIZE to (128*1024)
Daniel
Quoted 24 lines
Quoted 22 lines · Daniel Drake
Hi,
I'm experimenting with splice and have run into some unusual behaviour.
I am using the utilities in
git://brick.kernel.dk/data/git/splice.gitIn
splice.h, when changingSPLICE_SIZEfrom:#define
SPLICE_SIZE(64*1024)to
#define
SPLICE_SIZE((64*1024)+1)splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first
splice()call.Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.
Thanks!
Daniel
Don't top post, please.
Quoted 2 lines · Phillip Susi
While it should not simply hang, the splice size needs to be an even multiple of the page size.
No, that is incorrect.
Signature · 1 line
Jens Axboe
Quoted 12 lines · Daniel Drake
Hi,
I'm experimenting with splice and have run into some unusual behaviour.
I am using the utilities in
git://brick.kernel.dk/data/git/splice.gitIn
splice.h, when changingSPLICE_SIZEfrom:#define
SPLICE_SIZE(64*1024)to
Quoted 7 lines
#define
SPLICE_SIZE((64*1024)+1)splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first
splice()call.Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.
You can't, internally splice is using a pipe which is currently confined to 16 pages. The SPLICE_SIZE define isn't a suggestion in the code, it reflects that. You could fix splice-cp to not stall on changing that, however that still doesn't change the fact that you can only move chunks of 64kb (on your arch) right now.
Signature · 1 line
Jens Axboe
Quoted 18 lines · On Mon, 30 Oct 2006 20:54:27 +0100 Jens Axboe <[email protected]> wrote:
Quoted 17 lines · Daniel Drake
Hi,
I'm experimenting with splice and have run into some unusual behaviour.
I am using the utilities in
git://brick.kernel.dk/data/git/splice.gitIn
splice.h, when changingSPLICE_SIZEfrom:#define
SPLICE_SIZE(64*1024)to
#define
SPLICE_SIZE((64*1024)+1)splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first
splice()call.
Quoted 9 lines
Quoted 2 lines
Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.
You can't, internally splice is using a pipe which is currently confined to 16 pages. The
SPLICE_SIZEdefine isn't a suggestion in the code, it reflects that. You could fix splice-cp to not stall on changing that, however that still doesn't change the fact that you can only move chunks of 64kb (on your arch) right now.
It could accept larger values but only move SPLICE_SIZE, assuming caller checked for partial completions.
Signature · 1 line
Stephen Hemminger <[email protected]>
Quoted 31 lines · Stephen Hemminger
Quoted 18 lines · On Mon, 30 Oct 2006 20:54:27 +0100 Jens Axboe <[email protected]> wrote:
Quoted 17 lines · Daniel Drake
Hi,
I'm experimenting with splice and have run into some unusual behaviour.
I am using the utilities in
git://brick.kernel.dk/data/git/splice.gitIn
splice.h, when changingSPLICE_SIZEfrom:#define
SPLICE_SIZE(64*1024)to
#define
SPLICE_SIZE((64*1024)+1)splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first
splice()call.Quoted 9 lines
Quoted 2 lines
Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.
You can't, internally splice is using a pipe which is currently confined to 16 pages. The
SPLICE_SIZEdefine isn't a suggestion in the code, it reflects that. You could fix splice-cp to not stall on changing that, however that still doesn't change the fact that you can only move chunks of 64kb (on your arch) right now.
Quoted 2 lines
It could accept larger values but only move
SPLICE_SIZE, assuming caller checked for partial completions.
(one more time, for the list)
The caller has to check for partial completions, it would be like calling read(2) or write(2) and not checking the return value if you didn't. SPLICE_SIZE is the _maximum_ amount of data that current fits in a pipe, there's no way to ensure that you will actually be able to write that amount - the pipe may not be empty when you begin writing, or someone else could be filling it as well.
The initial mail said that splice-cp doesn't work when you change the SPLICE_SIZE define, and I'm not at all surprised that this is the case. That would be like saying program foo doesn't work when you randomly change some define or variable in the source.
Signature · 1 line
Jens Axboe