Skip to contentSkip to conversation

splice blocks indefinitely when len > 64k?

DISCUSSION Daniel Drake 4 participants 7 messages Oct 30, 2006 – Oct 31, 2006

lore
Share & tools
lore
#0 lore raw

Hi,

I'm experimenting with splice and have run into some unusual behaviour.

I am using the utilities in git://brick.kernel.dk/data/git/splice.git

In splice.h, when changing SPLICE_SIZE from:

#define SPLICE_SIZE (64*1024)

to

#define SPLICE_SIZE ((64*1024)+1)

splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.

Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.

Thanks!
Daniel

#1 lore raw

While it should not simply hang, the splice size needs to be an even multiple of the page size.

Quoted 22 lines · Daniel Drake

Hi,

I'm experimenting with splice and have run into some unusual behaviour.

I am using the utilities in git://brick.kernel.dk/data/git/splice.git

In splice.h, when changing SPLICE_SIZE from:

#define SPLICE_SIZE (64*1024)

to

#define SPLICE_SIZE ((64*1024)+1)

splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.

Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.

Thanks!
Daniel

#2 lore raw
Quoted 2 lines · Phillip Susi

While it should not simply hang, the splice size needs to be an even multiple of the page size.

I don't think that is true, at least on this level. Firstly, splice-cp does this:

#define BS SPLICE_SIZE
int this_len = min((off_t) BS, sb.st_size);
int ret = splice(in_fd, NULL, pfds[1], NULL, this_len, 0);

No considerations are made regarding page size.

Secondly, the problem still exists when I increase SPLICE_SIZE to (128*1024)

Daniel

Quoted 24 lines
Quoted 22 lines · Daniel Drake

Hi,

I'm experimenting with splice and have run into some unusual behaviour.

I am using the utilities in git://brick.kernel.dk/data/git/splice.git

In splice.h, when changing SPLICE_SIZE from:

#define SPLICE_SIZE (64*1024)

to

#define SPLICE_SIZE ((64*1024)+1)

splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.

Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.

Thanks!
Daniel

#3 lore raw
In reply to Phillip Susi

Don't top post, please.

Quoted 2 lines · Phillip Susi

While it should not simply hang, the splice size needs to be an even multiple of the page size.

No, that is incorrect.

Signature · 1 line
Jens Axboe
#4 lore raw
In reply to Daniel Drake
Quoted 12 lines · Daniel Drake

Hi,

I'm experimenting with splice and have run into some unusual behaviour.

I am using the utilities in git://brick.kernel.dk/data/git/splice.git

In splice.h, when changing SPLICE_SIZE from:

#define SPLICE_SIZE (64*1024)

to

Quoted 7 lines

#define SPLICE_SIZE ((64*1024)+1)

splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.

Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.

You can't, internally splice is using a pipe which is currently confined to 16 pages. The SPLICE_SIZE define isn't a suggestion in the code, it reflects that. You could fix splice-cp to not stall on changing that, however that still doesn't change the fact that you can only move chunks of 64kb (on your arch) right now.

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 18 lines · On Mon, 30 Oct 2006 20:54:27 +0100 Jens Axboe <[email protected]> wrote:
Quoted 17 lines · Daniel Drake

Hi,

I'm experimenting with splice and have run into some unusual behaviour.

I am using the utilities in git://brick.kernel.dk/data/git/splice.git

In splice.h, when changing SPLICE_SIZE from:

#define SPLICE_SIZE (64*1024)

to

#define SPLICE_SIZE ((64*1024)+1)

splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.

Quoted 9 lines
Quoted 2 lines

Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.

You can't, internally splice is using a pipe which is currently confined to 16 pages. The SPLICE_SIZE define isn't a suggestion in the code, it reflects that. You could fix splice-cp to not stall on changing that, however that still doesn't change the fact that you can only move chunks of 64kb (on your arch) right now.

It could accept larger values but only move SPLICE_SIZE, assuming caller checked for partial completions.

Signature · 1 line
Stephen Hemminger <[email protected]>
#6 lore raw
Quoted 31 lines · Stephen Hemminger
Quoted 18 lines · On Mon, 30 Oct 2006 20:54:27 +0100 Jens Axboe <[email protected]> wrote:
Quoted 17 lines · Daniel Drake

Hi,

I'm experimenting with splice and have run into some unusual behaviour.

I am using the utilities in git://brick.kernel.dk/data/git/splice.git

In splice.h, when changing SPLICE_SIZE from:

#define SPLICE_SIZE (64*1024)

to

#define SPLICE_SIZE ((64*1024)+1)

splice-cp hangs indefinitely when copying files sized 65537 bytes or more. It hangs on the first splice() call.

Quoted 9 lines
Quoted 2 lines

Is this a bug? I'd like to be able to copy much more than 64kb on a single splice call.

You can't, internally splice is using a pipe which is currently confined to 16 pages. The SPLICE_SIZE define isn't a suggestion in the code, it reflects that. You could fix splice-cp to not stall on changing that, however that still doesn't change the fact that you can only move chunks of 64kb (on your arch) right now.

Quoted 2 lines

It could accept larger values but only move SPLICE_SIZE, assuming caller checked for partial completions.

(one more time, for the list)

The caller has to check for partial completions, it would be like calling read(2) or write(2) and not checking the return value if you didn't. SPLICE_SIZE is the _maximum_ amount of data that current fits in a pipe, there's no way to ensure that you will actually be able to write that amount - the pipe may not be empty when you begin writing, or someone else could be filling it as well.

The initial mail said that splice-cp doesn't work when you change the SPLICE_SIZE define, and I'm not at all surprised that this is the case. That would be like saying program foo doesn't work when you randomly change some define or variable in the source.

Signature · 1 line
Jens Axboe