Skip to contentSkip to conversation

splice() on two pipes

DISCUSSION Max Kellermann 5 participants 7 messages Apr 29, 2009 – Apr 30, 2009

lore
Share & tools
lore
#0 lore raw

Hi,

when I read about the splice() system call, I thought it was obvious that it could copy data between two pipes. I was surprised that this assumption is wrong, it's not possible on 2.6.29, I get EINVAL. Can anybody please explain this limitation?

Background: I want to forward data between two subprocesses, which are connected to me with a pipe().

I have attached a small test program which prints a table of supported splice operations. Here's the output on 2.6.29.1:

 in\out  pipe    sock    reg     chr
 pipe    no      yes     yes     yes
 sock    no      no      no      no
 reg     yes     no      no      no
 chr     no      no      no      no

Max

/*
* Copyright (C) 2009 Max Kellermann <[email protected]>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer. *
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the
* distribution.
*/

/*
* This tiny program prints a matrix: which file descriptor
* combinations are supported by splice()?
*/

#define _GNU_SOURCE
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>

static struct {
    const char *const name;
    int in, out;
} fds[] = {
    { .name = "pipe", },
    { .name = "sock", },
    { .name = "reg", },
    { .name = "chr", },
};
enum {
    NUM_FDS = sizeof(fds) / sizeof(fds[0]),
};

int main(int argc, char **argv)

{
    int f[2], ret;
    unsigned x, y;
    char template1[] = "/tmp/test_splice.XXXXXX";
    char template2[] = "/tmp/test_splice.XXXXXX";

    (void)argc;
    (void)argv;

    /* open two file descriptors of each kind */

    fds[0].in = pipe(f) >= 0 ? f[0] : -1;
    fds[0].out = pipe(f) >= 0 ? f[1] : -1;
    fds[1].in = socketpair(AF_UNIX, SOCK_STREAM, 0, f) >= 0 ? f[0] : -1;
    fds[1].out = socketpair(AF_UNIX, SOCK_STREAM, 0, f) >= 0 ? f[0] : -1;
    fds[2].in = mkstemp(template1);
    fds[2].out = mkstemp(template2);
    fds[3].in = open("/dev/zero", O_RDONLY);
    fds[3].out = open("/dev/null", O_WRONLY);

    /* print table header */

    printf("in\\out");
    for (x = 0; x < NUM_FDS; ++x)
        printf("\t%s", fds[x].name);
    putchar('\n');

    for (y = 0; y < NUM_FDS; ++y) {
        fputs(fds[y].name, stdout);

        for (x = 0; x < NUM_FDS; ++x) {
            putchar('\t');

            if (fds[x].out < 0 || fds[y].in < 0) {
                fputs("n/a", stdout);
                continue;
            }

            ret = splice(fds[y].in, NULL, fds[x].out, NULL, 1,
                         SPLICE_F_NONBLOCK);
            if (ret >= 0 || errno == EAGAIN || errno == EWOULDBLOCK)
                /* EAGAIN or EWOULDBLOCK means that the kernel has
                   accepted this combination, but can't move pages
                   right now */
                fputs("yes", stdout);
            else if (errno == EINVAL)
                /* the kernel doesn't support this combination */
                fputs("no", stdout);
            else if (errno == ENOSYS)
                /* splice() isn't supported at all */
                fputs("ENOSYS", stdout);
            else
                /* an unexpected error code */
                fputs("err", stdout);
        }

        putchar('\n');
    }

    unlink(template1);
    unlink(template2);
}
#1 lore raw

Max Kellermann <[email protected]> writes:

I don't think splice is about handling all possible cases, but just cases where the kernel can do better than user space. I don't think that's the case here.

Quoted 2 lines

when I read about the splice() system call, I thought it was obvious that it could copy data between two pipes.

It would be more efficient if you used fd passing to pass the fd around to the other process and let it read directly.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#2 lore raw
Quoted 3 lines · Andi Kleen

I don't think splice is about handling all possible cases, but just cases where the kernel can do better than user space. I don't think that's the case here.

If splice() is about passing pointers of a pipe buffer, what's more trivial (and natural) than passing that pointer between two pipes?

Quoted 5 lines
Quoted 2 lines

when I read about the splice() system call, I thought it was obvious that it could copy data between two pipes.

It would be more efficient if you used fd passing to pass the fd around to the other process and let it read directly.

That's not so easy in my case. The header output of the one process has to be parsed before the rest of it (or part of the rest) is going to be forwarded to the second one. My master process would lose control over the transfer. splice() looks like the perfect solution.

Max

#3 lore raw
Quoted 11 lines · Max Kellermann
Quoted 3 lines · Andi Kleen

I don't think splice is about handling all possible cases, but just cases where the kernel can do better than user space. I don't think that's the case here.

If splice() is about passing pointers of a pipe buffer, what's more trivial (and natural) than passing that pointer between two pipes?

Quoted 3 lines
Quoted 2 lines

when I read about the splice() system call, I thought it was obvious that it could copy data between two pipes.

Quoted 7 lines
Quoted 2 lines

It would be more efficient if you used fd passing to pass the fd around to the other process and let it read directly.

That's not so easy in my case. The header output of the one process has to be parsed before the rest of it (or part of the rest) is going to be forwarded to the second one. My master process would lose control over the transfer. splice() looks like the perfect solution.

indeed, that could make sense. From what I have seen in the splicing code, I think tht implementing pipe to pipe should not be *that* hard, starting from existing code (eg: net to pipe). Maybe you could try to implement it since you have the code which makes use of it ? I think it is the kind of feature which can only improve step by step based on application needs.

BTW, I like your test program. Simple and easy. I have completed it to test tcp and udp, you can find it attached.

Regards,
Willy

/*
* Copyright (C) 2009 Max Kellermann <[email protected]>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* - Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer. *
* - Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the
* distribution.
*/

/*
* This tiny program prints a matrix: which file descriptor
* combinations are supported by splice()?
*/

#define _GNU_SOURCE
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>

static struct {
    const char *const name;
    int in, out;
} fds[] = {
    { .name = "pipe", },
    { .name = "reg", },
    { .name = "chr", },
    { .name = "unix", },
    { .name = "tcp", },
    { .name = "udp", },
};
enum {
    NUM_FDS = sizeof(fds) / sizeof(fds[0]),
};

int main(int argc, char **argv)

{
    int f[2], ret;
    unsigned x, y;
    char template1[] = "/tmp/test_splice.XXXXXX";
    char template2[] = "/tmp/test_splice.XXXXXX";

    (void)argc;
    (void)argv;

    /* open two file descriptors of each kind */

    fds[0].in = pipe(f) >= 0 ? f[0] : -1;
    fds[0].out = pipe(f) >= 0 ? f[1] : -1;
    fds[1].in = mkstemp(template1);
    fds[1].out = mkstemp(template2);
    fds[2].in = open("/dev/zero", O_RDONLY);
    fds[2].out = open("/dev/null", O_WRONLY);
    fds[3].in = socketpair(AF_UNIX, SOCK_STREAM, 0, f) >= 0 ? f[0] : -1;
    fds[3].out = socketpair(AF_UNIX, SOCK_STREAM, 0, f) >= 0 ? f[0] : -1;
    fds[4].in  = socket(AF_INET, SOCK_STREAM, 0);
    fds[4].out = socket(AF_INET, SOCK_STREAM, 0);
    fds[5].in  = socket(AF_INET, SOCK_DGRAM, 0);
    fds[5].out = socket(AF_INET, SOCK_DGRAM, 0);

    /* print table header */

    printf("in\\out");
    for (x = 0; x < NUM_FDS; ++x)
        printf("\t%s", fds[x].name);
    putchar('\n');

    for (y = 0; y < NUM_FDS; ++y) {
        fputs(fds[y].name, stdout);

        for (x = 0; x < NUM_FDS; ++x) {
            putchar('\t');

            if (fds[x].out < 0 || fds[y].in < 0) {
                fputs("n/a", stdout);
                continue;
            }

            ret = splice(fds[y].in, NULL, fds[x].out, NULL, 1,
                         SPLICE_F_NONBLOCK);
            if (ret >= 0 || errno == EAGAIN || errno == EWOULDBLOCK
		|| errno == ENOTCONN)
                /* EAGAIN or EWOULDBLOCK means that the kernel has
                   accepted this combination, but can't move pages
                   right now */
                fputs("yes", stdout);
            else if (errno == EINVAL)
                /* the kernel doesn't support this combination */
                fputs("no", stdout);
            else if (errno == ENOSYS)
                /* splice() isn't supported at all */
                fputs("ENOSYS", stdout);
            else
                /* an unexpected error code */
                fputs("err", stdout);
        }

        putchar('\n');
    }

    unlink(template1);
    unlink(template2);
}
#5 lore raw
In reply to Max Kellermann
Quoted 10 lines · Max Kellermann

Hi,

when I read about the splice() system call, I thought it was obvious that it could copy data between two pipes. I was surprised that this assumption is wrong, it's not possible on 2.6.29, I get EINVAL. Can anybody please explain this limitation?

Background: I want to forward data between two subprocesses, which are connected to me with a pipe().

Quoted 8 lines

I have attached a small test program which prints a table of supported splice operations. Here's the output on 2.6.29.1:

 in\out  pipe    sock    reg     chr
 pipe    no      yes     yes     yes
 sock    no      no      no      no
 reg     yes     no      no      no
 chr     no      no      no      no

See sys_tee().

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 1 line · Jens Axboe

See sys_tee().

Hm, so I could tee() from pipe1 to pipe2, then delete data from pipe1 by splicing it to /dev/null...

Max

#4 lore raw
In reply to Willy Tarreau
Quoted 1 line · Willy Tarreau

BTW, I like your test program. Simple and easy. I have completed it to test tcp and udp, you can find it attached.

I think the shown ability to splice to a unix socket may be misleading. Can someone offer some insight to my previous post (22nd April, "splice() on unix sockets")? Does splice to a unix socket actually result in zero copy?

  $ ./test_splice
  in\out  pipe    reg     chr     unix    tcp     udp
  pipe    no      yes     yes     yes     yes     yes
  reg     yes     no      no      no      no      no
  chr     no      no      no      no      no      no
  unix    no      no      no      no      no      no
  tcp     yes     no      no      no      no      no
  udp     no      no      no      no      no      no

Thanks

Signature · 1 line
Mark