zerocopy pipe, new version

Attached is a new version of the zerocopy pipe code.

A few months ago, Hubertus Franke found a severe performance problem with my last version. Now I figured out how I can solve it:
For good pipe performance, sys_read() must try to return as much data as possible with one syscall, even if the writer writes small bits. The current code uses PIPE_WAITING_WRITERS, but that doesn't work with nonblocking IO and is not that efficient.

I added a sched_yield() into that path, and that fixed the performance degradation:
if pipe_read made progress and the user wants even more data, then call sched_yield() and give the writers a chance to write additional data. After sched_yield() returns, try again until there is either no more data, or the user request is completely fullfilled. Then return to userspace.

Now I got +50% on UP with pipeflex -c2 -r32 -w1 with 2.5.2-pre8+zerocopy, SMP kernel over 2.4.2-UP.

Unfortunately the patch has virtually no effect on 4 kB write-4 kB read, and that's the most common case. (number of context switches cut in half, but a slight performance loss on K6, probably due to cache trashing)

The only solution I see for that problem are larger kernel buffers - more data must be queued in kernel. Either on-demand (kmalloc() up to <sysctl-limit, around 512 kB> if a request for <= 4096 bytes arrives and would block. If the allocation fails, then block), or at pipe creation like in Hubertus' patch.

--
Manfred

----- Original Message -----
From: "Manfred Spraul" <[email protected]>
To: <[email protected]>
Cc: <[email protected]>
Sent: Saturday, January 05, 2002 5:27 PM
Subject: [Lse-tech] zerocopy pipe, new version

my last

possible with

that

degradation:

sched_yield() and

returns, try again until

fullfilled. Then return to userspace.

SMP kernel

and that's the most

performance loss on K6,

data must be queued

if a request for

block), or at pipe creation

We can only echo Manfred's recent finding and improvements to the zerocopy patch he has submitted. Looks like a winner to us. As you might recall we had worked from the other end namely larger pipe buffers and a below we show that a combination of zero-copy and large pipe can basically deal with all the performance issues.

We show the performance gain/loss for various benchmarks on 2-way, 1-way and UP systems.

For this test we added our new large pipe patch over Manfred Spraul's Zero copy patch and all these test were ran on 2.4.17 kernel.

Here are our initial results which shows the performance improvement of large pipe buffer + zero copy over 2.4.17 vanilla kernel. The pipe size we used for this testing was 32K (ie. 8 pages).

Grep over 50mb file
-------------------

        2-way   1-way    UP
        -----   -----    --
%imp    206.76  8.02    -1.55

LMBench
-------
%imp

TS      2-way   1-way      UP
--      -----   -----      --
2k     102.61   -5.22    -7.25
4k     136       4.01    -5.02
6k      95.98  -22.15   -26.67
8k      29.75  -31.84   -44.23
12k      5.99  -14.09   -23.41
16k      8.49  -13.21   -22.19
24k     -0.92   -3.11   -15.39
32k      3.11   -3.14   -14.75
64k     26.55    0.99    25.76
128k    22.69   55.72    -7.43

Pipeflex -c2 -t 20 -x 500 -y 0 -r #TS -w 1 -o 0 -m 0
--------
%imp

TS      2-way   1-way    UP
--      -----   -----    --
2k      1.6     1.07     0.27
4k      0.14    0       -0.41
6k      21.08   0.29    -1.21
8k      15.01  -1.25    -3.07
12k     12.1   -3.81    -5.66
16k     27.02  -4.29    -5.8
24k     48.34  -4.86    -7.47
32k     27.89  -6.34    -8.24
64k     34.74  -6.67   -11.53
128k    58.23   2.09    -2.79

Here are the results of % improvement of Large Pipe(32k) + Zero copy over Zero copy.

Grep over 50mb file
-------------------

        2-way   1-way   UP
        -----   -----   --

%imp 96.62 -3.77 -4.64

LMBench
-------

TS      2-way    1-way   UP
--      -----    -----   --
2k      26.3    -20.19   -8.54
4k      26.98   -20.56  -29.92
6k      11.69   -47.83  -58.35
8k      -5.26   -53.8   -65.39
12k    -22.11   -28.19  -32.76
16k    -22.62   -30.1   -34.15
24k    -18.88   -27.02  -29.29
32k    -20.44   -28.92  -29.95
64k     -1.38   -29.75    1.92

128k 7.3 -2.02 -18.49

Pipeflex -c2 -t 20 -x 500 -y 0 -r #TS -w 1 -o 0 -m 0
----------------------------------------------------

TS      2-way    1-way    UP
--      -----    -----    --
2k       1.33    0.27    -0.53
4k       6.1     0.14    -0.54
6k      15.38   -0.47    -1.66
8k      22.69   -1.4     -3
12k     24.1    -3.25    -4.98
16k     25.49   -3.88    -5.96
24k     41.48   -5.22    -8.19
32k     21.84   -5.52    -9.04
64k     35.35   -8.32   -14.06
128k    42.8    -8.78   -16.16

Here are the results which shows the % improvement (degradation) of large pipe when added to the zero copy patch. For that we set the pipe size to the default 4k.

Grep
----

                2-way   1-way   UP
                -----   -----   --
%overhead       -0.73   -0.97   0

LMBench
-------
% overhead

TS      2-way    1-way   UP
--      -----    -----   --
2k      -9.59   13.51    1.75
4k       5.5    -1.22  -10.11
6k      -2.44    3.93   -2.92
8k      -1.61    0.51   -0.92
12k     -1.74    0.71   -0.85
16k     -0.04    0.44   -0.46
24k     -0.15    0.37    0.05
32k     -0.39   -0.02   -0.14
64k     -0.8    -1.05    1.52

128k 1.94 -7.46 -2.96

Pipeflex -c2 -t 20 -x 500 -y 0 -r #TS -w 1 -o 0 -m 0
--------

% overhead

TS      2-way   1-way    UP
--      -----   ----     --
2k      -0.27    0.27   -0.26
4k       1.45    0      -0.14
6k       2.35   -0.09   -0.09
8k       7.96   -0.37   -0.14
12k     13.89    0.37    0.05
16k     17.66   -1.96    0
24k     12.41   -0.92   -0.66
32k     16.12   -0.11   -0.64
64k     32.97   -0.35   -0.64
128k    67.87   -0.25   -0.89

Conclusion
----------
Manfred Spraul's new zero copy patch showed a very good performance improvement for pipeflex as well as grep on both 2-way and 1-way. By adding our large pipe support, performance improvement up to 42% for pipeflex and 96% for grep benchmarks on 2-way systems. No significant different has been observed for lmbench on 1-way systems. The right way to configure the system than would be for UP and 1-way to set the buffer size by default to 4K while for the SMP larger pipe buffers should be encouraged.
Below is the large pipe patch which has to be applied over manfred's latest zero copy patch.

diff -urbN linux-2.4.17-manfred-up/fs/pipe.c linux-2.4.17-pipe-up/fs/pipe.c
@@ -28,6 +28,8 @@
  * -- Julian Bradfield 1999-06-07.
  */
 
+struct pipe_stat_t pipe_stat;
+
 /* Drop the inode semaphore and wait for a pipe event, atomically */
 void pipe_wait(struct inode * inode)
 {

@@ -158,7 +160,7 @@
                 * case. Write into the internal buffer before
                 * checking for signals/error conditions.
                 */
-               size_t j = min((size_t)PIPE_SIZE, pio->len);
+               size_t j = min((size_t)PIPE_SIZE(*inode), pio->len);
                if (PIPE_LEN(*inode)) BUG();
                if (PIPE_START(*inode)) BUG();
                if (!copy_from_user(PIPE_BASE(*inode), buf + i, j)) {

@@ -201,8 +203,8 @@
                        int offset = PIPE_START(*inode)%PIPE_BUF;
                        if (chars > count)
                                chars = count;
-                       if (chars > PIPE_SIZE-offset)
-                               chars = PIPE_SIZE-offset;
+                       if (chars > PIPE_SIZE(*inode)-offset)
+                               chars = PIPE_SIZE(*inode)-offset;
                        if (unlikely(copy_to_user(buf, pipebuf+offset,

chars))) {
                                if (!read)
                                        read = -EFAULT;
@@ -326,11 +328,11 @@
                if (PIPE_PIOLEN(*inode))
                        goto skip_int_buf;
                /* write to internal buffer - could be cyclic */
-               while(start = PIPE_LEN(*inode),chars = PIPE_SIZE - start,
chars >= min) {
+               while(start = PIPE_LEN(*inode),chars = PIPE_SIZE(*inode) -
start, chars >= min) {
                        start += PIPE_START(*inode);

- start %= PIPE_SIZE;
- if (chars > PIPE_BUF - start)
- chars = PIPE_BUF - start;
+ start %= PIPE_SIZE(*inode);
+ if (chars > PIPE_SIZE(*inode) - start)
+ chars = PIPE_SIZE(*inode) - start;
if (chars > count)
chars = count;
if (unlikely(copy_from_user(PIPE_BASE(*inode)+start,
@@ -470,7 +472,7 @@

        if (!PIPE_READERS(*inode) && !PIPE_WRITERS(*inode)) {
                struct pipe_inode_info *info = inode->i_pipe;
                inode->i_pipe = NULL;
-               free_page((unsigned long) info->base);
+               free_pages((unsigned long) info->base, info->order);
                kfree(info);
        } else {
                wake_up_interruptible(PIPE_WAIT(*inode));
@@ -604,8 +606,12 @@
 struct inode* pipe_new(struct inode* inode)
 {
        unsigned long page;

+ int pipe_order = pipe_stat.pipe_size_order;
+
+ if (pipe_order > MAX_PIPE_ORDER)
+ pipe_order = MAX_PIPE_ORDER;

- page = __get_free_page(GFP_USER);

+       page = __get_free_pages(GFP_USER, pipe_order);
        if (!page)
                return NULL;

@@ -619,10 +625,11 @@

        PIPE_START(*inode) = PIPE_LEN(*inode) = PIPE_PIOLEN(*inode) = 0;
        PIPE_READERS(*inode) = PIPE_WRITERS(*inode) = 0;
        PIPE_RCOUNTER(*inode) = PIPE_WCOUNTER(*inode) = 1;

+ PIPE_ORDER(*inode) = pipe_order;

        return inode;
 fail_page:
-       free_page(page);
+       free_pages(page, pipe_order);
        return NULL;

}

@@ -737,7 +744,7 @@
close_f12_inode_i:

        put_unused_fd(i);
 close_f12_inode:
-       free_page((unsigned long) PIPE_BASE(*inode));
+       free_pages((unsigned long) PIPE_BASE(*inode), PIPE_ORDER(*inode));
        kfree(inode->i_pipe);
        inode->i_pipe = NULL;
        iput(inode);

diff -urbN linux-2.4.17-manfred-up/include/linux/pipe_fs_i.h linux-2.4.17-pipe-up/include/linux/pipe_fs_i.h
--- linux-2.4.17-manfred-up/include/linux/pipe_fs_i.h Mon Jan 7 15:54:00
2002
+++ linux-2.4.17-pipe-up/include/linux/pipe_fs_i.h Tue Jan 8 13:42:18
2002
@@ -2,6 +2,7 @@
#define _LINUX_PIPE_FS_I_H

#define PIPEFS_MAGIC 0x50495045
+#define MAX_PIPE_ORDER 3

 struct pipe_inode_info {
        wait_queue_head_t wait;
        char *base;
@@ -13,12 +14,21 @@
        unsigned int writers;
        unsigned int r_counter;
        unsigned int w_counter;

+ unsigned int order;
};

+struct pipe_stat_t{
+ int pipe_size_order;
+ int pipe_seg_order;
+};
+
+extern struct pipe_stat_t pipe_stat;
+
/* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual
memory allocation, whereas PIPE_BUF makes atomicity guarantees. */
-#define PIPE_SIZE PAGE_SIZE
+#define PIPE_SIZE(inode) ((1 << PIPE_ORDER(inode)) * PAGE_SIZE)

+#define PIPE_ORDER(inode) ((inode).i_pipe->order)
#define PIPE_SEM(inode) (&(inode).i_sem)
#define PIPE_WAIT(inode) (&(inode).i_pipe->wait)
#define PIPE_BASE(inode) ((inode).i_pipe->base)
@@ -31,8 +41,8 @@
#define PIPE_RCOUNTER(inode) ((inode).i_pipe->r_counter)
#define PIPE_WCOUNTER(inode) ((inode).i_pipe->w_counter)

-#define PIPE_FREE(inode) (PIPE_SIZE - PIPE_LEN(inode))
-#define PIPE_END(inode) ((PIPE_START(inode) + PIPE_LEN(inode)) &
(PIPE_SIZE-1))
+#define PIPE_FREE(inode) (PIPE_SIZE(inode) - PIPE_LEN(inode))
+#define PIPE_END(inode) ((PIPE_START(inode) + PIPE_LEN(inode)) &
(PIPE_SIZE(inode)-1))

/* Drop the inode semaphore and wait for a pipe event, atomically */ void pipe_wait(struct inode * inode);
diff -urbN linux-2.4.17-manfred-up/include/linux/sysctl.h linux-2.4.17-pipe-up/include/linux/sysctl.h
--- linux-2.4.17-manfred-up/include/linux/sysctl.h Tue Jan 8 16:16:55
2002
+++ linux-2.4.17-pipe-up/include/linux/sysctl.h Tue Jan 8 16:04:01 2002 @@ -543,6 +543,7 @@
FS_LEASES=13, /* int: leases enabled */
FS_DIR_NOTIFY=14, /* int: directory notification enabled */
FS_LEASE_TIME=15, /* int: maximum time to wait for a lease
break */
+ FS_PIPE_SIZE=16, /* int: number of pages allocated for PIPE
*/
};

/* CTL_DEBUG names: */
diff -urbN linux-2.4.17-manfred-up/kernel/sysctl.c linux-2.4.17-pipe-up/kernel/sysctl.c

@@ -307,6 +307,8 @@
         sizeof(int), 0644, NULL, &proc_dointvec},
        {FS_LEASE_TIME, "lease-break-time", &lease_break_time, sizeof(int),
         0644, NULL, &proc_dointvec},
+        {FS_PIPE_SIZE, "pipe-sz", &pipe_stat, 2*sizeof(int),
+         0644, NULL, &proc_dointvec},
        {0}
 };
 

Any conclusions are incomplete without mentioning the serious (30%+) degredation on UP systems under a good chunk of the benchmarks. Such aspects of the patch make it unsuitable as is for both the mainstream and vendor kernels.

-ben

From: "Benjamin LaHaise" <[email protected]>

My patch is definitively WIP - right now I again broke the -ENOMEM and -EFAULT handling.

--
Manfred

I am aware of that, but the lse-tech posting made it sound as if things are great now since the SMP numbers improved. Please folks, remember that UP numbers are important too.

-ben

Fish.

Ben, yes you are right, the lse-posting in a second reading is misleading. As reported previously http://lse.sourceforge.net/pipe/pipe-report
UP numbers see degradations for LM-Bench, other benchmarks are OK.

This is not solved either by an integration of zero-copy with large pipes. We have however shown that the for SMP systems adding larger pipes to zero-copy pipes makes sense and for UP and 1-way sticking with a 1-page pipe does not degradate Manfred's patch.

It hence boils down to a proper parameterization of the pipe dependent on the configuration.

-- Hubertus