Skip to contentSkip to conversation

sendfile+zerocopy: fairly sexy (nothing to do with ECN)

DISCUSSION Andrew Morton 20 participants 56 messages Jan 27, 2001 – Feb 5, 2001

lore
Share & tools
lore
#0 lore raw

(Please keep netdev copied, else Jamal will grump at you, and
you don't want that).

I've whacked together some tools to measure TCP throughput with both sendfile and read/write. I've tested with and without the zerocopy patch.

The CPU load figures are very accurate: the tool uses a `subtractive' algorithm which measures how much CPU is left over by the networking code, rather than trying to measure how much CPU load the networking stuff actually takes, if you see what I mean. This accounts accurately for CPU load, interrupts, softirq processing, memory bandwidth utilisation and cache pollution.

The client is a 650 MHz PIII. The NIC is a 3CCFE575CT Cardbus 3com. It supports Scatter/Gather and hardware checksums. The NIC's interrupt is shared with the Cardbus controller, so this will impact throughput slightly.

The kernels which were tested were 2.4.1-pre10 with and without the zerocopy patch. We only look at client load (the TCP sender).

The link throughput was 11.5 mbytes/sec at all times (saturated 100baseT)

2.4.1-pre10-vanilla, using sendfile(): 29.6% CPU
2.4.1-pre10-vanilla, using read()/write(): 34.5% CPU

2.4.1-pre10+zercopy, using sendfile(): 18.2% CPU
2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

2.4.1-pre10+zercopy, using sendfile():          22.9% CPU    * hardware tx checksums disabled
2.4.1-pre10+zercopy, using read()/write():      39.2% CPU    * hardware tx checksums disabled

What can we conclude?

- sendfile is 10% cheaper than read()-then-write() on 2.4.1-pre10.

- sendfile() with the zerocopy patch is 40% cheaper than sendfile() without the zerocopy patch.

- hardware Tx checksums don't make much difference. hmm...

Bear in mind that the 3c59x driver uses a one-interrupt-per-packet algorithm. Mitigation reduces this to 0.3 ints/packet. So we're absorbing 4,500 interrupts/sec while processing 12,000 packets/sec. gigE NICs do much better mitigation than this and the relative benefits of zerocopy will be much higher for these. Hopefully Jamal can do some testing.

BTW: I could not reproduce Jamal's oops when sending large files (2 gigs with sendfile()).

The test tool is, of course, documented [ :-)/2 ]. It's at

	http://www.uow.edu.au/~andrewm/linux/#zc

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#1 lore raw
Quoted 2 lines · Andrew Morton

2.4.1-pre10-vanilla, using read()/write(): 34.5% CPU
2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

Am I right to be bothered by this?

The majority of Unix network traffic is handled with read()/write(). Why would zerocopy slow that down?

If zerocopy is simply unoptimized, that's fine for now. But if the problem is inherent in the implementation or design, that might be a problem. Any patch which incurs a signifigant slowdown on traditional networking should be contraversial.

Aaron Lehmann

please ignore me if I don't know what I'm talking about.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
In reply to Andrew Morton
Quoted 7 lines · Aaron Lehmann
Quoted 2 lines · Andrew Morton

2.4.1-pre10-vanilla, using read()/write(): 34.5% CPU
2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

Am I right to be bothered by this?

Quoted 7 lines

The majority of Unix network traffic is handled with read()/write(). Why would zerocopy slow that down?

If zerocopy is simply unoptimized, that's fine for now. But if the problem is inherent in the implementation or design, that might be a problem. Any patch which incurs a signifigant slowdown on traditional networking should be contraversial.

Good point.

The figures I quoted for the no-hw-checksum case were still using scatter/gather. That can be turned off as well and it makes it a tiny bit quicker. So the table is now:

2.4.1-pre10-vanilla, using sendfile(): 29.6% CPU
2.4.1-pre10-vanilla, using read()/write(): 34.5% CPU

2.4.1-pre10+zercopy, using sendfile(): 18.2% CPU
2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

2.4.1-pre10+zercopy, using sendfile():          22.9% CPU    * hardware tx checksums disabled
2.4.1-pre10+zercopy, using read()/write():      39.2% CPU    * hardware tx checksums disabled
2.4.1-pre10+zercopy, using sendfile():          22.4% CPU    * hardware tx checksums and SG disabled
2.4.1-pre10+zercopy, using read()/write():      38.5% CPU    * hardware tx checksums and SG disabled

But that's not relevant.

I just retested everything. Yes, the zerocopy patch does appear to decrease the efficiency of TCP on non-SG+checksumming hardware by 5% - 10%. Others need to test...

With an RTL8139/8139too. CPU is 500MHz PII Celeron, uniprocessor:

2.4.1-pre10-vanilla, using sendfile(): 43.8% CPU
2.4.1-pre10-vanilla, using read()/write(): 54.1% CPU

2.4.1-pre10+zerocopy, using sendfile(): 43.1% CPU
2.4.1-pre10+zerocopy, using read()/write(): 55.5% CPU

Note that the 8139 only gets 10.8 Mbytes/sec here. it randomly jumps up to 11.5 occasionally, but spends most of its time at 10.8. Hard to know what to make of this. Of course, if you're using an 8139 you don't care about performance anyway :)

Contradictory results. rtl8139 doesn't do Rx checksums, and I think has an extra copy in the driver, so caching effects may be obscuring things here.

I can test with eepro100 in a couple of days.

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 3 lines · Andrew Morton

The figures I quoted for the no-hw-checksum case were still using scatter/gather. That can be turned off as well and it makes it a tiny bit quicker.

Hmm. Are you sure the differences are not just noise? Unless you modified the zerocopy patch yourself, it won't use SG without checksums...

In fact it would be interesting to revert that policy and see how much SG alone helps. Probably not much, since the CPU checksumming is close to onecopy.

Ion

Signature · 6 lines
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#4 lore raw
In reply to Andrew Morton
Quoted 8 lines · Ion Badulescu
Quoted 3 lines · Andrew Morton

The figures I quoted for the no-hw-checksum case were still using scatter/gather. That can be turned off as well and it makes it a tiny bit quicker.

Hmm. Are you sure the differences are not just noise?

I don't think so. It's all pretty repeatable.

Quoted 3 lines

Unless you
modified the zerocopy patch yourself, it won't use SG without checksums...

I believe it in fact does use SG when hardware tx checksums are unavailable, but this capability wil be removed RSN because userspace can scribble on the pagecache after the checksum has been calculated, and before the frame has hit the wire.

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
In reply to Andrew Morton

The "more expensive" write/send in zerocopy is a known cost of paged SKBs. This cost may be decreased a bit with some fine tuning, but not eliminated entirely.

What do we get for this cost?

Basically, the big win is not that the card checksums the packet. We could get that for free while copying the data from userspace into the kernel pages during the sendmsg(), using the combined "copy+checksum" hand-coded assembly routines we already have.

It is in fact the better use of memory. Firstly, we use page allocations, only single ones. With linear buffers SLAB could use multiple pages which strain the memory subsystem quite a bit at times. Secondly, we fill pages with socket data precisely whereas SLAB can only get as tight packing as any general purpose memory allocator can.

This, I feel, outweighs the slight performance decrease. And I would wager a bet that the better usage of memory will result in better all around performance.

The problem with microscopic tests is that you do not see the world around the thing being focused on. I feel Andrew/Jamal's test are very valuable, but lets keep things in perspective when doing cost analysis.

Finally, please do some tests on loopback. It is usually a great way to get "pure software overhead" measurements of our TCP stack.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#6 lore raw
In reply to Andrew Morton
Quoted 4 lines · David S. Miller

The "more expensive" write/send in zerocopy is a known cost of paged SKBs. This cost may be decreased a bit with some fine tuning, but not eliminated entirely.

Can you say what causes the difference? I had a brief poke around - generic_copy_from_user() dominates in both cases of course, but nothing really stood out when comparing the zerocopy kernel's profile with non-zc.

Varying the value of MAXPGS (all the way down to 1) and also the amount of data which is sent with send() does change the throughput, but not the ratio wrt non-zc.

Quoted 14 lines

What do we get for this cost?

Basically, the big win is not that the card checksums the packet. We could get that for free while copying the data from userspace into the kernel pages during the sendmsg(), using the combined "copy+checksum" hand-coded assembly routines we already have.

It is in fact the better use of memory. Firstly, we use page allocations, only single ones. With linear buffers SLAB could use multiple pages which strain the memory subsystem quite a bit at times. Secondly, we fill pages with socket data precisely whereas SLAB can only get as tight packing as any general purpose memory allocator can.

Quoted 3 lines

This, I feel, outweighs the slight performance decrease. And I would wager a bet that the better usage of memory will result in better all around performance.

ie: inappropriate test coverage. Not surprising. What additional scenarios need to be tested? Zillions of connections?

If anyone really needs that 10% they can use the `hw_checksums=0' module parm, but SG+xsum is enabled by default - we need the testing.

Quoted 7 lines

The problem with microscopic tests is that you do not see the world around the thing being focused on. I feel Andrew/Jamal's test are very valuable, but lets keep things in perspective when doing cost analysis.

Finally, please do some tests on loopback. It is usually a great way to get "pure software overhead" measurements of our TCP stack.

Will do.

BTW: can you suggest why I'm not observing any change in NFS client efficiency?

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#7 lore raw
Quoted 2 lines · Andrew Morton

BTW: can you suggest why I'm not observing any change in NFS client efficiency?

As in "filecopy speed" or "cpu usage while copying a file"?

The current fragmentation code eliminates a full SKB allocation and data copy on the NFS file data receive path in the client, CPU has to be saved compared to pre-zerocopy or something is very wrong.

File copy speed, well you should be link speed limited as even without the zerocopy patches you ought to have enough cpu to keep it busy.

Later,
David S. Miller
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#8 lore raw
In reply to Andrew Morton
Quoted 5 lines · David S. Miller
Quoted 2 lines · Andrew Morton

BTW: can you suggest why I'm not observing any change in NFS client efficiency?

Quoted 9 lines

As in "filecopy speed" or "cpu usage while copying a file"?

The current fragmentation code eliminates a full SKB allocation and data copy on the NFS file data receive path in the client, CPU has to be saved compared to pre-zerocopy or something is very wrong.

File copy speed, well you should be link speed limited as even without the zerocopy patches you ought to have enough cpu to keep it busy.

Mount the server rsize=wsize=8192. `cp' a 102,400,000 byte file from the NFS server to /dev/null. The file is fully cached on the server. unmount and remount the server between runs to eliminate client caching. The copy takes 8.654 seconds. That's 11.8 megabytes/sec.

Client is 2.4.1-vanilla:                   29.8% CPU
Client is 2.4.1-zc:                        28.2% CPU

Client is 2.4.1-zc, non-SG+xsum NIC: 27.7% CPU

So I was mistaken - there is an improvement. (A 2% CPU change is easily measurable with this setup).

It may be a little better than this - cyclesoak I think will underestimate the benefit of saving on memory traffic. It only generates 10,000 cacheline writebacks per second per CPU. But winding it up to 80,000 doesn't affect the above figures much at all.

The box has 130 mbyte/sec memory write bandwidth, so saving
a copy should save 10% of this. (Wanders away, scratching
head...)

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#9 lore raw
Quoted 3 lines · Andrew Morton

The box has 130 mbyte/sec memory write bandwidth, so saving
a copy should save 10% of this. (Wanders away, scratching
head...)

Are you sure your measurment program will account properly for all system cycles spent in softnet processing? This is where the bulk of the cpu cycle savings will occur.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#10 lore raw
In reply to Andrew Morton
Quoted 10 lines · David S. Miller
Quoted 3 lines · Andrew Morton

The box has 130 mbyte/sec memory write bandwidth, so saving
a copy should save 10% of this. (Wanders away, scratching
head...)

Are you sure your measurment program will account properly for all system cycles spent in softnet processing? This is where the bulk of the cpu cycle savings will occur.

It tries to. It runs n_cpus instances of this:

static void busyloop(int instance)

{
        int idx;

        for ( ; ; ) {
                for (idx = 0; idx < busyloop_size; idx++) {
                        int thumb;

                        busyloop_buf[idx]++;                    /* Dirty a cacheline */
                        for (thumb = 0; thumb < 200; thumb++)
                                ;                               /* twiddle */
                        busyloop_progress[instance * CACHE_LINE_SIZE]++;
                }
        }
}

At minimum priority.

And it measures how much these threads are slowed
down, wrt an unloaded system. So interrupt work
is definitely accounted for.

It needs work. It should walk the buffer in cacheline-sized strides, should have tunable read-versus-write ratios, should be scheduled with `idle' priority, should be bondable to CPUs and should create PCI traffic. That means a in-kernel implementation.

But tweaking this thing thus far has made only very small differences in output.

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#11 lore raw
In reply to Andrew Morton

On Wed, Jan 31, 2001 at 01:58:44AM +1100, Andrew Morton wrote:

Mount the server rsize=wsize=8192. `cp' a 102,400,000 byte file from the NFS server to /dev/null. The file is fully cached on the server. unmount and remount the server between runs to eliminate client caching. The copy takes 8.654 seconds. That's 11.8 megabytes/sec.

What server are you using here? Using NetApp filers I don't see anything like this, probably only 8.5MB/s at most and this number is fairly noisy.

--cw
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#12 lore raw
Quoted 3 lines · Chris Wedgwood

What server are you using here? Using NetApp filers I don't see anything like this, probably only 8.5MB/s at most and this number is fairly noisy.

8.5MB/sec sounds like half-duplex 100baseT. Positive you are running at full duplex all the way to the netapp, and if so how many switches sit between you and this netapp?

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#13 lore raw
In reply to Andrew Morton

On Tue, Jan 30, 2001 at 02:17:57PM -0800, David S. Miller wrote:

8.5MB/sec sounds like half-duplex 100baseT.

No; I'm 100% its FD; HD gives 40k/sec TCP because of collisions and such like.

Positive you are running at full duplex all the way to the netapp, and if so how many switches sit between you and this netapp?

It's FD all the way (we hardwire everything to 100-FD and never trust auto-negotiate); I see no errors or such like anywhere.

There are ... <pause> ... 3 switches between four switches in between, mostly linked via GE. I'm not sure if latency might be an issue here, is it was critical I can imagine 10 km of glass might be a problem but it's not _that_ far...

--cw

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#14 lore raw
Quoted 4 lines · Chris Wedgwood

There are ... <pause> ... 3 switches between four switches in between, mostly linked via GE. I'm not sure if latency might be an issue here, is it was critical I can imagine 10 km of glass might be a problem but it's not _that_ far...

Other than this, I don't know what to postulate. Really, most reports and my own experimentation (directly connected Linux knfsd to 2.4.x nfs client) supports the fact that our client can saturate 100baseT rather fully.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#15 lore raw
In reply to Andrew Morton
Quoted 4 lines · David S. Miller

...
Finally, please do some tests on loopback. It is usually a great way to get "pure software overhead" measurements of our TCP stack.

Here we are. TCP and NFS/UDP over lo.

Machine is a dual-PII. I didn't bother running CPU utilisation testing while benchmarking loopback, although this may be of some interest for SMP. I just looked at the throughput.

Machine is a dual 500MHz PII (again). Memory read bandwidth is 320 meg/sec. Write b/w is 130 meg/sec. The working set is 60 ~300k files, everything cached. We run the following tests:

1: sendfile() to localhost, sender and receiver pinned to
separate CPUs

2: sendfile() to localhost, sender and receiver pinned to
the same CPU

3: sendfile() to localhost, no explicit pinning.

4, 5, 6: same as above, except we use send() in 8kbyte
chunks.

Repeat with and without zerocopy patch 2.4.1-2.

The receiver reads 64k hunks and throws them away. sendfile() sends the entire file.

Also, do an NFS mount of localhost, rsize=wsize=8192, see how long it takes to cp' a 100 meg file from the "server" to /dev/null. The file is cached on the "server". Do this for the three pinning cases as well - all the NFS kernel processes were pinned as a group and cp' was the other group.

sendfile() send(8k) NFS
Mbyte/s Mbyte/s Mbyte/s

No explicit bonding

  2.4.1:                          66600        70000     25600
  2.4.1-zc:                      208000        69000     25000

Bond client and server to separate CPUs

  2.4.1:                          66700        68000     27800
  2.4.1-zc:                      213047        66000     25700

Bond client and server to same CPU:

  2.4.1:                          56000        57000     23300
  2.4.1-zc:                      176000        55000     22100

Much the same story. Big increase in sendfile() efficiency, small drop in send() and NFS unchanged.

The relative increase in sendfile() efficiency is much higher than with a real NIC, presumably because we've factored out the constant (and large) cost of the device driver.

All the bits and pieces to reproduce this are at

	http://www.uow.edu.au/~andrewm/linux/#zc

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#16 lore raw

I have been watching this thread with interest for a while now, but am wondering about the real-world use of this, given the performance penalty for write()

As I see it there are two basic cases you are saying this will help in.

1. webservers

2. other fileservers

I also freely admit that I don't know a lot about sendfile() so it may have some capability that makes my concerns meaningless, if so please let me know.

1a. for webservers that server static content (and can therefor use sendfile) I don't see this as significant becouse as your tests have been showing, even a modest machine can saturate your network (unless you are useing gigE at which time it takes a skightly larger machine)

1b. for webservers that are not primarily serving static content, they have to use write() for the output from cgi's, etc and therefor pay the performance penalty without being able to use sendfile() much to get the advantages. These machines are the ones that really need the performance as the cgi's take a significant amount of your cpu.

2. for other fileservers sendfile() sounds like it would be useful if the client is reading the entire file, but what about the cases where the client is reading part of the file, or is writing to the file. In both of these cases it seems that the fileserver is back to the write() penalty. does anyone have stats on the types of requests that fileservers are being asked for?

David Lang

Quoted 77 lines · Andrew Morton

Date: Fri, 02 Feb 2001 21:12:50 +1100
From: Andrew Morton <[email protected]>
To: David S. Miller <[email protected]>
Cc: lkml <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 4 lines · David S. Miller

...
Finally, please do some tests on loopback. It is usually a great way to get "pure software overhead" measurements of our TCP stack.

Here we are. TCP and NFS/UDP over lo.

Machine is a dual-PII. I didn't bother running CPU utilisation testing while benchmarking loopback, although this may be of some interest for SMP. I just looked at the throughput.

Machine is a dual 500MHz PII (again). Memory read bandwidth is 320 meg/sec. Write b/w is 130 meg/sec. The working set is 60 ~300k files, everything cached. We run the following tests:

1: sendfile() to localhost, sender and receiver pinned to
separate CPUs

2: sendfile() to localhost, sender and receiver pinned to
the same CPU

3: sendfile() to localhost, no explicit pinning.

4, 5, 6: same as above, except we use send() in 8kbyte
chunks.

Repeat with and without zerocopy patch 2.4.1-2.

The receiver reads 64k hunks and throws them away. sendfile() sends the entire file.

Also, do an NFS mount of localhost, rsize=wsize=8192, see how long it takes to cp' a 100 meg file from the "server" to /dev/null. The file is cached on the "server". Do this for the three pinning cases as well - all the NFS kernel processes were pinned as a group and cp' was the other group.

sendfile() send(8k) NFS
Mbyte/s Mbyte/s Mbyte/s

No explicit bonding

  2.4.1:                          66600        70000     25600
  2.4.1-zc:                      208000        69000     25000

Bond client and server to separate CPUs

  2.4.1:                          66700        68000     27800
  2.4.1-zc:                      213047        66000     25700

Bond client and server to same CPU:

  2.4.1:                          56000        57000     23300
  2.4.1-zc:                      176000        55000     22100

Much the same story. Big increase in sendfile() efficiency, small drop in send() and NFS unchanged.

The relative increase in sendfile() efficiency is much higher than with a real NIC, presumably because we've factored out the constant (and large) cost of the device driver.

All the bits and pieces to reproduce this are at

	http://www.uow.edu.au/~andrewm/linux/#zc
(quoted diff) +0 -2
-
-
Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#17 lore raw
Quoted 4 lines · David Lang

1a. for webservers that server static content (and can therefor use sendfile) I don't see this as significant becouse as your tests have been showing, even a modest machine can saturate your network (unless you are useing gigE at which time it takes a skightly larger machine)

Start using more than one interface, then it begins to become interesting.

Quoted 5 lines

1b. for webservers that are not primarily serving static content, they have to use write() for the output from cgi's, etc and therefor pay the performance penalty without being able to use sendfile() much to get the advantages. These machines are the ones that really need the performance as the cgi's take a significant amount of your cpu.

CGI's can be cached btw if the implementation is clever (f.e. CGI tells the web server that if the file used as input to the CGI does not change then the output from the CGI will not change, meaning CGI output is based solely on input, therefore CGI output can be cached by the web server).

Quoted 6 lines

2. for other fileservers sendfile() sounds like it would be useful if the client is reading the entire file, but what about the cases where the client is reading part of the file, or is writing to the file. In both of these cases it seems that the fileserver is back to the write() penalty. does anyone have stats on the types of requests that fileservers are being asked for?

It helps no matter what part of the file the client reads.

sendfile() can be used on an arbitrary offset+len portion of a file, it is not limited to just sending an entire fire.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#18 lore raw

Thanks, that info on sendfile makes sense for the fileserver situation. for webservers we will have to see (many/most CGI's look at stuff from the client so I still have doubts as to how much use cacheing will be)

David Lang

Quoted 36 lines · David S. Miller

Date: Fri, 2 Feb 2001 14:46:07 -0800 (PST)
From: David S. Miller <[email protected]>
To: David Lang <[email protected]>
Cc: Andrew Morton <[email protected]>, lkml <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 4 lines · David Lang

1a. for webservers that server static content (and can therefor use sendfile) I don't see this as significant becouse as your tests have been showing, even a modest machine can saturate your network (unless you are useing gigE at which time it takes a skightly larger machine)

Start using more than one interface, then it begins to become interesting.

Quoted 5 lines

1b. for webservers that are not primarily serving static content, they have to use write() for the output from cgi's, etc and therefor pay the performance penalty without being able to use sendfile() much to get the advantages. These machines are the ones that really need the performance as the cgi's take a significant amount of your cpu.

CGI's can be cached btw if the implementation is clever (f.e. CGI tells the web server that if the file used as input to the CGI does not change then the output from the CGI will not change, meaning CGI output is based solely on input, therefore CGI output can be cached by the web server).

Quoted 6 lines

2. for other fileservers sendfile() sounds like it would be useful if the client is reading the entire file, but what about the cases where the client is reading part of the file, or is writing to the file. In both of these cases it seems that the fileserver is back to the write() penalty. does anyone have stats on the types of requests that fileservers are being asked for?

Quoted 9 lines

It helps no matter what part of the file the client reads.

sendfile() can be used on an arbitrary offset+len portion of a file, it is not limited to just sending an entire fire.

Later,
David S. Miller
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#19 lore raw
Quoted 3 lines · David Lang

Thanks, that info on sendfile makes sense for the fileserver situation. for webservers we will have to see (many/most CGI's look at stuff from the client so I still have doubts as to how much use cacheing will be)

Also note that the decreased CPU utilization resulting from zerocopy sendfile leaves more CPU available for CGI execution.

This was a point I forgot to make.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#20 lore raw

right, assuming that there is enough sendfile() benifit to overcome the write() penalty from the stuff that can't be cached or sent from a file.

my question was basicly are there enough places where sendfile would actually be used to make it a net gain.

David Lang

Quoted 13 lines · David S. Miller

Date: Fri, 2 Feb 2001 15:09:13 -0800 (PST)
From: David S. Miller <[email protected]>
To: David Lang <[email protected]>
Cc: Andrew Morton <[email protected]>, lkml <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 3 lines · David Lang

Thanks, that info on sendfile makes sense for the fileserver situation. for webservers we will have to see (many/most CGI's look at stuff from the client so I still have doubts as to how much use cacheing will be)

Quoted 9 lines

Also note that the decreased CPU utilization resulting from zerocopy sendfile leaves more CPU available for CGI execution.

This was a point I forgot to make.

Later,
David S. Miller
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#21 lore raw

Let's see.... all the work being done for clustering would definitely benefit... all the static images on your webserver--and static images makes up most of the bandwidth from web servers (images, activeX controls, java apps, sound clips...)... NFS servers, Samba servers (both of which are used more than you may think)... email servers...

Once Real Networks patches their Realserver to use sendfile (which shouldn't bee all that hard), then that would help too....

I think that sendfile can be used in a LOT of applications, and the only ones that wouldn't benefit are mostly low-bandwidth anyway (CGI apps almost always return either a small html file or a small image file, then there's telnet and other interactive utilities...).

Most applications that use a lot of bandwidth (and thus a lot of CPU time sending the packets) are capable of being patched to use sendfile.

Quoted 33 lines · David Lang

right, assuming that there is enough sendfile() benifit to overcome the write() penalty from the stuff that can't be cached or sent from a file.

my question was basicly are there enough places where sendfile would actually be used to make it a net gain.

David Lang

Quoted 22 lines · David S. Miller

Date: Fri, 2 Feb 2001 15:09:13 -0800 (PST)
From: David S. Miller <[email protected]>
To: David Lang <[email protected]>
Cc: Andrew Morton <[email protected]>, lkml <[email protected]>,
"[email protected]" <[email protected]>
Subject: Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 3 lines · David Lang

Thanks, that info on sendfile makes sense for the fileserver situation. for webservers we will have to see (many/most CGI's look at stuff from the client so I still have doubts as to how much use cacheing will be)

Also note that the decreased CPU utilization resulting from zerocopy sendfile leaves more CPU available for CGI execution.

This was a point I forgot to make.

Later,
David S. Miller
[email protected]

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#22 lore raw
In reply to David Lang
Quoted 5 lines · David Lang

right, assuming that there is enough sendfile() benifit to overcome the write() penalty from the stuff that can't be cached or sent from a file.

my question was basicly are there enough places where sendfile would actually be used to make it a net gain.

There are non-performance issues as well (really, all of these points have been mentioned in this thread btw). One is that since paged SKBs use only single-order page allocations, the memory allocation subsystem is stressed less than the current scheme where SLAB allocates multi-order pages to satisfy allocations of linear SKB data buffers.

This has consequences and benefits system wide.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#23 lore raw
In reply to David Lang
Quoted 3 lines · David Lang

Thanks, that info on sendfile makes sense for the fileserver situation. for webservers we will have to see (many/most CGI's look at stuff from the client so I still have doubts as to how much use cacheing will be)

CGI performance isn't directly affected by this - the whole point is to reduce the "cost" of handling static requests to zero (at least, as close as possible) leaving as much CPU as possible for the CGI to use.

So sendfile won't help your CGI directly - it will just give your CGI more resources to work with.

James.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#24 lore raw
In reply to Andrew Morton
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Andrew Morton <[email protected]> writes:

> Much the same story. Big increase in sendfile() efficiency, > small drop in send() and NFS unchanged.

This is normal. The server doesn't do zero copy reads, but instead copies from the page cache into an NFS-specific buffer using file.f_op->read(). Alexey and Dave's changes are therefore unlikely to register on NFS performance (other than on CPU use as has been mentioned before) until we implement a sendfile-like scheme for knfsd over TCP.
I've been wanting to start doing that (and also to finish the client conversion to use the TCP zero-copy), but I'm pretty pressed for time at the moment.

Cheers,
Trond
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#25 lore raw
In reply to Andrew Morton
Quoted 10 lines · Andrew Morton

The client is a 650 MHz PIII. The NIC is a 3CCFE575CT Cardbus 3com. It supports Scatter/Gather and hardware checksums. The NIC's interrupt is shared with the Cardbus controller, so this will impact throughput slightly.

The kernels which were tested were 2.4.1-pre10 with and without the zerocopy patch. We only look at client load (the TCP sender).

The link throughput was 11.5 mbytes/sec at all times (saturated 100baseT)

Quoted 8 lines

2.4.1-pre10-vanilla, using sendfile(): 29.6% CPU
2.4.1-pre10-vanilla, using read()/write(): 34.5% CPU

2.4.1-pre10+zercopy, using sendfile(): 18.2% CPU
2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

2.4.1-pre10+zercopy, using sendfile():          22.9% CPU    * hardware tx checksums disabled
2.4.1-pre10+zercopy, using read()/write():      39.2% CPU    * hardware tx checksums disabled

750MHz PIII, Adaptec Starfire NIC, driver modified to use hardware sg+csum (both Tx/Rx), and Intel i82559 (eepro100), no hardware csum support, vanilla driver.

The box has 512MB of RAM, and I'm using a 100MB file, so it's entirely cached.

starfire:
2.4.1-pre10+zerocopy, using sendfile(): 9.6% CPU
2.4.1-pre10+zerocopy, using read()/write(): 18.3%-29.6% CPU * why so much variance?

2.4.1-pre10+zerocopy, using sendfile(): 17.4% CPU * hardware csum disabled
2.4.1-pre10+zerocopy, using read()/write(): 16.5%-26.8% CPU * idem, again why so much variance?

2.4.1-pre10-vanilla, using sendfile(): 16.5% CPU
2.4.1-pre10-vanilla, using read()/write(): 14.5%-24.5% CPU * high variance again

eepro100:
2.4.1-pre10+zerocopy, using sendfile(): 16.0% CPU
2.4.1-pre10+zerocopy, using read()/write(): 15.0%-24.5% CPU * why so much variance?

2.4.1-pre10-vanilla, using sendfile(): 16.7% CPU
2.4.1-pre10-vanilla, using read()/write(): 14.5%-24.6% CPU * high variance again

The read+write case is really weird. I'm getting results like this:

CPU load: 27.9491
CPU load: 25.4763
CPU load: 15.8544
CPU load: 25.455
CPU load: 25.2072
CPU load: 15.8677
CPU load: 25.4896
CPU load: 25.2791
CPU load: 15.8837

i.e. 2 slow, 1 fast, 2 slow, 1 fast, and so on so forth.

Quoted 3 lines

What can we conclude?

- sendfile is 10% cheaper than read()-then-write() on 2.4.1-pre10.

Hard to tell, with such inconclusive results...

Quoted 2 lines

- sendfile() with the zerocopy patch is 40% cheaper than sendfile() without the zerocopy patch.

Indeed. Close to 50% in fact.

Quoted 1 line

- hardware Tx checksums don't make much difference. hmm...

Actually it makes all the difference in the world for the starfire. Interesting...

Quoted 6 lines

Bear in mind that the 3c59x driver uses a one-interrupt-per-packet algorithm. Mitigation reduces this to 0.3 ints/packet. So we're absorbing 4,500 interrupts/sec while processing 12,000 packets/sec. gigE NICs do much better mitigation than this and the relative benefits of zerocopy will be much higher for these. Hopefully Jamal can do some testing.

Hmm.. the starfire also has quite advanced interrupt mitigation, but I have not played with it. Maybe tomorrow. So these results are with one-interrupt-per-packet.

P.S. The starfire still doesn't like tinygrams (skb's with 1-byte fragments). Fortunately your test program doesn't seem to generate them. :-)

Ion

Signature · 6 lines
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#26 lore raw
Quoted 11 lines · Ion Badulescu

750MHz PIII, Adaptec Starfire NIC, driver modified to use hardware sg+csum (both Tx/Rx), and Intel i82559 (eepro100), no hardware csum support, vanilla driver.

The box has 512MB of RAM, and I'm using a 100MB file, so it's entirely cached.

starfire:
2.4.1-pre10+zerocopy, using sendfile(): 9.6% CPU
2.4.1-pre10+zerocopy, using read()/write(): 18.3%-29.6% CPU * why so much variance?

What are your throughput numbers?

Could you also, please, test using:

http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz

post both sender and receiver data. Repeat each test about 5 times.

cheers,
jamal

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#27 lore raw
Quoted 6 lines · jamal
Quoted 4 lines

starfire:
2.4.1-pre10+zerocopy, using sendfile(): 9.6% CPU
2.4.1-pre10+zerocopy, using read()/write(): 18.3%-29.6% CPU * why so much variance?

What are your throughput numbers?

11.5kBps, quite consistently.

BTW, Andrew's new tool (with 8k reads/writes) has shown the load in the read/write case to be essentially the lower margin of the intervals I got in the first mail.

Quoted 6 lines

Could you also, please, test using:

http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz

post both sender and receiver data. Repeat each test about 5 times.

I've tried it, but I'm not really sure what I can report. ttcp's measurements are clearly misleading, so I used Andrew's cyclesoak instead. The numbers are (with 2.4.1-pre10+zerocopy):

[starfire, hw csum & sg enabled]
sending with sendfile: 10.0-10.2%
sending with send/write: 13.5-13.7%
receiving: 20.0-20.2%

[starfire, hw csum & sg disabled]
sending with sendfile: 18.1-18.3%
sending with send/write: 13.9-14.1%
receiving: 24.3-24.5%

[eepro100, i82559, no hw fancies]
sending with sendfile: 16.2-16.4%
sending with send/write: 12.0-12.2%
receiving: 21.5-21.7%

Same tests, this time with 2.4.1-pre10 vanilla:

[starfire]
sending with sendfile: 18.1-18.3%
sending with send/write: 12.5-12.7%
receiving: 23.0-23.1%

[eepro100, i82559]
sending with sendfile: 16.7-16.9%
sending with send/write: 12.0-12.2%
receiving: 20.8-20.9%

Ion

Signature · 7 lines
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#28 lore raw
Quoted 1 line · Ion Badulescu

11.5kBps, quite consistently.

This gige card is really sick. Are you sure? Please double check.

Quoted 3 lines

I've tried it, but I'm not really sure what I can report. ttcp's measurements are clearly misleading, so I used Andrew's cyclesoak instead.

The ttcp CPU (times()) measurements are misleading. In particular when doing sendfile. All they say is how much time ttcp spent in kernel space vs user space. So all CPU measurement i have posted in the past should be considered bogus. It is interesting to note, however, that the trend reported by ttcp's CPU measurements as well as Andrew (and yourself) are similar;->
But the point is: CPU is not the only measure that is of interest. Throughput is definetly one of those that is of extreme importance. 100Mbps is not exciting. You seem to have gigE. I think your 11KB looks suspiciously wrong. Can you double check please?

cheers,
jamal

PS:- another important parameter is latency, but that might not be as important in file serving (maybe in short file tranfers ala http).

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#29 lore raw
Quoted 3 lines · jamal
Quoted 1 line

11.5kBps, quite consistently.

This gige card is really sick. Are you sure? Please double check.

Umm.. the starfire chipset is 100Mbit only. So 11.5MBps (sorry, that was a typo, it's mega not kilo) is really all I'd expect out of it.

Ion

Signature · 7 lines
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#30 lore raw

Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 9 lines · Ion Badulescu
Quoted 3 lines · jamal
Quoted 1 line

11.5kBps, quite consistently.

This gige card is really sick. Are you sure? Please double check.

Umm.. the starfire chipset is 100Mbit only. So 11.5MBps (sorry, that was a typo, it's mega not kilo) is really all I'd expect out of it.

not good.

So far all the tests have been around CPU. The general trend seems to be:
- sendfile + ZC good for CPU
- write() + ZC not good for CPU
(i might have forgotten something from Andrew's results). This happens (even with my bogus cpu measure) to be similar. That seems to be explainable.

** I reported that there was also an oddity in throughput values, unfortunately since no one (other than me) seems to have access to a gige cards in the ZC list, nobody can confirm or disprove what i posted. Here again as a reminder:

Kernel | tput | sender-CPU | receiver-CPU |
-------------------------------------------------

2.4.0-pre3 | 99MB/s |   87%      |  23%         |
NSF        |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 86MB/s |   100%     |  17%         |
SF         |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 66.2   |   60%      |  11%         |
+ZC        | MB/s   |            |              |
-------------------------------------------------
2.4.0-pre3 | 68     |   8%       |  8%          |
+ZC  SF    | MB/s   |            |              |

-------------------------------------------------

Just ignore the CPU readings, focus on throughput. And could someone plese post results?

cheers,
jamal

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#31 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 8 lines · jamal

Kernel | tput | sender-CPU | receiver-CPU |
-------------------------------------------------

2.4.0-pre3 | 99MB/s |   87%      |  23%         |
NSF        |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 68     |   8%       |  8%          |
+ZC  SF    | MB/s   |            |              |

-------------------------------------------------

isnt the CPU utilization difference amazing? :-)

a couple of questions:

- is this UDP or TCP based? (UDP i guess)

- what wsize/rsize are you using? How do these requests look like on the network, ie. are they suffieciently MTU-sized?

- what happens if you run multiple instances of the testcode, does it saturate bandwidth (or CPU)?

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#32 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 12 lines · Ingo Molnar
Quoted 8 lines · jamal

Kernel | tput | sender-CPU | receiver-CPU |
-------------------------------------------------

2.4.0-pre3 | 99MB/s |   87%      |  23%         |
NSF        |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 68     |   8%       |  8%          |
+ZC  SF    | MB/s   |            |              |

-------------------------------------------------

Quoted 2 lines

isnt the CPU utilization difference amazing? :-)

With a caveat, sadly ;-> ttcp uses times() system call (or a diff of times() one at the beggining and another at the end). So the cpu measurements are not reflective.

Quoted 4 lines

a couple of questions:

- is this UDP or TCP based? (UDP i guess)

TCP

Quoted 2 lines

- what wsize/rsize are you using? How do these requests look like on the network, ie. are they suffieciently MTU-sized?

yes. writes vary from 8K->64K but not much difference over the long period of time.

Quoted 3 lines

- what happens if you run multiple instances of the testcode, does it saturate bandwidth (or CPU)?

This is something of great interest. I havent tried it. I should. I suspect this would be where the value of the ZC changes will become evident.

cheers,
jamal

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#33 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 3 lines · jamal
Quoted 2 lines

- is this UDP or TCP based? (UDP i guess)

TCP

well then i'd suggest to do:

	echo 100000 100000 100000 > /proc/sys/net/ipv4/tcp_wmem

does this make any difference?

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#34 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 5 lines · Ingo Molnar
Quoted 2 lines · jamal
Quoted 2 lines

- is this UDP or TCP based? (UDP i guess)

Quoted 7 lines
Quoted 1 line

TCP

well then i'd suggest to do:

	echo 100000 100000 100000 > /proc/sys/net/ipv4/tcp_wmem

does this make any difference?

According to my notes, i dont see this.
however, 262144 into /proc/sys/net/core/*mem_max/default.

I have access to my h/ware this weekend. Hopefully i should get something better than ttcp to use.

cheers,
jamal

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#35 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

In reply to jamal
Quoted 5 lines · Ingo Molnar
Quoted 2 lines · jamal
Quoted 2 lines

- is this UDP or TCP based? (UDP i guess)

Quoted 7 lines
Quoted 1 line

TCP

well then i'd suggest to do:

	echo 100000 100000 100000 > /proc/sys/net/ipv4/tcp_wmem

does this make any difference?

For the last week I've been benchmarking Linux network and I/O on a couple of machines with 3c985 gigabit cards and some other stuff (see below). One of the things I tried yesterday was a beta test version of a secure ftpd written by Chris Evans which happens to use sendfile() making it a convenient extra benchmark. I'd already put net.core.{r,w}mem_max up to 262144 for the sake of gensink and other benchmarks which raise SO_{SND,RCV}BUF. I hadn't however, tried raising tcp_wmem as per your suggestion above.

Currently the systems are linked back to back with fibre with jumbo frames (MTU 9000) on and running pure kernel 2.4.1. I transferred a 300 MByte file repeatedly from the server to the client with an ftp "get" client-side. The file will have been completely in page cache on the server (both machines have 512MB RAM) and was written to /dev/null on the client side. (Yes, I checked the client was doing ordinary read/write and not throwing it away).

Without the raised tcp_wmem setting I was getting 81 MByte/s. With tcp_wmem set as above I got 86 MByte/s. Nice increase. Any other setting I can tweak apart from {r,w}mem_max and tcp_{w,r}mem? The CPU on the client (350 MHz PII) is the bottleneck: gensink4 maxes out at 69 Mbyte/s pulling TCP from the server and 94 Mbyte/s pushing. (The other system, 733 MHz PIII pushes >100MByte/s UDP with ttcp but the client drops most of it).

I'll be following up Dave Miller's "please benchmark zerocopy" request when I've got some more numbers written down since I've only just put the zerocopy patch in and haven't rebooted yet.

If anyone wants any other specific benchmarks done (I/O or network) I may get some time to do them: the PIII system has an 8-port Escalade card with 8 x 46GB disks (117 MByte/s block writes as measured by Bonnie on a RAID1/0 mixed RAIDset) and there are also four dual-port eepro fast ethernet cards, a Cisco 8-port 3508G gigabit switch and a 24-port 3524 fast ethernet switch (gigastack linked to the 3508G). I'm benchmarking and looking into the possibility of a DIY NAS or SAN-type thing.

--Malcolm

Signature · 7 lines
Malcolm Beattie <[email protected]>
Unix Systems Programmer
Oxford University Computing Services
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#36 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN)

Quoted 7 lines · Malcolm Beattie

Without the raised tcp_wmem setting I was getting 81 MByte/s. With tcp_wmem set as above I got 86 MByte/s. Nice increase. Any other setting I can tweak apart from {r,w}mem_max and tcp_{w,r}mem? The CPU on the client (350 MHz PII) is the bottleneck: gensink4 maxes out at 69 Mbyte/s pulling TCP from the server and 94 Mbyte/s pushing. (The other system, 733 MHz PIII pushes >100MByte/s UDP with ttcp but the client drops most of it).

you can speed up the client significantly by using the MSG_TRUNC option ('truncate message'). It will zap incoming data without copying it into user-space. (you can use this for the 'bulk transfer' part - the initial protocol handling code needs to see the actual data.) This way you should be able to saturate the server even more.

Ingo

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#37 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to dowith ECN)

In reply to jamal
Quoted 18 lines

** I reported that there was also an oddity in throughput values, unfortunately since no one (other than me) seems to have access to a gige cards in the ZC list, nobody can confirm or disprove what i posted. Here again as a reminder:

Kernel | tput | sender-CPU | receiver-CPU |
-------------------------------------------------

2.4.0-pre3 | 99MB/s |   87%      |  23%         |
NSF        |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 86MB/s |   100%     |  17%         |
SF         |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 66.2   |   60%      |  11%         |
+ZC        | MB/s   |            |              |
-------------------------------------------------
2.4.0-pre3 | 68     |   8%       |  8%          |
+ZC  SF    | MB/s   |            |              |
Quoted 4 lines

-------------------------------------------------

Just ignore the CPU readings, focus on throughput. And could someone plese post results?

In the spirit of the socratic method :)

Is your gige card based on Alteon?

How does ZC/SG change the nature of the packets presented to the NIC?

How well does the NIC do with that changed nature?

rick jones

sometimes, performance tuning is like squeezing a balloon. one part gets smaller, but then you start to see the rest of the balloon...

Signature · 8 lines
ftp://ftp.cup.hp.com/dist/networking/misc/rachel/
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, OR post, but please do NOT do BOTH...
my email address is raj in the cup.hp.com domain...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#38 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing to dowith ECN)

Quoted 20 lines · Rick Jones
Quoted 20 lines

** I reported that there was also an oddity in throughput values, unfortunately since no one (other than me) seems to have access to a gige cards in the ZC list, nobody can confirm or disprove what i posted. Here again as a reminder:

Kernel | tput | sender-CPU | receiver-CPU |
-------------------------------------------------

2.4.0-pre3 | 99MB/s |   87%      |  23%         |
NSF        |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 86MB/s |   100%     |  17%         |
SF         |        |            |              |
-------------------------------------------------
2.4.0-pre3 | 66.2   |   60%      |  11%         |
+ZC        | MB/s   |            |              |
-------------------------------------------------
2.4.0-pre3 | 68     |   8%       |  8%          |
+ZC  SF    | MB/s   |            |              |

-------------------------------------------------

Quoted 4 lines
Quoted 2 lines

Just ignore the CPU readings, focus on throughput. And could someone plese post results?

In the spirit of the socratic method :)

;->

Quoted 2 lines

Is your gige card based on Alteon?

Yes, sir, it is. To be precise:

** Sender: SMP-PII-450Mhz, ASUS m/board; 3com version of acenic
- 1M version
** receiver: same hardware; acenic alteon card - 1M version

Quoted 1 line

How does ZC/SG change the nature of the packets presented to the NIC?

what do you mean? I am _sure_ you know how SG/ZC work. So i am suspecting more than socratic view on life here. Could be influence from Aristotle;->

Quoted 2 lines

How well does the NIC do with that changed nature?

Hard question to answer ;-> I havent done any analysis at that level

cheers,
jamal

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#39 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing todowith ECN)

Quoted 4 lines
Quoted 1 line

How does ZC/SG change the nature of the packets presented to the NIC?

what do you mean? I am _sure_ you know how SG/ZC work. So i am suspecting more than socratic view on life here. Could be influence from Aristotle;->

Well, I don't know the specifics of Linux, but I gather from what I've read on the list thusfar, that prior to implementing SG support, Linux NIC drivers would copy packets into single contiguous buffers that were then sent to the NIC yes?

If so, the implication is with SG going, that copy no longer takes place, and so a chain of buffers is given to the NIC.

Also, if one is fully ZC :) pesky things like protocol headers can naturally end-up in separate buffers.

So, now you have to ask how well any given NIC follows chains of buffers. At what number of buffers is the overhead in the NIC of following the chains enough to keep it from achieving link-rate?

One way to try and deduce that would be to meld some of the SG and preSG behaviours and copy packets into varying numbers of buffers per packet and measure the resulting impact on throughput through the NIC.

rick jones

As time marches on, the orders of magnitude of the constants may change, but basic concepts still remain, and the "lessons" learned in the past by one generation tend to get relearned in the next :) for example - there is no such a thing as a free lunch... :)

Signature · 8 lines
ftp://ftp.cup.hp.com/dist/networking/misc/rachel/
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, OR post, but please do NOT do BOTH...
my email address is raj in the cup.hp.com domain...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#40 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing todowith ECN)

Quoted 10 lines · Rick Jones
Quoted 4 lines
Quoted 1 line

How does ZC/SG change the nature of the packets presented to the NIC?

what do you mean? I am _sure_ you know how SG/ZC work. So i am suspecting more than socratic view on life here. Could be influence from Aristotle;->

Well, I don't know the specifics of Linux, but I gather from what I've read on the list thusfar, that prior to implementing SG support, Linux NIC drivers would copy packets into single contiguous buffers that were then sent to the NIC yes?

yes.

Quoted 3 lines

If so, the implication is with SG going, that copy no longer takes place, and so a chain of buffers is given to the NIC.

yes.

Quoted 3 lines

Also, if one is fully ZC :) pesky things like protocol headers can naturally end-up in separate buffers.

yes.

Quoted 4 lines

So, now you have to ask how well any given NIC follows chains of buffers. At what number of buffers is the overhead in the NIC of following the chains enough to keep it from achieving link-rate?

hmmm... not sure how you would enforce this today or why you would want that. Alexey, Dave?
The kernel should be able to break it into two buffers(with netperf, for example -- header + data).
Ok, probably with tux-http 3 (header, data, trailler).

Quoted 4 lines

One way to try and deduce that would be to meld some of the SG and preSG behaviours and copy packets into varying numbers of buffers per packet and measure the resulting impact on throughput through the NIC.

If only time were on my hands i'd love to do this. Alas. NOTE also, that effect would also be an effect of the specif NIC.

Quoted 6 lines

rick jones

As time marches on, the orders of magnitude of the constants may change, but basic concepts still remain, and the "lessons" learned in the past by one generation tend to get relearned in the next :) for example - there is no such a thing as a free lunch... :)

;->
BTW, i am reading one of your papers (circa 1993 ;->, "we go fast with a little help from your apps") in which you make an interesting observation. That (figure 2) there is "a considerable increase in efficiency but not a considerable increase in throughput" .... I "scanned" to the end of the paper and dont see an explanation.
I've made a somehow similar observation with the current zc patches and infact observed that throughput goes down with the linux zc patches. [This is being contested but no-one else is testing at gigE, so my word is the only truth].
Of course your paper doesnt talk about sendfile rather the page pinning + COW tricks (which are considered taboo in Linux) but i do sense a relationship.

cheers,
jamal

PS:- I dont have "my" machines yet and i have a feeling it will be a while before i re-run the tests; however, i have created a patch for linux-sendfile with netperf. Please take a look at it at:
http://www.cyberus.ca/~hadi/patch-nperf-sfile-linux.gz
tell me if is missing anything and if it is ok, could you please merge in your tree?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#41 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing todowith ECN)

Quoted 10 lines · jamal
Quoted 4 lines

So, now you have to ask how well any given NIC follows chains of buffers. At what number of buffers is the overhead in the NIC of following the chains enough to keep it from achieving link-rate?

hmmm... not sure how you would enforce this today or why you would want that. Alexey, Dave?
The kernel should be able to break it into two buffers(with netperf, for example -- header + data).
Ok, probably with tux-http 3 (header, data, trailler).

First, just to make sure Jamal understands what Rick Jones is trying to make note of. He is trying to say that the cost of dealing with extra TX descriptor ring entries can begin to nullify the gains of zerocopy, depending upon HW implementation (both at the NIC and the PCI controller).

Back to today, it is possible that this is an issue if your machine is near PCI bandwidth saturation before zerocopy for these tests. I think this may be one of the factors causing Jamal to see results Alexey cannot reproduce. Get two people with identical PCI host bridges, Acenic in identical PCI slot, I bet the numbers begin to jive.

Currently, you get "1 + ((MTU + PAGE_SIZE - 1) / PAGE_SIZE)" buffers per packet when going over a zerocopy device using TCP.

Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#42 lore raw

Re: Still not sexy! (Re: sendfile+zerocopy: fairly sexy (nothing todowith ECN)

In reply to jamal
Quoted 11 lines
Quoted 4 lines

As time marches on, the orders of magnitude of the constants may change, but basic concepts still remain, and the "lessons" learned in the past by one generation tend to get relearned in the next :) for example - there is no such a thing as a free lunch... :)

;->
BTW, i am reading one of your papers (circa 1993 ;->, "we go fast with a little help from your apps") in which you make an interesting observation. That (figure 2) there is "a considerable increase in efficiency but not a considerable increase in throughput" .... I "scanned" to the end of the paper and dont see an explanation.

That would be the copyavoidance paper using the very old G30 with the HP-PB (sometimes called PeanutButter) bus :)
(http://ftp.cup.hp.com/dist/networking/briefs/)

No, back then we were not going to describe the dirty laundry of the G30 hardware :) The limiter appears to have been the bus converter from the SGC (?) main bus of the Novas (8x7,F,G,H,I) to the HP-PB bus. The chip was (apropriately enough) codenamed "BOA" and it was a constrictor :)

I never had a chance to carry-out the tests on an older 852 system - those have slower CPU's, but HP-PB was _the_ bus in the system. Prototypes leading to the HP-PB FDDI card achieved 10 MB/s on an 832 system using UDP - this was back in the 1988-1989 timeframe iirc.

Quoted 7 lines

I've made a somehow similar observation with the current zc patches and infact observed that throughput goes down with the linux zc patches. [This is being contested but no-one else is testing at gigE, so my word is the only truth].
Of course your paper doesnt talk about sendfile rather the page pinning + COW tricks (which are considered taboo in Linux) but i do sense a relationship.

Well, the HP-PB FDDI card did follow buffer chains rather well, and there was no mapping overhead on a Nova - it was a non-coherent I/O subsystem and DMA was done exclusively with physical addresses (and requisite pre-DMA flushes on outbound, and purges on inbound - another reason why copy-avoidance was such a win overheadwise).

Also, there was no throughput drop when going to copyavoidance in that stuff. So, I'd say that while somethings might "feel" similar, it does not go much deeper than that.

rick

Quoted 6 lines

PS:- I dont have "my" machines yet and i have a feeling it will be a while before i re-run the tests; however, i have created a patch for linux-sendfile with netperf. Please take a look at it at:
http://www.cyberus.ca/~hadi/patch-nperf-sfile-linux.gz
tell me if is missing anything and if it is ok, could you please merge in your tree?

I will take a look.

Signature · 8 lines
ftp://ftp.cup.hp.com/dist/networking/misc/rachel/
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, OR post, but please do NOT do BOTH...
my email address is raj in the cup.hp.com domain...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#43 lore raw
In reply to Andrew Morton
Quoted 2 lines · Ion Badulescu

2.4.1-pre10+zerocopy, using read()/write(): 18.3%-29.6% CPU * why so much variance?

The variance is presumably because of the naive read/write implementation. It sucks in 16 megs and writes out out again. With a 100 megabyte file you'll get aliasing effects between the sampling interval and the client's activity.

You will get more repeatable results using smaller files. I'm just sending /usr/local/bin/* ten times, with

./zcc -s otherhost -c /usr/local/bin/* -n10 -N2 -S

Maybe that 16 meg buffer should be shorter... Yes, making it smaller smooths things out.

Heh, look at this. It's a simple read-some, send-some loop. Plot CPU utilisation against the transfer size:

Size %CPU

256             31
512             25
1024            22
2048            18
4096            17
8192            16
16384           18
32768           19
65536           21
128k            22
256k            22.5

8192 bytes is best.

I've added the `-b' option to zcc to set the transfer size. Same URL.

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#44 lore raw
In reply to Andrew Morton
Quoted 3 lines · Andrew Morton

(Please keep netdev copied, else Jamal will grump at you, and
you don't want that).

Thanks, Andrew ;-> Isnt netdev where networking stuff should be discussed? I think i give up and will join lk, RSN ;->

Quoted 15 lines

The kernels which were tested were 2.4.1-pre10 with and without the zerocopy patch. We only look at client load (the TCP sender).

The link throughput was 11.5 mbytes/sec at all times (saturated 100baseT)

2.4.1-pre10-vanilla, using sendfile(): 29.6% CPU
2.4.1-pre10-vanilla, using read()/write(): 34.5% CPU

2.4.1-pre10+zercopy, using sendfile(): 18.2% CPU
2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

2.4.1-pre10+zercopy, using sendfile():          22.9% CPU    * hardware tx checksums disabled
2.4.1-pre10+zercopy, using read()/write():      39.2% CPU    * hardware tx checksums disabled
Quoted 7 lines

What can we conclude?

- sendfile is 10% cheaper than read()-then-write() on 2.4.1-pre10.

- sendfile() with the zerocopy patch is 40% cheaper than sendfile() without the zerocopy patch.

It is also useful to have both client and server stats. BTW, since the laptop (with the 3C card) is the client, the SG shouldnt kick in at all.

Quoted 9 lines

- hardware Tx checksums don't make much difference. hmm...

Bear in mind that the 3c59x driver uses a one-interrupt-per-packet algorithm. Mitigation reduces this to 0.3 ints/packet. So we're absorbing 4,500 interrupts/sec while processing 12,000 packets/sec. gigE NICs do much better mitigation than this and the relative benefits of zerocopy will be much higher for these. Hopefully Jamal can do some testing.

I dont have my babies right now, but as soon as i can get access to them

Quoted 2 lines

BTW: I could not reproduce Jamal's oops when sending large files (2 gigs with sendfile()).

Alexey was concerned about this. Good. But maybe it will still happen with my setupo. We'll see.

Quoted 5 lines

The test tool is, of course, documented [ :-)/2 ]. It's at

	http://www.uow.edu.au/~andrewm/linux/#zc

I'll give this a shot later. Can you try with the sendfiled-ttcp?
http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz
Anyways, you are NIC-challenged ;-> Get GigE. 100Mbps doesnt give much information.

cheers,
jamal

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#45 lore raw
Quoted 2 lines

I'll give this a shot later. Can you try with the sendfiled-ttcp?
http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz

I guess I need to "leverage" some bits for netperf :)

WRT getting data with links that cannot saturate a system, having something akin to the netperf service demand measure can help. Nothing terribly fancy - simply a conversion of the CPU utilization and throughput to a microseconds of CPU to transfer a KB of data.

As for CKO and avoiding copies and such, if past experience is any guide (ftp://ftp.cup.hp.com/dist/networking/briefs/copyavoid.ps) you get a very nice synergistic effect once the last "access" of data is removed. CKO gets you say 10%, avoiding the copy gets you say 10%, but doing both at the same time gets you 30%.

rick jones
http://www.netperf.org/

Signature · 8 lines
ftp://ftp.cup.hp.com/dist/networking/misc/rachel/
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, OR post, but please do NOT do BOTH...
my email address is raj in the cup.hp.com domain...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
#46 lore raw
In reply to Andrew Morton
Quoted 5 lines · jamal

..
It is also useful to have both client and server stats. BTW, since the laptop (with the 3C card) is the client, the SG shouldnt kick in at all.

The `client' here is doing the sendfiling, so yes, the gathering occurs on the client.

Quoted 8 lines

...

Quoted 4 lines

The test tool is, of course, documented [ :-)/2 ]. It's at

      http://www.uow.edu.au/~andrewm/linux/#zc

I'll give this a shot later. Can you try with the sendfiled-ttcp?
http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz

hmm.. I didn't bother with TCP_CORK because the files being sent are "much" larger than a frame. Guess I should.

The problem with things like ttcp is the measurement of CPU load. If your network is so fast that your machine can't keep up then fine, raw throughput is a good measure. But if the link is saturated then normal process accounting doesn't cut it.

For example, at 100 mbps, `top' says ttcp is chewing 4% CPU. But guess what? A low-priority process running on the same machine is in fact slowed down by 30%. top lies. Most of the cost of the networking layer is being accounted to swapper, and lost. And who accounts for cache eviction, bus utilisation, etc. We're better off measuring what's left behind, rather than measuring what is consumed.

You can in fact do this with ttcp: run it with a super-high priority and run a little task in the background (dummyload.c in the above tarball does this). See how much the dummy task is slowed down wrt an unloaded system. It gets tricky on SMP though.

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#47 lore raw
Quoted 10 lines · Andrew Morton
Quoted 5 lines · jamal

..
It is also useful to have both client and server stats. BTW, since the laptop (with the 3C card) is the client, the SG shouldnt kick in at all.

The `client' here is doing the sendfiling, so yes, the gathering occurs on the client.

OK, semantics. Maybe we should stick to sender and receiver. (server normally will translate to "serve" files)

Quoted 5 lines
Quoted 2 lines

I'll give this a shot later. Can you try with the sendfiled-ttcp?
http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz

hmm.. I didn't bother with TCP_CORK because the files being sent are "much" larger than a frame. Guess I should.

It doesnt make much sense to use sendfile without TCP_CORK.

Quoted 4 lines

The problem with things like ttcp is the measurement of CPU load. If your network is so fast that your machine can't keep up then fine, raw throughput is a good measure. But if the link is saturated then normal process accounting doesn't cut it.

ttcp's CPU measure is not the best. Part of my plan was to change that. It uses times(). So the measurement is not good. It is infact not very reflective on SMP. The way to do it there is to break it down by CPU.
Throughput: 100Mbps is really nothing. Linux never had a problem with 4-500Mbps file serving. So throughput is an important number. so is end to end latency, but in file serving case, latency might not be a big deal so ignore it.

Quoted 7 lines

For example, at 100 mbps, `top' says ttcp is chewing 4% CPU. But guess what? A low-priority process running on the same machine is in fact slowed down by 30%. top lies. Most of the cost of the networking layer is being accounted to swapper, and lost. And who accounts for cache eviction, bus utilisation, etc. We're better off measuring what's left behind, rather than measuring what is consumed.

Quoted 5 lines

You can in fact do this with ttcp: run it with a super-high priority and run a little task in the background (dummyload.c in the above tarball does this). See how much the dummy task is slowed down wrt an unloaded system. It gets tricky on SMP though.

The best way to do CPU measurement is via /proc. The way top does it. You measure it from within your nettest program. This does measure what is "left behind" since your proggie is in user space. Actually, it shouldnt matter whether you do it from your test program or from dummyload.c. With dummyload you might have to sigkill the program every time a test terminates.
You also should break down utilization by CPU.

cheers,
jamal

PS:- can you try it out with the ttcp testcode i posted?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#48 lore raw
In reply to Andrew Morton
Quoted 2 lines · jamal

PS:- can you try it out with the ttcp testcode i posted?

Yup. See below. The numbers are almost the same as with zcs' and zcc'.

The CPU utilisation code which was in zcc' has been broken out into a standalone tool, so the new cyclesoak' app is a general-purpose system load measurement tool. It's fascinating to play with, if you're into that sort of thing.

`cyclesoak' was used to measure ttcp-sf and NFS/UDP client and server throughput. The times()-based instrumentation inside ttcp-sf doesn't (can't) give correct numbers. 2-4% CPU at 100 mbps? We wish :)

The zerocopy patch doesn't seem to affect NFS efficiency at all. Confused.

Excerpt from the rapidly swelling README:

NFS/UDP client results
======================

Reading a 100 meg file across 100baseT. The file is fully cached on the server. The client is the above machine. You need to unmount the server between runs to avoid client-side caching.

The server is mounted with various rsize and wsize options.

Kernel rsize wsize mbyte/sec CPU

  2.4.1-pre10+zc   1024  1024     2.4         10.3%
  2.4.1-pre10+zc   2048  2048     3.7         11.4%
  2.4.1-pre10+zc   4096  4096     10.1        29.0%
  2.4.1-pre10+zc   8199  8192     11.9        28.2%
  2.4.1-pre10+zc  16384 16384     11.9        28.2%
  2.4.1-pre10      1024  1024      2.4         9.7%
  2.4.1-pre10      2048  2048      3.7        11.8%
  2.4.1-pre10      4096  4096     10.7        33.6%
  2.4.1-pre10      8199  8192     11.9        29.5%

2.4.1-pre10 16384 16384 11.9 29.2%

Small diff at 8192.

NFS/UDP server results
======================

Reading a 100 meg file across 100baseT. The file is fully cached on the server. The server is the above machine.

Kernel rsize wsize mbyte/sec CPU

  2.4.1-pre10+zc   1024  1024      2.6        19.1%
  2.4.1-pre10+zc   2048  2048      3.9        18.8%
  2.4.1-pre10+zc   4096  4096     10.0        34.5%
  2.4.1-pre10+zc   8199  8192     11.8        28.9%
  2.4.1-pre10+zc  16384 16384     11.8        29.0%
  2.4.1-pre10      1024  1024      2.6        18.5%
  2.4.1-pre10      2048  2048      3.9        18.6%
  2.4.1-pre10      4096  4096     10.9        33.8%
  2.4.1-pre10      8199  8192     11.8        29.0%
  2.4.1-pre10     16384 16384     11.8        29.0%

No diff.

ttcp-sf Results
===============

Jamal Hadi Salim has taught ttcp to use sendfile. See
http://www.cyberus.ca/~hadi/ttcp-sf.tar.gz

Using the same machine as above, and the following commands:

Sender:    ./ttcp-sf -t -c -l 32768 -v receiver_host
Receiver:  ./ttcp-sf -c -r -l 32768 -v sender_host
                                                        CPU

    2.4.1-pre10-zerocopy, sending with ttcp-sf:        10.5%
    2.4.1-pre10-zerocopy, receiving with ttcp-sf:      16.1%

    2.4.1-pre10-vanilla, sending with ttcp-sf:         18.5%
    2.4.1-pre10-vanilla, receiving with ttcp-sf:       16.0%

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#49 lore raw
In reply to Andrew Morton
Quoted 7 lines · [email protected] wrote:

Hello!

Quoted 1 line

2.4.1-pre10+zercopy, using read()/write(): 38.1% CPU

write() on zc card is worse than normal write() by definition. It generates split buffers.

yes. The figures below show this. Disabling SG+checksums speeds up write() and send().

Quoted 4 lines

Split buffers are more expensive and we have to pay for this. You have paid too much for slow card though. 8)

Do you measure load correctly?

Yes. Quite confident about this. Here's the algorithm:

1: Run a cycle-soaker on each CPU on an otherwise unloaded
system. See how much "work" they all do per second.

2: Run the cycle-soakers again, but with network traffic happening.
See how much their "work" is reduced. Deduce networking CPU load from this difference.

The networking code all runs SCHED_FIFO or in interrupt context, so the cycle-soakers have no effect upon the network code's access to the CPU.

The "cycle-soakers" just sit there spinning and dirtying 10,000 cachelines per second.

Quoted 4 lines
Quoted 1 line

2.4.1-pre10+zercopy, using read()/write(): 39.2% CPU * hardware tx checksums disabled

This is illegal combination of parameters. You force two memory accesses, doing this. The fact that it does not add to load is dubious. 8)8)

mm.. Perhaps with read()/write() the data is already in cache?

Anyway, I've tweaked up the tool again so it can do send() or write() (then I looked at the implementation and wondered why I'd bothered). It also does TCP_CORK now.

I ran another set of tests. The zerocopy patch improves sendfile() hugely but slows down send()/write() significantly, with a 3c905C:

http://www.uow.edu.au/~andrewm/linux/#zc

The kernels which were tested were 2.4.1-pre10 with and without the zerocopy patch. We only look at client load (the TCP sender).

In all tests the link throughput was 11.5 mbytes/sec at all times (saturated 100baseT) unless otherwise noted.

The client (the thing which sends data) is a dual 500MHz PII with a 3c905C.

For the write() and send() tests, the chunk size was 64 kbytes.

The workload was 63 files with an average length of 350 kbytes.

                                                     CPU

    2.4.1-pre10+zerocopy, using sendfile():          9.6%
    2.4.1-pre10+zerocopy, using send():             24.1%
    2.4.1-pre10+zerocopy, using write():            24.2%

    2.4.1-pre10+zerocopy, using sendfile():         16.2%       * checksums and SG disabled
    2.4.1-pre10+zerocopy, using send():             21.5%       * checksums and SG disabled
    2.4.1-pre10+zerocopy, using write():            21.5%       * checksums and SG disabled



    2.4.1-pre10-vanilla, using sendfile():          17.1%
    2.4.1-pre10-vanilla, using send():              21.1%
    2.4.1-pre10-vanilla, using write():             21.1%

Bearing in mind that a large amount of the load is in the device driver, the zerocopy patch makes a large improvement in sendfile efficiency. But read() and send() performance is decreased by 10% - more than this if you factor out the constant device driver overhead.

TCP_CORK makes no difference. The files being sent are much larger than a single frame.

Conclusions:

For a NIC which cannot do scatter/gather/checksums, the zerocopy patch makes no change in throughput in all case.

For a NIC which can do scatter/gather/checksums, sendfile() efficiency is improved by 40% and send() efficiency is decreased by 10%. The increase and decrease caused by the zerocopy patch will in fact be significantly larger than these two figures, because the measurements here include a constant base load caused by the device driver.

-
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#50 lore raw
In reply to Andrew Morton

Thus spake Andrew Morton ([email protected]):

Quoted 11 lines

Conclusions:

For a NIC which cannot do scatter/gather/checksums, the zerocopy patch makes no change in throughput in all case.

For a NIC which can do scatter/gather/checksums, sendfile() efficiency is improved by 40% and send() efficiency is decreased by 10%. The increase and decrease caused by the zerocopy patch will in fact be significantly larger than these two figures, because the measurements here include a constant base load caused by the device driver.

What is missing here is a good authoritative web ressource that tells people which NIC to buy.

I have a tulip NIC because a few years ago that apparently was the NIC of choice. It has good multicast (which is important to me), but AFAIK it has neither scatter-gather nor hardware checksumming.

Is there such a web page already?
If not, I volunteer to create amd maintain one.

Felix
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#51 lore raw
Quoted 6 lines · Felix von Leitner

What is missing here is a good authoritative web ressource that tells people which NIC to buy.
I have a tulip NIC because a few years ago that apparently was the NIC of choice. It has good multicast (which is important to me), but AFAIK it has neither scatter-gather nor hardware checksumming.
Is there such a web page already?

http://www.anime.net/~goemon/cardz/

Based on discussions I've had with Donald Becker about chipsets.

For 100bt, 3c905C is the most efficient card at the moment. I've no idea about gigglebit ethernet.

-Dan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#52 lore raw
In reply to Andrew Morton
Quoted 9 lines · Felix von Leitner

What is missing here is a good authoritative web ressource that tells people which NIC to buy.

I have a tulip NIC because a few years ago that apparently was the NIC of choice. It has good multicast (which is important to me), but AFAIK it has neither scatter-gather nor hardware checksumming.

Is there such a web page already?
If not, I volunteer to create amd maintain one.

Here a try for FastEthernet. Corrections/additions welcome.

Currently the 3c9xx cards look like the best commonly affordable ones, at least when you care about zero copy networking. The newer ones have all the necessary toys for it.
Don't use them with the 3com vendor driver though, their driver is crap.

eepro100 seems to have mostly the same facilities, but Intel doesn't document it fully so it is not usable in the standard Linux driver. They have an own driver available (e100.c) which does more, but it of course lags in progress to the normal stack.

I don't know about starfire, but it seems to be hard to even buy them anyways.

Sun HME seems to be on similar level as 3c9xx, but near impossible to buy or very expensive.

Realtek is ok for low cost and being relatively hazzle free, but they miss lots of useful facilities and basically require a copy for RX.

SMC epic/100 is handicapped by not being able to receive to unaligned addresses, requiring in linux a driver level copy (that may change in 2.5 though, zero copy has the necessary infrastructure to only copy the header in this case, not the whole packet)

AMD pcnet afaik doesn't do hardware checksums.

Tulip doesn't do hardware checksum and is a bit constrained by the required long word alignment in RX (causing problems with misaligned IP headers, see above on epic100)

Advantage of Tulip and AMD is that they perform much better in my experience on half duplex Ethernet than other cards because they a modified patented backoff scheme. Without it Linux 2.1+ tends to suffer badly from ethernet congestion by colliding with the own acks, probably because it sends too fast.

-Andi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#53 lore raw
In reply to Andrew Morton
Quoted 16 lines · Felix von Leitner

Thus spake Andrew Morton ([email protected]):

Quoted 11 lines

Conclusions:

For a NIC which cannot do scatter/gather/checksums, the zerocopy patch makes no change in throughput in all case.

For a NIC which can do scatter/gather/checksums, sendfile() efficiency is improved by 40% and send() efficiency is decreased by 10%. The increase and decrease caused by the zerocopy patch will in fact be significantly larger than these two figures, because the measurements here include a constant base load caused by the device driver.

What is missing here is a good authoritative web ressource that tells people which NIC to buy.

Quoted 6 lines

I have a tulip NIC because a few years ago that apparently was the NIC of choice. It has good multicast (which is important to me), but AFAIK it has neither scatter-gather nor hardware checksumming.

Is there such a web page already?
If not, I volunteer to create amd maintain one.

Additionally, it would be useful to have some boot messages comment on the abilities of cards. I am sick and tired of dealing with people telling me that 'Linux performance sucks' when they keep putting Linux on systems with pci 8139 adaptors.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#54 lore raw

Choosing Linux NICs (was: Re: sendfile+zerocopy: fairly sexy (nothing to do with ECN))

In reply to Andrew Morton

Thus spake Felix von Leitner ([email protected]):

Quoted 2 lines

What is missing here is a good authoritative web ressource that tells people which NIC to buy.

I started one now.

It's at http://www.fefe.de/linuxeth/, but there is not much content yet. Please contribute!

Felix
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/

#55 lore raw
In reply to Andrew Morton

Hi!

Quoted 5 lines

Advantage of Tulip and AMD is that they perform much better in my experience on half duplex Ethernet than other cards because they a modified patented backoff scheme. Without it Linux 2.1+ tends to suffer badly from ethernet congestion by colliding with the own acks, probably because it sends too fast.

Is that real problem? If so, some strategic delay loop should do the trick...
Pavel

Signature · 6 lines
I'm [email protected]. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/