Skip to contentSkip to conversation

Rough VJ Channel Implementation - vj_core.patch

PATCH New Caitlin Bestler 5 participants 18 messages Apr 28, 2006 – May 2, 2006 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw
Quoted 15 lines · Evgeniy Polyakov
Quoted 6 lines · Caitlin Bestler
Quoted 2 lines

Btw, how is it supposed to work without header split capabale hardware?

Hardware that can classify packets is obviously capable of doing header data separation, but that does not mean that it has to do so.

Quoted 6 lines

If the host wants header data separation it's real value is that when packets arrive in order that fewer distinct copies are required to move the data to the user buffer (because separated data can be placed back-to-back in a data-only ring). But that's an optimization, it's not needed to make the idea worth doing, or even necessarily in the first implementation.

Quoted 7 lines

If there is dataflow, not flow of packets or flow of data with holes, it could be possible to modify recv() to just return the right pointer, so in theory userspace modifications would be minimal.
With copy in place it completely does not differ from current design with copy_to_user() being used since memcpy() is just slightly faster than copy*user().

If the app is really ready to use a modified interface we might as well just give them a QP/CQ interface. But I suppose "receive by pointer" interfaces don't really stretch the sockets interface all that badly. The key is that you have to decide how the buffer is released, is it the next call? Or a separate call? Does releasing buffer N+2 release buffers N and N+1? What you want to avoid
is having to keep a scoreboard of which buffers have been released.

But in context, header/data separation would allow in order packets to have the data be placed back to back, which could allow a single recv to report the payload of multiple successive TCP segments. So the benefit of header/data separation remains the same, and I still say it's a optimization that should not be made a requirement. The benefits of vj_channels exist even without them. When the packet classifier runs on the host, header/data separation would not be free. I want to enable hardware offloads, not make the kernel bend over backwards to emulate how hardware would work. I'm just hoping that we can agree to let hardware do its work without being forced to work the same way the kernel does (i.e., running down a long list of arbitrary packet filter rules on a per packet basis).

#1 lore raw
Quoted 24 lines · On Fri, 28 Apr 2006 10:02:10 -0700 "Caitlin Bestler" <[email protected]>…
Quoted 15 lines · Evgeniy Polyakov
Quoted 6 lines · Caitlin Bestler
Quoted 2 lines

Btw, how is it supposed to work without header split capabale hardware?

Hardware that can classify packets is obviously capable of doing header data separation, but that does not mean that it has to do so.

Quoted 6 lines

If the host wants header data separation it's real value is that when packets arrive in order that fewer distinct copies are required to move the data to the user buffer (because separated data can be placed back-to-back in a data-only ring). But that's an optimization, it's not needed to make the idea worth doing, or even necessarily in the first implementation.

Quoted 7 lines

If there is dataflow, not flow of packets or flow of data with holes, it could be possible to modify recv() to just return the right pointer, so in theory userspace modifications would be minimal.
With copy in place it completely does not differ from current design with copy_to_user() being used since memcpy() is just slightly faster than copy*user().

Quoted 9 lines

If the app is really ready to use a modified interface we might as well just give them a QP/CQ interface. But I suppose "receive by pointer" interfaces don't really stretch the sockets interface all that badly. The key is that you have to decide how the buffer is released, is it the next call? Or a separate call? Does releasing buffer N+2 release buffers N and N+1? What you want to avoid
is having to keep a scoreboard of which buffers have been released.

Please just use existing AIO interface. We don't need another interface. The number of interfaces increases the exposed bug surface geometrically. Which means for each new interface, it means testing and fixing bugs in every possible usage.

Quoted 19 lines

But in context, header/data separation would allow in order packets to have the data be placed back to back, which could allow a single recv to report the payload of multiple successive TCP segments. So the benefit of header/data separation remains the same, and I still say it's a optimization that should not be made a requirement. The benefits of vj_channels exist even without them. When the packet classifier runs on the host, header/data separation would not be free. I want to enable hardware offloads, not make the kernel bend over backwards to emulate how hardware would work. I'm just hoping that we can agree to let hardware do its work without being forced to work the same way the kernel does (i.e., running down a long list of arbitrary packet filter rules on a per packet basis).

-
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 4 lines · Stephen Hemminger

Please just use existing AIO interface. We don't need another interface. The number of interfaces increases the exposed bug surface geometrically. Which means for each new interface, it means testing and fixing bugs in every possible usage.

Networking AIO? Like [1] :)
That would be really good.

1. http://tservice.net.ru/~s0mbre/old/?section=projects&item=naio

Signature · 1 line
	Evgeniy Polyakov
#3 lore raw
Quoted 11 lines · On Fri, 28 Apr 2006 21:29:32 +0400 Evgeniy Polyakov <[email protected]> wrote:
Quoted 4 lines · Stephen Hemminger

Please just use existing AIO interface. We don't need another interface. The number of interfaces increases the exposed bug surface geometrically. Which means for each new interface, it means testing and fixing bugs in every possible usage.

Networking AIO? Like [1] :)
That would be really good.

1. http://tservice.net.ru/~s0mbre/old/?section=projects&item=naio

The existing infrastructure is there in the syscall layer, it just isn't really AIO for sockets. That naio project has two problems, first they require driver changes, and he is doing it on the stupidest of hardware, optimizing a 8139too is foolish. Second, introducing kevents, seems unnecessary and hasn't been accepted in the mainline.

The existing linux AIO model seems sufficient:
http://lse.sourceforge.net/io/aio.html

There is work to put true Posix AIO on top of this.

#4 lore raw
Quoted 12 lines · Stephen Hemminger
Quoted 9 lines · On Fri, 28 Apr 2006 21:29:32 +0400 Evgeniy Polyakov <[email protected]> wrote:
Quoted 4 lines · Stephen Hemminger

Please just use existing AIO interface. We don't need another interface. The number of interfaces increases the exposed bug surface geometrically. Which means for each new interface, it means testing and fixing bugs in every possible usage.

Networking AIO? Like [1] :)
That would be really good.

Quoted 7 lines
Quoted 2 lines

1. http://tservice.net.ru/~s0mbre/old/?section=projects&item=naio

The existing infrastructure is there in the syscall layer, it just isn't really AIO for sockets. That naio project has two problems, first they require driver changes, and he is doing it on the stupidest of hardware, optimizing a 8139too is foolish.

No, it does not. You confuse it with receiving zero-copy support which allows to DMA data directly into VFS cache [1].
NAIO works for any kind of hardware and was tested with e1000 and showed noticeble win in both CPU usage and network performance.

Quoted 2 lines

Second, introducing
kevents, seems unnecessary and hasn't been accepted in the mainline.

kevent was never sent to lkml@ although it showed over 40% win over epoll for test web server. Sending it to lkml@ is just jumping into ... not into technical world, so I posted it first here, but without much attention though.

Quoted 4 lines

The existing linux AIO model seems sufficient:
http://lse.sourceforge.net/io/aio.html

There is work to put true Posix AIO on top of this.

There are a lot of discussions about combining AIO with epoll and combine them into something similar to kevent which allows to monitor level and edge triggered events, to create proper state machine for AIO compeltions. kevent [2] does exactly that. AIO works not as state machine, but it's repeated-check design is more like postponing work from one context to special thread.

1. receiving zero-copy support
http://tservice.net.ru/~s0mbre/old/?section=projects&item=recv_zero_copy

2. kevent system
http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent

Signature · 1 line
	Evgeniy Polyakov
maintainer
#5 lore raw

From: Evgeniy Polyakov <[email protected]>
Date: Fri, 28 Apr 2006 21:55:39 +0400

Quoted 8 lines · Stephen Hemminger
Quoted 2 lines

Second, introducing
kevents, seems unnecessary and hasn't been accepted in the mainline.

kevent was never sent to lkml@ although it showed over 40% win over epoll for test web server. Sending it to lkml@ is just jumping into ... not into technical world, so I posted it first here, but without much attention though.

Frankly I found kevents to be a very strong idea.

#6 lore raw
Quoted 7 lines · On Fri, 28 Apr 2006 12:16:36 -0700 (PDT) "David S. Miller"…

From: Evgeniy Polyakov <[email protected]>
Date: Fri, 28 Apr 2006 21:55:39 +0400

Quoted 4 lines · Stephen Hemminger
Quoted 2 lines

Second, introducing
kevents, seems unnecessary and hasn't been accepted in the mainline.

Quoted 6 lines
Quoted 4 lines

kevent was never sent to lkml@ although it showed over 40% win over epoll for test web server. Sending it to lkml@ is just jumping into ... not into technical world, so I posted it first here, but without much attention though.

Frankly I found kevents to be a very strong idea.

But there is this huge semantic overload of kevent, poll, epoll, aio, regular sendmsg/recv, posix aio, etc.

Perhaps a clean break with the socket interface is needed. Otherwise, there are nasty complications with applications that mix old socket calls and new interface on the same connection.

#7 lore raw
Quoted 6 lines · Stephen Hemminger

But there is this huge semantic overload of kevent, poll, epoll, aio, regular sendmsg/recv, posix aio, etc.

Perhaps a clean break with the socket interface is needed. Otherwise, there are nasty complications with applications that mix old socket calls and new interface on the same connection.

kevent can be used as poll without any changes to the socket code. There are two types of network related kevents - socket events (recv/send/accept) and network aio, which can be turned completely off in config.
There are following events which are supported by kevent:
o usual poll/select notifications
o inode notifications (create/remove)
o timer notifications
o socket notifications (send/recv/accept)
o network aio system
o fs aio (project closed, aio_sendfile() is being developed instead)

Any of the above can be turned off by config option.

Signature · 1 line
	Evgeniy Polyakov
maintainer
#8 lore raw

From: Evgeniy Polyakov <[email protected]>
Date: Fri, 28 Apr 2006 23:59:30 +0400

Quoted 12 lines

kevent can be used as poll without any changes to the socket code. There are two types of network related kevents - socket events (recv/send/accept) and network aio, which can be turned completely off in config.
There are following events which are supported by kevent:
o usual poll/select notifications
o inode notifications (create/remove)
o timer notifications
o socket notifications (send/recv/accept)
o network aio system
o fs aio (project closed, aio_sendfile() is being developed instead)

Quoted 1 line

Any of the above can be turned off by config option.

Feel free to post the current version of your kevent patch here so we can discuss something concrete.

Maybe you have even some toy example user applications that use kevent that people can look at too? That might help in understanding how it's supposed to be used.

#9 lore raw
Quoted 15 lines · David S. Miller

From: Evgeniy Polyakov <[email protected]>
Date: Fri, 28 Apr 2006 23:59:30 +0400

Quoted 12 lines

kevent can be used as poll without any changes to the socket code. There are two types of network related kevents - socket events (recv/send/accept) and network aio, which can be turned completely off in config.
There are following events which are supported by kevent:
o usual poll/select notifications
o inode notifications (create/remove)
o timer notifications
o socket notifications (send/recv/accept)
o network aio system
o fs aio (project closed, aio_sendfile() is being developed instead)

Quoted 8 lines
Quoted 1 line

Any of the above can be turned off by config option.

Feel free to post the current version of your kevent patch here so we can discuss something concrete.

Maybe you have even some toy example user applications that use kevent that people can look at too? That might help in understanding how it's supposed to be used.

There are several at project's homepage [1] and in archive [2]:
evserver_epoll.c - epoll-based web server (pure epoll) evserver_kevent.c - kevent-based web server (socket notifications) evserver_poll.c - web server which uses kevent-based poll (poll/select notifications)
evtest.c - can wait for any type of events. It was used to test timer notifications.
naio_recv.c/naio_send.c - network AIO sending and receiving benchmarks (sync/async)
aio_sendfile.c - aio sendfile benchmark (sendfile/aio_sendfile). Kernel implementation is not 100% ready, pages are only asynchronously propagated into VFS cache, but are not sent yet.

There are also links to benchmark results, comparison with FreeBSD kqueue, some conclusions on kevent homepage [1].
Network AIO [3] homepage also contains additional NAIO benchmarks with some graphs.

1. kevent project home page.
http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent

2. kevent archive
http://tservice.net.ru/~s0mbre/archive/kevent/

3. Network AIO
http://tservice.net.ru/~s0mbre/old/?section=projects&item=naio

Current development kevent patchset (against 2.6.15-rc7, but could be applied against later trees too) attached gzipped, sory if you get this twice.

Signature · 1 line
	Evgeniy Polyakov

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 8 lines · David S. Miller

From: Evgeniy Polyakov <[email protected]>
Date: Sat, 29 Apr 2006 16:44:51 +0400

Quoted 1 line

I understand how in some ways this is work in progress, but direct calls into ext3 from the kevent code? I'd like stuff like that cleaned up before reviewing :-)

Well, this only requires per address space ->get_block() callback, which is what ext3_get_block() is.

I will update and resend patchset today.

Thank you.

Signature · 1 line
	Evgeniy Polyakov
#13 lore raw

PATCH 1/1 Kevent subsystem.

In reply to David S. Miller

Kevent subsystem incorporates several AIO/kqueue design notes and ideas. Kevent can be used both for edge and level triggered notifications.

It supports:
o socket notifications (accept, receiving and sending)
o network AIO (aio_send(), aio_recv() and aio_sendfile()) [3]
o inode notifications (create/remove)
o generic poll()/select() notifications
o timer notifications

More info, design notes, benchmarks (web server based on epoll, kevent, kevent_poll; naio_send() vs. send(), naio_recv() vs. recv() with different number of sending/receiving users) can be found on project's homepage [1].

Userspace interface was greatly described in LWN article [2].

1. kevent homepage.
http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent

2. LWN article about kevent.
http://lwn.net/Articles/172844/

3. Network AIO (aio_send(), aio_recv(), aio_sendfile()).
http://tservice.net.ru/~s0mbre/old/?section=projects&item=naio

Signature · 1 line
	Evgeniy Polyakov
#14 lore raw
In reply to David S. Miller
Quoted 7 lines · David S. Miller

From: Evgeniy Polyakov <[email protected]>
Date: Fri, 28 Apr 2006 21:55:39 +0400

Quoted 4 lines · Stephen Hemminger
Quoted 2 lines

Second, introducing
kevents, seems unnecessary and hasn't been accepted in the mainline.

Quoted 6 lines
Quoted 4 lines

kevent was never sent to lkml@ although it showed over 40% win over epoll for test web server. Sending it to lkml@ is just jumping into ... not into technical world, so I posted it first here, but without much attention though.

Frankly I found kevents to be a very strong idea.

Glad to hear this.
I probably should resend patches netdev@ and (mar my karma) send it to lkml@...?

Signature · 1 line
	Evgeniy Polyakov
maintainer
#15 lore raw
In reply to Stephen Hemminger

From: Stephen Hemminger <[email protected]>
Date: Fri, 28 Apr 2006 10:18:33 -0700

Quoted 1 line

Please just use existing AIO interface.

I totally disagree, the existing AIO interface is garbage.

We need new APIs to do this right, to get the ring buffer and the zero-copy'ness correct.

#16 lore raw
Quoted 10 lines · David S. Miller

From: Stephen Hemminger <[email protected]>
Date: Fri, 28 Apr 2006 10:18:33 -0700

Quoted 1 line

Please just use existing AIO interface.

I totally disagree, the existing AIO interface is garbage.

We need new APIs to do this right, to get the ring buffer and the zero-copy'ness correct.
-

Heh, like PF_RING? Just mmap a socket and read out some structures?

#17 lore raw
In reply to Caitlin Bestler
Quoted 9 lines · Caitlin Bestler

If the app is really ready to use a modified interface we might as well just give them a QP/CQ interface. But I suppose "receive by pointer" interfaces don't really stretch the sockets interface all that badly. The key is that you have to decide how the buffer is released, is it the next call? Or a separate call? Does releasing buffer N+2 release buffers N and N+1? What you want to avoid
is having to keep a scoreboard of which buffers have been released.

Quoted 13 lines

But in context, header/data separation would allow in order packets to have the data be placed back to back, which could allow a single recv to report the payload of multiple successive TCP segments. So the benefit of header/data separation remains the same, and I still say it's a optimization that should not be made a requirement. The benefits of vj_channels exist even without them. When the packet classifier runs on the host, header/data separation would not be free. I want to enable hardware offloads, not make the kernel bend over backwards to emulate how hardware would work. I'm just hoping that we can agree to let hardware do its work without being forced to work the same way the kernel does (i.e., running down a long list of arbitrary packet filter rules on a per packet basis).

I see your point, and respectfully disagree.
The more complex userspace interface we create the less users it will have. It is completely unconvenient to read 100 bytes and receive only 80, since 20 were eaten by header. And what if we need only 20, but packet contains 100, introduce per packet head pointer?
For purpose of benchmarking it works perfectly -
read the whole packet, one can event touch that data to emulate real work, but for the real world it becomes practically unusabl.

But what we are talking about right now is a research project, not production system, so we can create any interface we like since the main goal, IMHO, is searching for the bottlenecks in the current stack and ways of it's removal even by introducing new complex interface. I would definitely like to see how your approach works for some kind of real workloads and does it allow to
create faster and generally better systems.

Signature · 1 line
	Evgeniy Polyakov
maintainer
#18 lore raw

From: Evgeniy Polyakov <[email protected]>
Date: Fri, 28 Apr 2006 21:25:36 +0400

Quoted 3 lines

The more complex userspace interface we create the less users it will have. It is completely unconvenient to read 100 bytes and receive only 80, since 20 were eaten by header.

These bytes are charged to socket anyways, and allowing the headers to be there is the only clean way to finesse the whole zero-copy problem.

User can manage his data any way he likes. He can decide to take advantage of the zero-copy layout we've provided, or he can copy to put things into a format he is more happy with at the cost of the copy.