TX_RING and packet mmap
Subsystems (4)
PATCH 1/1 TX_RING and packet mmap
From: Johann Baudy <[email protected]>
New packet socket feature that makes packet socket more efficient for transmission.
- It reduces number of system call through a PACKET_TX_RING mechanism, based on PACKET_RX_RING (Circular buffer allocated in kernel space which is mmapped from user space).
- It minimizes CPU copy using fragmented SKB (almost zero copy).
--
As I start to receive few mails from people that are now using this feature, I would like to discuss with you if it is possible to integrate this patch in the kernel under CONFIG_PACKET_MMAP feature flag. More details (example of using, patch 2.4,2.6 ...) can be found at: http://wiki.ipxwarzone.com/index.php5?title=Linux_packet_mmap
This feature is based on PACKET_RX_RING mechanism. I've implemented a parallel mechanism for TX that can send multiple packets to device in one system call thanks to mmapped memory. It uses skb destructor and mmapped memory (status flag) to indicate user that packet has been sent and buffer is now free for new sending.
It also adds some abstraction functions/structures to prevent code redundancy.
Documentation/networking/packet_mmap.txt | 139 +++++++- include/linux/if_packet.h | 2 + net/packet/af_packet.c | 561 ++++++++++++++++++++++++------ 3 files changed, 579 insertions(+), 123 deletions(-)
Documentation/networking/packet_mmap.txt +120 -19
@@ -4,16 +4,18 @@ This file documents the CONFIG_PACKET_MMAP option available with the PACKET socket interface on 2.4 and 2.6 kernels. This type of sockets is used for -capture network traffic with utilities like tcpdump or any other that uses -the libpcap library. - -You can find the latest version of this document at +capture network traffic with utilities like tcpdump or any other that needs +raw access to network interface. +You can find the latest version of this document at: http://pusa.uv.es/~ulisses/packet_mmap/ -Please send me your comments to +Howto can be found at: + http://wiki.gnu-log.net (packet_mmap) +Please send your comments to Ulisses Alonso Camaró <[email protected]> + Johann Baudy <[email protected]> ------------------------------------------------------------------------------- + Why use PACKET_MMAP
@@ -25,19 +27,24 @@ to capture each packet, it requires two if you want to get packet's timestamp (like libpcap always does). In the other hand PACKET_MMAP is very efficient. PACKET_MMAP provides a size -configurable circular buffer mapped in user space. This way reading packets just -needs to wait for them, most of the time there is no need to issue a single -system call. By using a shared buffer between the kernel and the user -also has the benefit of minimizing packet copies. - -It's fine to use PACKET_MMAP to improve the performance of the capture process, -but it isn't everything. At least, if you are capturing at high speeds (this -is relative to the cpu speed), you should check if the device driver of your -network interface card supports some sort of interrupt load mitigation or -(even better) if it supports NAPI, also make sure it is enabled. +configurable circular buffer mapped in user space that can be used to either +send or receive packets. This way reading packets just needs to wait for them, +most of the time there is no need to issue a single system call. Concerning +transmission, multiple packets can be sent through one system call to get the +highest bandwidth. +By using a shared buffer between the kernel and the user also has the benefit +of minimizing packet copies. + +It's fine to use PACKET_MMAP to improve the performance of the capture and +transmission process, but it isn't everything. At least, if you are capturing +at high speeds (this is relative to the cpu speed), you should check if the +device driver of your network interface card supports some sort of interrupt +load mitigation or (even better) if it supports NAPI, also make sure it is +enabled. For transmission, check the MTU (Maximum Transmission Unit) used and +supported by devices of your network. -------------------------------------------------------------------------------- -+ How to use CONFIG_PACKET_MMAP ++ How to use CONFIG_PACKET_MMAP to improve capture process -------------------------------------------------------------------------------- From the user standpoint, you should use the higher level libpcap library, which
@@ -57,7 +64,7 @@ the low level details or want to improve libpcap by including PACKET_MMAP support. -------------------------------------------------------------------------------- -+ How to use CONFIG_PACKET_MMAP directly ++ How to use CONFIG_PACKET_MMAP directly to improve capture process -------------------------------------------------------------------------------- From the system calls stand point, the use of PACKET_MMAP involves
@@ -66,6 +73,7 @@ the following process: [setup] socket() -------> creation of the capture socket setsockopt() ---> allocation of the circular buffer (ring) + option: PACKET_RX_RING mmap() ---------> mapping of the allocated buffer to the user process
@@ -97,13 +105,75 @@ also the mapping of the circular buffer in the user process and the use of this buffer. -------------------------------------------------------------------------------- ++ How to use CONFIG_PACKET_MMAP directly to improve transmission process +-------------------------------------------------------------------------------- +Transmission process is similar to capture as shown below. + +[setup] socket() -------> creation of the transmission socket + setsockopt() ---> allocation of the circular buffer (ring) + option: PACKET_TX_RING + bind() ---------> bind transmission socket with a network interface + mmap() ---------> mapping of the allocated buffer to the + user process + +[transmission] poll() ---------> wait for free packets (optional) + send() ---------> send all packets that are set as ready in + the ring + The flag MSG_DONTWAIT can be used to return + before end of transfer. + +[shutdown] close() --------> destruction of the transmission socket and + deallocation of all associated resources. + +Binding the socket to your network interface is mandatory (with zero copy) to +know the header size of frames used in the circular buffer. + +As capture, each frame contains two parts: + + -------------------- +| struct tpacket_hdr | Header. It contains the status of +| | of this frame +|--------------------| +| data buffer | +. . Data that will be sent over the network interface. +. . + -------------------- + + bind() associates the socket to your network interface thanks to + sll_ifindex parameter of struct sockaddr_ll. + + Initialization example: + + struct sockaddr_ll my_addr; + struct ifreq s_ifr; + ... + + strncpy (s_ifr.ifr_name, "eth0", sizeof(s_ifr.ifr_name)); + + /* get interface index of eth0 */ + ioctl(this->socket, SIOCGIFINDEX, &s_ifr); + + /* fill sockaddr_ll struct to prepare binding */ + my_addr.sll_family = AF_PACKET; + my_addr.sll_protocol = ETH_P_ALL; + my_addr.sll_ifindex = s_ifr.ifr_ifindex; + + /* bind socket to eth0 */ + bind(this->socket, (struct sockaddr *)&my_addr, sizeof(struct sockaddr_ll)); + + A complete tutorial is available at: http://wiki.gnu-log.net/ + +-------------------------------------------------------------------------------- + PACKET_MMAP settings -------------------------------------------------------------------------------- To setup PACKET_MMAP from user level code is done with a call like + - Capture process setsockopt(fd, SOL_PACKET, PACKET_RX_RING, (void *) &req, sizeof(req)) + - Transmission process + setsockopt(fd, SOL_PACKET, PACKET_TX_RING, (void *) &req, sizeof(req)) The most significant argument in the previous call is the req parameter, this parameter must to have the following structure:
@@ -117,11 +187,11 @@ this parameter must to have the following structure: }; This structure is defined in /usr/include/linux/if_packet.h and establishes a -circular buffer (ring) of unswappable memory mapped in the capture process. +circular buffer (ring) of unswappable memory. Being mapped in the capture process allows reading the captured frames and related meta-information like timestamps without requiring a system call. -Captured frames are grouped in blocks. Each block is a physically contiguous +Frames are grouped in blocks. Each block is a physically contiguous region of memory and holds tp_block_size/tp_frame_size frames. The total number of blocks is tp_block_nr. Note that tp_frame_nr is a redundant parameter because
@@ -336,6 +406,7 @@ struct tpacket_hdr). If this field is 0 means that the frame is ready to be used for the kernel, If not, there is a frame the user can read and the following flags apply: ++++ Capture process: from include/linux/if_packet.h #define TP_STATUS_COPY 2
@@ -391,6 +462,36 @@ packets are in the ring: It doesn't incur in a race condition to first check the status value and then poll for frames. + +++ Transmission process +Those defines are also used for transmission: + + #define TP_STATUS_KERNEL 0 // Frame is available + #define TP_STATUS_USER 1 // Frame will be sent on next send() + #define TP_STATUS_COPY 2 // Frame is currently in transmission + +First, the kernel initializes all frames to TP_STATUS_KERNEL. To send a packet, +the user fills a data buffer of an available frame, sets tp_len to current +data buffer size and sets its status field to TP_STATUS_USER. This can be done +on multiple frames. Once the user is ready to transmit, it calls send(). +Then all buffers with status equal to TP_STATUS_USER are forwarded to the +network device. The kernel updates each status of sent frames with +TP_STATUS_COPY until the end of transfer. +At the end of each transfer, buffer status returns to TP_STATUS_KERNEL. + + header->tp_len = in_i_size; + header->tp_status = TP_STATUS_USER; + retval = send(this->socket, NULL, 0, 0); + +The user can also use poll() to check if a buffer is available: +(status == TP_STATUS_KERNEL) + + struct pollfd pfd; + pfd.fd = fd; + pfd.revents = 0; + pfd.events = POLLOUT; + retval = poll(&pfd, 1, timeout); + -------------------------------------------------------------------------------- + THANKS --------------------------------------------------------------------------------
include/linux/if_packet.h +2 -0
@@ -46,6 +46,8 @@ struct sockaddr_ll#define PACKET_VERSION 10#define PACKET_HDRLEN 11#define PACKET_RESERVE 12 +#define PACKET_TX_RING 13 +#define PACKET_LOSS 14struct tpacket_stats {
net/packet/af_packet.c +457 -104
763 lines not on this page · raw message
net/packet/af_packet.c ring_is_full: · line 838
Quoted 10 lines · Johann Baudy
net/packet/af_packet.c +9
-0+static void tpacket_destruct_skb (struct sk_buff *skb) +{ +struct packet_sock *po =pkt_sk (skb->sk); +void * ph; + +BUG_ON (skb ==NULL ); + +if (likely (po->tx_ring.pg_vec)) { + ph =packet_lookup_frame (po, &po->tx_ring, skb->mark,TP_STATUS_COPY );
What if something modifies skb->mark after it's sent? For now the only thing I can think of is a tc netfilter action. So you could also solve the problem by removing that feature since IMHO it is pretty dodgy :)
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 4 lines · Herbert Xu
What if something modifies
skb->markafter it's sent?
For now the only thing I can think of is a tc netfilter action. So you could also solve the problem by removing that feature since IMHO it is pretty dodgy :)
I see it as being non-trivial even with tc/mark. What did you have in mind?
If it was possible to have a different skb->cookie (other than skb->mark, sort of "global cb") which the sender sets and the kernel never touches then this would be easy. The skb->destructor can then be sure that it was the original skb in the lookup. Such a field could serve other purposes like notify user space that the packet has been really sent out (eg in the case of tun). But that would require a new field on the skb.
[For a fleeting latte-moment there i thought i had a clever idea: I wanted to say that the numeric (32/64 bit) allocated address of the skb may be useful for the cookie but i think that could change in the skb's kernel lifetime (skb_expand etc).]
cheers,
jamal
Quoted 3 lines · jamal
I see it as being non-trivial even with tc/mark. What did you have in mind?
Well through tc ipt actions you can invoke arbitrary netfilter targets, including MARK --set-mark.
In fact I've never really liked tc ipt actions since it doesn't do any sanity checking on the packet before invoking the target while normally netfitler does quite a bit of sanity checking.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 2 lines · Herbert Xu
Well through tc ipt actions you can invoke arbitrary netfilter targets, including MARK --set-mark.
I thought you were saying this issue can be solved by tc/mark but you meant tc/mark is one culprit;-> Solvable by making sure tc/mark is not used in the system you are running an app using this feature.
But the superset of what Johann wants seems to be a common
issue: identifying _precisely_ when the skb sent out is done with - in his case, so he can use some field to lookup a table. Other times i have seen people ask for it is for tun/tap notification in user space when a packet sent actually went out.
Quoted 3 lines
In fact I've never really liked tc ipt actions since it doesn't do any sanity checking on the packet before invoking the target while normally netfitler does quite a bit of sanity checking.
more could be done in user space if you really need it.
cheers,
jamal
Quoted 9 lines · jamal
Quoted 2 lines · Herbert Xu
Well through tc ipt actions you can invoke arbitrary netfilter targets, including MARK --set-mark.
I thought you were saying this issue can be solved by tc/mark but you meant tc/mark is one culprit;-> Solvable by making sure tc/mark is not used in the system you are running an app using this feature.
Actually it's not just the tc ipt action. If the target device is a bridge then that too can invoke netfilter. On the other hand this is yet another feature that I've never really liked :)
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtHi All,
Many thanks for your replies!
Indeed, I've tried to find a solution to store data in skb (valid until destructor).
I've chosen this solution according to previous email on this subject:
http://lists.openwall.net/netdev/2008/11/11/93
I thought skb->mark was not used anymore at this level. So I've used it to forward buffer index.
Do you think this solution is not acceptable?
Thanks,
Johann
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Johann Baudy
Hi All,
Many thanks for your replies!Indeed, I've tried to find a solution to store data in skb (valid until destructor).
Makes sense for what you are trying to do (or someone else a long while back who wanted to notify user space of a sent skb). Any skb metadata can mutate along its path. Actually even if you used a field off the skb->data it too could be changed somewhere along the path before destructor is invoked. There maybe a "hard way" to achieve your goal: use the address of the skb to derive your index; i am not 100% sure if your destructor will always be called (check skb_expand() for example).
cheers,
jamal
Quoted 8 lines · jamal
Makes sense for what you are trying to do (or someone else a long while back who wanted to notify user space of a sent skb). Any skb metadata can mutate along its path. Actually even if you used a field off the
skb->datait too could be changed somewhere along the path before destructor is invoked. There maybe a "hard way" to achieve your goal: use the address of the skb to derive your index; i am not 100% sure if your destructor will always be called (checkskb_expand()for example).
It should, I actually do not see any sending path which does not invoke original skb destructor with the new data. It does not change the fact though, that effectively any other skb field can be modified during skb lifecycle no matter at which level it was allocated.
Having a data pointer as an index could work though, especially it looks promising for fragments placed in own pages.
Signature · 1 line
Evgeniy Polyakov
When you said "skb address", did you mean skb pointer value or fragments data address?
If fragment data address, I can use first fragment page address to deduce header address (in destructor)
As there is a constant offset between header address and data address.
However I'm afraid of issues once it is linearized. Is there a sending path that will lose fragments info?
Best regards,
Johann
Hi Evgeniy,
Quoted 6 lines
It should, I actually do not see any sending path which does not invoke original skb destructor with the new data. It does not change the fact though, that effectively any other skb field can be modified during skb lifecycle no matter at which level it was allocated.
Having a data pointer as an index could work though, especially it looks promising for fragments placed in own pages.
Do you mean that skb_shinfo(skb)->frags[i].page will be valid until destructor? even if linearized, cloned ....?
Thanks in advance for your help,
Johann
Hi Johann.
Sorry for long reply.
Quoted 2 lines · Johann Baudy
Do you mean that skb_shinfo(skb)->frags[i].page will be valid until destructor? even if linearized, cloned ....?
Page itself will live until destruction time, when skb is linearized page is freed, so your own destructor will find that given page is no longer used, and appropriate memory area can be overwritten.
When skb is cloned, page is untouched, only shared info reference counter is increased.
Signature · 1 line
Evgeniy Polyakov
Thanks Evgeniy,
Quoted 3 lines
Page itself will live until destruction time, when skb is linearized page is freed, so your own destructor will find that given page is no longer used, and appropriate memory area can be overwritten.
Ok, so if my understanding is correct, fragment data is copied, frags info (page, offset, size ...) of skb_shinfo(skb) remain available and skb->data_len is equal to 0 (to tag it as linear).
Then I can use address of skb_shinfo(n)->frags[0].page to derive my header whatever sending path.
Is it correct?
Best regards,
Johann
Signature · 2 lines
Johann Baudy [email protected]
Quoted 8 lines · Johann Baudy
Quoted 3 lines
Page itself will live until destruction time, when skb is linearized page is freed, so your own destructor will find that given page is no longer used, and appropriate memory area can be overwritten.
Ok, so if my understanding is correct, fragment data is copied, frags info (page, offset, size ...) of
skb_shinfo(skb)remain available andskb->data_lenis equal to 0 (to tag it as linear).
Then I can use address of skb_shinfo(n)->frags[0].page to derive my header whatever sending path.
No, I was wrong, linearization may silently put pages without calling a destructor.
Signature · 1 line
Evgeniy Polyakov
I've seen that during skb_linearize(), skb_shinfo(skb)->frags[i].page is not lost even if pages has been put. So I can get back original frag address from this info in skb destructor. Can I rely on skb_shinfo(skb)->frags[i].page on more complicated sending paths ... ?
Thanks in advance,
Johann
Quoted 16 lines · Evgeniy Polyakov
Quoted 8 lines · Johann Baudy
Quoted 3 lines
Page itself will live until destruction time, when skb is linearized page is freed, so your own destructor will find that given page is no longer used, and appropriate memory area can be overwritten.
Ok, so if my understanding is correct, fragment data is copied, frags info (page, offset, size ...) of skb_shinfo(skb) remain available and
skb->data_lenis equal to 0 (to tag it as linear).
Then I can use address of skb_shinfo(n)->frags[0].page to derive my header whatever sending path.No, I was wrong, linearization may silently put pages without calling a destructor.
--
Evgeniy Polyakov
Signature · 2 lines
Johann Baudy [email protected]
Quoted 4 lines · Johann Baudy
I've seen that during
skb_linearize(), skb_shinfo(skb)->frags[i].page is not lost even if pages has been put. So I can get back original frag address from this info in skb destructor. Can I rely on skb_shinfo(skb)->frags[i].page on more complicated sending paths ... ?
skb_shinfo(skb)->frags[i] can be overwritten, for example in __pskb_pull_tail() at pull_pages: label.
Signature · 1 line
Evgeniy Polyakov
Where do you see that skb_shinfo(skb)->frags[i].page (not other fields) can be overwritten ?
Thanks,
Johann
Quoted 12 lines · Evgeniy Polyakov
Quoted 4 lines · Johann Baudy
I've seen that during
skb_linearize(), skb_shinfo(skb)->frags[i].page is not lost even if pages has been put. So I can get back original frag address from this info in skb destructor. Can I rely on skb_shinfo(skb)->frags[i].page on more complicated sending paths ... ?skb_shinfo(skb)->frags[i] can be overwritten, for example in
__pskb_pull_tail()atpull_pages: label.--
Evgeniy Polyakov
Signature · 2 lines
Johann Baudy [email protected]
Quoted 2 lines · Johann Baudy
Where do you see that skb_shinfo(skb)->frags[i].page (not other fields) can be overwritten ?
pull_pages:
eat = delta;
k = 0;
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
if (skb_shinfo(skb)->frags[i].size <= eat) {
put_page(skb_shinfo(skb)->frags[i].page);
eat -= skb_shinfo(skb)->frags[i].size;
} else {
skb_shinfo(skb)->frags[k] = skb_shinfo(skb)->frags[i];Signature · 1 line
Evgeniy Polyakov
Thanks Evgeniy,
So if my understanding is correct, there is no way to get original fragment address in destructor using skb fragment page/data.
Then,
I can't use fragments due to skb_linearize
I can't hide pointer into skb data due to skb_copy.
I can't rely on other fields of skb.
IMHO, using skb pointer requires too much cpu resources (parsing headers to identify the right buffer...)
So what can I do except using a new field?
What do you think about adding a new field that is always linked to destructor? I mean adding a generic new field skb->destructor_arg.
Currently if someone want to change destructor, it stores old destructor before substitution; and executes it at the end of new destructor. (ex: dev_gso_skb_destructor(struct sk_buff *skb))
Can we just add same mechanism for a new argument?
If someone needs destructor_arg, it saves the old value somewhere; and restores it before calling old destructor (in the new destructor). This way everybody can forward data to his destructor properly.
Is it conceivable?
Thanks for your help,
Johann
Signature · 2 lines
Johann Baudy [email protected]
Hi Johann.
Quoted 10 lines · Johann Baudy
So if my understanding is correct, there is no way to get original fragment address in destructor using skb fragment page/data.
Then,
I can't use fragments due toskb_linearize
I can't hide pointer into skb data due toskb_copy.
I can't rely on other fields of skb.
IMHO, using skb pointer requires too much cpu resources (parsing headers to identify the right buffer...)
Quoted 3 lines
So what can I do except using a new field?
What do you think about adding a new field that is always linked to destructor? I mean adding a generic new fieldskb->destructor_arg.
It can work but do not place it into skb itself, but into shared area, which is slab allocated and does not suffer from size restrictions. You can also add shared-info destructor, which could be used by other
Signature · 1 line
Evgeniy Polyakov
Thanks for your reply, Evgeniy,
Quoted 2 lines
It can work but do not place it into skb itself, but into shared area, which is slab allocated and does not suffer from size restrictions.
Ok
Quoted 1 line
You can also add shared-info destructor, which could be used by other
Kind of:
struct skb_shared_info {
atomic_t dataref;....
+void (*destructor)(struct sk_buff *skb, void *destructor_arg);
+void * destructor_arg;
};
Should I call this new destructor from skb_release_data() before kfree(skb->head) ?
or in skb_release_head_state()? close to current destructor call
Best regards,
Johann
Signature · 2 lines
Johann Baudy [email protected]
Quoted 8 lines · Johann Baudy
+void (*destructor)(struct
sk_buff*skb, void *destructor_arg);
+void *destructor_arg;
};Should I call this new destructor from
skb_release_data()beforekfree(skb->head)?
or inskb_release_head_state()? close to current destructor call
I think it should only be called when shared area is about to be freed and not when attached skb is freed. Thus you can put your pages into fraglist and reuse them after the last reference to the shared area is gone.
Signature · 1 line
Evgeniy Polyakov
Quoted 2 lines
I think it should only be called when shared area is about to be freed and not when attached skb is freed.
During skb_linearize(). shared info is memcopied and freed in pskb_expand_head() if not enough memory is available in current head. Then, Do you expect this new destructor to be called twice? (during skb_linearize() and kfree_skb())
Quoted 3 lines
Thus you can put your pages into
fraglist and reuse them after the last reference to the shared area is gone.
If my understanding is correct, skb can be linearized without calling skb_release_data() (if enough space is available in head). Hence, In this case, I will not have access to original frags[] from skb_shared_info destructor.
Thanks again,
Johann
Hi Johann.
Quoted 6 lines · Johann Baudy
Quoted 2 lines
I think it should only be called when shared area is about to be freed and not when attached skb is freed.
During
skb_linearize(). shared info is memcopied and freed inpskb_expand_head()if not enough memory is available in current head. Then, Do you expect this new destructor to be called twice? (duringskb_linearize()andkfree_skb())
It cals skb_release_data() which should invoke shared area destructor before being freed.
Quoted 8 lines
Quoted 3 lines
Thus you can put your pages into
fraglist and reuse them after the last reference to the shared area is gone.If my understanding is correct, skb can be linearized without calling
skb_release_data()(if enough space is available in head). Hence, In this case, I will not have access to original frags[] fromskb_shared_infodestructor.
Yes, that's right.
Signature · 1 line
Evgeniy Polyakov
Hi Evgeniy,
Quoted 2 lines
It cals
skb_release_data()which should invoke shared area destructor before being freed.
Yes but in this case, it will be called twice, as shared info (that contains destructor) is memcopied to new expanded head. shared info destructor will be excuted two times on "same" data (during skb_expand_head() and during kfree_skb()) Actually, I want to know if this behavior is intended with a new shared info destructor.
Thanks
Signature · 1 line
Johann Baudy
Quoted 6 lines · Johann Baudy
Yes but in this case, it will be called twice, as shared info (that contains destructor) is memcopied to new expanded head. shared info destructor will be excuted two times on "same" data (during
skb_expand_head()and duringkfree_skb()) Actually, I want to know if this behavior is intended with a new shared info destructor.
It depends on your task... You can always store a pointer in the tree/hash and check it in the destructor, you can play some games in the expand helper not to call destructor: like messing with the reference counter and reusing the old area, or silently freeing old area without destruction invocation (presumably with some new helper).
Moreover you can allocate skb so that no matter what but it could not be reused by the underlying layers, so it could be fully copied. In this case usual destructor is enough.
Signature · 1 line
Evgeniy Polyakov
Quoted 2 lines · Johann Baudy
Do you think this solution is not acceptable?
Not unless you can figure out a way to prevent netfilter from being run on your packet. The two hurdles you have to overcome are act_ipt and bridge netfilter.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txtQuoted 5 lines · Johann Baudy
+++ Transmission process
Documentation/networking/packet_mmap.txt +4
-0+Those defines are also used for transmission: + + #define TP_STATUS_KERNEL 0 // Frame is available + #define TP_STATUS_USER 1 // Frame will be sent on next send()
Documentation/networking/packet_mmap.txt packets are in the ring: · line 480
Quoted 10 lines
Documentation/networking/packet_mmap.txt +10
-0+ #define TP_STATUS_COPY 2 // Frame is currently in transmission + +First, the kernel initializes all frames to TP_STATUS_KERNEL. To send a packet, +the user fills a data buffer of an available frame, sets tp_len to current +data buffer size and sets its status field to TP_STATUS_USER. This can be done +on multiple frames. Once the user is ready to transmit, it calls send(). +Then all buffers with status equal to TP_STATUS_USER are forwarded to the +network device. The kernel updates each status of sent frames with +TP_STATUS_COPY until the end of transfer. +At the end of each transfer, buffer status returns to TP_STATUS_KERNEL.
Could you clean then states up a bit to reflect what they actually mean?
TP_STATUS_AVAILABLE => Frame is availableTP_STATUS_SEND_REQUEST => Frame waits for sendingTP_STATUS_SENDING => Frame is being sent.
Also can you ensure that send() continues to send if I concurrently set the status to TP_STATUS_SEND_REQUEST from another thread? How it is serialized anyways? Status is an atomic value? Or do you rely on status only being modified while send() is running?
Hi Christoph,
Quoted 5 lines
Could you clean then states up a bit to reflect what they actually mean?
TP_STATUS_AVAILABLE=> Frame is availableTP_STATUS_SEND_REQUEST=> Frame waits for sendingTP_STATUS_SENDING=> Frame is being sent.
Ok
Quoted 4 lines
Also can you ensure that
send()continues to send if I concurrently set the status toTP_STATUS_SEND_REQUESTfrom another thread? How it is serialized anyways? Status is an atomic value? Or do you rely on status only being modified whilesend()is running?
TP_STATUS_KERNEL => TP_STATUS_SEND_REQUEST: only performed by user. TP_STATUS_SEND_REQUEST => TP_STATUS_SENDING only performed by kernel TP_STATUS_SENDING => TP_STATUS_KERNEL only performed by kernel.
Only one thread is allowed to change status values from user space. This way, you can take advantage of smp. One thread is filling the buffer changing status from TP_STATUS_KERNEL to TP_STATUS_SEND_REQUEST , another is calling send() in loop (MSG_DONTBLOCK option can be used).
You can also perform filling and send() sequentially
An example can be found at:
http://wiki.gnu-log.net/index.php5?title=Linux_packet_mmap
It can customize almost all parameters, use multi_thread, use DGRAM ...
Thanks,
Johann
Quoted 8 lines
Quoted 6 lines
Could you clean then states up a bit to reflect what they actually mean?
TP_STATUS_AVAILABLE=> Frame is availableTP_STATUS_SEND_REQUEST=> Frame waits for sendingTP_STATUS_SENDING=> Frame is being sent.Ok
Do you suggest to replace previous TP_STATUS_KERNEL with TP_STATUS_AVAILABLE? or Add a new status?
Thanks in advance,
Johann
Quoted 10 lines · Johann Baudy
Quoted 7 lines
Quoted 5 lines
Could you clean then states up a bit to reflect what they actually mean?
TP_STATUS_AVAILABLE=> Frame is availableTP_STATUS_SEND_REQUEST=> Frame waits for sendingTP_STATUS_SENDING=> Frame is being sent.Ok
Do you suggest to replace previous
TP_STATUS_KERNELwithTP_STATUS_AVAILABLE? or Add a new status?
Well yes. But I guess you know best what to call them. Just make them more decriptive of their function.
Quoted 2 lines
Well yes. But I guess you know best what to call them. Just make them more decriptive of their function.
Yes, I understand, but I'm just afraid of backward compatibility with libraries that are currently using packet_mmap for RX ring. If I rename those TP_STATUS_KERNEL into TP_STATUS_AVAILABLE. I will break this compatibility.
(for compilation).
Quoted 8 lines · Johann Baudy
Quoted 2 lines
Well yes. But I guess you know best what to call them. Just make them more decriptive of their function.
Yes, I understand, but I'm just afraid of backward compatibility with libraries that are currently using
packet_mmapfor RX ring. If I rename thoseTP_STATUS_KERNELintoTP_STATUS_AVAILABLE. I will break this compatibility.
(for compilation).
You can do
#define TP_STATUS_AVAILABLE TP_STATUS_KERNEL
Quoted 10 lines · Johann Baudy
Quoted 4 lines
Also can you ensure that
send()continues to send if I concurrently set the status toTP_STATUS_SEND_REQUESTfrom another thread? How it is serialized anyways? Status is an atomic value? Or do you rely on status only being modified whilesend()is running?
TP_STATUS_KERNEL=>TP_STATUS_SEND_REQUEST: only performed by user.TP_STATUS_SEND_REQUEST=>TP_STATUS_SENDINGonly performed by kernelTP_STATUS_SENDING=>TP_STATUS_KERNELonly performed by kernel.Only one thread is allowed to change status values from user space.
Duh. So I cannot concurrently operate with multiple threads on the structure.
The kernel synchronizes with itself via the socket?
Quoted 4 lines
This way, you can take advantage of smp. One thread is filling the buffer changing status from
TP_STATUS_KERNELtoTP_STATUS_SEND_REQUEST, another is callingsend()in loop (MSG_DONTBLOCKoption can be used).
Ah ok.
Quoted 2 lines
Duh. So I cannot concurrently operate with multiple threads on the structure.
Except if you can ensure that multiple threads will not use the same buffer.
Quoted 1 line
The kernel synchronizes with itself via the socket?
Not sure to understand?
Both below rules + cache/mem flushing synchronize User and Kernel spaces TP_STATUS_KERNEL => TP_STATUS_SEND_REQUEST: only performed by user. TP_STATUS_SENDING => TP_STATUS_KERNEL only performed by kernel.
Quoted 3 lines · Johann Baudy
Quoted 2 lines
Duh. So I cannot concurrently operate with multiple threads on the structure.
Except if you can ensure that multiple threads will not use the same buffer.
Hmmm... Ok.
Quoted 2 lines
Quoted 1 line
The kernel synchronizes with itself via the socket?
Not sure to understand?
Doesnt the kernel need some way to synchronize the two transitions made by the kernel? From "send request" to "sending" and then to "available"? Otherwise the kernel may set the request to "available" before the change from "send request" to "sending" is complete.
Quoted 4 lines
Doesnt the kernel need some way to synchronize the two transitions made by the kernel? From "send request" to "sending" and then to "available"? Otherwise the kernel may set the request to "available" before the change from "send request" to "sending" is complete.
This synchronization is performed through skb mechanism. "send request" to "sending" is done before submitting skb to device. "sending" to "available" will be performed in skb destructor.