Skip to contentSkip to conversation

i386 vs x86_64 struct tpacket_hdr layout

DISCUSSION Norbert van Bolhuis 4 participants 17 messages Dec 20, 2013 – Jan 8, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I'm compiling my 32bit application with -m32 on a x86_64 system/kernel.

Things aren't working because for my application tp_len is at offset 4 but for the kernel it is at offset 8.

struct tpacket_hdr {
        unsigned long   tp_status;
        unsigned int    tp_len;
        unsigned int    tp_snaplen;
        unsigned short  tp_mac;
        unsigned short  tp_net;
        unsigned int    tp_sec;
        unsigned int    tp_usec;
};

How is this suppose to work ?

thanks,
Norbert van Bolhuis

maintainer
#1 lore raw

From: Norbert van Bolhuis <[email protected]>
Date: Fri, 20 Dec 2013 14:30:06 +0100

Quoted 17 lines

I'm compiling my 32bit application with -m32 on a x86_64 system/kernel.

Things aren't working because for my application tp_len is at offset 4 but for the kernel it is at offset 8.

struct tpacket_hdr {
        unsigned long   tp_status;
        unsigned int    tp_len;
        unsigned int    tp_snaplen;
        unsigned short  tp_mac;
        unsigned short  tp_net;
        unsigned int    tp_sec;
        unsigned int    tp_usec;
};
Quoted 1 line

How is this suppose to work ?

This is why you should use tpacket layout v2 or v3, rather than v1, they fix these issues.

#2 lore raw
Quoted 20 lines · David Miller

From: Norbert van Bolhuis <[email protected]>
Date: Fri, 20 Dec 2013 14:30:06 +0100

Quoted 17 lines

I'm compiling my 32bit application with -m32 on a x86_64 system/kernel.

Things aren't working because for my application tp_len is at offset 4 but for the kernel it is at offset 8.

struct tpacket_hdr {
         unsigned long   tp_status;
         unsigned int    tp_len;
         unsigned int    tp_snaplen;
         unsigned short  tp_mac;
         unsigned short  tp_net;
         unsigned int    tp_sec;
         unsigned int    tp_usec;
};
Quoted 4 lines
Quoted 1 line

How is this suppose to work ?

This is why you should use tpacket layout v2 or v3, rather than v1, they fix these issues.

Norbert, please also read Documentation/networking/packet_mmap.txt

Thanks !

#3 lore raw
Quoted 21 lines · Daniel Borkmann
Quoted 20 lines · David Miller

From: Norbert van Bolhuis <[email protected]>
Date: Fri, 20 Dec 2013 14:30:06 +0100

Quoted 17 lines

I'm compiling my 32bit application with -m32 on a x86_64 system/kernel.

Things aren't working because for my application tp_len is at offset 4 but for the kernel it is at offset 8.

struct tpacket_hdr {
unsigned long tp_status;
unsigned int tp_len;
unsigned int tp_snaplen;
unsigned short tp_mac;
unsigned short tp_net;
unsigned int tp_sec;
unsigned int tp_usec;
};

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

How is this suppose to work ?

This is why you should use tpacket layout v2 or v3, rather than v1, they fix these issues.

Norbert, please also read Documentation/networking/packet_mmap.txt

Thanks !

Ah... thanks guys, now I see. struct tpacket2_hdr has fixed layout. I will use v2 (or v3).

I could not find this info in [linux-3.6.8/]Documentation/networking/packet_mmap.txt though.

---
Norbert.

#4 lore raw

single process receives own frames due to PACKET_MMAP

In reply to Norbert van Bolhuis

Our application uses raw AF_PACKET socket to send and receive on one particular ethernet interface.

Recently we started using PACKET_MMAP (TPACKET_V2). This makes the Appl use a TX socket and a RX socket.
Both sockets are bound to the same (eth) interface. I noticed the RX socket receives all frames that are sent via the TX socket (same process, different thread). This I do not want.

I know it is supposed to happen for different processes (otherwise wireshark won't work), but I did not expect it to happen for one single process (with different threads).

I can filter them out in user-space (PACKET_OUTGOING) or via kernel packet filter (SO_ATTACH_FILTER), but performance is critical.

I wonder whether this (PACKET_MMAP) behaviour is OK.

It did not happen before (with a non-PACKET_MMAP AF_PACKET socket which was used by both threads of the same Appl process). So why is it happening now ?
I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

If I'm not doing something wrong, this means this behaviour causes my CPU to be loaded much more (since all transmitted frames have to be filtered out).

Let me know what you think.

thanks,
Norbert van Bolhuis

#5 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 14 lines · Norbert van Bolhuis

Our application uses raw AF_PACKET socket to send and receive on one particular ethernet interface.

Recently we started using PACKET_MMAP (TPACKET_V2). This makes the Appl use a TX socket and a RX socket.
Both sockets are bound to the same (eth) interface. I noticed the RX socket receives all frames that are sent via the TX socket (same process, different thread). This I do not want.

I know it is supposed to happen for different processes (otherwise wireshark won't work), but I did not expect it to happen for one single process (with different threads).

Quoted 5 lines

I can filter them out in user-space (PACKET_OUTGOING) or via kernel packet filter (SO_ATTACH_FILTER), but performance is critical.

I wonder whether this (PACKET_MMAP) behaviour is OK.

For your use-case, we recently introduced in d346a3fae3ff1 ("packet: introduce PACKET_QDISC_BYPASS socket option") a bypass of dev_queue_xmit() (that internally invokes dev_queue_xmit_nit()).

Quoted 3 lines

It did not happen before (with a non-PACKET_MMAP AF_PACKET socket which was used by both threads of the same Appl process). So why is it happening now ?

Can you elaborate a bit on which kernel versions that behaviour changed?

Quoted 16 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

If I'm not doing something wrong, this means this behaviour causes my CPU to be loaded much more (since all transmitted frames have to be filtered out).

Let me know what you think.

thanks,
Norbert van Bolhuis

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 26 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann <[email protected]> wrote:
Quoted 14 lines · Norbert van Bolhuis

Our application uses raw AF_PACKET socket to send and receive on one particular ethernet interface.

Recently we started using PACKET_MMAP (TPACKET_V2). This makes the Appl use a TX socket and a RX socket.
Both sockets are bound to the same (eth) interface. I noticed the RX socket receives all frames that are sent via the TX socket (same process, different thread). This I do not want.

I know it is supposed to happen for different processes (otherwise wireshark won't work), but I did not expect it to happen for one single process (with different threads).

Quoted 5 lines

I can filter them out in user-space (PACKET_OUTGOING) or via kernel packet filter (SO_ATTACH_FILTER), but performance is critical.

I wonder whether this (PACKET_MMAP) behaviour is OK.

For your use-case, we recently introduced in d346a3fae3ff1 ("packet: introduce PACKET_QDISC_BYPASS socket option") a bypass of dev_queue_xmit() (that internally invokes dev_queue_xmit_nit()).

Quoted 9 lines
Quoted 3 lines

It did not happen before (with a non-PACKET_MMAP AF_PACKET socket which was used by both threads of the same Appl process). So why is it happening now ?

Can you elaborate a bit on which kernel versions that behaviour changed?

Quoted 2 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

Quoted 8 lines
Quoted 8 lines

If I'm not doing something wrong, this means this behaviour causes my CPU to be loaded much more (since all transmitted frames have to be filtered out).

Let me know what you think.

thanks,
Norbert van Bolhuis

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#7 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 29 lines · Jesper Dangaard Brouer
Quoted 26 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 14 lines · Norbert van Bolhuis

Our application uses raw AF_PACKET socket to send and receive on one particular ethernet interface.

Recently we started using PACKET_MMAP (TPACKET_V2). This makes the Appl use a TX socket and a RX socket.
Both sockets are bound to the same (eth) interface. I noticed the RX socket receives all frames that are sent via the TX socket (same process, different thread). This I do not want.

I know it is supposed to happen for different processes (otherwise wireshark won't work), but I did not expect it to happen for one single process (with different threads).

Quoted 5 lines

I can filter them out in user-space (PACKET_OUTGOING) or via kernel packet filter (SO_ATTACH_FILTER), but performance is critical.

I wonder whether this (PACKET_MMAP) behaviour is OK.

For your use-case, we recently introduced in d346a3fae3ff1 ("packet: introduce PACKET_QDISC_BYPASS socket option") a bypass of dev_queue_xmit() (that internally invokes dev_queue_xmit_nit()).

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

It did not happen before (with a non-PACKET_MMAP AF_PACKET socket which was used by both threads of the same Appl process). So why is it happening now ?

Can you elaborate a bit on which kernel versions that behaviour changed?

Sorry, I wasn't very clear. This does not regard kernel (versions). With "before" I mean our previous version of our application which used an AF_PACKET socket, but not the PACKET_MMAP option.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that. In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

Thanks a lot for your answers.

---
Norbert

#8 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 33 lines · Norbert van Bolhuis
Quoted 32 lines · Jesper Dangaard Brouer
Quoted 30 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 14 lines · Norbert van Bolhuis

Our application uses raw AF_PACKET socket to send and receive on one particular ethernet interface.

Recently we started using PACKET_MMAP (TPACKET_V2). This makes the Appl use a TX socket and a RX socket.
Both sockets are bound to the same (eth) interface. I noticed the RX socket receives all frames that are sent via the TX socket (same process, different thread). This I do not want.

I know it is supposed to happen for different processes (otherwise wireshark won't work), but I did not expect it to happen for one single process (with different threads).

Quoted 5 lines

I can filter them out in user-space (PACKET_OUTGOING) or via kernel packet filter (SO_ATTACH_FILTER), but performance is critical.

I wonder whether this (PACKET_MMAP) behaviour is OK.

For your use-case, we recently introduced in d346a3fae3ff1 ("packet: introduce PACKET_QDISC_BYPASS socket option") a bypass of dev_queue_xmit() (that internally invokes dev_queue_xmit_nit()).

Quoted 3 lines

It did not happen before (with a non-PACKET_MMAP AF_PACKET socket which was used by both threads of the same Appl process). So why is it happening now ?

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

Can you elaborate a bit on which kernel versions that behaviour changed?

Sorry, I wasn't very clear. This does not regard kernel (versions). With "before" I mean our previous version of our application which used an AF_PACKET socket, but not the PACKET_MMAP option.

Hmm, taking a look into 2.6.35.14, it doesn't seem to be so much of a difference.

W/o seeing how you've invoked mmap vs non-mmap API, it's hard to say. Could you try with latest upstream resp. net-next kernel?

Quoted 18 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

Quoted 7 lines

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that. In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Yep, kernel is definitely too old for that.

Quoted 10 lines

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

Thanks a lot for your answers.

---
Norbert
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw

Re: single process receives own frames due to PACKET_MMAP

In reply to Norbert van Bolhuis
Quoted 6 lines · On Tue, 07 Jan 2014 14:16:03 +0100 Norbert van Bolhuis…
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 2 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 1 line · Norbert van Bolhuis

[...]

Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

Quoted 8 lines

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that.

The easiest way is to:
cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at the result with "perf report") for a lock contention, and "expand" the spin_lock and see if packet_rcv() is calling this spin lock.

Quoted 2 lines

In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Be careful not to start a wireshark/tcpdump, at the sametime, as this will slow you down.

Quoted 1 line

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not catch these packets, remember that.

Signature · 5 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer
#10 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 36 lines · Jesper Dangaard Brouer
Quoted 6 lines · On Tue, 07 Jan 2014 14:16:03 +0100 Norbert van Bolhuis<[email protected]>…
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 2 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 1 line · Norbert van Bolhuis

[...]

Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:

   ring->s_ll.sll_protocol = 0
Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:

   https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:

   https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
Quoted 8 lines

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that.

Quoted 8 lines

The easiest way is to:
cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at the result with "perf report") for a lock contention, and "expand" the spin_lock and see if packet_rcv() is calling this spin lock.

I checked the easy way.
Even on my old FC14 system the "protocol=0 patch" seems to make a difference for trafgen.
Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in /proc/net/ptype.
With the patch I see no additional "packet_rcv entry".

It could be my Appl is wrong or maybe the "protocol=0 patch" does not help. I think the latter, afterall my Appl has, unlike trafgen, another RX (AF_PACKET) socket.

Quoted 4 lines
Quoted 3 lines

In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Quoted 8 lines

Be careful not to start a wireshark/tcpdump, at the sametime, as this will slow you down.

Quoted 1 line

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not catch these packets, remember that.

Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.

So I will try my Appl with the latest net-next kernel to see if that makes it work. Hopefully I can find some time in the next coming days, I will keep you informed.

---
Norbert

#11 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 46 lines · Norbert van Bolhuis
Quoted 35 lines · Jesper Dangaard Brouer
Quoted 6 lines · On Tue, 07 Jan 2014 14:16:03 +0100 Norbert van Bolhuis<[email protected]>…
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 2 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 1 line · Norbert van Bolhuis

[...]

Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:

   ring->s_ll.sll_protocol = 0
Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:

   https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:

   https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1
Quoted 8 lines

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that.

Quoted 8 lines

The easiest way is to:
cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at the result with "perf report") for a lock contention, and "expand" the spin_lock and see if packet_rcv() is calling this spin lock.

Quoted 6 lines

I checked the easy way.
Even on my old FC14 system the "protocol=0 patch" seems to make a difference for trafgen.
Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in /proc/net/ptype.
With the patch I see no additional "packet_rcv entry".

Yes, that is expected behaviour. ;-) See more below.

Quoted 19 lines

It could be my Appl is wrong or maybe the "protocol=0 patch" does not help. I think the latter, afterall my Appl has, unlike trafgen, another RX (AF_PACKET) socket.

Quoted 4 lines
Quoted 3 lines

In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Quoted 8 lines

Be careful not to start a wireshark/tcpdump, at the sametime, as this will slow you down.

Quoted 1 line

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not catch these packets, remember that.

Quoted 5 lines

Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.

So I will try my Appl with the latest net-next kernel to see if that makes it work. Hopefully I can find some time in the next coming days, I will keep you informed.

As long as there's at least one single PF_PACKET receive socket open and you do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.

If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then packets generated by that socket will not hit the dev_queue_xmit_nit() path back to other possible rx listeners that are present on your system (w/ the side-effects for tx as described in [1]).

[1] Documentation/networking/packet_mmap.txt +960

Quoted 2 lines

---
Norbert

#12 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 85 lines · Daniel Borkmann
Quoted 46 lines · Norbert van Bolhuis
Quoted 35 lines · Jesper Dangaard Brouer
Quoted 6 lines · On Tue, 07 Jan 2014 14:16:03 +0100 Norbert van Bolhuis<[email protected]>…
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 2 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 1 line · Norbert van Bolhuis

[...]

Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

Quoted 8 lines

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that.

Quoted 8 lines

The easiest way is to:
cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at the result with "perf report") for a lock contention, and "expand" the spin_lock and see if packet_rcv() is calling this spin lock.

Quoted 6 lines

I checked the easy way.
Even on my old FC14 system the "protocol=0 patch" seems to make a difference for trafgen.
Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in /proc/net/ptype.
With the patch I see no additional "packet_rcv entry".

Yes, that is expected behaviour. ;-) See more below.

Quoted 19 lines

It could be my Appl is wrong or maybe the "protocol=0 patch" does not help. I think the latter, afterall my Appl has, unlike trafgen, another RX (AF_PACKET) socket.

Quoted 4 lines
Quoted 3 lines

In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Quoted 8 lines

Be careful not to start a wireshark/tcpdump, at the sametime, as this will slow you down.

Quoted 1 line

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not catch these packets, remember that.

Quoted 5 lines

Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.

So I will try my Appl with the latest net-next kernel to see if that makes it work. Hopefully I can find some time in the next coming days, I will keep you informed.

As long as there's at least one single PF_PACKET receive socket open and you do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.

Quoted 7 lines

If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then packets generated by that socket will not hit the dev_queue_xmit_nit() path back to other possible rx listeners that are present on your system (w/ the side-effects for tx as described in [1]).

[1] Documentation/networking/packet_mmap.txt +960

Ok, that's clear.

But this means my PF_PACKET socket application performs worse because of using PACKET_MMAP. I expected the opposit.

Afterall my old PF_PACKET socket application (which does not use PACKET_MMAP) uses only one PF_PACKET socket (for TX and RX). Because packets are never sent back to the socket they originated from, my old PF_PACKET socket application performs better.

Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

---
Norbert

#13 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 100 lines · Norbert van Bolhuis
Quoted 85 lines · Daniel Borkmann
Quoted 46 lines · Norbert van Bolhuis
Quoted 35 lines · Jesper Dangaard Brouer
Quoted 6 lines · On Tue, 07 Jan 2014 14:16:03 +0100 Norbert van Bolhuis<[email protected]>…
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 2 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 1 line · Norbert van Bolhuis

[...]

Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

Quoted 8 lines

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that.

Quoted 8 lines

The easiest way is to:
cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at the result with "perf report") for a lock contention, and "expand" the spin_lock and see if packet_rcv() is calling this spin lock.

Quoted 6 lines

I checked the easy way.
Even on my old FC14 system the "protocol=0 patch" seems to make a difference for trafgen.
Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in /proc/net/ptype.
With the patch I see no additional "packet_rcv entry".

Yes, that is expected behaviour. ;-) See more below.

Quoted 19 lines

It could be my Appl is wrong or maybe the "protocol=0 patch" does not help. I think the latter, afterall my Appl has, unlike trafgen, another RX (AF_PACKET) socket.

Quoted 4 lines
Quoted 3 lines

In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Quoted 8 lines

Be careful not to start a wireshark/tcpdump, at the sametime, as this will slow you down.

Quoted 1 line

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not catch these packets, remember that.

Quoted 5 lines

Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.

So I will try my Appl with the latest net-next kernel to see if that makes it work. Hopefully I can find some time in the next coming days, I will keep you informed.

As long as there's at least one single PF_PACKET receive socket open and you do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.

Quoted 7 lines

If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then packets generated by that socket will not hit the dev_queue_xmit_nit() path back to other possible rx listeners that are present on your system (w/ the side-effects for tx as described in [1]).

[1] Documentation/networking/packet_mmap.txt +960

Ok, that's clear.

But this means my PF_PACKET socket application performs worse because of using PACKET_MMAP. I expected the opposit.

Quoted 6 lines

Afterall my old PF_PACKET socket application (which does not use PACKET_MMAP) uses only one PF_PACKET socket (for TX and RX). Because packets are never sent back to the socket they originated from, my old PF_PACKET socket application performs better.

Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

Yep:

http://thread.gmane.org/gmane.linux.network/269129/focus=269188

Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt I think could be useful for others as well.

#14 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 106 lines · Daniel Borkmann
Quoted 105 lines · Norbert van Bolhuis
Quoted 85 lines · Daniel Borkmann
Quoted 46 lines · Norbert van Bolhuis
Quoted 35 lines · Jesper Dangaard Brouer
Quoted 6 lines · On Tue, 07 Jan 2014 14:16:03 +0100 Norbert van Bolhuis<[email protected]>…
Quoted 5 lines · Jesper Dangaard Brouer
Quoted 2 lines · On Tue, 07 Jan 2014 10:32:01 +0100 Daniel Borkmann<[email protected]> wrote:
Quoted 1 line · Norbert van Bolhuis

[...]

Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

I'd say it makes no sense to make the same process receive its own transmitted frames on that same interface (unless its lo).

Have you setup:
ring->s_ll.sll_protocol = 0

Quoted 8 lines

This is what I did in trafgen to avoid this problem.

See line 55 in netsniff-ng/ring.c:
https://github.com/borkmann/netsniff-ng/blob/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1/ring.c#L55

Commit:
https://github.com/borkmann/netsniff-ng/commit/c3602a995b21e8133c7f4fd1fb1e7e21b6a844f1

Quoted 8 lines

No I did not do that, I was checking my code against netsniff-ng-0.5.8-rc4.

But I just tried it, I believe I do the same as netsniff-ng-0.5.8-rc5, but it doesn't work for me. Maybe because I have an old FC14 system (kernel 2.6.35.14-106.fc14.x86_64).

So I tried to see whether netsniff-ng-0.5.8-rc5/trafgen still makes the kernel call packet_rcv() on my FC14 system. So I build and run it, but I'm not sure how to (easily) check that.

Quoted 8 lines

The easiest way is to:
cat /proc/net/ptype
And look if someone registered a proto handler/function: packet_rcv (or tpacket_rcv).

The more exact method is, to run "perf record -a -g" and then look (at the result with "perf report") for a lock contention, and "expand" the spin_lock and see if packet_rcv() is calling this spin lock.

Quoted 6 lines

I checked the easy way.
Even on my old FC14 system the "protocol=0 patch" seems to make a difference for trafgen.
Without the patch I see for each CPU in use by trafgen a "packet_rcv entry" in /proc/net/ptype.
With the patch I see no additional "packet_rcv entry".

Yes, that is expected behaviour. ;-) See more below.

Quoted 19 lines

It could be my Appl is wrong or maybe the "protocol=0 patch" does not help. I think the latter, afterall my Appl has, unlike trafgen, another RX (AF_PACKET) socket.

Quoted 4 lines
Quoted 3 lines

In anyway, Wireshark does capture the trafgen generated frames, does that say anything ?

Quoted 8 lines

Be careful not to start a wireshark/tcpdump, at the sametime, as this will slow you down.

Quoted 1 line

In the future, I can at least use PACKET_QDISC_BYPASS as a "workaround".

And in the future with PACKET_QDISC_BYPASS, your wireshark will not catch these packets, remember that.

Quoted 5 lines

Yes, this is why I would love to see the "protocol=0 patch" work for my Appl.

So I will try my Appl with the latest net-next kernel to see if that makes it work. Hopefully I can find some time in the next coming days, I will keep you informed.

As long as there's at least one single PF_PACKET receive socket open and you do not make use of PACKET_QDISC_BYPASS on your tx socket, then those packets go back the dev_queue_xmit_nit() path, even if your tx socket uses protocol=0.

Quoted 7 lines

If you make use of PACKET_QDISC_BYPASS [1] for your particular tx socket, then packets generated by that socket will not hit the dev_queue_xmit_nit() path back to other possible rx listeners that are present on your system (w/ the side-effects for tx as described in [1]).

[1] Documentation/networking/packet_mmap.txt +960

Ok, that's clear.

But this means my PF_PACKET socket application performs worse because of using PACKET_MMAP. I expected the opposit.

Afterall my old PF_PACKET socket application (which does not use PACKET_MMAP) uses only one PF_PACKET socket (for TX and RX). Because packets are never sent back to the socket they originated from, my old PF_PACKET socket application performs better.

Quoted 8 lines
Quoted 1 line

Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

Yep:

http://thread.gmane.org/gmane.linux.network/269129/focus=269188

Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt I think could be useful for others as well.

Good, it all works fine now, though performance is still not as good as I'd hoped.

I will sent a doc patch soon.

thanks for all help!

---
Norbert.

#15 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 11 lines · Norbert van Bolhuis
Quoted 8 lines
Quoted 1 line

Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

Yep:

http://thread.gmane.org/gmane.linux.network/269129/focus=269188

Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt I think could be useful for others as well.

Good, it all works fine now, though performance is still not as good as I'd hoped.

Does your use case allow for TPACKET_V3 or for using fanout ?

Quoted 1 line

I will sent a doc patch soon.

Sounds great, thanks!

Quoted 1 line

thanks for all help!

#16 lore raw

Re: single process receives own frames due to PACKET_MMAP

Quoted 8 lines · Daniel Borkmann
Quoted 6 lines · Norbert van Bolhuis
Quoted 6 lines
Quoted 1 line

Is there a way to use one PF_PACKET socket for both TX and RX and use PACKET_MMAP ?

Yep:

http://thread.gmane.org/gmane.linux.network/269129/focus=269188

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Feel free to make a patch and add this to Documentation/networking/packet_mmap.txt I think could be useful for others as well.

Good, it all works fine now, though performance is still not as good as I'd hoped.

Does your use case allow for TPACKET_V3 or for using fanout ?

no, unfortunately not.

Though currently our performance is limited by application/system stuff, so at the moment it doesn't really matter.

--
Norbert