Skip to contentSkip to conversation

How to do TCP tx checksums

DISCUSSION Ben Greear 5 participants 10 messages Nov 17, 2015

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Ben Greear

author
#0 lore raw

Hello!

I'm hacking on (my already hacked) pktgen, trying to get it to send TCP frames.

And, having issues getting checksums to work properly.

I'm trying this:

	struct iphdr *iph = ip_hdr(skb);
	struct net_device *odev = pkt_dev->odev;

	if (pkt_dev->flags & F_TCP) {
		if (odev->features & NETIF_F_V4_CSUM) {
			skb->ip_summed = CHECKSUM_PARTIAL;
		} else {
			skb->ip_summed = CHECKSUM_NONE;
		}
		skb->csum = 0;
		__tcp_v4_send_check(skb, iph->saddr, iph->daddr);

I added an export so I could call that __tcp_v4_send_check method w/out having to put a fake socket struct on the stack.

But, the receiving NIC reports 100% checksum failure, so obviously I'm not doing it correct.

Any suggestions on what I might be doing wrong or the proper method(s) to call?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#1 lore raw
Quoted 20 lines · Ben Greear

Hello!

I'm hacking on (my already hacked) pktgen, trying to get it to send TCP frames.

And, having issues getting checksums to work properly.

I'm trying this:

        struct iphdr *iph = ip_hdr(skb);
        struct net_device *odev = pkt_dev->odev;

        if (pkt_dev->flags & F_TCP) {
                if (odev->features & NETIF_F_V4_CSUM) {
                        skb->ip_summed = CHECKSUM_PARTIAL;
                } else {
                        skb->ip_summed = CHECKSUM_NONE;
                }
                skb->csum = 0;
                __tcp_v4_send_check(skb, iph->saddr, iph->daddr);
Quoted 9 lines

I added an export so I could call that __tcp_v4_send_check method w/out having to put a fake socket struct on the stack.

But, the receiving NIC reports 100% checksum failure, so obviously I'm not doing it correct.

Any suggestions on what I might be doing wrong or the proper method(s) to call?

It's really hard to tell what is happening without seeing the full patch your using. Maybe you're not setting the TCP correctly or transport header is not set right in skb.

Tom

Quoted 12 lines

Thanks,
Ben

--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 3 lines · Tom Herbert

It's really hard to tell what is happening without seeing the full patch your using. Maybe you're not setting the TCP correctly or transport header is not set right in skb.

BTW, using trafgen from netsniff-ng would save you Ben lots of headaches.

No need for kernel patches...

author
#3 lore raw
In reply to Tom Herbert
Quoted 27 lines · Tom Herbert
Quoted 26 lines · Ben Greear

Hello!

I'm hacking on (my already hacked) pktgen, trying to get it to send TCP frames.

And, having issues getting checksums to work properly.

I'm trying this:

         struct iphdr *iph = ip_hdr(skb);
         struct net_device *odev = pkt_dev->odev;

         if (pkt_dev->flags & F_TCP) {
                 if (odev->features & NETIF_F_V4_CSUM) {
                         skb->ip_summed = CHECKSUM_PARTIAL;
                 } else {
                         skb->ip_summed = CHECKSUM_NONE;
                 }
                 skb->csum = 0;
                 __tcp_v4_send_check(skb, iph->saddr, iph->daddr);

I added an export so I could call that __tcp_v4_send_check method w/out having to put a fake socket struct on the stack.

But, the receiving NIC reports 100% checksum failure, so obviously I'm not doing it correct.

Quoted 6 lines
Quoted 3 lines

Any suggestions on what I might be doing wrong or the proper method(s) to call?

It's really hard to tell what is happening without seeing the full patch your using. Maybe you're not setting the TCP correctly or transport header is not set right in skb.

Unfortunately, my pktgen has diverged significantly from upstream, so a patch is probably not so useful.

The full pktgen.[ch] files are here:

http://www.candelatech.com/~greearb/patches/

If you look for 'F_TCP' you can find the pertinent bits that I'm mucking with. I have changed it a bit since I last posted the patch above...but still the receiving NIC is reporting 100% csum failure, so I must still be doing something wrong.

I have also tweaked how UDP is done, but that seems to be working.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
author
#4 lore raw
In reply to Tom Herbert
Quoted 27 lines · Tom Herbert
Quoted 26 lines · Ben Greear

Hello!

I'm hacking on (my already hacked) pktgen, trying to get it to send TCP frames.

And, having issues getting checksums to work properly.

I'm trying this:

         struct iphdr *iph = ip_hdr(skb);
         struct net_device *odev = pkt_dev->odev;

         if (pkt_dev->flags & F_TCP) {
                 if (odev->features & NETIF_F_V4_CSUM) {
                         skb->ip_summed = CHECKSUM_PARTIAL;
                 } else {
                         skb->ip_summed = CHECKSUM_NONE;
                 }
                 skb->csum = 0;
                 __tcp_v4_send_check(skb, iph->saddr, iph->daddr);

I added an export so I could call that __tcp_v4_send_check method w/out having to put a fake socket struct on the stack.

But, the receiving NIC reports 100% checksum failure, so obviously I'm not doing it correct.

Quoted 6 lines
Quoted 3 lines

Any suggestions on what I might be doing wrong or the proper method(s) to call?

It's really hard to tell what is happening without seeing the full patch your using. Maybe you're not setting the TCP correctly or transport header is not set right in skb.

And in case it helps, here is the output from dmesg.

This is an Ethernet frame, and in this case csum-start is the index of the first byte of the TCP header as far as I can tell.

[64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#5 lore raw
Quoted 7 lines · Ben Greear

And in case it helps, here is the output from dmesg.

This is an Ethernet frame, and in this case csum-start is the index of the first byte of the TCP header as far as I can tell.

[64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514

You forgot to clear tcp header @check before calling __tcp_v4_send_check(), I think.

( th->check = 0; )

Anyway, just use trafgen and get ~6 Mpps, more easily than with pktgen.

author
#6 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Ben Greear

And in case it helps, here is the output from dmesg.

This is an Ethernet frame, and in this case csum-start is the index of the first byte of the TCP header as far as I can tell.

Quoted 7 lines
Quoted 2 lines
[64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514

You forgot to clear tcp header @check before calling __tcp_v4_send_check(), I think.

( th->check = 0; )

From printk I added to e1000e, it looks like I have bad offsets somehow. I'll try clearing check as well, however.

Quoted 1 line

Anyway, just use trafgen and get ~6 Mpps, more easily than with pktgen.

I do plan to look at this...but need to make sure I can get similar features before I ditch pktgen. It's not just raw throughput that is of interest to me, but also randomizing, sequential, etc header fields to test various network equipment (routers, MAC tables in switches, etc).

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#7 lore raw
In reply to Ben Greear
Quoted 19 lines · Ben Greear
Quoted 7 lines · Eric Dumazet
Quoted 5 lines · Ben Greear

And in case it helps, here is the output from dmesg.

This is an Ethernet frame, and in this case csum-start is the index of the first byte of the TCP header as far as I can tell.

Quoted 7 lines
Quoted 2 lines
[64967.194251] check: 0xf51b  csum-start: 34  offset: 16 summed: 0x3  saddr: 0x201010a  daddr: 0x101010a len: 1514

You forgot to clear tcp header @check before calling __tcp_v4_send_check(), I think.

( th->check = 0; )

From printk I added to e1000e, it looks like I have bad offsets
somehow. I'll try clearing check as well, however.

Quoted 6 lines
Quoted 1 line

Anyway, just use trafgen and get ~6 Mpps, more easily than with pktgen.

I do plan to look at this...but need to make sure I can get similar features before I ditch pktgen. It's not just raw throughput that is of interest to me, but also randomizing, sequential, etc header fields to test various network equipment (routers, MAC tables in switches, etc).

trafgen has a number of these things you mentioned. It started out as a hobby project longer time ago as a simple way to tx raw frames, which was needed to test some kernel project back in the days. Of course, nothing is perfect and patches are always welcome. And if we can also get AF_PACKET into a better shape by this, even better. ;)

author
#9 lore raw
In reply to Tom Herbert
Quoted 27 lines · Tom Herbert
Quoted 26 lines · Ben Greear

Hello!

I'm hacking on (my already hacked) pktgen, trying to get it to send TCP frames.

And, having issues getting checksums to work properly.

I'm trying this:

         struct iphdr *iph = ip_hdr(skb);
         struct net_device *odev = pkt_dev->odev;

         if (pkt_dev->flags & F_TCP) {
                 if (odev->features & NETIF_F_V4_CSUM) {
                         skb->ip_summed = CHECKSUM_PARTIAL;
                 } else {
                         skb->ip_summed = CHECKSUM_NONE;
                 }
                 skb->csum = 0;
                 __tcp_v4_send_check(skb, iph->saddr, iph->daddr);

I added an export so I could call that __tcp_v4_send_check method w/out having to put a fake socket struct on the stack.

But, the receiving NIC reports 100% checksum failure, so obviously I'm not doing it correct.

Quoted 6 lines
Quoted 3 lines

Any suggestions on what I might be doing wrong or the proper method(s) to call?

It's really hard to tell what is happening without seeing the full patch your using. Maybe you're not setting the TCP correctly or transport header is not set right in skb.

Ok, I have this working now.. at least in my current setup.

Main problem is that TCP stack uses skb->len before the other transport headers are counted in skb->len, it appears.

So, in pktgen I need to subtract out the pkt prefix. This at least makes the hw-accel path work...not sure about software csum path yet...

static void pg_do_csum(struct pktgen_dev *pkt_dev, struct sk_buff *skb) {
	struct iphdr *iph = ip_hdr(skb);
	struct udphdr *uh;
	struct net_device *odev = pkt_dev->odev;

	if (pkt_dev->flags & F_TCP) {
		struct tcphdr *th = tcp_hdr(skb);
		unsigned int prefix_len = (unsigned int)((unsigned char*)th - skb->data);

		if (odev->features & NETIF_F_V4_CSUM) {
			skb->ip_summed = CHECKSUM_PARTIAL;
			/* Subtract out IP hdr and before */
			th->check = ~tcp_v4_check(skb->len - prefix_len, iph->saddr, iph->daddr, 0);
			skb->csum_start = skb_transport_header(skb) - skb->head;
			skb->csum_offset = offsetof(struct tcphdr, check);
		} else {
			skb->ip_summed = CHECKSUM_NONE;
			th->check = 0;
			skb->csum = 0;
			th->check = tcp_v4_check(skb->len - prefix_len, iph->saddr, iph->daddr,
						 csum_partial(th, th->doff << 2, skb->csum));
		}
	} else {

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com