pktgen patch available for perusal.

I've completed the first pass of my changes to pktgen in 2.6.18. Many of these features are probably DOA based on previous conversations, but perhaps this will help someone....

Changes:
* use a nano-second timer based on the scheduler timer (TSC) for relative times, instead of get_time_of_day.
* Don't busy-spin:
* add hook to netdevice wake tx queue to re-start when queue was stopped.
* add logic to pktgen to register/unregister this hook.
* accumulate delay and sleep interruptible on a queue when 1ms of sleep has accumulated or when the associated netdev tx-queues for pktgen devices needing to transmit are asleep.
(This all assumes that HZ is 1000, could probably work with other HZ with minor work.)
* Add receive logic to dev.c so that pktgen can receive packets and report on various stats including latency, packet loss, jitter, etc.
* Remove the if-lock, primarily by making the 'add-interface' logic deferred to the thread's main loop. It's likely I missed a corner case or two here, hopefully I'll
find them all in testing...
* Add ioctl to grab all of the pktgen stats in one call.

The pktgen.[ch] patch is here: http://www.candelatech.com/oss/pktgen.patch

The full patch, including the callback hooks (and several other features) is located here: http://www.candelatech.com/oss/candela_2.6.18.patch

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Thanks. Well sometimes there is a need to capture and drop pkts and various points, so it handy to have code fragments ready and just add the hook when it's needed in driver or rx softirq etc.

Any chance you extract this one so we can compare with get_time_of_day. This pops up in the profiles and TX numbers are not what used to be.

Cheers.

					--ro

I'd be thrilled to have the receive logic go into pktgen, even if it was #if 0 with a comment showing how to patch dev.c to get it working. It would make my out-of-tree patch smaller and should help others who are doing research and driver development...

If you'd accept a patch like this, please let me know.

I'll work on that.

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Just an idea... RX part is pretty separate from TX. How about if we keep the this code in a small seprate optional module? It can developed into some general RX probe.

Cheers.

						--ro

It requires a hook in dev.c, or at least that is the only way I can think to implement it. I don't think that hook is going to be allowed into the kernel, so the best I was hoping for was to have the code in pktgen with #if 0 and let users patch their kernel to add the hook...

Ben

Ben Greear <[email protected]> Candela Technologies Inc  http://www.candelatech.com

Well the hook be placed along the packet path even in drivers. In tulip I didn't even take packet of the ring in some experiments.

And there plenty of existing hooks already i.e PRE_ROUTE?

Right so what I was trying to say was rather having #if 0 and dead code in pktgen it could be kept separate.

Cheers.

						--ro

For my particular application the hook needs to be right after the bridge hook. My dev.c hook looks like this:

#if defined(CONFIG_NET_PKTGEN) || defined(CONFIG_NET_PKTGEN_MODULE) #include "pktgen.h"

#warning "Compiling dev.c for pktgen.";

int (*handle_pktgen_hook)(struct sk_buff *skb) = NULL; EXPORT_SYMBOL(handle_pktgen_hook);

static __inline__ int handle_pktgen_rcv(struct sk_buff* skb) {
	if (handle_pktgen_hook) {
		return handle_pktgen_hook(skb);
	}
	return -1;
}

#endif

.....

#if defined(CONFIG_NET_PKTGEN) || defined(CONFIG_NET_PKTGEN_MODULE)
	if ((skb->dev->pkt_dev) &&
	    (handle_pktgen_rcv(skb) >= 0)) {
		/* Pktgen may consume the packet, no need to send
		 * to further protocols.
		 */
		goto out;
	}

#endif

If the rx-hook logic is already in pktgen module, and it's symbol exported, perhaps a second hook module could be written that would just be the bridge between a driver or a PRE-ROUTE hook and pktgen? The hook module would probably not be much larger than the code above...

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

I use pktgen extensively these days for ipsec testing. I also record stats for pkts i receive using a tc action drop. Very simple and works great.
Ben, you keep insisting on doing this hook (every 3 months) because you dont want to invest time to do it with netlink (I thought you sold this as part of your product - the investment of your time is really not that high). If you tell me what the packet trigger is, I will send you a script ;->

Robert, I have started writing some generic netlink messaging to replace the /proc that i will send your way. I need to get consensus on some tunnel-mode IPSEC patches first then i will send the about three patchsets your way (to replace the old ones i sent earlier).

cheers,
jamal

Please do send a script. I match based on this method below. Probably you only
need the part that checks for the MAGIC, the rest can be in a method called after you match in your script?

/* Returns < 0 if the skb is not a pktgen buffer. */

int pktgen_receive(struct sk_buff* skb) {
       /* See if we have a pktgen packet */

/* TODO: Add support for detecting IPv6, TCP packets too. This
will only
* catch UDP at the moment. --Ben
*/
/*printk("pktgen-rcv, skb->len: %d\n", skb->len);*/

       if ((skb->len >= (20 + 8 + sizeof(struct pktgen_hdr))) &&
           (skb->protocol == __constant_htons(ETH_P_IP))) {
       struct pktgen_hdr* pgh;
              
               /* It's IP, and long enough, lets check the magic number.
                * TODO:  This is a hack not always guaranteed to catch 
the right
                * packets.
                */
      
       /*printk("Length & protocol passed, skb->data: %p, raw: %p\n",
         skb->data, skb->h.raw);*/
     
               pgh = (struct pktgen_hdr*)(skb->data + 20 + 8);
      
               /*
               tmp = (char*)(skb->data);
               for (i = 0; i<90; i++) {
                       printk("%02hx ", tmp[i]);
                       if (((i + 1) % 15) == 0) {
                               printk("\n");
                       }
               }
               printk("\n");
               */
              
               if (pgh->pgh_magic == __constant_ntohl(PKTGEN_MAGIC)) {
                       struct net_device* dev = skb->dev;
                       struct pktgen_dev* pkt_dev;
                       __u32 seq = ntohl(pgh->seq_num);

           // TODO:  Need lock..maybe
           pkt_dev = dev->pkt_dev;
                      
                       if (!pkt_dev) {
               return -1;
                       }

                       pkt_dev->pkts_rcvd++;
                       pkt_dev->bytes_rcvd += ((skb->tail - 

skb->mac.raw) + 4); /* +4 for the checksum */

                       /* Check for out-of-sequence packets */
                       if (pkt_dev->last_seq_rcvd == seq) {
                               pkt_dev->dup_rcvd++;
                               pkt_dev->dup_since_incr++;
                       }
                       else {
                               __s64 rx;
                               __s64 tx;
                               struct timeval txtv;
               if (skb->tstamp.off_sec || skb->tstamp.off_usec) {
                   skb_get_timestamp(skb, &txtv);
               }
               else {
                   do_gettimeofday(&txtv);
                   skb_set_timestamp(skb, &txtv);
               }
                               rx = tv_to_us(&txtv);

                               txtv.tv_usec = ntohl(pgh->tv_usec);
                               txtv.tv_sec = ntohl(pgh->tv_sec);
                               tx = tv_to_us(&txtv);
                               record_latency(pkt_dev, rx - tx);
                              
                               if ((pkt_dev->last_seq_rcvd + 1) == seq) {
                                       if ((pkt_dev->peer_clone_skb > 1) &&
                                           (pkt_dev->peer_clone_skb > 
(pkt_dev->dup_since_incr + 1))) {
                                              
                                               pkt_dev->seq_gap_rcvd += 

(pkt_dev->peer_clone_skb -

pkt_dev->dup_since_incr - 1);
                                       }
                                       /* Great, in order...all is well */
                               }
                               else if (pkt_dev->last_seq_rcvd < seq) {
                                       /* sequence gap, means we 
dropped a pkt most likely */
                                       if (pkt_dev->peer_clone_skb > 1) {
                                               /* We dropped more than 

one sequence number's worth,
* and if we're using
clone_skb, then this is quite
* a few. This number
still will not be exact, but

                                                * it will be closer.
                                                */
                                               pkt_dev->seq_gap_rcvd += 

(((seq - pkt_dev->last_seq_rcvd) *

pkt_dev->peer_clone_skb) -

pkt_dev->dup_since_incr);
                                       }
                                       else {
                                               pkt_dev->seq_gap_rcvd += 
(seq - pkt_dev->last_seq_rcvd - 1);
                                       }
                               }
                               else {
                                       pkt_dev->ooo_rcvd++; /* 
out-of-order */
                               }
                              
                               pkt_dev->dup_since_incr = 0;
                       }
                       pkt_dev->last_seq_rcvd = seq;
                       kfree_skb(skb);
                       if (debug > 1) {
                               printk("done with pktgen_receive, free'd 
pkt\n");
                       }
                       return 0;
               }
       }
       return -1; /* Let another protocol handle it, it's not for us! */
}/* pktgen_receive */

Ben Greear <[email protected]> Candela Technologies Inc  http://www.candelatech.com

What i do in my case is send to the SUT to UDP port 9. The SUT loops back the packet to me after processing and i just have a simple check in the traffic generator of:

a) Packet = IPV4, IP address matches what i want, port =9,
b) If this matches i account and drop it. This means all other traffic (like ssh etc goes through fine).

I dont really need to check for length because _i know_ udp port 9 to the SUT is test traffic.

For such a setup, the script would be as simple as (assuming SUT hooked to eth0):

--
#interested in incoming packets on eth0
tc qdisc add dev eth0 ingress
#interested in packets coming from SUT(10.0.0.21) on UDP port 9 tc filter add dev eth0 parent ffff: protocol ip prio 6 u32 \
match ip src 10.0.0.21/32 \
match udp port 9 0xffff \
flowid 1:16 \
action drop
---

When you do a listing of this filter you will see accounting info. You list by saying:
---
tc -s filter ls dev eth0 parent ffff:
---

You can ofcourse add many of these based on other header info.

It seems to me your magic header is inside the UDP packet, correct? In that case you will have to play with matches since you can specify arbitraty offsets and values inside the packet.

Of course you can do this from your application instead of using tc. I think that will be the best place to control this and sync with pktgen sending.

If the packet/byte counters that the drop action provides are not good enough, you can write yourself a little module that will do the accounting the way you want it. For example this will be necessary to the out-of-sequence cheques below. Timestamps are already being updated Look at net/sched/act_simple.c and its associated user space code in iproute2.

Now, Ben - are you listening really this time or am i wasting my time for the nth time giving you all these details? ;->

If you are listening then start with:

1) Do a simple test with just udp traffic as above, doing simple accounting. This helps you to get a feel on how things work.
2) modify the matching rules to match your magic cookie
3) write a simple action invoked by your matching rules and use tc to add it to the policy.
4) integrate in your app now that you know what you are doing.

If you follow these steps or a variant-of i will spend time and help.

cheers,
jamal

Yes, I want to be able to randomize source/dest MAC, IP, and IP Port, so I need to match only on the magic cookie. I'll soon add support to deal with TCP packets as well, but that is only going to require calculating a different offset to look for the magic value.

You mean the equiv of calling system(tc filter .....) from the app, or do you mean something else entirely?

You always give details that are pertinent to your setup and not to mine, and you always have some 'small' step that is 'write a kernel module and hack on tc' if you want it to do something other than the generic thing.

These first two do not look too difficult. I certainly need more control/stats than what the generic counters would be, however, so I assume I need to move to step 3.

I have exactly no idea of what you mean by this. How do I get this tc framework to send the packet with 32-bit value FOO at offset X to some method called pktgen_rx_pkt(skb) in the pktgen module? That is all I want in a hook: The ability to match on a pkt and send it to my kernel module and to let my kernel module and have it traverse the stack no farther unless my kernel module decides to re-insert it for some reason.

If this takes significant work, then please don't waste more time trying to talk me into this, as my small patch to dev.c already does exactly what I need and the code is very easy to understand.

If this is easy, how about you write the module and make the tc changes. I'll work on any changes needed in pktgen and polish up a nice patch for Robert et al.... Then you can get the latency distributions, OOO, Dup, Dropped and other counters that my pktgen logic provides for your own testing purposes...

Thanks,
Ben

Ben Greear <[email protected]> Candela Technologies Inc  http://www.candelatech.com

Yes. Sounds like simple and general solution. No call-backs, no #ifdef's no extra modules. Just a little recipe in pktgen.txt

Cheers.

					--ro

Seems I missed to set tsc as clocksource. It makes a difference. Performance is normal and I'm less confused.

e1000 82546GB @ 1.6 GHz Opteron. Kernel 2.6.19-rc1_Bifrost_-g18e199c6-dirty

echo acpi_pm > /sys/devices/system/clocksource/clocksource0/current_clocksource

psize pps
---------
60 556333

124  526942
252  452981
508  234996

1020 119748
1496 82248

echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource

psize pps
---------
60 819914

124  747286
252  452975
508  234993

1020 119749
1496 82247

Cheers.

						--ro