pktgen question

The pktgen module provides a way to "clone" the skb its using for transmission, and allows passing N clones of the originally created skb
to the driver under test. However, it doesn't really use skb_clone(),
but rather it just bumps the skb->users count for each "clone" and passes the same skb ptr to the driver.

Q: Is that a valid use of skb->users or should pktgen really be cloning the skbuff?

Hi Steve.

Quoted 3 lines · Steve Wise

The pktgen module provides a way to "clone" the skb its using for transmission, and allows passing N clones of the originally created skb to the driver under test. However, it doesn't really use skb_clone(), but rather it just bumps the skb->users count for each "clone" and passes the same skb ptr to the driver.

Q: Is that a valid use of skb->users or should pktgen really be cloning the skbuff?

It's a hack, but since skb is owned by pktgen only (no copies in some outside queues or some other access) it is allowed just to bump reference counter (i.e. 'share' skb in usual notation).

Its a hack that breaks cxgb3 because cxgb3 uses the skb->cb area for each skb passed down. So cxgb3 is at fault then? IE a driver cannot use the skb->cb field if the users count is > 1? Or maybe a driver can _never_ use the cb field?

Quoted 2 lines · Steve Wise

Its a hack that breaks cxgb3 because cxgb3 uses the skb->cb area for each skb passed down. So cxgb3 is at fault then? IE a driver cannot use the skb->cb field if the users count is > 1? Or maybe a driver can _never_ use the cb field?

any layer can use the cb structure whichever way they wish. There are violations, e.g:
the vlan code also uses the cb field to pass vlan details for hardware acceleration. How does pktgen affect it though, clone() will just copy the cb field and pktgen doesnt touch it.
In retrospect, pktgen may have to use clone - ccing Robert Olsson.

Quoted 6 lines · jamal

Quoted 2 lines · Steve Wise

Its a hack that breaks cxgb3 because cxgb3 uses the skb->cb area for each skb passed down. So cxgb3 is at fault then? IE a driver cannot use the skb->cb field if the users count is > 1? Or maybe a driver can _never_ use the cb field?

Quoted 7 lines

any layer can use the cb structure whichever way they wish. There are violations, e.g:
the vlan code also uses the cb field to pass vlan details for hardware acceleration. How does pktgen affect it though, clone() will just copy the cb field and pktgen doesnt touch it.
In retrospect, pktgen may have to use clone - ccing Robert Olsson.

Pktgen abuses the ref count, as far as I can tell. It works with most ethernet drivers, but that multi-pkt will also fail with things like vlans because the skb->dev is changed as it is transmitted (to the lower-level device).

I'd say just don't use the multi-pkt with pktgen on devices that can't handle it.

Thanks,
Ben

I think pktgen should be cloning the skbs using skb_clone(). Then it will work for all devices, eh?

Quoted 1 line · Steve Wise

I think pktgen should be cloning the skbs using skb_clone(). Then it will work for all devices, eh?

That might work, but it would decrease performance slightly (or, increase CPU load at least).

Maybe a new option: multi_clone

Quoted 3 lines · Ben Greear

Quoted 1 line · Steve Wise

I think pktgen should be cloning the skbs using skb_clone(). Then it will work for all devices, eh?

That might work, but it would decrease performance slightly (or, increase CPU load at least).

Perf-wise, you could clone the skbs up front, then deliver them to the nic in a tight loop. This would mitigate the added overhead introduced by calling skb_clone() in the loop doing transmits...

Quoted 3 lines

Maybe a new option: multi_clone

If the current code is busted, I think it should be fixed.

Quoted 1 line

Perf-wise, you could clone the skbs up front, then deliver them to the nic in a tight loop. This would mitigate the added overhead introduced by calling skb_clone() in the loop doing transmits...

That only works if you are sending a small number of skbs. You can't pre-clone several minutes worth of 10Gbe traffic with any normal amount of RAM.

Quoted 4 lines

Quoted 1 line

Perf-wise, you could clone the skbs up front, then deliver them to the nic in a tight loop. This would mitigate the added overhead introduced by calling skb_clone() in the loop doing transmits...

That only works if you are sending a small number of skbs. You can't pre-clone several minutes worth of 10Gbe traffic with any normal amount of RAM.

Does pktgen really need to allocate anything more than some smallish fraction more than the depth of the driver's transmit queue?

Quoted 6 lines · Rick Jones

Quoted 4 lines

Quoted 1 line

Perf-wise, you could clone the skbs up front, then deliver them to the nic in a tight loop. This would mitigate the added overhead introduced by calling skb_clone() in the loop doing transmits...

That only works if you are sending a small number of skbs. You can't pre-clone several minutes worth of 10Gbe traffic with any normal amount of RAM.

Does pktgen really need to allocate anything more than some smallish fraction more than the depth of the driver's transmit queue?

If you want to send sustained high rates of traffic, for more than just a trivial amount of time, then you either have to play the current trick with the skb_get(), or you have to allocate a real packet each time (maybe with skb_clone() or similar, but it's still more overhead than the skb_get which only bumps a reference count.)

I see no other way, but if you can think of one, please let me know.

Thanks,
Ben

Quoted 14 lines · Ben Greear

Quoted 6 lines · Rick Jones

Quoted 4 lines

Quoted 1 line

Perf-wise, you could clone the skbs up front, then deliver them to the nic in a tight loop. This would mitigate the added overhead introduced by calling skb_clone() in the loop doing transmits...

That only works if you are sending a small number of skbs. You can't pre-clone several minutes worth of 10Gbe traffic with any normal amount of RAM.

Does pktgen really need to allocate anything more than some smallish fraction more than the depth of the driver's transmit queue?

If you want to send sustained high rates of traffic, for more than just a trivial amount of time, then you either have to play the current trick with the skb_get(), or you have to allocate a real packet each time (maybe with skb_clone() or similar, but it's still more overhead than the skb_get which only bumps a reference count.)

Quoted 2 lines

I see no other way, but if you can think of one, please let me know.

You can keep freed skb's that were cloned on a free list, then reuse them once freed. You can detect when the driver frees them by adding a destroy function to the skb. So what will happen is the set of cloned skbs needed will eventually settled down to a constent amount and the amount will be based on the latency involved in transmitting a single skb. And it should be bounded by the max txq depth. Yes? (or am I all wet :)

So you would pay the overhead of cloning only until you hit this steady state.

Whatchathink?

This skb recycling can certainly work and has been done several times before. It never gets into the kernel though.

From: Ben Greear <[email protected]>
Date: Mon, 08 Oct 2007 14:57:13 -0700

Quoted 2 lines

This skb recycling can certainly work and has been done several times before. It never gets into the kernel though.

Because it doesn't work.

A socket can hang onto a receive packet essentially forever.

You cannot therefore rely upon the network stack to give you the packet back in some reasonable finite amount of time. This is simply the nature of the beast.

Which means that you either:

1) Starve and stop receiving packets when the recycling ring runs out because all of those packets are stuck in socket buffers. This is easily DoS'able by users on your system

2) End up allocating new packets anyways, but then what's the point of the recycling ring? It's just a hack that works when everything goes as planned, and in real life that is rarely the case.

It also makes the driver locking more complicated. Packet input occurs in NAPI drivers via netif_receive_skb() which would be capable of recycling packets back to the same driver in a recycling scheme. But the recycling can occur from other contexts too. The netif_receive_skb() caller already has atomic access to the receive queue, but those other callback cases do not.

That locking issue is just the tip of the iceberg. Once you start discussing solutions, all sorts of new issues begin to pop up.

SKB recycling, just say no.

We're talking about just for pktgen...eh?

I think pktgen should be cloning the skbs using skb_clone(). Then it will work for all devices, eh?

Perf-wise, you could clone the skbs up front, then deliver them to the nic in a tight loop. This would mitigate the added overhead introduced by calling skb_clone() in the loop doing transmits...

I think pktgen should be cloning the skbs using skb_clone(). Then it will work for all devices, eh?