Skip to contentSkip to conversation

netif_stop_queue() and multiple hardware queues

DISCUSSION Jeremy Jackson 2 participants 3 messages Jul 3, 2006 – Jul 6, 2006 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

I'd like to raise the question again. How can the network stack's flow control function with hardware that has multiple physical queues? How can these queues be exposed to tc?

After scanning the netdev mailing list activity for the last 6 months, I can see no furthur discussion since my original post. I did notice some churn in several 802.11 layer stacks trying to deal with multiple hardware queues, the IPW2200 in particular.

I propose that this is an architecture/framework/API issue that the Linux core networking code should be dealing with.

There is some fragmentation already; compare the QoS *policy* implemented by Madwifi vs IPW2200 vs devicescape's 80211 stack. It's worth noting that at the 80211 layer, the first 2 listed above have a "priority" argument to (80211)_hard_start_xmit().

It would seem ideal to expose the queues of Ethernet layer hardware (provide mechanism) and let userspace tc tools do the classifying (sysadmin can set the policy).

Jeremy

Quoted 14 lines · Jeremy Jackson

Hi,

I posted this briefly on linux-net, before being redirected here.

Two pieces of hardware now have Linux driver support for multiple hardware queues: Intel's e1000 (two queues from what I can see in the code) and Atheros's 5212 and up, in support of 802.11e and WME (four hardware queues). In the GigE case, this just reduces latency due to hardware queueing. In the WiFi case, the queues actually have significance in access to the shared medium. (ACKs can be disabled as well) It is also worthy of note that ADSL2, VDSL, and ADSL2+ have 4 different latency "queues". These last two are significant; real-time applications suffer the most from low speed, shared, and/or non-deterministic media. I wonder where DOCSIS 2 is in this regard. Anyone? Beuler?

So my question is, what's it going to take to get dev->hard_start_xmit() to hook up tc queues directly to hardware/driver queues?

Right now, it seems no matter how elaborate a tc setup you have, everything funnels through one queue, where the only thing that survives from the classifying/queueing is skb-> priority (ie nothing). The hardware driver can then try to reclassify packets. I suppose you could hack up an iptables classifyer to set skb->priority...

The Atheros driver tries to do it's own classifying by first wiping out skb->priority, then hard-coding a mapping (tsk - policy is for the sysadmin!) between VLAN tag priority, IP TOS/DSCP, and skb->priority, then to one of the 4 queues and ACK states, blithely ignoring any fine work done by tc.

It's be sweet to head this nonsense off at the pass, before others discover the rabbit trail and make it into a trade route.

maintainer
#2 lore raw

From: Jeremy Jackson <[email protected]>
Date: Sun, 02 Jul 2006 22:36:03 -0400

Quoted 2 lines

I propose that this is an architecture/framework/API issue that the Linux core networking code should be dealing with.

Proposals are worthless if nobody wants to do the work to actually implement the proposal.

Your tone seems to suggest that you feel if you propose something, someone will just jump out of their seat and implement your proposal.

That's not how this works :)

#3 lore raw
Quoted 5 lines · David Miller

From: Jeremy Jackson <[email protected]>
Date: Sun, 02 Jul 2006 22:36:03 -0400

Quoted 1 line

I propose that this is an architecture/framework/API issue that the Linux core networking code should be dealing with.

Quoted 7 lines

Proposals are worthless if nobody wants to do the work to actually implement the proposal.

Your tone seems to suggest that you feel if you propose something, someone will just jump out of their seat and implement your proposal.

That's not how this works :)

I apologize for the tone. I *am* interested working on this. I'm just surprised nobody has jumped out of their seat to do it :)

I have changed a line or two of kernel code but am by no means an expert, so I guess I was hoping for some kind of feedback about the proposal. I guess submit a patch for discussion? If it's gross I guess that would generate some feedback.

What's the best Git tree to base this on? (maybe it's name means make svn and and cvs users feel like a stupid git?)

In the current 2.6 network driver interface, the TX path seems to be based on mydev->hard_start_xmit(). The device driver initiates flow control with netif_stop_queue().

netif_stop_queue() and other functions (macros in netdevice.h) touch mydev->state, the bit __LINK_STATE_XOFF:

netif_schedule()
netif_start_queue()
netif_wake_queue()
netif_stop_queue()
netif_queue_stopped()

There is only one queue now, implied by default. Adding a "hwqueue" parameter to all of the above would be a start.

It would be ideal if mydev->xmit_lock could be separated out, one for each hwqueue. A quick hack to implement this would be for the device to set the mydev->features NETIF_F_LLTX, and implement per-queue locking.

Teaching qdiscs to use the multiple queues looks tricky in comparison, a quick hack might be to have prio's bands map 1:1 with the hardware queues.

Regards,

Jeremy