Optimizing performance for lots of virtual stations.
I've been doing some performance testing, and having lots of
stations causes quite a drag: total throughput with 1 station: 250Mbps TCP throughput, total with 50 stations: 225 Mbps, and with 128 stations: 20-40Mbps (it varies a lot..not so sure why).
I poked around in the rx logic and it seems the rx-data path is fairly clean for data packets. But, from what I can tell, each beacon is going to cause an skb_copy() call and a queued work-item for each station interface, and there are going to be lots of beacons per second in most scenarios...
I was wondering if this could be optimized a bit to special case beacons and not make a new copy (or possibly move some of the beacon handling logic up to the radio object and out of the sdata).
And of course, it could be there are more important optimizations...I'm curious if anyone is aware of any other code that should be optimized to have better performance with lots of stations...
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 9 lines · Ben Greear
I've been doing some performance testing, and having lots of
stations causes quite a drag: total throughput with 1 station: 250Mbps TCP throughput, total with 50 stations: 225 Mbps, and with 128 stations: 20-40Mbps (it varies a lot..not so sure why).I poked around in the rx logic and it seems the rx-data path is fairly clean for data packets. But, from what I can tell, each beacon is going to cause an
skb_copy()call and a queued work-item for each station interface, and there are going to be lots of beacons per second in most scenarios...
Quoted 7 lines
I was wondering if this could be optimized a bit to special case beacons and not make a new copy (or possibly move some of the beacon handling logic up to the radio object and out of the sdata).
And of course, it could be there are more important optimizations...I'm curious if anyone is aware of any other code that should be optimized to have better performance with lots of stations...
How about doing some profiling with lots of stations - that should hopefully reveal where the real bottleneck is.
By the way, with that many stations and low throughput, is the CPU usage on your system significantly higher, or could it just be some extra latency introduced somewhere else in the code?
- Felix
Quoted 14 lines · Felix Fietkau
Quoted 13 lines · Ben Greear
I've been doing some performance testing, and having lots of
stations causes quite a drag: total throughput with 1 station: 250Mbps TCP throughput, total with 50 stations: 225 Mbps, and with 128 stations: 20-40Mbps (it varies a lot..not so sure why).I poked around in the rx logic and it seems the rx-data path is fairly clean for data packets. But, from what I can tell, each beacon is going to cause an
skb_copy()call and a queued work-item for each station interface, and there are going to be lots of beacons per second in most scenarios...I was wondering if this could be optimized a bit to special case beacons and not make a new copy (or possibly move some of the beacon handling logic up to the radio object and out of the sdata).
Quoted 8 lines
Quoted 3 lines
And of course, it could be there are more important optimizations...I'm curious if anyone is aware of any other code that should be optimized to have better performance with lots of stations...
How about doing some profiling with lots of stations - that should hopefully reveal where the real bottleneck is.
By the way, with that many stations and low throughput, is the CPU usage on your system significantly higher, or could it just be some extra latency introduced somewhere else in the code?
CPU load is fairly high, but doesn't seem to just be CPU bound. Maybe lots and lots of work items all piled up or something like that...
I'll work on some profiling as soon as I get a chance.
I'm suspicious that the the management frame handling will need some optimization though..I think it basically copies the skb and broadcasts all mgt frames to all running stations....
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 24 lines · Ben Greear
Quoted 14 lines · Felix Fietkau
Quoted 13 lines · Ben Greear
I've been doing some performance testing, and having lots of
stations causes quite a drag: total throughput with 1 station: 250Mbps TCP throughput, total with 50 stations: 225 Mbps, and with 128 stations: 20-40Mbps (it varies a lot..not so sure why).I poked around in the rx logic and it seems the rx-data path is fairly clean for data packets. But, from what I can tell, each beacon is going to cause an
skb_copy()call and a queued work-item for each station interface, and there are going to be lots of beacons per second in most scenarios...I was wondering if this could be optimized a bit to special case beacons and not make a new copy (or possibly move some of the beacon handling logic up to the radio object and out of the sdata).
Quoted 8 lines
Quoted 3 lines
And of course, it could be there are more important optimizations...I'm curious if anyone is aware of any other code that should be optimized to have better performance with lots of stations...
How about doing some profiling with lots of stations - that should hopefully reveal where the real bottleneck is.
By the way, with that many stations and low throughput, is the CPU usage on your system significantly higher, or could it just be some extra latency introduced somewhere else in the code?
Quoted 8 lines
CPU load is fairly high, but doesn't seem to just be CPU bound. Maybe lots and lots of work items all piled up or something like that...
I'll work on some profiling as soon as I get a chance.
I'm suspicious that the the management frame handling will need some optimization though..I think it basically copies the skb and broadcasts all mgt frames to all running stations....
Here's another thing that might be negatively affecting your tests. The driver has a 128-packet buffer limit per hardware queue for aggregation. With too many stations, they will be competing for a very limited number of buffers, making aggregation a lot less effective. Increasing the number of buffers is a bad idea here, as it will harm environments with fewer stations due to bufferbloat.
What's required to fix this properly is better queue management, something that will require some bigger changes to the ath9k tx path and some mac80211 changes as well. It's on my TODO list, but I don't know when I'll get around to implementing it.
- Felix
Quoted 40 lines · Felix Fietkau
Quoted 24 lines · Ben Greear
Quoted 14 lines · Felix Fietkau
Quoted 13 lines · Ben Greear
I've been doing some performance testing, and having lots of
stations causes quite a drag: total throughput with 1 station: 250Mbps TCP throughput, total with 50 stations: 225 Mbps, and with 128 stations: 20-40Mbps (it varies a lot..not so sure why).I poked around in the rx logic and it seems the rx-data path is fairly clean for data packets. But, from what I can tell, each beacon is going to cause an
skb_copy()call and a queued work-item for each station interface, and there are going to be lots of beacons per second in most scenarios...I was wondering if this could be optimized a bit to special case beacons and not make a new copy (or possibly move some of the beacon handling logic up to the radio object and out of the sdata).
Quoted 8 lines
Quoted 3 lines
And of course, it could be there are more important optimizations...I'm curious if anyone is aware of any other code that should be optimized to have better performance with lots of stations...
How about doing some profiling with lots of stations - that should hopefully reveal where the real bottleneck is.
By the way, with that many stations and low throughput, is the CPU usage on your system significantly higher, or could it just be some extra latency introduced somewhere else in the code?Quoted 8 lines
CPU load is fairly high, but doesn't seem to just be CPU bound. Maybe lots and lots of work items all piled up or something like that...
I'll work on some profiling as soon as I get a chance.
I'm suspicious that the the management frame handling will need some optimization though..I think it basically copies the skb and broadcasts all mgt frames to all running stations....
Here's another thing that might be negatively affecting your tests. The driver has a 128-packet buffer limit per hardware queue for aggregation. With too many stations, they will be competing for a very limited number of buffers, making aggregation a lot less effective. Increasing the number of buffers is a bad idea here, as it will harm environments with fewer stations due to bufferbloat.
Quoted 4 lines
What's required to fix this properly is better queue management, something that will require some bigger changes to the ath9k tx path and some mac80211 changes as well. It's on my TODO list, but I don't know when I'll get around to implementing it.
I thought of that as well, but I saw something that made me think rx might be a big part of it as well:
With 50 stations each trying to transmit a 5Mbps TCP stream, I get around 210-220Mbps of total TCP throughput. But, if I simply add another 78 associated stations and do not run any traffic on them, throughput drops to about 80Mbps.
But, when I add traffic on those extra 78 stations, total throughput does drop down to around 20-40Mbps, so that part could easily be tx aggregation issues...
Would the tx-bytes-all / xmit-ampdus ratio give an idea of how well aggregation is working? (As reported by the ath9k xmit debugfs file).
I think I'll be better at trying to optimize the rx path than the tx path, as I get endlessly confused when trying to figure out the ath9k xmit path, but I can almost start to understand the mac80211 rx path after a while :)
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
I probably should have done this first of course..but here is a 'perf top' on the station machine (50 TCP streams transmitting on each of 50 stations, and 78 associated-but-mostly-idle stations:
Looks like sta_info_get would be a good place to start :)
---------------------------------------------------------------------------------------------------------------------------
PerfTop: 1890 irqs/sec kernel:87.9% exact: 0.0% [1000Hz cycles], (all, 2 CPUs)
---------------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ _______________________________ ______________
2261.00 20.8% sta_info_get [mac80211]
1707.00 15.7% ieee80211_tx_status [mac80211]
1192.00 11.0% intel_idle [kernel]
462.00 4.3% __ieee80211_recalc_idle [mac80211]
414.00 3.8% ieee80211_prepare_and_rx_handle [mac80211]
240.00 2.2% dev_queue_xmit_nit [kernel]
199.00 1.8% ieee80211_rx [mac80211]
154.00 1.4% ieee80211_find_sta_by_ifaddr [mac80211]
124.00 1.1% read_hpet [kernel]
101.00 0.9% _raw_spin_lock_irqsave [kernel]
92.00 0.8% __netif_receive_skb [kernel]
80.00 0.7% tg_load_down [kernel]
76.00 0.7% fget_light [kernel]
75.00 0.7% ieee80211_propagate_queue_wake [mac80211]
75.00 0.7% memcpy [kernel]
71.00 0.7% __ieee80211_stop_queue [mac80211]
70.00 0.6% ipt_do_table [kernel]
66.00 0.6% csum_partial_copy_generic [kernel]
65.00 0.6% datagram_poll [kernel]
63.00 0.6% _raw_spin_lock_bh [kernel]
61.00 0.6% ath_get_rate [ath9k]
49.00 0.5% ieee80211_subif_start_xmit [mac80211]
48.00 0.4% tcp_poll [kernel]Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 7 lines · Ben Greear
I probably should have done this first of course..but here is a 'perf top' on the station machine (50 TCP streams transmitting on each of 50 stations, and 78 associated-but-mostly-idle stations:
Looks like
sta_info_getwould be a good place to start :)
Quoted 8 lines
---------------------------------------------------------------------------------------------------------------------------
PerfTop: 1890 irqs/sec kernel:87.9% exact: 0.0% [1000Hz cycles], (all, 2 CPUs)---------------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO _______ _____ _______________________________ ______________ 2261.00 20.8% sta_info_get [mac80211]
Ahh, crap...I see the problem. The 'sta->addr' is the MAC of the VAP, so if I have 100 stations all connected to the same AP, then the hashing is worthless and just ends up being a linear search.
Probably not going to be fun to fix that!
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com