Ath9k in funky state after adding 130 STA interfaces.
This on wireless-testing from Friday, plus a few debugfs patches I posted recently.
I was running two STA just fine, and then tried to add 128 more.
After that, nothing will authenticate (timed out).
I don't see any obvious errors in the logs, other than DMA failing to stop after 10ms, but this debugfs information looks quite suspect:
[root at atom lanforge]# cat /debug/ath9k/phy0/wiphy
primary: phy0 (ACTIVE chan=0 ht=0)
addr: ef:be:ad:de:ef:be
addrmask: ef:be:ad:de:ef:be
rfilt: 0xdeadbfef UCAST MCAST BCAST CONTROL PROM PROBEREQ PHYERR MYBEACON COMP_BAR PHYRADAR MCAST_BCAST_ALL
It seems some sort of race..if I add and configure these interfaces manually, it seems to work. It may also be a problem with bringing up the interface before applying the wifi settings, as that was the last change I made to my user-space application that manages these interfaces.
I briefly saw the deadbeef when configuring manually..but the single wlan0 seemed to be associated OK, and when I added another station, the values went back to normal.
Any ideas what this might indicate?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 4 lines · Ben Greear
This on wireless-testing from Friday, plus a few debugfs patches I posted recently.
I was running two STA just fine, and then tried to add 128 more.
You're serious right? I mean I'm happy your serious, but whoa, you want 130 STAs on one interface working fine?
Luis
Quoted 7 lines · Luis R. Rodriguez
Quoted 4 lines
This on wireless-testing from Friday, plus a few debugfs patches I posted recently.
I was running two STA just fine, and then tried to add 128 more.
You're serious right? I mean I'm happy your serious, but whoa, you want 130 STAs on one interface working fine?
Come on. If it is impossible to work because of whatever limitations then wouldn't it make sense for the driver to throw an error instead of not throwing an error and just not working?
//Peter
Quoted 4 lines · Peter Stuge
Quoted 3 lines · Luis R. Rodriguez
Quoted 3 lines
This on wireless-testing from Friday, plus a few debugfs patches I posted recently.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
I was running two STA just fine, and then tried to add 128 more.
You're serious right? I mean I'm happy your serious, but whoa, you want 130 STAs on one interface working fine?
Come on. If it is impossible to work because of whatever limitations then wouldn't it make sense for the driver to throw an error instead of not throwing an error and just not working?
You're right but if you are using the ath9k virtual wiphy stuff please not that code is intended to be removed, it was experimental code and this is also why we don't have good documentation for it. It was just a hack as a proof of concept. The real solution needs to take place on cfg80211/mac80211.
Luis
Quoted 14 lines · Luis R. Rodriguez
Quoted 4 lines · Peter Stuge
Quoted 3 lines · Luis R. Rodriguez
Quoted 3 lines
This on wireless-testing from Friday, plus a few debugfs patches I posted recently.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
I was running two STA just fine, and then tried to add 128 more.
You're serious right? I mean I'm happy your serious, but whoa, you want 130 STAs on one interface working fine?
Come on. If it is impossible to work because of whatever limitations then wouldn't it make sense for the driver to throw an error instead of not throwing an error and just not working?
Quoted 5 lines
You're right but if you are using the ath9k virtual wiphy stuff please not that code is intended to be removed, it was experimental code and this is also why we don't have good documentation for it. It was just a hack as a proof of concept. The real solution needs to take place on cfg80211/mac80211.
I'm not using virtual wiphy..just one PHY and lots of STA devices on top of it.
I'd be more than happy to see the wiphy stuff dissappear, as it seems useless to me. Using lots of STA and AP devices on a single phy is much more useful, however.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 8 lines · Luis R. Rodriguez
Quoted 4 lines · Ben Greear
This on wireless-testing from Friday, plus a few debugfs patches I posted recently.
I was running two STA just fine, and then tried to add 128 more.
You're serious right? I mean I'm happy your serious, but whoa, you want 130 STAs on one interface working fine?
Oh, I'm hoping for 256+ :)
Ath5k can do 128 ok, with very minimal traffic load, at least.
Our primary interest is load testing APs and such, so the more the merrier.
Ath9k seems a bit more touchy. I'm testing today on a different system, but still ath9k. The system has been panic-ing, and locking hard. Here is results of a mostly-hard-lock.
I was hoping the 'deadbeef' registers indicated a particular error in the NIC that I might could use for further debugging.
Oct 4 10:08:32 localhost kernel: sta25: authenticate with 00:14:d1:c6:d2:54 (try 1) Oct 4 10:08:32 localhost kernel: ath: timeout (100000 us) on reg 0x9860: 0x0000002f & 0x00000001 != 0x00000000 Oct 4 10:08:32 localhost kernel: ath: Unable to reset channel (2437 MHz), reset status -5 Oct 4 10:08:32 localhost kernel: ath: Unable to set channel Oct 4 10:08:32 localhost kernel: sta119: authenticate with 00:14:d1:c6:d2:54 (try 1) Oct 4 10:08:32 localhost kernel: sta25: authenticate with 00:14:d1:c6:d2:54 (try 2) Oct 4 10:08:32 localhost kernel: sta119: authenticate with 00:14:d1:c6:d2:54 (try 2) Oct 4 10:08:32 localhost kernel: sta25: authenticate with 00:14:d1:c6:d2:54 (try 3) Oct 4 10:08:32 localhost kernel: sta2: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta13: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta57: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta74: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta96: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta97: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta110: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta117: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta120: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 1 flags: 0x2 Oct 4 10:08:32 localhost kernel: sta119: authenticate with 00:14:d1:c6:d2:54 (try 3) Oct 4 10:08:32 localhost kernel: sta25: authentication with 00:14:d1:c6:d2:54 timed out Oct 4 10:08:32 localhost kernel: sta119: authentication with 00:14:d1:c6:d2:54 timed out Oct 4 10:08:33 localhost kernel: sta2: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta13: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta57: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta74: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta96: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta97: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta110: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta117: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: sta120: No probe response from AP 00:14:d1:c6:d2:54 after 500ms, try 0 flags: 0x2 Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:33 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 4 10:08:33 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:08:33 localhost kernel: ath: Chip reset failed Oct 4 10:08:33 localhost kernel: ath: Unable to reset hardware; reset status -22 Oct 4 10:08:33 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:33 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:33 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:08:33 localhost kernel: ath: Chip reset failed Oct 4 10:08:33 localhost kernel: ath: Unable to reset channel (2412 MHz), reset status -22 Oct 4 10:08:33 localhost kernel: ath: Unable to set channel Oct 4 10:08:33 localhost kernel: sta97: deauthenticating from 00:14:d1:c6:d2:54 by local choice (reason=3) Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:08:34 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 4 10:08:34 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:34 localhost kernel: ath: timeout (100000 us) on reg 0x9860: 0xdeadbeef & 0x00000001 != 0x00000000 Oct 4 10:08:34 localhost kernel: ath: Unable to reset hardware; reset status -5 Oct 4 10:08:34 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:34 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:08:34 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:08:34 localhost kernel: ath: Chip reset failed Oct 4 10:08:34 localhost kernel: ath: Unable to reset channel (2437 MHz), reset status -22 Oct 4 10:08:34 localhost kernel: ath: Unable to set channel Oct 4 10:08:34 localhost kernel: ieee80211 phy0: Removed STA 00:14:d1:c6:d2:54 Oct 4 10:08:34 localhost kernel: ieee80211 phy0: Destroyed STA 00:14:d1:c6:d2:54
... Lots more spewage, then no more kernel spewage, ..seems all HD access is fried..tried to pipe dmesg to disk, but command hung.
Oct 4 10:09:01 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:09:01 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:09:02 localhost kernel: ath: timeout (100000 us) on reg 0x7000: 0xdeadbeef & 0x00000003 != 0x00000000 Oct 4 10:09:02 localhost kernel: ath: Chip reset failed Oct 4 10:09:02 localhost kernel: ath: Unable to reset channel (2437 MHz), reset status -22 Oct 4 10:09:02 localhost kernel: ath: Unable to set channel Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA in 100 msec after killing last frame Oct 4 10:09:02 localhost kernel: ath: Failed to stop TX DMA. Resetting hardware! Oct 4 10:09:02 localhost kernel: ath: DMA failed to stop in 10 ms AR_CR=0xdeadbeef AR_DIAG_SW=0xdeadbeef Oct 4 10:09:02 localhost kernel: ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
SysRq : Show Locks Held
Thanks,
Ben
Quoted 2 lines
Luis
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 2 lines · Ben Greear
I was hoping the 'deadbeef' registers indicated a particular error in the NIC that I might could use for further debugging.
I'd guess it's a general error rather than something specific. I get it on most boots and the driver needs some jerking around before the card will start working. Sometimes it doesn't matter what I do and power cycle is all that seems to help. I haven't spent much time on thorough research yet though, in part because there's so little response here. :\
//Peter
Quoted 10 lines · Peter Stuge
Quoted 2 lines · Ben Greear
I was hoping the 'deadbeef' registers indicated a particular error in the NIC that I might could use for further debugging.
I'd guess it's a general error rather than something specific. I get it on most boots and the driver needs some jerking around before the card will start working. Sometimes it doesn't matter what I do and power cycle is all that seems to help. I haven't spent much time on thorough research yet though, in part because there's so little response here. :\
yeah the deadbeef stuff typically is a misprogramming on the driver to not wake the chip up for a read/write operation. We have a lot of fixes for this over the last few kernels and if you using 2.6.35 chances are a few fixes may not have trickled down yet. For development purposes you are best to use the latest always.
Luis
Quoted 12 lines · Luis R. Rodriguez
Quoted 10 lines · Peter Stuge
Quoted 2 lines · Ben Greear
I was hoping the 'deadbeef' registers indicated a particular error in the NIC that I might could use for further debugging.
I'd guess it's a general error rather than something specific. I get it on most boots and the driver needs some jerking around before the card will start working. Sometimes it doesn't matter what I do and power cycle is all that seems to help. I haven't spent much time on thorough research yet though, in part because there's so little response here. :\
Quoted 6 lines
yeah the deadbeef stuff typically is a misprogramming on the driver to not wake the chip up for a read/write operation. We have a lot of fixes for this over the last few kernels and if you using 2.6.35 chances are a few fixes may not have trickled down yet. For development purposes you are best to use the latest always.
I'm on wireless-testing, totally up-to-date as I can get.
It seems trivial to kill the system using lots of STA devices on ath9k, so if anyone needs help reproducing this, I'm more than happy to help. In general, I'd guess that the problems I hit very quickly could also be hit by other users if they got unlucky, so it may be worth some time to debug this in detail.
Basically, create 130 (don't think exact number matters) STA devices, configure a wpa_supplicant for each, and stand back and watch things go south.
I do have a few extra patches..like to restrict scanning to the single associated channel, but I doubt that has much impact on the bug (and it works fine in ath5k).
I have lockdep turned on, and pre-empt enabled in the kernel configs.
If someone prefers, I can make my entire tree available through git and post .config files, etc.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 28 lines · Ben Greear
Quoted 12 lines · Luis R. Rodriguez
Quoted 10 lines · Peter Stuge
Quoted 2 lines · Ben Greear
I was hoping the 'deadbeef' registers indicated a particular error in the NIC that I might could use for further debugging.
I'd guess it's a general error rather than something specific. I get it on most boots and the driver needs some jerking around before the card will start working. Sometimes it doesn't matter what I do and power cycle is all that seems to help. I haven't spent much time on thorough research yet though, in part because there's so little response here. :\
Quoted 6 lines
yeah the deadbeef stuff typically is a misprogramming on the driver to not wake the chip up for a read/write operation. We have a lot of fixes for this over the last few kernels and if you using 2.6.35 chances are a few fixes may not have trickled down yet. For development purposes you are best to use the latest always.
I'm on wireless-testing, totally up-to-date as I can get.
It seems trivial to kill the system using lots of STA devices on ath9k, so if anyone needs help reproducing this, I'm more than happy to help. ?In general, I'd guess that the problems I hit very quickly could also be hit by other users if they got unlucky, so it may be worth some time to debug this in detail.
Quoted 7 lines
Basically, create 130 (don't think exact number matters) STA devices, configure a
wpa_supplicantfor each, and stand back and watch things go south.I do have a few extra patches..like to restrict scanning to the single associated channel, but I doubt that has much impact on the bug (and it works fine in ath5k).
ath5k has no power save support.
Quoted 5 lines
I have lockdep turned on, and pre-empt enabled in the kernel configs.
If someone prefers, I can make my entire tree available through git and post .config files, etc.
Try:
iw dev wlan0 dev set power_save off
for each wireless interface and see if you see these go away.
Luis
Quoted 11 lines · Luis R. Rodriguez
Quoted 3 lines · Ben Greear
I do have a few extra patches..like to restrict scanning to the single associated channel, but I doubt that has much impact on the bug (and it works fine in ath5k).
ath5k has no power save support.
Quoted 3 lines
I have lockdep turned on, and pre-empt enabled in the kernel configs.
Quoted 8 lines
Quoted 2 lines
If someone prefers, I can make my entire tree available through git and post .config files, etc.
Try:
iw dev wlan0 dev set
power_saveofffor each wireless interface and see if you see these go away.
That seems to be helping...most of the interfaces won't actually associate (probes to AP time-out and such), but at least it isn't crashing so often.
Should we always disable power-save if there are > 1 VIF?
Thanks,
Ben
Quoted 2 lines
Luis
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 19 lines · Ben Greear
Quoted 18 lines · Luis R. Rodriguez
Quoted 3 lines · On Mon, Oct 4, 2010 at 10:48 AM, Ben Greear<[email protected]> ?wrote:
I do have a few extra patches..like to restrict scanning to the single associated channel, but I doubt that has much impact on the bug (and it works fine in ath5k).
ath5k has no power save support.
Quoted 5 lines
I have lockdep turned on, and pre-empt enabled in the kernel configs.
If someone prefers, I can make my entire tree available through git and post .config files, etc.
Try:
iw dev wlan0 dev set
power_saveoff
Quoted 7 lines
Quoted 1 line
for each wireless interface and see if you see these go away.
That seems to be helping...most of the interfaces won't actually associate (probes to AP time-out and such), but at least it isn't crashing so often.
Should we always disable power-save if there are > 1 VIF?
Well I can see us wanting to actually support PS in the case of WiFi Direct but for that I can see us in the worst case wanting us to use two interfaces, one STA on one band, and another GO (AP) interface on another band. After that, yeah it seems to make sense to just disable PS.
Luis