Thousands of ath9k warnings on dmesg before laptop froze
I'm testing the latest kernel 3.3.0-rc2+ I pulled from git this morning.
My laptop just froze, and when I rebooted I noticed that /var/log/messages contained 48 thousand (!) warnings coming from ath9k since a few hours ago. I'm pasting the first one:
------------[ cut here ]------------ WARNING: at /home/mafra/linux-2.6/drivers/net/wireless/ath/ath9k/rc.c:697 ath_rc_get_highest_rix+0x156/0x210 [ath9k]() Hardware name: VPCEB4X1E Modules linked in: dvb_usb_dib0700 dib0090 dib7000p dib7000m dib0070 dib8000 dib3000mc dibx000_common af_packet snd_pcm_oss snd_mixer_oss microcode arc4 sr_mod cdrom sg ath9k mac80211 uvcvideo ath9k_common ath9k_hw videobuf2_core videobuf2_vmalloc videobuf2_memops ath i2c_i801 cfg80211 sony_laptop rfkill radeon ttm drm_kms_helper drm i2c_algo_bit Pid: 48, comm: kworker/u:6 Not tainted 3.3.0-rc2+ #6 Call Trace: [<ffffffff81032f2a>] warn_slowpath_common+0x7a/0xb0 [<ffffffff81032f75>] warn_slowpath_null+0x15/0x20 [<ffffffffa0298eb6>] ath_rc_get_highest_rix+0x156/0x210 [ath9k] [<ffffffffa0298ff3>] ath_get_rate+0x83/0x480 [ath9k] [<ffffffffa024f006>] rate_control_get_rate+0x86/0x160 [mac80211] [<ffffffffa025a112>] invoke_tx_handlers+0x7b2/0x1420 [mac80211] [<ffffffff81061a00>] ? load_balance+0x90/0x8a0 [<ffffffffa025af22>] ieee80211_tx+0x82/0xf0 [mac80211] [<ffffffffa025b030>] ieee80211_xmit+0xa0/0x100 [mac80211] [<ffffffffa025c23a>] ieee80211_tx_skb_tid+0x5a/0x70 [mac80211] [<ffffffffa0246f56>] ieee80211_send_nullfunc+0x56/0x70 [mac80211] [<ffffffffa0248460>] ieee80211_dynamic_ps_enable_work+0x240/0x310 [mac80211] [<ffffffffa0248220>] ? ieee80211_dynamic_ps_disable_work+0x50/0x50 [mac80211] [<ffffffff81048d67>] process_one_work+0x117/0x390 [<ffffffff81049899>] worker_thread+0x169/0x350 [<ffffffff81049730>] ? manage_workers+0x240/0x240 [<ffffffff8104f096>] kthread+0x96/0xa0 [<ffffffff814f7714>] kernel_thread_helper+0x4/0x10 [<ffffffff8104f000>] ? flush_kthread_worker+0xb0/0xb0 [<ffffffff814f7710>] ? gs_change+0xb/0xb ---[ end trace 7a4892c93f508f3a ]--- ------------[ cut here ]------------
I must say that until the laptop froze, I didn't notice anything unusual to the point of checking my dmesg. Sometimes the internet was slow, but I thought it was unrelated to the kernel.
Is there any other info I should provide or the above warning is enough?
Quoted 9 lines · On Mon, 6 Feb 2012 00:29:07 +0000 "Carlos R. Mafra" <[email protected]> wrote:
I'm testing the latest kernel 3.3.0-rc2+ I pulled from git this morning.
My laptop just froze, and when I rebooted I noticed
that/var/log/messagescontained 48 thousand (!) warnings coming from ath9k since a few hours ago. I'm pasting the first one:
Quoted 4 lines
------------[ cut here ]------------ WARNING: at /home/mafra/linux-2.6/drivers/net/wireless/ath/ath9k/rc.c:697 ath_rc_get_highest_rix+0x156/0x210 [ath9k]() Hardware name: VPCEB4X1E
I believe I found a solution for this today. Please see this bug
tracker: https://bugzilla.redhat.com/show_bug.cgi?id=768639
While Fedora users report a warning, I've seen panic reports in the list. It's a memory corruption bug, so it can manifest in different ways. Please test the latest patch (attached).
Here's my comment to the patch:
This patch is based on my analysis of printk() output I added to the ath9k driver. I didn't have a chance to test the patch, so testing would be greatly appreciated.
The corruption must be happening in ath_debug_stat_rc(), which is given the result of ath_rc_get_rateindex(). ath_rc_get_rateindex() can return -1, which causes ath_debug_stat_rc() to increment the value that lies 16 bytes before rcstats in struct ath_rate_priv. On 64-bit systems, that happens to be rate_table. Once the rate_table pointer is incremented, all data there becomes invalid, which leads to the warning. On 32-bit systems, the corruption should happen in neg_ht_rates.
The -1 value of idx in struct ieee80211_tx_rate is described in net/mac80211.h. I don't know why we have -1 there and how to reproduce the problem reliably. But -1 can be there and ath9k has no checks for it.
The patch introduces two protections: ath_rc_get_rateindex() never returns a negative value and ath_debug_stat_rc() checks the array bounds.
It may not be good enough for the kernel, but it may be good enough for Fedora.
Signature · 9 lines
Regards, Pavel Roskin -------------- next part -------------- A non-text attachment was scrubbed... Name: 01-rix-check.patch Type: text/x-patch Size: 1280 bytes Desc: not available Url : http://lists.ath9k.org/pipermail/ath9k-devel/attachments/20120206/721598be/attachment.bin
Quoted 44 lines · Pavel Roskin
Quoted 9 lines · On Mon, 6 Feb 2012 00:29:07 +0000 "Carlos R. Mafra" <[email protected]> wrote:
I'm testing the latest kernel 3.3.0-rc2+ I pulled from git this morning.
My laptop just froze, and when I rebooted I noticed
that/var/log/messagescontained 48 thousand (!) warnings coming from ath9k since a few hours ago. I'm pasting the first one:Quoted 4 lines
------------[ cut here ]------------ WARNING: at /home/mafra/linux-2.6/drivers/net/wireless/ath/ath9k/rc.c:697 ath_rc_get_highest_rix+0x156/0x210 [ath9k]() Hardware name: VPCEB4X1EI believe I found a solution for this today. Please see this bug
tracker: https://bugzilla.redhat.com/show_bug.cgi?id=768639While Fedora users report a warning, I've seen panic reports in the list. It's a memory corruption bug, so it can manifest in different ways. Please test the latest patch (attached).
Here's my comment to the patch:
This patch is based on my analysis of
printk()output I added to the ath9k driver. I didn't have a chance to test the patch, so testing would be greatly appreciated.The corruption must be happening in
ath_debug_stat_rc(), which is given the result ofath_rc_get_rateindex().ath_rc_get_rateindex()can return -1, which causesath_debug_stat_rc()to increment the value that lies 16 bytes before rcstats in structath_rate_priv. On 64-bit systems, that happens to berate_table. Once therate_tablepointer is incremented, all data there becomes invalid, which leads to the warning. On 32-bit systems, the corruption should happen inneg_ht_rates.The -1 value of idx in struct
ieee80211_tx_rateis described innet/mac80211.h. I don't know why we have -1 there and how to reproduce the problem reliably. But -1 can be there and ath9k has no checks for it.
Quoted 6 lines
The patch introduces two protections:
ath_rc_get_rateindex()never returns a negative value andath_debug_stat_rc()checks the array bounds.It may not be good enough for the kernel, but it may be good enough for Fedora.
Thanks for the link to the bugzilla and for the attached patch.
I'm currently testing it, and so far so good.
Quoted 42 lines
Prevent memory corruption in ath9k rate control algorithm
From: Pavel Roskin <[email protected]>
Check
final_rateinath_debug_stat_rc(). Don't return negative values fromath_rc_get_rateindex(), callers don't expect it.Signed-off-by Pavel Roskin <[email protected]>---
drivers/net/wireless/ath/ath9k/rc.c | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)drivers/net/wireless/ath/ath9k/rc.c +10
-0@@ -385,6 +385,11 @@ static int ath_rc_get_rateindex(const struct ath_rate_table *rate_table,int rix =0 , i =0 ;static const int mcs_rix_off[] = {7 ,15 ,20 ,21 ,22 ,23 }; +if (rate->idx <0 ) { +printk (KERN_ERR "%s: rate->idx = %d\n" , __func__, rate->idx); +return 0 ; + } +if (!(rate->flags &IEEE80211_TX_RC_MCS ))return rate->idx;@@ -1324,6 +1329,11 @@ static void ath_debug_stat_rc(struct ath_rate_priv *rc, int final_rate) {struct ath_rc_stats *stats; +if (final_rate <0 || final_rate >=RATE_TABLE_SIZE ) { +printk (KERN_ERR "%s: invalid final_rate: %d\n" , __func__, + final_rate); +return ; + } stats = &rc->rcstats[final_rate]; stats->success++; }
Calvin Owens
Tested-byI've been having this issue on x86_64 with 3.3-rc2+ as well. I got a panic a couple days ago, but I haven't been able to reproduce it.
I can, however, reproduce the WARN_ON()'s on my laptop by pinging an external server with a high frequency (.1s works well) while walking in circles around the 2nd floor of a nearby library. (It's kind of hilarious, but that's the only way I seem to be able to get it).
I can confirm that Pavel's patch resolves the problem. Feel free to add:
Thanks,
Calvin