Something goes wrong with timer statistics.
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpg
If i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Quoted 7 lines · Ian Kumlien
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 8 lines
Quoted 6 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
(Where is my magic ball?)
(Lost? =))
Quoted 1 line
Let me guess - Pentium 4 with HT?
Core 2 Duo, without HT =)
Quoted 2 lines
Also, please try the latest hrt tree
http://tglx.de/projects/hrtimers/2.6.22-rc3/
If you still think i should try that, i will, this box is not in production yet for a while so... =)
I switched to using 64 bit hpet after the last crash, will this influence the results?
Quoted 3 lines
Regards,
Michal
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 3 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Added more wizards :)
Quoted 11 lines · Ian Kumlien
Quoted 10 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpg
This was reported before. It's incredibly hard to reproduce.
Ian, is this happening reproducible ?
Quoted 6 lines
Quoted 2 lines
If i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
(Where is my magic ball?)
Let me guess - Pentium 4 with HT?
I might be wrong, but it's a newer core duo thingy.
tglx
Quoted 15 lines · Thomas Gleixner
Quoted 3 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Added more wizards :)
Quoted 8 lines · Ian Kumlien
Quoted 7 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgThis was reported before. It's incredibly hard to reproduce.
Ian, is this happening reproducible ?
Seems like it, i haven't actually TRIED to crash yet... But i can try if i don't have to crash in the future =)
(The machine is just a motherboard lying on the floor with some disks attached, so i have to bring out my pocket knife and short circuit the right pins =))
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
If i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
(Where is my magic ball?)
Let me guess - Pentium 4 with HT?
I might be wrong, but it's a newer core duo thingy.
tglx
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 15 lines · Thomas Gleixner
Quoted 3 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Added more wizards :)
Quoted 8 lines · Ian Kumlien
Quoted 7 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgThis was reported before. It's incredibly hard to reproduce.
Hm, tstats_write() only grabs show_mutex, but it calls reset_entries() which happily messes with the entries table, shouldn't the table_lock also be held in that case?
Björn
Quoted 20 lines · Björn Steinbrink
Quoted 19 lines · Thomas Gleixner
Quoted 3 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Added more wizards :)
Quoted 11 lines · Ian Kumlien
Quoted 10 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpg
Quoted 5 lines
Quoted 1 line
This was reported before. It's incredibly hard to reproduce.
Hm,
tstats_write()only grabsshow_mutex, but it callsreset_entries()which happily messes with the entries table, shouldn't thetable_lockalso be held in that case?
Hrmpf, nevermind, I missed the active flag...
Sorry,
Björn
Quoted 15 lines · Thomas Gleixner
Quoted 3 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Added more wizards :)
Quoted 8 lines · Ian Kumlien
Quoted 7 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgThis was reported before. It's incredibly hard to reproduce.
OK, second try, this time with a patch. In timer_stats_update_stats, input is allocated on the stack, so it is uninitialized, in particular the "next" field is random. Now in tstat_lookup, the new entry "curr" is initialized with the values from "input" (passed as "entry") and "next" is set to NULL _after_ it is added to the list, so if a second CPU is running the fastpath lookup while we're inserting the new element, it might get the garbage value in "next". The patch below fixes that.
Björn
Initialize the "next" field of a timer stats entry before it is inserted into the list to avoid a race with the fastpath lookup.
---
kernel/time/timer_stats.c +1 -1
@@ -202,12 +202,12 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ; + curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN );if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);
Quoted 48 lines · On Wed, 30 May 2007 13:38:25 +0200 Björn Steinbrink <[email protected]> wrote:
Quoted 15 lines · Thomas Gleixner
Quoted 3 lines · Michal Piotrowski
Hi Ian,
(Thomas "The Wizard of Time" added to CC :))
Added more wizards :)
Quoted 8 lines · Ian Kumlien
Quoted 7 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgThis was reported before. It's incredibly hard to reproduce.
OK, second try, this time with a patch. In
timer_stats_update_stats, input is allocated on the stack, so it is uninitialized, in particular the "next" field is random. Now intstat_lookup, the new entry "curr" is initialized with the values from "input" (passed as "entry") and "next" is set to NULL _after_ it is added to the list, so if a second CPU is running the fastpath lookup while we're inserting the new element, it might get the garbage value in "next". The patch below fixes that.Björn
Initialize the "next" field of a timer stats entry before it is inserted into the list to avoid a race with the fastpath lookup.
Signed-off-by Björn Steinbrink <[email protected]>---
kernel/time/timer_stats.c +0
-0@@ -202,12 +202,12 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ;
kernel/time/timer_stats.c static struct entry *tstat_lookup(struct entry *entry, char *comm) · line 213
Quoted 10 lines
kernel/time/timer_stats.c +1
-1+ curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN );if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);
Your analysis might be right, not the fix.
You *cannot* assume curr->next = NULL will be done before insert.
You probably also need a memory barrier.
curr->next = NULL; memcpy(curr->comm, comm, TASK_COMM_LEN); smp_mb(); /* commit all writes to curr before insert */ if (prev) prev->next = curr; else *head = curr;
Quoted 58 lines · Eric Dumazet
Quoted 42 lines · On Wed, 30 May 2007 13:38:25 +0200 Björn Steinbrink <[email protected]> wrote:
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · Michal Piotrowski
Quoted 7 lines · Ian Kumlien
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgThis was reported before. It's incredibly hard to reproduce.
OK, second try, this time with a patch. In
timer_stats_update_stats, input is allocated on the stack, so it is uninitialized, in particular the "next" field is random. Now intstat_lookup, the new entry "curr" is initialized with the values from "input" (passed as "entry") and "next" is set to NULL _after_ it is added to the list, so if a second CPU is running the fastpath lookup while we're inserting the new element, it might get the garbage value in "next". The patch below fixes that.Björn
Initialize the "next" field of a timer stats entry before it is inserted into the list to avoid a race with the fastpath lookup.
Signed-off-by Björn Steinbrink <[email protected]>---
kernel/time/timer_stats.c +0
-0@@ -202,12 +202,12 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ;kernel/time/timer_stats.c static struct entry *tstat_lookup(struct entry *entry, char *comm) · line 213
Quoted 10 lines
kernel/time/timer_stats.c +1
-1+ curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN );if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);Your analysis might be right, not the fix.
Quoted 3 lines
You *cannot* assume
curr->next= NULL will be done before insert.You probably also need a memory barrier.
Ehrm, right. I somehow thought of the spinlock being enough to satisfy that constraint, but if that was the case, the whole problem wouldn't exist in the first place. D'oh!
Thanks,
Björn
Initialize the "next" field of a timer stats entry before it is inserted into the list to avoid a race with the fastpath lookup.
Thanks to Eric Dumazet for reminding me of the memory barrier.
---
kernel/time/timer_stats.c +4 -1
@@ -202,12 +202,15 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ; + curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN ); + +smp_mb ();/* Ensure that curr is initialized before insert */ +if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · On 2007.05.31 12:20:47 +0200, [email protected] wrote:
Quoted 2 lines · Björn Steinbrink
Initialize the "next" field of a timer stats entry before it is inserted into the list to avoid a race with the fastpath lookup.
Sorry to say... but this does not fix my problem, however, i can't reach that machine at all atm, but i will post the oops later today.
Ok, three times is a charm, right? ;-) The previous patch did fix a race, but there's still one left. The hash table is never cleared, so it can still point into the entries array.
As long as we do not race with the insertion, that's not a directly visible problem as all fields are 0, so we get no match and the lookup finishes right away. We probably _can_ get weird results though, as the all-zero entry isn't marked as used, but is used as "head" and "prev" in this case and even multiple hash entries might point to it in some cases. That would eventually lead to lost entries because the "next" field gets overwritten when alloc_entry() finally claims the entry.
But when we race, we can still end up between "*curr = *entry" (makes "next" contain garbage) and "curr->next = NULL", while we're doing the fastpath lookup, because the insertion basically already took place! Replacing "curr->next = NULL" with "entry->next = NULL" and moving it up is pointless, as "*curr = *entry" isn't atomic, so instead, we have to simply clean the hash table when the entries are cleared.
Björn
Fix two races in the timer stats lookup code. One by ensuring that the initialization of a new entry is finished upon insertion of that entry. The other by cleaning up the hash table when the entries array is cleared, so that we don't have "pre-inserted" entries.
Thanks to Eric Dumazet for reminding me of the memory barrier.
---
kernel/time/timer_stats.c +20 -16
@@ -117,21 +117,6 @@ static struct entry entries[MAX_ENTRIES];static atomic_t overflow_count; -static void reset_entries (void ) -{ - nr_entries =0 ; -memset (entries,0 ,sizeof (entries)); -atomic_set (&overflow_count,0 ); -} - -static struct entry *alloc_entry (void ) -{ -if (nr_entries >=MAX_ENTRIES ) -return NULL ; - -return entries + nr_entries++; -} -/* * The entries are in a hash-table, for fast lookup: */
@@ -149,6 +134,22 @@ static struct entry *alloc_entry(void)static struct entry *tstat_hash_table[TSTAT_HASH_SIZE ]__read_mostly ; +static void reset_entries (void ) +{ + nr_entries =0 ; +memset (entries,0 ,sizeof (entries)); +memset (tstat_hash_table,0 ,sizeof (tstat_hash_table)); +atomic_set (&overflow_count,0 ); +} + +static struct entry *alloc_entry (void ) +{ +if (nr_entries >=MAX_ENTRIES ) +return NULL ; + +return entries + nr_entries++; +} +static int match_entries (struct entry *entry1,struct entry *entry2) { return entry1->timer == entry2->timer &&
@@ -202,12 +203,15 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ; + curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN ); + +smp_mb ();/* Ensure that curr is initialized before insert */ +if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);
Quoted 96 lines · On Thu, 31 May 2007 16:25:22 +0200 Björn Steinbrink <[email protected]> wrote:
Quoted 6 lines · On 2007.05.31 12:20:47 +0200, [email protected] wrote:
Quoted 2 lines · Björn Steinbrink
Initialize the "next" field of a timer stats entry before it is inserted into the list to avoid a race with the fastpath lookup.
Sorry to say... but this does not fix my problem, however, i can't reach that machine at all atm, but i will post the oops later today.
Ok, three times is a charm, right? ;-) The previous patch did fix a race, but there's still one left. The hash table is never cleared, so it can still point into the entries array.
As long as we do not race with the insertion, that's not a directly visible problem as all fields are 0, so we get no match and the lookup finishes right away. We probably _can_ get weird results though, as the all-zero entry isn't marked as used, but is used as "head" and "prev" in this case and even multiple hash entries might point to it in some cases. That would eventually lead to lost entries because the "next" field gets overwritten when
alloc_entry()finally claims the entry.But when we race, we can still end up between "*curr = *entry" (makes "next" contain garbage) and "
curr->next= NULL", while we're doing the fastpath lookup, because the insertion basically already took place! Replacing "curr->next= NULL" with "entry->next= NULL" and moving it up is pointless, as "*curr = *entry" isn't atomic, so instead, we have to simply clean the hash table when the entries are cleared.Björn
Fix two races in the timer stats lookup code. One by ensuring that the initialization of a new entry is finished upon insertion of that entry. The other by cleaning up the hash table when the entries array is cleared, so that we don't have "pre-inserted" entries.
Thanks to Eric Dumazet for reminding me of the memory barrier.
Signed-off-by Björn Steinbrink <[email protected]>---
kernel/time/timer_stats.c +19
-15@@ -117,21 +117,6 @@ static struct entry entries[MAX_ENTRIES];static atomic_t overflow_count; -static void reset_entries (void ) -{ - nr_entries =0 ; -memset (entries,0 ,sizeof (entries)); -atomic_set (&overflow_count,0 ); -} - -static struct entry *alloc_entry (void ) -{ -if (nr_entries >=MAX_ENTRIES ) -return NULL ; - -return entries + nr_entries++; -} -/* * The entries are in a hash-table, for fast lookup: */ @@ -149,6 +134,22 @@ static struct entry *alloc_entry(void)static struct entry *tstat_hash_table[TSTAT_HASH_SIZE ]__read_mostly ; +static void reset_entries (void ) +{ + nr_entries =0 ; +memset (entries,0 ,sizeof (entries)); +memset (tstat_hash_table,0 ,sizeof (tstat_hash_table)); +atomic_set (&overflow_count,0 ); +} + +static struct entry *alloc_entry (void ) +{ +if (nr_entries >=MAX_ENTRIES ) +return NULL ; + +return entries + nr_entries++; +} +static int match_entries (struct entry *entry1,struct entry *entry2) { return entry1->timer == entry2->timer &&@@ -202,12 +203,15 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ; + curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN ); + +smp_mb ();/* Ensure that curr is initialized before insert */
kernel/time/timer_stats.c static struct entry *tstat_lookup(struct entry *entry, char *comm)
Quoted 10 lines
kernel/time/timer_stats.c +1
-2+if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock); -
Well... :) , there is still a memory barrier missing it seems.
Another cpu might see a bad value if 'active=1' is set before tstat_hash_table is really cleared.
diff -urp linux/kernel/time/timer_stats.c.orig linux/kernel/time/timer_stats.c
time/timer_stats.c renamed from time/timer_stats.c.orig +1 -0
@@ -361,6 +361,7 @@ static ssize_t tstats_write(struct fileif (!active) {reset_entries (); time_start =ktime_get (); +smp_mb (); active =1 ; }break ;
Quoted 3 lines · Eric Dumazet
Well... :) , there is still a memory barrier missing it seems.
Another cpu might see a bad value if 'active=1' is set before
tstat_hash_tableis really cleared.
Hm, that even makes the assumption in my first try valid ;-) Just for the record, this time I thought that the barrier from the spinlock in timer_stats_update_stats (right before the check for active) would be enough, but that's obviously running on the wrong cpu if we race... *sigh*
Thanks,
Björn
Fix two races in the timer stats lookup code. One by ensuring that the initialization of a new entry is finished upon insertion of that entry. The other by cleaning up the hash table when the entries array is cleared, so that we don't have "pre-inserted" entries.
Thanks to Eric Dumazet for reminding me of the memory barriers.
---
kernel/time/timer_stats.c +21 -16
@@ -117,21 +117,6 @@ static struct entry entries[MAX_ENTRIES];static atomic_t overflow_count; -static void reset_entries (void ) -{ - nr_entries =0 ; -memset (entries,0 ,sizeof (entries)); -atomic_set (&overflow_count,0 ); -} - -static struct entry *alloc_entry (void ) -{ -if (nr_entries >=MAX_ENTRIES ) -return NULL ; - -return entries + nr_entries++; -} -/* * The entries are in a hash-table, for fast lookup: */
@@ -149,6 +134,22 @@ static struct entry *alloc_entry(void)static struct entry *tstat_hash_table[TSTAT_HASH_SIZE ]__read_mostly ; +static void reset_entries (void ) +{ + nr_entries =0 ; +memset (entries,0 ,sizeof (entries)); +memset (tstat_hash_table,0 ,sizeof (tstat_hash_table)); +atomic_set (&overflow_count,0 ); +} + +static struct entry *alloc_entry (void ) +{ +if (nr_entries >=MAX_ENTRIES ) +return NULL ; + +return entries + nr_entries++; +} +static int match_entries (struct entry *entry1,struct entry *entry2) { return entry1->timer == entry2->timer &&
@@ -202,12 +203,15 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ; + curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN ); + +smp_mb ();/* Ensure that curr is initialized before insert */ +if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);
@@ -360,6 +364,7 @@ static ssize_t tstats_write(struct file *file, const char __user *buf,if (!active) {reset_entries (); time_start =ktime_get (); +smb_mb (); active =1 ; }break ;
Quoted 16 lines · Björn Steinbrink
Quoted 3 lines · Eric Dumazet
Well... :) , there is still a memory barrier missing it seems.
Another cpu might see a bad value if 'active=1' is set before
tstat_hash_tableis really cleared.Hm, that even makes the assumption in my first try valid ;-) Just for the record, this time I thought that the barrier from the spinlock in
timer_stats_update_stats(right before the check for active) would be enough, but that's obviously running on the wrong cpu if we race... *sigh*Thanks,
Björn
Quoted 6 lines
Fix two races in the timer stats lookup code. One by ensuring that the initialization of a new entry is finished upon insertion of that entry. The other by cleaning up the hash table when the entries array is cleared, so that we don't have "pre-inserted" entries.
Thanks to Eric Dumazet for reminding me of the memory barriers.
Fix the comment below and you can add:
It's currently been running for the longest period ever, ie, 11 minutes =)
I'm gonna leave it running during the night and send a status update when the evil daystar reaches it's peak CET. (i haven't been able to stop since linus mentioned it... damn it... =))
kernel/time/timer_stats.c static ssize_t tstats_write(struct file *file, const char __user *buf, · line 367
Quoted 8 lines
Signed-off-by Björn Steinbrink <[email protected]>---
kernel/time/timer_stats.c +1
-0@@ -360,6 +364,7 @@ static ssize_t tstats_write(struct file *file, const char __user *buf,if (!active) {reset_entries (); time_start =ktime_get (); +smb_mb ();
smb? you mean smp =)
Quoted 3 lines
kernel/time/timer_stats.c +0
-0active =1 ; }break ;
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Quoted 13 lines · Ian Kumlien
Quoted 11 lines · Björn Steinbrink
Quoted 3 lines · Eric Dumazet
Well... :) , there is still a memory barrier missing it seems.
Another cpu might see a bad value if 'active=1' is set before
tstat_hash_tableis really cleared.Hm, that even makes the assumption in my first try valid ;-) Just for the record, this time I thought that the barrier from the spinlock in
timer_stats_update_stats(right before the check for active) would be enough, but that's obviously running on the wrong cpu if we race... *sigh*
Quoted 5 lines
Fix the comment below and you can add:
Signed-off-by Ian Kumlien <[email protected]>It's currently been running for the longest period ever, ie, 11 minutes =)
Finally! :-)
Quoted 11 lines
I'm gonna leave it running during the night and send a status update when the evil daystar reaches it's peak CET. (i haven't been able to stop since linus mentioned it... damn it... =))
Review static ssize_t tstats_write(struct file *file, const char __user *buf, · line 367
Quoted 5 lines
(quoted diff) +1
-0@@ -360,6 +364,7 @@ static ssize_t tstats_write(struct file *file, const char __user *buf, if (!active) { reset_entries(); time_start = ktime_get(); + smb_mb();smb? you mean smp =)
No idea how I managed to break that after test compiling it... :-/ Thanks for catching it :-)
So unless I messed up something again, here's the final patch.
Thanks,
Björn
Fix two races in the timer stats lookup code. One by ensuring that the initialization of a new entry is finished upon insertion of that entry. The other by cleaning up the hash table when the entries array is cleared, so that we don't have any "pre-inserted" entries.
Thanks to Eric Dumazet for reminding me of the memory barriers.
---
kernel/time/timer_stats.c +21 -16
@@ -117,21 +117,6 @@ static struct entry entries[MAX_ENTRIES];static atomic_t overflow_count; -static void reset_entries (void ) -{ - nr_entries =0 ; -memset (entries,0 ,sizeof (entries)); -atomic_set (&overflow_count,0 ); -} - -static struct entry *alloc_entry (void ) -{ -if (nr_entries >=MAX_ENTRIES ) -return NULL ; - -return entries + nr_entries++; -} -/* * The entries are in a hash-table, for fast lookup: */
@@ -149,6 +134,22 @@ static struct entry *alloc_entry(void)static struct entry *tstat_hash_table[TSTAT_HASH_SIZE ]__read_mostly ; +static void reset_entries (void ) +{ + nr_entries =0 ; +memset (entries,0 ,sizeof (entries)); +memset (tstat_hash_table,0 ,sizeof (tstat_hash_table)); +atomic_set (&overflow_count,0 ); +} + +static struct entry *alloc_entry (void ) +{ +if (nr_entries >=MAX_ENTRIES ) +return NULL ; + +return entries + nr_entries++; +} +static int match_entries (struct entry *entry1,struct entry *entry2) { return entry1->timer == entry2->timer &&
@@ -202,12 +203,15 @@ static struct entry *tstat_lookup(struct entry *entry, char *comm)if (curr) { *curr = *entry; curr->count =0 ; + curr->next =NULL ;memcpy (curr->comm, comm,TASK_COMM_LEN ); + +smp_mb ();/* Ensure that curr is initialized before insert */ +if (prev) prev->next = curr;else *head = curr; - curr->next =NULL ; } out_unlock:spin_unlock (&table_lock);
@@ -360,6 +364,7 @@ static ssize_t tstats_write(struct file *file, const char __user *buf,if (!active) {reset_entries (); time_start =ktime_get (); +smp_mb (); active =1 ; }break ;
Ingo Molnar
Acked-byQuoted 7 lines · Björn Steinbrink
Quoted 5 lines
Fix the comment below and you can add:
Signed-off-by Ian Kumlien <[email protected]>It's currently been running for the longest period ever, ie, 11 minutes =)
Finally! :-)
cool! Great detective work Björn!
Quoted 4 lines
Thanks to Eric Dumazet for reminding me of the memory barriers.
Signed-off-by Björn Steinbrink <[email protected]>Signed-off-by Ian Kumlien <[email protected]>
Ingo
From: Ian Kumlien <[email protected]>
Date: Tue, 29 May 2007 23:38:48 +0200
Quoted 11 lines
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
I see this same exact problem on sparc64.
Quoted 8 lines · David Miller
From: Ian Kumlien <[email protected]>
Date: Tue, 29 May 2007 23:38:48 +0200Quoted 5 lines
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 8 lines
Quoted 6 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
I see this same exact problem on sparc64.
100% reproducible ?
tglx
From: Thomas Gleixner <[email protected]>
Date: Wed, 30 May 2007 00:42:12 +0200
Quoted 4 lines · David Miller
Quoted 1 line
I see this same exact problem on sparc64.
100% reproducible ?
I think so, I'll try to find time to tinker with it a bit more.
Quoted 10 lines · David Miller
From: Thomas Gleixner <[email protected]>
Date: Wed, 30 May 2007 00:42:12 +0200Quoted 4 lines · David Miller
Quoted 1 line
I see this same exact problem on sparc64.
100% reproducible ?
I think so, I'll try to find time to tinker with it a bit more.
It seems to be a subtle bug in the timer stats code, but I did not spot it.
tglx
A message that is not in the archive (sent off-list or not yet received): [email protected]
From: "Michal Piotrowski" <[email protected]>
Date: Wed, 30 May 2007 00:41:46 +0200
Quoted 13 lines · David Miller
Quoted 12 lines
From: Ian Kumlien <[email protected]>
Date: Tue, 29 May 2007 23:38:48 +0200Quoted 9 lines
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpg
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
If i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
I see this same exact problem on sparc64.
Have you tried this patch?
http://lkml.org/lkml/2007/5/29/392
Of course, I worked with Thomas on the fix :-)
Quoted 19 lines · David Miller
From: "Michal Piotrowski" <[email protected]>
Date: Wed, 30 May 2007 00:41:46 +0200Quoted 16 lines · David Miller
Quoted 15 lines
From: Ian Kumlien <[email protected]>
Date: Tue, 29 May 2007 23:38:48 +0200Quoted 11 lines
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I see this same exact problem on sparc64.
Have you tried this patch?
http://lkml.org/lkml/2007/5/29/392Of course, I worked with Thomas on the fix :-)
I dunno if this applies to me, i run a 32 bit userspace...
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
From: Ian Kumlien <[email protected]>
Date: Wed, 30 May 2007 00:51:52 +0200
Quoted 20 lines · David Miller
Quoted 19 lines
From: "Michal Piotrowski" <[email protected]>
Date: Wed, 30 May 2007 00:41:46 +0200Quoted 16 lines · David Miller
Quoted 15 lines
From: Ian Kumlien <[email protected]>
Date: Tue, 29 May 2007 23:38:48 +0200Quoted 11 lines
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I see this same exact problem on sparc64.
Have you tried this patch?
http://lkml.org/lkml/2007/5/29/392Of course, I worked with Thomas on the fix :-)
I dunno if this applies to me, i run a 32 bit userspace...
That patch fixes a problem on 64-bit kernels, regardless of userspace, when NOHZ is enabled.
Quoted 28 lines · David Miller
From: Ian Kumlien <[email protected]>
Date: Wed, 30 May 2007 00:51:52 +0200Quoted 25 lines · David Miller
Quoted 24 lines
From: "Michal Piotrowski" <[email protected]>
Date: Wed, 30 May 2007 00:41:46 +0200Quoted 20 lines · David Miller
Quoted 8 lines
From: Ian Kumlien <[email protected]>
Date: Tue, 29 May 2007 23:38:48 +0200Quoted 5 lines
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 8 lines
Quoted 6 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
I see this same exact problem on sparc64.
Have you tried this patch?
http://lkml.org/lkml/2007/5/29/392
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Of course, I worked with Thomas on the fix :-)
I dunno if this applies to me, i run a 32 bit userspace...
That patch fixes a problem on 64-bit kernels, regardless of userspace, when NOHZ is enabled.
Sorry, i run all 32 bit, i was on the phone while typing that, so a wee bit distracted. Anyways, i run a nohz 32 bit kernel.
I dunno how good a intel T7200 is when it comes to 64 bit kernel/userspace.. (Haven't really read up on it)
Signature · 1 line
Ian Kumlien <pomac () vapor ! com> -- http://pomac.netswarm.net
Le Tue, May 29, 2007 at 11:38:48PM +0200, Ian Kumlien écrivait :
Quoted 7 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 6 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
Same here, P4 HT, if I run powertop for 5-10 minutes the system crash :( It happens with 2.6.22-rc{2,3} at least :( I didn't try before.
I don't have a oops since the last crash was in X, but next time I will try to catch it. If there is any patch around to fix this I am willing to test.
Regards
Signature · 3 lines
Stéphane Casset LOGIDÉE sàrl Se faire plaisir d'apprendre 1a, rue Pasteur Tel : +33 388 23 69 77 [email protected] F-67540 OSTWALD Fax : +33 388 23 69 77 http://logidee.com
Quoted 15 lines · Stephane Casset
Le Tue, May 29, 2007 at 11:38:48PM +0200, Ian Kumlien écrivait :
Quoted 7 lines
Hi,
As the daystar sets, i try to play some with my new would be firewall/server, but since this will be running for quite some time i have been experimenting with powertop to find out what i can do to limit it's power usage.
Quoted 6 lines
But, if i run powertop for too long or a few times to many... this happens:
http://pomac.netswarm.net/pics/kernel_panic.jpgIf i don't run powertop, it is rock solid... Compiling for hours, running memtest for hours etc etc...
Quoted 6 lines
Same here, P4 HT, if I run powertop for 5-10 minutes the system crash :( It happens with 2.6.22-rc{2,3} at least :( I didn't try before.
I don't have a oops since the last crash was in X, but next time I will try to catch it. If there is any patch around to fix this I am willing to test.
There is finally one that seems to work:
http://lkml.org/lkml/2007/5/31/394
I'd be glad if you could give it a try.
Thanks,
Björn