Skip to contentSkip to conversation

livelock with hrtimer cpu_base->lock

DISCUSSION Sodagudi Prasad 2 participants 4 messages Oct 9, 2018 – Oct 12, 2018

lore
Share & tools
lore
#0 lore raw

Hi Will,

This is regarding - thread "try to fix contention between expire_timers and try_to_del_timer_sync".
https://lkml.org/lkml/2017/7/28/172

I think this live lockup issue was discussed earlier but the final set of changes were not concluded.
I would like to check whether you have new updates on this issue or not.
This problem is observed with 4.14 .64 stable kernel too.
We see this problem 2 times in overnight testing.

I have to add the following code to avoid live lock. I am thinking that fixing this at the cpu_relax() level.

+++ b/kernel/time/hrtimer.c
@@ -52,6 +52,7 @@
#include <linux/timer.h>
#include <linux/freezer.h>
#include <linux/compat.h>
+#include <linux/delay.h>

  #include <linux/uaccess.h>

@@ -152,6 +153,7 @@ struct hrtimer_clock_base *lock_hrtimer_base(const struct hrtimer *timer,

raw_spin_unlock_irqrestore(&base->cpu_base->lock, *flags);
                 }
                 cpu_relax();
+               udelay(1);
         }

}

@@ -1067,6 +1069,7 @@ int hrtimer_cancel(struct hrtimer *timer)

                 if (ret >= 0)
                         return ret;
                 cpu_relax();
+               udelay(1);
         }

}
EXPORT_SYMBOL_GPL(hrtimer_cancel);

Note:- Timer event streaming is enabled and still live lock observed.

-Thanks, Prasad

Signature · 3 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum,
Linux Foundation Collaborative Project
#1 lore raw

Hi Prasad,

Quoted 6 lines · Sodagudi Prasad

This is regarding - thread "try to fix contention between expire_timers and try_to_del_timer_sync".
https://lkml.org/lkml/2017/7/28/172

I think this live lockup issue was discussed earlier but the final set of changes were not concluded.

Well we basically need a way to pick a value for CPU_RELAX_WFE_THRESHOLD. Do you have any ideas? It could be determined at runtime if necessary.

Quoted 26 lines

I would like to check whether you have new updates on this issue or not. This problem is observed with 4.14 .64 stable kernel too.
We see this problem 2 times in overnight testing.

I have to add the following code to avoid live lock. I am thinking that fixing this at the cpu_relax() level.

+++ b/kernel/time/hrtimer.c

(quoted diff) +2 -0
@@ -52,6 +52,7 @@
 #include <linux/timer.h>
 #include <linux/freezer.h>
 #include <linux/compat.h>
+#include <linux/delay.h>
 
 #include <linux/uaccess.h>
 
@@ -152,6 +153,7 @@ struct hrtimer_clock_base *lock_hrtimer_base(const
 struct hrtimer *timer,
 raw_spin_unlock_irqrestore(&base->cpu_base->lock, *flags);
                }
                cpu_relax();
+               udelay(1);
        }
 }

Review int hrtimer_cancel(struct hrtimer *timer) · line 1075

Quoted 7 lines
(quoted diff) +1 -0
@@ -1067,6 +1069,7 @@ int hrtimer_cancel(struct hrtimer *timer)
                if (ret >= 0)
                        return ret;
                cpu_relax();
+               udelay(1);
        }
 }
 EXPORT_SYMBOL_GPL(hrtimer_cancel);

This is just another bodge and likely to hurt in places where 1us is excessive because there isn't contention.

Will

#2 lore raw
Quoted 8 lines · Will Deacon

Hi Prasad,

Quoted 5 lines · Sodagudi Prasad

This is regarding - thread "try to fix contention between expire_timers and
try_to_del_timer_sync".
https://lkml.org/lkml/2017/7/28/172

Quoted 8 lines
Quoted 3 lines

I think this live lockup issue was discussed earlier but the final set of
changes were not concluded.

Well we basically need a way to pick a value for CPU_RELAX_WFE_THRESHOLD.
Do you have any ideas? It could be determined at runtime if necessary.

Hi Will,

Please share what are values need to be tried for
CPU_RELAX_WFE_THRESHOLD.

It would be great if it can be determined from runtime. Please let me
know
if any testing need to be done with dynamic detection patch.

-Thanks, Prasad

Quoted 40 lines
Quoted 28 lines

I would like to check whether you have new updates on this issue or not.
This problem is observed with 4.14 .64 stable kernel too. We see this problem 2 times in overnight testing.

I have to add the following code to avoid live lock. I am thinking that
fixing this at the cpu_relax() level.

+++ b/kernel/time/hrtimer.c

(quoted diff) +2 -0
@@ -52,6 +52,7 @@
 #include <linux/timer.h>
 #include <linux/freezer.h>
 #include <linux/compat.h>
+#include <linux/delay.h>
 
 #include <linux/uaccess.h>
 
@@ -152,6 +153,7 @@ struct hrtimer_clock_base *lock_hrtimer_base(const
 struct hrtimer *timer,
 raw_spin_unlock_irqrestore(&base->cpu_base->lock, *flags);
                }
                cpu_relax();
+               udelay(1);
        }
 }

Review int hrtimer_cancel(struct hrtimer *timer) · line 1075

Quoted 7 lines
(quoted diff) +1 -0
@@ -1067,6 +1069,7 @@ int hrtimer_cancel(struct hrtimer *timer)
                if (ret >= 0)
                        return ret;
                cpu_relax();
+               udelay(1);
        }
 }
 EXPORT_SYMBOL_GPL(hrtimer_cancel);

This is just another bodge and likely to hurt in places where 1us is excessive because there isn't contention.

Will

Signature · 3 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum,
Linux Foundation Collaborative Project
#3 lore raw
Quoted 11 lines · Sodagudi Prasad
Quoted 10 lines · Will Deacon
Quoted 8 lines · Sodagudi Prasad

This is regarding - thread "try to fix contention between expire_timers and
try_to_del_timer_sync".
https://lkml.org/lkml/2017/7/28/172

I think this live lockup issue was discussed earlier but the final set of
changes were not concluded.

Quoted 8 lines
Quoted 3 lines

Well we basically need a way to pick a value for CPU_RELAX_WFE_THRESHOLD. Do you have any ideas? It could be determined at runtime if necessary.

Please share what are values need to be tried for CPU_RELAX_WFE_THRESHOLD.

It would be great if it can be determined from runtime. Please let me know
if any testing need to be done with dynamic detection patch.

I was actually hoping you'd have some ideas about how to write the patch. I don't have a good feel for how to determine this threshold, and you have hardware that needs it...

One avenue of exploration might be to tune the fallback to WFE based on what percentage of the eventstream period we've been polling, e.g. 50%.

Will