Enable arm_global_timer for Zynq brakes boot

Quoted 29 lines · Stephen Boyd

Quoted 28 lines · S?ren Brinkmann

Quoted 27 lines · Stephen Boyd

Quoted 15 lines · S?ren Brinkmann

Hi Stephen,

Quoted 12 lines · Stephen Boyd

Quoted 11 lines · S?ren Brinkmann

Hi all,

I'm trying to enable the arm_global_timer on Zynq platforms with the attached patch. Unfortunately that patch breaks booting up. It hangs when handing over to init/early userspace (see attached boot.log).

The funny thing is, if I remove either the global timer or the arm,cortex-a9-twd-timer node from my dts, it works. So, it looks like the two timer (drivers) interfere somehow. Does anybody have an idea of what is going on and probably even how to resolve it?

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

The patch is based on commit c0d15cc in Linus' tree.

If you boot with one CPU does it hang? It looks like secondary CPUs aren't getting interrupts but I'm not sure why. Maybe you can try this patch and/or put some prints in the timer interrupt handler to see if interrupts are firing on secondary CPUs.

Your proposed patch does not seem to make a difference, but adding 'maxcpus=1' to the kernel command line makes the system boot.

Hmm I guess that doesn't really confirm much because TWD doesn't register itself on maxcpus=1 systems, so it's the same as removing the node from DT. Probably easier to put a printk in the interrupt handler and confirm that you're receiving interrupts on secondary CPUs.

Quoted 6 lines

Quoted 3 lines

Turns out it does work when I disable Zynq's cpuidle driver. I think I can blame that driver.

Hmm.. Perhaps the arm_global_timer driver also needs FEAT_C3_STOP added to it. Do you know if that timer is reset during low power modes?

Our cpudidle driver is not powering off anything, AFAIK. I think it just executes 'wfi' on the CPU. I don't know how the timer core handles it, but I'd expect the CPU should find the timer just a it was left before entering idle (well, the timer continues to run I assume, but other than that).
I'll do some debugging and see if I can find out what exactly causes the hang.

Hi,

adding LKML and LAKML (which I forgot on the original email, sorry)

Quoted 37 lines · S?ren Brinkmann

Quoted 29 lines · Stephen Boyd

Quoted 28 lines · S?ren Brinkmann

Quoted 27 lines · Stephen Boyd

Quoted 15 lines · S?ren Brinkmann

Hi Stephen,

Quoted 12 lines · Stephen Boyd

Quoted 11 lines · S?ren Brinkmann

Hi all,

I'm trying to enable the arm_global_timer on Zynq platforms with the attached patch. Unfortunately that patch breaks booting up. It hangs when handing over to init/early userspace (see attached boot.log).

The funny thing is, if I remove either the global timer or the arm,cortex-a9-twd-timer node from my dts, it works. So, it looks like the two timer (drivers) interfere somehow. Does anybody have an idea of what is going on and probably even how to resolve it?

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

The patch is based on commit c0d15cc in Linus' tree.

If you boot with one CPU does it hang? It looks like secondary CPUs aren't getting interrupts but I'm not sure why. Maybe you can try this patch and/or put some prints in the timer interrupt handler to see if interrupts are firing on secondary CPUs.

Your proposed patch does not seem to make a difference, but adding 'maxcpus=1' to the kernel command line makes the system boot.

Hmm I guess that doesn't really confirm much because TWD doesn't register itself on maxcpus=1 systems, so it's the same as removing the node from DT. Probably easier to put a printk in the interrupt handler and confirm that you're receiving interrupts on secondary CPUs.

Quoted 6 lines

Quoted 3 lines

Turns out it does work when I disable Zynq's cpuidle driver. I think I can blame that driver.

Hmm.. Perhaps the arm_global_timer driver also needs FEAT_C3_STOP added to it. Do you know if that timer is reset during low power modes?

Quoted 7 lines

Our cpudidle driver is not powering off anything, AFAIK. I think it just executes 'wfi' on the CPU. I don't know how the timer core handles it, but I'd expect the CPU should find the timer just a it was left before entering idle (well, the timer continues to run I assume, but other than that).
I'll do some debugging and see if I can find out what exactly causes the hang.

So, what I found:
The Zynq cpuidle driver provides two idle states, which are both basically just ARM wfi states. But the second one set's these flags:
.flags = CPUIDLE_FLAG_TIME_VALID | CPUIDLE_FLAG_TIMER_STOP,

I don't know what these flags cause in detail. But the CPUIDLE_FLAG_TIMER_STOP seemed suspicious, since wfi does not have any effect on the timer. So, I removed that one and things are working.

I also tried the other approach: Leaving cpuidle as is and adding the C3STOP flag to the global timer. That solves it too.

Does anybody know what the correct solution is?
In case the C3STOP flag is considered to be corret for the timer, I could prepare a patch for that and bundle it with the one to enable the timer for Zynq?

@Michal: What do we do about the cpuidle driver? If you asked me, we should rip out the second state completely. We have it as disabled placeholder in our vendor tree and it seems to break things having it enabled in mainline.

S?ren

Quoted 59 lines · S?ren Brinkmann

Hi,

adding LKML and LAKML (which I forgot on the original email, sorry)

Quoted 37 lines · S?ren Brinkmann

Quoted 29 lines · Stephen Boyd

Quoted 28 lines · S?ren Brinkmann

Quoted 27 lines · Stephen Boyd

Quoted 15 lines · S?ren Brinkmann

Hi Stephen,

Quoted 12 lines · Stephen Boyd

Quoted 11 lines · S?ren Brinkmann

Hi all,

I'm trying to enable the arm_global_timer on Zynq platforms with the attached patch. Unfortunately that patch breaks booting up. It hangs when handing over to init/early userspace (see attached boot.log).

The funny thing is, if I remove either the global timer or the arm,cortex-a9-twd-timer node from my dts, it works. So, it looks like the two timer (drivers) interfere somehow. Does anybody have an idea of what is going on and probably even how to resolve it?

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

The patch is based on commit c0d15cc in Linus' tree.

If you boot with one CPU does it hang? It looks like secondary CPUs aren't getting interrupts but I'm not sure why. Maybe you can try this patch and/or put some prints in the timer interrupt handler to see if interrupts are firing on secondary CPUs.

Your proposed patch does not seem to make a difference, but adding 'maxcpus=1' to the kernel command line makes the system boot.

Hmm I guess that doesn't really confirm much because TWD doesn't register itself on maxcpus=1 systems, so it's the same as removing the node from DT. Probably easier to put a printk in the interrupt handler and confirm that you're receiving interrupts on secondary CPUs.

Quoted 6 lines

Quoted 3 lines

Turns out it does work when I disable Zynq's cpuidle driver. I think I can blame that driver.

Hmm.. Perhaps the arm_global_timer driver also needs FEAT_C3_STOP added to it. Do you know if that timer is reset during low power modes?

Quoted 7 lines

Our cpudidle driver is not powering off anything, AFAIK. I think it just executes 'wfi' on the CPU. I don't know how the timer core handles it, but I'd expect the CPU should find the timer just a it was left before entering idle (well, the timer continues to run I assume, but other than that).
I'll do some debugging and see if I can find out what exactly causes the hang.

So, what I found:
The Zynq cpuidle driver provides two idle states, which are both basically just ARM wfi states. But the second one set's these flags:
.flags = CPUIDLE_FLAG_TIME_VALID | CPUIDLE_FLAG_TIMER_STOP,

I don't know what these flags cause in detail. But the CPUIDLE_FLAG_TIMER_STOP seemed suspicious, since wfi does not have any effect on the timer. So, I removed that one and things are working.

Quoted 7 lines

I also tried the other approach: Leaving cpuidle as is and adding the C3STOP flag to the global timer. That solves it too.

Does anybody know what the correct solution is?
In case the C3STOP flag is considered to be corret for the timer, I could prepare a patch for that and bundle it with the one to enable the timer for Zynq?

Hi Soren,

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

The C3STOP flag has a similar semantic than the CPUIDLE_FLAG_TIMER_STOP, that is the timer can be shutdown with a specific idle state. This flag is used by the tick broadcast code.

If the C3STOP flag is not set for a local timer, the CPUIDLE_FLAG_TIMER_STOP does not make sense because it will be ignored by the tick-broadcast code.

If the local timer could be shutdown at idle time, you *must* specify this flag.

If the idle state shutdowns the cpu with its local timer, you *must* specify the CPUIDLE_FLAG_TIMER_STOP flag for this specific state.

At the first glance, the idle state #2 is aimed to do DDR self refresh and to switch to WFI, so no power gating, then no local timer down. The CPUIDLE_FLAG_TIMER_STOP shouldn't be used here.

IIUC, the global timer does not belong to the CPU and the cluster power domains, so it can't be shutdown: the C3STOP shouldn't be used.

I hope that helps.

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

In case of a vanilla kernel and with my patches for enabling the global timer (I removed the wrongly set flag from the C2 state + adding the DT fragment to use the GT), this is what I see (full output from timer_list attached):
Tick Device: mode: 1
Broadcast device
Clock Event Device: xttcps_clockevent

And the local timer seems to be the arm twd timer (also in both cases).

I don't think I can gather this information for the actual broken case, but AFAIK, there shouldn't be any other timer be capable of this for Zynq.

S?ren

Timer List Version: v0.7
HRTIMER_MAX_CLOCK_BASES: 3
now at 76779317814 nsecs

cpu: 0
clock 0:
.base: c14a0c90
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <c14a2b50>, menu_hrtimer_notify, S:01, hrtimer_start, swapper/0/0

 # expires at 76779328789-76779328789 nsecs [in 10975 to 10975 nsecs]
 #1: <c14a0dd8>, tick_sched_timer, S:01, hrtimer_start, swapper/0/0
 # expires at 76800000000-76800000000 nsecs [in 20682186 to 20682186 nsecs]
 #2: <ed956e40>, timerfd_tmrproc, S:01, hrtimer_start, systemd/1
 # expires at 93249172026-93249172026 nsecs [in 16469854212 to 16469854212 nsecs]
 #3: <ed956c00>, timerfd_tmrproc, S:01, hrtimer_start, systemd/1
 # expires at 900000055806-900000055806 nsecs [in 823220737992 to 823220737992 nsecs]
 clock 1:
  .base:       c14a0cc8
  .index:      1
  .resolution: 1 nsecs
  .get_time:   ktime_get_real
  .offset:     975789465870609368 nsecs
active timers:
 clock 2:
  .base:       c14a0d00
  .index:      2
  .resolution: 1 nsecs
  .get_time:   ktime_get_boottime
  .offset:     0 nsecs
active timers:
  .expires_next   : 76779492678 nsecs
  .hres_active    : 1
  .nr_events      : 2449
  .nr_retries     : 66
  .nr_hangs       : 0
  .max_hang_time  : 0 nsecs
  .nohz_mode      : 2
  .last_tick      : 76780000000 nsecs
  .tick_stopped   : 1
  .idle_jiffies   : 4294944973
  .idle_calls     : 3114
  .idle_sleeps    : 981
  .idle_entrytime : 76779354678 nsecs
  .idle_waketime  : 76779336246 nsecs
  .idle_exittime  : 76769862116 nsecs
  .idle_sleeptime : 70735903990 nsecs
  .iowait_sleeptime: 18432 nsecs
  .last_jiffies   : 4294944973
  .next_jiffies   : 4294944976
  .idle_expires   : 76800000000 nsecs
jiffies: 4294944973

cpu: 1
clock 0:
.base: c14a9c90
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <c14a9dd8>, tick_sched_timer, S:01, hrtimer_start_range_ns, swapper/1/0

 # expires at 76780000000-76780000000 nsecs [in 682186 to 682186 nsecs]
 clock 1:
  .base:       c14a9cc8
  .index:      1
  .resolution: 1 nsecs
  .get_time:   ktime_get_real
  .offset:     975789465870609368 nsecs
active timers:
 clock 2:
  .base:       c14a9d00
  .index:      2
  .resolution: 1 nsecs
  .get_time:   ktime_get_boottime
  .offset:     0 nsecs
active timers:
  .expires_next   : 76780000000 nsecs
  .hres_active    : 1
  .nr_events      : 2257
  .nr_retries     : 61
  .nr_hangs       : 0
  .max_hang_time  : 0 nsecs
  .nohz_mode      : 2
  .last_tick      : 76700000000 nsecs
  .tick_stopped   : 0
  .idle_jiffies   : 4294944965
  .idle_calls     : 2998
  .idle_sleeps    : 444
  .idle_entrytime : 76771944950 nsecs
  .idle_waketime  : 76768498136 nsecs
  .idle_exittime  : 76768498136 nsecs
  .idle_sleeptime : 72124544054 nsecs
  .iowait_sleeptime: 4276262 nsecs
  .last_jiffies   : 4294944973
  .next_jiffies   : 4294944974
  .idle_expires   : 78880000000 nsecs
jiffies: 4294944973

Tick Device: mode: 1
Broadcast device
Clock Event Device: xttcps_clockevent

 max_delta_ns:   1207932479
 min_delta_ns:   18432
 mult:           233015
 shift:          32
 mode:           1
 next_event:     9223372036854775807 nsecs
 set_next_event: xttcps_set_next_event
 set_mode:       xttcps_set_mode
 event_handler:  tick_handle_oneshot_broadcast
 retries:        0

tick_broadcast_mask: 00000000
tick_broadcast_oneshot_mask: 00000000

Tick Device: mode: 1
Per CPU device: 0
Clock Event Device: local_timer

 max_delta_ns:   12884902005
 min_delta_ns:   1000
 mult:           715827876
 shift:          31
 mode:           3
 next_event:     76800000000 nsecs
 set_next_event: twd_set_next_event
 set_mode:       twd_set_mode
 event_handler:  hrtimer_interrupt
 retries:        0

Tick Device: mode: 1
Per CPU device: 1
Clock Event Device: local_timer

 max_delta_ns:   12884902005
 min_delta_ns:   1000
 mult:           715827876
 shift:          31
 mode:           3
 next_event:     76780000000 nsecs
 set_next_event: twd_set_next_event
 set_mode:       twd_set_mode
 event_handler:  hrtimer_interrupt
 retries:        0

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Hmm, interesting. Can you give the ouput of /proc/interrupts also with the global timer ?

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Is it possible the global timer driver simply does not work ? So when the cpuidle driver switches to it, the system stays stuck with no interrupt.

Removing the CPUIDLE_FLAG_TIMER_STOP prevents to use the broadcast timer (aka arm global timer), so the problem does not appear.

And when the C3STOP feature flag is added to the global timer, this one can't be a broadcast timer, so another clock is selected for that (I guess cadence_ttc). So again the problem does not appear.

I am more and more convinced the problem is not coming from the cpuidle driver. The cpuidle flag has just spotted a problem somewhere else and I suspect the arm_global_timer is not working for zynq.

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

	 next_event:     108080000000 nsecs
	 set_next_event: gt_clockevent_set_next_event
	 set_mode:       gt_clockevent_set_mode
	 event_handler:  tick_handle_oneshot_broadcast
	 retries:        0
	
	tick_broadcast_mask: 00000001
	tick_broadcast_oneshot_mask: 00000000
	
	Tick Device: mode:     1
	Per CPU device: 0
	Clock Event Device: local_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3
	 next_event:     106900000000 nsecs
	 set_next_event: twd_set_next_event
	 set_mode:       twd_set_mode
	 event_handler:  hrtimer_interrupt
	 retries:        0

# cat /proc/interrupts 
	           CPU0       
	 27:        252       GIC  27  gt
	 29:        626       GIC  29  twd
	 43:          0       GIC  43  ttc_clockevent
	 82:        410       GIC  82  xuartps
	IPI0:          0  CPU wakeup interrupts
	IPI1:          0  Timer broadcast interrupts
	IPI2:          0  Rescheduling interrupts
	IPI3:          0  Function call interrupts
	IPI4:          0  Single function call interrupts
	IPI5:          0  CPU stop interrupts
	Err:          0

S?ren

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

	[ 4689.992658] CPU1: Booted secondary processor
	[ 4690.986295] CPU1: failed to come online
	sh: write error: Input/output error
	# [ 4691.045945] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
	[ 4691.045986] 
	[ 4691.052972] ===============================
	[ 4691.057349] [ INFO: suspicious RCU usage. ]
	[ 4691.061413] 3.11.0-rc3-00001-gc14f576-dirty #139 Not tainted
	[ 4691.067026] -------------------------------
	[ 4691.071129] kernel/sched/fair.c:5477 suspicious rcu_dereference_check() usage!
	[ 4691.078292] 
	[ 4691.078292] other info that might help us debug this:
	[ 4691.078292] 
	[ 4691.086209] 
	[ 4691.086209] RCU used illegally from offline CPU!
	[ 4691.086209] rcu_scheduler_active = 1, debug_locks = 0
	[ 4691.097216] 1 lock held by swapper/1/0:
	[ 4691.100968]  #0:  (rcu_read_lock){.+.+..}, at: [<c00679b4>] set_cpu_sd_state_idle+0x0/0x1e4
	[ 4691.109250] 
	[ 4691.109250] stack backtrace:
	[ 4691.113531] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.11.0-rc3-00001-gc14f576-dirty #139
	[ 4691.121755] [<c0016a88>] (unwind_backtrace+0x0/0x128) from [<c0012d58>] (show_stack+0x20/0x24)
	[ 4691.130263] [<c0012d58>] (show_stack+0x20/0x24) from [<c045bd50>] (dump_stack+0x80/0xc4)
	[ 4691.138264] [<c045bd50>] (dump_stack+0x80/0xc4) from [<c007ad78>] (lockdep_rcu_suspicious+0xdc/0x118)
	[ 4691.147371] [<c007ad78>] (lockdep_rcu_suspicious+0xdc/0x118) from [<c0067ac0>] (set_cpu_sd_state_idle+0x10c/0x1e4)
	[ 4691.157605] [<c0067ac0>] (set_cpu_sd_state_idle+0x10c/0x1e4) from [<c0078238>] (tick_nohz_idle_enter+0x48/0x80)
	[ 4691.167583] [<c0078238>] (tick_nohz_idle_enter+0x48/0x80) from [<c006dc5c>] (cpu_startup_entry+0x28/0x388)
	[ 4691.177127] [<c006dc5c>] (cpu_startup_entry+0x28/0x388) from [<c0014acc>] (secondary_start_kernel+0x12c/0x144)
	[ 4691.187013] [<c0014acc>] (secondary_start_kernel+0x12c/0x144) from [<000081ec>] (0x81ec)

S?ren

Quoted 73 lines · S??ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S??ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S??ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S??ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local timer will be stopped when entering to the idle state. In this case, the cpuidle framework will call clockevents_notify(ENTER) and switches to a broadcast timer and will call clockevents_notify(EXIT) when exiting the idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Quoted 121 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

Quoted 133 lines · S?ren Brinkmann

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 26 lines · Daniel Lezcano

On 07/30/2013 02:03 AM, S?ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Quoted 7 lines

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

This is not necessary related to the bug. If the patch you sent broke the cadence_ttc driver, when you use it (with the TIMER_STOP), you will be stuck. Removing the flag, may signifies you don't use the broadcast timer, hence the bug is not surfacing.

Going back to the bug with the arm_global_timer, what is observed is the broadcast timer is *shutdown* when the second cpu is online.

I have to dig into the kernel/time/clockevents.c|tick-*.c because IMO the issue is coming from there but before I have to reproduce the bug, so find a board I have where I can add the arm_global_timer.

Quoted 6 lines

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

did you try to dump the stacks with magic-sysrq ? Or git bisect ?

Quoted 161 lines · Daniel Lezcano

Quoted 133 lines · Sören Brinkmann

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · Sören Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · Sören Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · Sören Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · Sören Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · Sören Brinkmann

On Tue, Jul 30, 2013 at 10:47:15AM +0200, Daniel Lezcano wrote:
On 07/30/2013 02:03 AM, Sören Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as
broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to
enable the global timer, the twd timers are still used as local timers
and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to
obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Quoted 7 lines

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

This is not necessary related to the bug. If the patch you sent broke the cadence_ttc driver, when you use it (with the TIMER_STOP), you will be stuck. Removing the flag, may signifies you don't use the broadcast timer, hence the bug is not surfacing.

Going back to the bug with the arm_global_timer, what is observed is the broadcast timer is *shutdown* when the second cpu is online.

I have to dig into the kernel/time/clockevents.c|tick-*.c because IMO the issue is coming from there but before I have to reproduce the bug, so find a board I have where I can add the arm_global_timer.

Quoted 6 lines

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

Quoted 1 line

did you try to dump the stacks with magic-sysrq ? Or git bisect ?

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

Quoted 162 lines · Michal Simek

Quoted 161 lines · Daniel Lezcano

Quoted 133 lines · S?ren Brinkmann

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 48 lines · Daniel Lezcano

On 07/31/2013 12:34 AM, S?ren Brinkmann wrote:
On Tue, Jul 30, 2013 at 10:47:15AM +0200, Daniel Lezcano wrote:
On 07/30/2013 02:03 AM, S?ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as
broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to
enable the global timer, the twd timers are still used as local timers
and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Again, since boot hangs in the actually broken case, I don't see way to
obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Quoted 7 lines

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

This is not necessary related to the bug. If the patch you sent broke the cadence_ttc driver, when you use it (with the TIMER_STOP), you will be stuck. Removing the flag, may signifies you don't use the broadcast timer, hence the bug is not surfacing.

Going back to the bug with the arm_global_timer, what is observed is the broadcast timer is *shutdown* when the second cpu is online.

I have to dig into the kernel/time/clockevents.c|tick-*.c because IMO the issue is coming from there but before I have to reproduce the bug, so find a board I have where I can add the arm_global_timer.

Quoted 6 lines

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

Quoted 5 lines

Quoted 1 line

did you try to dump the stacks with magic-sysrq ? Or git bisect ?

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

I tried to download qemu for zynq but it fails:

git clone git://git.xilinx.com/qemu-xarm.git
Cloning into 'qemu-xarm'...
fatal: The remote end hung up unexpectedly

I am also looking for the option specified for the kernel:

"The kernel needs to be built with this feature turned on (in menuconfig, System Type->Xilinx Specific Features -> Device Tree At Fixed Address)."

ps : apart that, well documented website !

Quoted 169 lines · Daniel Lezcano

Quoted 162 lines · Michal Simek

Quoted 161 lines · Daniel Lezcano

Quoted 133 lines · Sören Brinkmann

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · Sören Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · Sören Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · Sören Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 65 lines · Sören Brinkmann

On Wed, Jul 31, 2013 at 10:49:06PM +0200, Daniel Lezcano wrote:
On 07/31/2013 12:34 AM, Sören Brinkmann wrote:
On Tue, Jul 30, 2013 at 10:47:15AM +0200, Daniel Lezcano wrote:
On 07/30/2013 02:03 AM, Sören Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as
broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to
enable the global timer, the twd timers are still used as local timers
and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Again, since boot hangs in the actually broken case, I don't see way to
obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that
it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and
the gt enabled and having maxcpus=1 set.

/proc/timer_list:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer
max_delta_ns: 12884902005
min_delta_ns: 1000
mult: 715827876
shift: 31
mode: 3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Quoted 7 lines

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

This is not necessary related to the bug. If the patch you sent broke the cadence_ttc driver, when you use it (with the TIMER_STOP), you will be stuck. Removing the flag, may signifies you don't use the broadcast timer, hence the bug is not surfacing.

Going back to the bug with the arm_global_timer, what is observed is the broadcast timer is *shutdown* when the second cpu is online.

I have to dig into the kernel/time/clockevents.c|tick-*.c because IMO the issue is coming from there but before I have to reproduce the bug, so find a board I have where I can add the arm_global_timer.

Quoted 6 lines

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

Quoted 5 lines

Quoted 1 line

did you try to dump the stacks with magic-sysrq ? Or git bisect ?

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

Quoted 5 lines

I tried to download qemu for zynq but it fails:

git clone git://git.xilinx.com/qemu-xarm.git
Cloning into 'qemu-xarm'...
fatal: The remote end hung up unexpectedly

Not sure which site have you found but
it should be just qemu.git
https://github.com/Xilinx/qemu

or github clone.

Quoted 5 lines

I am also looking for the option specified for the kernel:

"The kernel needs to be built with this feature turned on (in menuconfig, System Type->Xilinx Specific Features -> Device Tree At Fixed Address)."

This also sound like a very ancient tree.
This is the latest kernel tree - master-next is the latest devel branch.
https://github.com/Xilinx/linux-xlnx

Or there should be an option to use the latest kernel from kernel.org. (I think Soren is using it)

Zynq is the part of multiplatfrom kernel and cadence ttc is there, dts is also in the mainline kernel.

Quoted 1 line

ps : apart that, well documented website !

Can you send me the link to it?

This should be the main page for it.
http://www.wiki.xilinx.com/

Quoted 176 lines · Michal Simek

Quoted 169 lines · Daniel Lezcano

Quoted 162 lines · Michal Simek

Quoted 161 lines · Daniel Lezcano

Quoted 133 lines · S?ren Brinkmann

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

On 07/31/2013 10:58 PM, S?ren Brinkmann wrote:
On Wed, Jul 31, 2013 at 10:49:06PM +0200, Daniel Lezcano wrote:
On 07/31/2013 12:34 AM, S?ren Brinkmann wrote:
On Tue, Jul 30, 2013 at 10:47:15AM +0200, Daniel Lezcano wrote:
On 07/30/2013 02:03 AM, S?ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as
broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to
enable the global timer, the twd timers are still used as local timers
and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Again, since boot hangs in the actually broken case, I don't see way to
obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that
it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and
the gt enabled and having maxcpus=1 set.

/proc/timer_list:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer
max_delta_ns: 12884902005
min_delta_ns: 1000
mult: 715827876
shift: 31
mode: 3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending
cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the
cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Quoted 7 lines

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

This is not necessary related to the bug. If the patch you sent broke the cadence_ttc driver, when you use it (with the TIMER_STOP), you will be stuck. Removing the flag, may signifies you don't use the broadcast timer, hence the bug is not surfacing.

Going back to the bug with the arm_global_timer, what is observed is the broadcast timer is *shutdown* when the second cpu is online.

I have to dig into the kernel/time/clockevents.c|tick-*.c because IMO the issue is coming from there but before I have to reproduce the bug, so find a board I have where I can add the arm_global_timer.

Quoted 6 lines

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

Quoted 5 lines

Quoted 1 line

did you try to dump the stacks with magic-sysrq ? Or git bisect ?

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

Quoted 5 lines

I tried to download qemu for zynq but it fails:

git clone git://git.xilinx.com/qemu-xarm.git
Cloning into 'qemu-xarm'...
fatal: The remote end hung up unexpectedly

Quoted 5 lines

Not sure which site have you found but
it should be just qemu.git
https://github.com/Xilinx/qemu

or github clone.

Ok, cool I was able to clone it.

Quoted 5 lines

Quoted 5 lines

I am also looking for the option specified for the kernel:

"The kernel needs to be built with this feature turned on (in menuconfig, System Type->Xilinx Specific Features -> Device Tree At Fixed Address)."

Ok.

Quoted 3 lines

This also sound like a very ancient tree.
This is the latest kernel tree - master-next is the latest devel branch.
https://github.com/Xilinx/linux-xlnx

Ok, cool. I have the right one.

Quoted 6 lines

Or there should be an option to use the latest kernel from kernel.org. (I think Soren is using it)

Zynq is the part of multiplatfrom kernel and cadence ttc is there, dts is also in the mainline kernel.

Quoted 3 lines

Quoted 1 line

ps : apart that, well documented website !

Can you send me the link to it?

http://xilinx.wikidot.com/zynq-qemu
http://xilinx.wikidot.com/zynq-linux

Quoted 3 lines

This should be the main page for it.
http://www.wiki.xilinx.com/

Thanks Michal !

Quoted 24 lines

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

I tried to download qemu for zynq but it fails:

git clone git://git.xilinx.com/qemu-xarm.git
Cloning into 'qemu-xarm'...
fatal: The remote end hung up unexpectedly

Quoted 5 lines

Not sure which site have you found but
it should be just qemu.git
https://github.com/Xilinx/qemu

or github clone.

Ok, cool I was able to clone it.

Quoted 5 lines

Quoted 5 lines

I am also looking for the option specified for the kernel:

"The kernel needs to be built with this feature turned on (in menuconfig, System Type->Xilinx Specific Features -> Device Tree At Fixed Address)."

Quoted 7 lines

Ok.

Quoted 3 lines

This also sound like a very ancient tree.
This is the latest kernel tree - master-next is the latest devel branch.
https://github.com/Xilinx/linux-xlnx

Ok, cool. I have the right one.

Quoted 7 lines

Quoted 7 lines

Or there should be an option to use the latest kernel from kernel.org. (I think Soren is using it)

Zynq is the part of multiplatfrom kernel and cadence ttc is there, dts is also in the mainline kernel.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

ps : apart that, well documented website !

Can you send me the link to it?

http://xilinx.wikidot.com/zynq-qemu
http://xilinx.wikidot.com/zynq-linux

I will find out information why it is still there. I think it was moved to the new location.

Quoted 3 lines

Quoted 3 lines

This should be the main page for it.
http://www.wiki.xilinx.com/

On 08/06/2013 03:18 PM, Michal Simek wrote:

[ ... ]

Quoted 24 lines

Quoted 24 lines

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

I tried to download qemu for zynq but it fails:

git clone git://git.xilinx.com/qemu-xarm.git
Cloning into 'qemu-xarm'...
fatal: The remote end hung up unexpectedly

Quoted 5 lines

Not sure which site have you found but
it should be just qemu.git
https://github.com/Xilinx/qemu

or github clone.

Ok, cool I was able to clone it.

Quoted 5 lines

Quoted 5 lines

I am also looking for the option specified for the kernel:

"The kernel needs to be built with this feature turned on (in menuconfig, System Type->Xilinx Specific Features -> Device Tree At Fixed Address)."

Quoted 7 lines

Quoted 7 lines

Ok.

Quoted 3 lines

This also sound like a very ancient tree.
This is the latest kernel tree - master-next is the latest devel branch.
https://github.com/Xilinx/linux-xlnx

Ok, cool. I have the right one.

Following the documentation, I was able to boot a kernel with qemu for the linux-xlnx and qemu-xilinx.

But this kernel is outdated regarding the upstream one, so I tried to boot a 3.11-rc4 kernel without success, I did the following:

I used the default config file from linux-xlnx for the upstream kernel.

I compiled the kernel with:

make -j 5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- UIMAGE_LOADADDR=0x8000 uImage

I generated the dtb with:

make -j 5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- dtbs

For qemu, I started qemu with:

./arm-softmmu/qemu-system-arm -M arm-generic-fdt -nographic -smp 2 -machine linux=on -serial mon:stdio -dtb zynq-zed.dtb -kernel kernel/zImage -initrd filesystem/ramdisk.img

I tried with the dtb available for the upstream kernel:

zynq-zc706.dtb, zynq-zc702.dtb and zynq-zed.dtb

Did I miss something ?

On 08/06/2013 03:18 PM, Michal Simek wrote:

[ ... ]

Quoted 24 lines

Quoted 24 lines

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

Soren: Are you able to replicate this issue on QEMU?
If yes, it should be the best if you can provide Qemu, kernel .config/ rootfs and simple manual to Daniel how to reach that fault.

I tried to download qemu for zynq but it fails:

git clone git://git.xilinx.com/qemu-xarm.git
Cloning into 'qemu-xarm'...
fatal: The remote end hung up unexpectedly

Quoted 5 lines

Not sure which site have you found but
it should be just qemu.git
https://github.com/Xilinx/qemu

or github clone.

Ok, cool I was able to clone it.

Quoted 5 lines

Quoted 5 lines

I am also looking for the option specified for the kernel:

"The kernel needs to be built with this feature turned on (in menuconfig, System Type->Xilinx Specific Features -> Device Tree At Fixed Address)."

Quoted 7 lines

Quoted 7 lines

Ok.

Quoted 3 lines

This also sound like a very ancient tree.
This is the latest kernel tree - master-next is the latest devel branch.
https://github.com/Xilinx/linux-xlnx

Ok, cool. I have the right one.

Following the documentation, I was able to boot a kernel with qemu for the linux-xlnx and qemu-xilinx.

But this kernel is outdated regarding the upstream one, so I tried to boot a 3.11-rc4 kernel without success, I did the following:

I used the default config file from linux-xlnx for the upstream kernel.

I compiled the kernel with:

make -j 5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- UIMAGE_LOADADDR=0x8000 uImage

I generated the dtb with:

make -j 5 ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- dtbs

For qemu, I started qemu with:

./arm-softmmu/qemu-system-arm -M arm-generic-fdt -nographic -smp 2 -machine linux=on -serial mon:stdio -dtb zynq-zed.dtb -kernel kernel/zImage -initrd filesystem/ramdisk.img

I tried with the dtb available for the upstream kernel:

zynq-zc706.dtb, zynq-zc702.dtb and zynq-zed.dtb

Did I miss something ?

Quoted 133 lines · S?ren Brinkmann

Hi Daniel,

Quoted 122 lines · Daniel Lezcano

Quoted 121 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 26 lines · Daniel Lezcano

On 07/30/2013 02:03 AM, S?ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 3 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Quoted 7 lines

Quoted 2 lines

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I will check later (kid duty) :)

Quoted 7 lines

I was actually waiting for an update from your side and did something else, but I seem to have run into this again. I was overhauling the cadence_ttc (patch attached, based on tip/timers/core). And it seems to show the same behavior as enabling the global_timer. With cpuidle off, it works. With cpuidle, on it hangs. Removing the TIMER_STOP flag from the C2 state makes it boot again.
It works just fine on our 3.10 kernel.

This is not necessary related to the bug. If the patch you sent broke the cadence_ttc driver, when you use it (with the TIMER_STOP), you will be stuck. Removing the flag, may signifies you don't use the broadcast timer, hence the bug is not surfacing.

Going back to the bug with the arm_global_timer, what is observed is the broadcast timer is *shutdown* when the second cpu is online.

I have to dig into the kernel/time/clockevents.c|tick-*.c because IMO the issue is coming from there but before I have to reproduce the bug, so find a board I have where I can add the arm_global_timer.

Quoted 6 lines

Another thing I noticed - probably unrelated but hard to tell: On 3.11-rc1 and later my system stops for quite some time at the hand off to userspace. I.e. I see the 'freeing unused kernel memory...' line and sometimes the following 'Welcome to Buildroot...' and then it stops and on good kernels it continues after a while and boots through and on bad ones it just hangs there.

did you try to dump the stacks with magic-sysrq ? Or git bisect ?

Quoted 118 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

Quoted 6 lines

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag would make the timer core switch to a broadcast device even though it wouldn't be necessary. But shouldn't it still work? It sounds like we do something useless, but nothing wrong in a sense that it should result in breakage. I guess I'm missing something obvious. This timer system will always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this 'broadcast timer'. I guess that is some clockevent device which is common to all cores? (that would be the cadence_ttc for Zynq). Is the hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 1 line

Quoted 5 lines

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

Hi Daniel,

Quoted 119 lines · Daniel Lezcano

Quoted 118 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

      Tick Device: mode:     1
      Broadcast device
      Clock Event Device: arm_global_timer
       max_delta_ns:   12884902005
       min_delta_ns:   1000
       mult:           715827876
       shift:          31
       mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

        # echo 1 > online && cat /proc/timer_list

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c index 38959c8..3ab11c1 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,

    */
   void clockevents_shutdown(struct clock_event_device *dev)
   {
  +       pr_info("ce->name:%s\n", dev->name);
  +       dump_stack();
          clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
          dev->next_event.tv64 = KTIME_MAX;

}

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 1 line

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I have an idea, but no real evidence to prove it:
Some of the registers in the arm_global_timer are banked per CPU. I.e. some code must be executed on the CPU the timer is associated with (struct clock_event_device.cpumask) to have the intended effect As far as I can tell, there is no guarantee, that the set_mode() and program_next_event() calls execute on the correct CPU.

If this was correct, shutting down the timer for the CPU entering idle might actually shut down the timer for the running CPU, if set_mode() executes on the CPU which is _not_ about to enter idle.

I tried to prove this by adding some really ugly smp_call_any() wrappers in kernel/time/clockevents.c for the calls to set_mode() and program_net_event() but that ends in all kinds of dead locks.

S?ren

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Quoted 115 lines · S?ren Brinkmann

Hi Daniel,

Quoted 112 lines · Daniel Lezcano

Quoted 102 lines · S?ren Brinkmann

Quoted 101 lines · Daniel Lezcano

Quoted 100 lines · S?ren Brinkmann

Quoted 75 lines · Daniel Lezcano

Quoted 63 lines · S?ren Brinkmann

Quoted 62 lines · Daniel Lezcano

Quoted 51 lines · S?ren Brinkmann

Quoted 40 lines · Daniel Lezcano

Quoted 39 lines · S?ren Brinkmann

Quoted 23 lines · Daniel Lezcano

On 07/30/2013 02:03 AM, S?ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)
the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.
I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?
If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 3 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

      Tick Device: mode:     1
      Broadcast device
      Clock Event Device: arm_global_timer
       max_delta_ns:   12884902005
       min_delta_ns:   1000
       mult:           715827876
       shift:          31
       mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

Quoted 8 lines

Quoted 5 lines

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

        # echo 1 > online && cat /proc/timer_list

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c index 38959c8..3ab11c1 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,

    */
   void clockevents_shutdown(struct clock_event_device *dev)
   {
  +       pr_info("ce->name:%s\n", dev->name);
  +       dump_stack();
          clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
          dev->next_event.tv64 = KTIME_MAX;

}

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Quoted 7 lines

Quoted 1 line

Thanks for the traces.

Sure.

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

Quoted 6 lines

I have an idea, but no real evidence to prove it:
Some of the registers in the arm_global_timer are banked per CPU. I.e. some code must be executed on the CPU the timer is associated with (struct clock_event_device.cpumask) to have the intended effect As far as I can tell, there is no guarantee, that the set_mode() and program_next_event() calls execute on the correct CPU.

I believe the core clockevents code enforces that, or all other percpu clockevent_device drivers would be horrifically broken.

Quoted 116 lines · Mark Rutland

Quoted 115 lines · S?ren Brinkmann

Hi Daniel,

Quoted 112 lines · Daniel Lezcano

Quoted 102 lines · S?ren Brinkmann

Quoted 101 lines · Daniel Lezcano

Quoted 100 lines · S?ren Brinkmann

Quoted 75 lines · Daniel Lezcano

Quoted 63 lines · S?ren Brinkmann

Quoted 62 lines · Daniel Lezcano

Quoted 51 lines · S?ren Brinkmann

Quoted 40 lines · Daniel Lezcano

Quoted 39 lines · S?ren Brinkmann

On Tue, Jul 30, 2013 at 10:47:15AM +0200, Daniel Lezcano wrote:
On 07/30/2013 02:03 AM, S?ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)
the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.
I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?
If you look at the /proc/timer_list, which timer is used for broadcasting ?
So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as
broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to
enable the global timer, the twd timers are still used as local timers
and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 3 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to
obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

      Tick Device: mode:     1
      Broadcast device
      Clock Event Device: arm_global_timer
       max_delta_ns:   12884902005
       min_delta_ns:   1000
       mult:           715827876
       shift:          31
       mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

Quoted 8 lines

Quoted 5 lines

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

        # echo 1 > online && cat /proc/timer_list

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c index 38959c8..3ab11c1 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,

    */
   void clockevents_shutdown(struct clock_event_device *dev)
   {
  +       pr_info("ce->name:%s\n", dev->name);
  +       dump_stack();
          clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
          dev->next_event.tv64 = KTIME_MAX;

}

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Quoted 7 lines

Quoted 1 line

Thanks for the traces.

Sure.

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

Quoted 8 lines

Quoted 6 lines

I have an idea, but no real evidence to prove it:
Some of the registers in the arm_global_timer are banked per CPU. I.e. some code must be executed on the CPU the timer is associated with (struct clock_event_device.cpumask) to have the intended effect As far as I can tell, there is no guarantee, that the set_mode() and program_next_event() calls execute on the correct CPU.

I believe the core clockevents code enforces that, or all other percpu clockevent_device drivers would be horrifically broken.

Maybe the problem here is that a per-cpu device is being used for the broadcast source? I can't recall but I think the broadcast programming can bounce around CPUs depending on which CPU is the one to enter broadcast mode first? At least I don't think this configuration has ever been tested (for example, look at how tick_do_broadcast_on_off() enables the broadcast timer on whatever CPU goes into deep idle first).

Quoted 131 lines · S??ren Brinkmann

Hi Daniel,

Quoted 119 lines · Daniel Lezcano

Quoted 118 lines · S??ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S??ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S??ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S??ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S??ren Brinkmann

Quoted 26 lines · Daniel Lezcano

On 07/30/2013 02:03 AM, S??ren Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

      Tick Device: mode:     1
      Broadcast device
      Clock Event Device: arm_global_timer
       max_delta_ns:   12884902005
       min_delta_ns:   1000
       mult:           715827876
       shift:          31
       mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

        # echo 1 > online && cat /proc/timer_list

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c index 38959c8..3ab11c1 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,

    */
   void clockevents_shutdown(struct clock_event_device *dev)
   {
  +       pr_info("ce->name:%s\n", dev->name);
  +       dump_stack();
          clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
          dev->next_event.tv64 = KTIME_MAX;

}

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 1 line

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

Quoted 6 lines

I have an idea, but no real evidence to prove it:
Some of the registers in the arm_global_timer are banked per CPU. I.e. some code must be executed on the CPU the timer is associated with (struct clock_event_device.cpumask) to have the intended effect As far as I can tell, there is no guarantee, that the set_mode() and program_next_event() calls execute on the correct CPU.

I believe the core clockevents code enforces that, or all other percpu clockevent_device drivers would be horrifically broken.

Hi Daniel,

Quoted 119 lines · Daniel Lezcano

Quoted 118 lines · S?ren Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · S?ren Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · S?ren Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · S?ren Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · S?ren Brinkmann

Quoted 20 lines · Daniel Lezcano

Quoted 19 lines · S?ren Brinkmann

Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Quoted 6 lines

Quoted 4 lines

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 1 line

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

I have an idea, but no real evidence to prove it:
Some of the registers in the arm_global_timer are banked per CPU. I.e. some code must be executed on the CPU the timer is associated with (struct clock_event_device.cpumask) to have the intended effect As far as I can tell, there is no guarantee, that the set_mode() and program_next_event() calls execute on the correct CPU. If this was correct, shutting down the timer for the CPU entering idle might actually shut down the timer for the running CPU, if set_mode() executes on the CPU which is _not_ about to enter idle.

I tried to prove this by adding some really ugly smp_call_any() wrappers in kernel/time/clockevents.c for the calls to set_mode() and program_net_event() but that ends in all kinds of dead locks.

S?ren

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Quoted 131 lines · Sören Brinkmann

Hi Daniel,

Quoted 119 lines · Daniel Lezcano

Quoted 118 lines · Sören Brinkmann

Quoted 108 lines · Daniel Lezcano

Quoted 107 lines · Sören Brinkmann

Quoted 84 lines · Daniel Lezcano

Quoted 73 lines · Sören Brinkmann

Quoted 72 lines · Daniel Lezcano

Quoted 57 lines · Sören Brinkmann

Quoted 44 lines · Daniel Lezcano

Quoted 43 lines · Sören Brinkmann

Quoted 26 lines · Daniel Lezcano

On 07/30/2013 02:03 AM, Sören Brinkmann wrote:
Hi Daniel,

On Mon, Jul 29, 2013 at 02:51:49PM +0200, Daniel Lezcano wrote:
(snip)

the CPUIDLE_FLAG_TIMER_STOP flag tells the cpuidle framework the local
timer will be stopped when entering to the idle state. In this case, the
cpuidle framework will call clockevents_notify(ENTER) and switches to a
broadcast timer and will call clockevents_notify(EXIT) when exiting the
idle state, switching the local timer back in use.

I've been thinking about this, trying to understand how this makes my
boot attempts on Zynq hang. IIUC, the wrongly provided TIMER_STOP flag
would make the timer core switch to a broadcast device even though it
wouldn't be necessary. But shouldn't it still work? It sounds like we do
something useless, but nothing wrong in a sense that it should result in
breakage. I guess I'm missing something obvious. This timer system will
always remain a mystery to me.

Actually this more or less leads to the question: What is this
'broadcast timer'. I guess that is some clockevent device which is
common to all cores? (that would be the cadence_ttc for Zynq). Is the
hang pointing to some issue with that driver?

If you look at the /proc/timer_list, which timer is used for broadcasting ?

So, the correct run results (full output attached).

The vanilla kernel uses the twd timers as local timers and the TTC as broadcast device:
Tick Device: mode: 1
Broadcast device
Clock Event Device: ttc_clockevent

When I remove the offending CPUIDLE flag and add the DT fragment to enable the global timer, the twd timers are still used as local timers and the broadcast device is the global timer:
Tick Device: mode: 1
Broadcast device
Clock Event Device: arm_global_timer

Quoted 4 lines

Quoted 2 lines

Again, since boot hangs in the actually broken case, I don't see way to obtain this information for that case.

Can't you use the maxcpus=1 option to ensure the system to boot up ?

Right, that works. I forgot about that option after you mentioned, that it is most likely not that useful.

Anyway, this are those sysfs files with an unmodified cpuidle driver and the gt enabled and having maxcpus=1 set.

/proc/timer_list:

Quoted 8 lines

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: arm_global_timer
	 max_delta_ns:   12884902005
	 min_delta_ns:   1000
	 mult:           715827876
	 shift:          31
	 mode:           3

Here the mode is 3 (CLOCK_EVT_MODE_ONESHOT)

The previous timer_list output you gave me when removing the offending cpuidle flag, it was 1 (CLOCK_EVT_MODE_SHUTDOWN).

Quoted 6 lines

Quoted 2 lines

Is it possible you try to get this output again right after onlining the cpu1 in order to check if the broadcast device switches to SHUTDOWN ?

How do I do that? I tried to online CPU1 after booting with maxcpus=1 and that didn't end well:

	# echo 1 > online && cat /proc/timer_list 

Hmm, I was hoping to have a small delay before the kernel hangs but apparently this is not the case... :(

Quoted 6 lines

I suspect the global timer is shutdown at one moment but I don't understand why and when.

Can you add a stack trace in the "clockevents_shutdown" function with the clockevent device name ? Perhaps, we may see at boot time an interesting trace when it hangs.

I did this change:

	diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
	index 38959c8..3ab11c1 100644
	--- a/kernel/time/clockevents.c
	+++ b/kernel/time/clockevents.c
	@@ -92,6 +92,8 @@ void clockevents_set_mode(struct clock_event_device *dev,
	  */
	 void clockevents_shutdown(struct clock_event_device *dev)
	 {
	+       pr_info("ce->name:%s\n", dev->name);
	+       dump_stack();
	        clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
	        dev->next_event.tv64 = KTIME_MAX;
	 }

Quoted 5 lines

Quoted 3 lines

It is hit a few times during boot, so I attach a full boot log. I really don't know what to look for, but I hope you can spot something in it. I really appreciate you taking the time.

Thanks for the traces.

Sure.

Quoted 1 line

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

If you try without the ttc_clockevent configured in the kernel (but with twd and gt), does it boot ?

Absence of the TTC doesn't seem to make any difference. It hangs at the same location.

Ok, IMO there is a problem with the broadcast device registration (may be vs twd).

Quoted 9 lines

I have an idea, but no real evidence to prove it:
Some of the registers in the arm_global_timer are banked per CPU. I.e. some code must be executed on the CPU the timer is associated with (struct clock_event_device.cpumask) to have the intended effect As far as I can tell, there is no guarantee, that the set_mode() and program_next_event() calls execute on the correct CPU. If this was correct, shutting down the timer for the CPU entering idle might actually shut down the timer for the running CPU, if set_mode() executes on the CPU which is _not_ about to enter idle.

Hi Sören,
Am able to reproduce similar issue on StiH415 SOC by enabling both global_timer and twd and using cpuidle driver like zynq.

When CPU0 goes to idle, I noticed that the global timer used for boardcast is actually scheduled on wrong cpu.
My traces for printk like this

	printk("DEBUG: %s on CPU:%d CPUMASK:%s\n", __FUNCTION__,

smp_processor_id(), scpumask);

shows:

DEBUG: gt_clockevent_set_mode on CPU:1 CPUMASK: 0
DEBUG: gt_clockevent_set_next_event on CPU:1 CPUMASK:0

Which indicates that setting the mode and next_event for a clkevent with cpumask 0 is scheduled on cpu1, this will generate an global timer interrupt on cpu1 rather than cpu0.

This might be the reason for cpu0 not coming out of the cpu_idle_loop.

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

Quoted 6 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 6 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

Quoted 7 lines

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

I'm not sure I'm getting this right. But neither the cadence_ttc nor the arm_global_timer have the FEAT_C3_STOP flag set. So, shouldn't they be treated equally even with your change?

Quoted 6 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

---8<----

kernel/time/tick-broadcast.c +0 -0

index 218bcb5..d3539e5 100644

kernel/time/tick-broadcast.c +0 -0

 	return !curdev || newdev->rating > curdev->rating;
 }
 

kernel/time/tick-common.c +0 -0

index 64522ec..1628b9f 100644

kernel/time/tick-common.c +2 -0

+
+	/*
 	 * Use the higher rated one, but prefer a CPU local device with a lower
 	 * rating than a non-CPU local device
 	 */

Quoted 23 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

---8<----

kernel/time/tick-broadcast.c +0 -0

index 218bcb5..d3539e5 100644
@@ -77,6 +77,9 @@ static bool tick_check_broadcast_device(struct clock_event_device *curdev,
 	    !(newdev->features & CLOCK_EVT_FEAT_ONESHOT))
 		return false;
 

kernel/time/tick-broadcast.c static bool tick_check_broadcast_device(struct clock_event_device *curdev, · line 81

Quoted 2 lines

kernel/time/tick-broadcast.c +2 -0

+	if (cpumask_equal(newdev->cpumask, cpumask_of(smp_processor_id())))
+		return false;

Yes, that makes sense to prevent local timer devices to be a broadcast one.

In the case of the arm global timer, each cpu will register their timer, so the test will be ok but is it possible the cpu0 registers the timers for the other cpus ?

Quoted 11 lines

kernel/time/tick-broadcast.c +0 -0

 	return !curdev || newdev->rating > curdev->rating;
 }
 

kernel/time/tick-common.c +0 -0

index 64522ec..1628b9f 100644
@@ -245,6 +245,15 @@ static bool tick_check_preferred(struct clock_event_device *curdev,
 	}
 

kernel/time/tick-common.c static bool tick_check_preferred(struct clock_event_device *curdev, · line 254

Quoted 8 lines

kernel/time/tick-common.c +7 -0

 	/*
+	 * Prefer tick devices that don't suffer from FEAT_C3_STOP
+	 * regardless of their rating
+	 */
+	if (curdev && cpumask_equal(curdev->cpumask, newdev->cpumask) &&
+	    !(newdev->features & CLOCK_EVT_FEAT_C3STOP) &&
+	    (curdev->features & CLOCK_EVT_FEAT_C3STOP))
+		return true;

That sounds reasonable, but what is the acceptable gap between the ratings ? I am wondering if there isn't too much heuristic in the tick code...

Quoted 28 lines · Daniel Lezcano

Quoted 20 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

---8<----

kernel/time/tick-broadcast.c +0 -0

index 218bcb5..d3539e5 100644

kernel/time/tick-broadcast.c static bool tick_check_broadcast_device(struct clock_event_device *curdev, · line 81

Quoted 5 lines

kernel/time/tick-broadcast.c +2 -0

@@ -77,6 +77,9 @@ static bool tick_check_broadcast_device(struct clock_event_device *curdev,
 	    !(newdev->features & CLOCK_EVT_FEAT_ONESHOT))
 		return false;
 
+	if (cpumask_equal(newdev->cpumask, cpumask_of(smp_processor_id())))
+		return false;

Yes, that makes sense to prevent local timer devices to be a broadcast one.

Quoted 3 lines

In the case of the arm global timer, each cpu will register their timer, so the test will be ok but is it possible the cpu0 registers the timers for the other cpus ?

As far as I know every tick device has to be registered on the CPU that it will be used on. See tick_check_percpu().

Quoted 10 lines

Quoted 10 lines

kernel/time/tick-broadcast.c +0 -0

 	return !curdev || newdev->rating > curdev->rating;
 }
 

kernel/time/tick-common.c +0 -0

index 64522ec..1628b9f 100644
@@ -245,6 +245,15 @@ static bool tick_check_preferred(struct clock_event_device *curdev,
 	}
 

Quoted 11 lines

kernel/time/tick-common.c static bool tick_check_preferred(struct clock_event_device *curdev, · line 254

Quoted 8 lines

kernel/time/tick-common.c +7 -0

 	/*
+	 * Prefer tick devices that don't suffer from FEAT_C3_STOP
+	 * regardless of their rating
+	 */
+	if (curdev && cpumask_equal(curdev->cpumask, newdev->cpumask) &&
+	    !(newdev->features & CLOCK_EVT_FEAT_C3STOP) &&
+	    (curdev->features & CLOCK_EVT_FEAT_C3STOP))
+		return true;

That sounds reasonable, but what is the acceptable gap between the ratings ? I am wondering if there isn't too much heuristic in the tick code...

Yes I wonder if we should just change the ratings of the clockevents. But it feels to me like the rating should only matter if the two are equal in features. Otherwise we can use the features to determine what we want.

Quoted 6 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 8 lines · Stephen Boyd

Quoted 6 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

Quoted 7 lines

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Quoted 17 lines · S?ren Brinkmann

Quoted 15 lines · Stephen Boyd

Quoted 6 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

[ ... ]

Can you boot with maxcpus=1 and then give the result of timer_list ?

Quoted 6 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

Quoted 7 lines

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Quoted 14 lines · S?ren Brinkmann

Quoted 13 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

Quoted 17 lines · Stephen Boyd

Quoted 16 lines · S?ren Brinkmann

Quoted 6 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

Quoted 7 lines

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Quoted 3 lines

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

What is the base for your patch? I based my GT enable patch on 3.11-rc3 and for consistency in our debugging, I didn't move it elsewhere since. Your patch doesn't apply cleanly on it. I see if I can work it out, just let me know if it depends on something not available in 3.11-rc3.

Quoted 21 lines · S?ren Brinkmann

Quoted 20 lines · Stephen Boyd

Quoted 14 lines · S?ren Brinkmann

Quoted 6 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

Quoted 7 lines

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

Quoted 4 lines

What is the base for your patch? I based my GT enable patch on 3.11-rc3 and for consistency in our debugging, I didn't move it elsewhere since. Your patch doesn't apply cleanly on it. I see if I can work it out, just let me know if it depends on something not available in 3.11-rc3.

I applied this on 3.11-rc4. I don't think anything has changed there between rc3 and rc4. so you're probably running into the whitespace problem.

Quoted 14 lines · S?ren Brinkmann

Quoted 13 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

Quoted 15 lines · Stephen Boyd

Quoted 14 lines · S?ren Brinkmann

Quoted 13 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 6 lines

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

That did it! Booted straight into the system.

The broadcast device is
the TTC instead of GT, now.

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: ttc_clockevent
	 max_delta_ns:   1207932479
	 min_delta_ns:   18432
	 mult:           233015
	 shift:          32
	 mode:           1
	 next_event:     9223372036854775807 nsecs
	 set_next_event: ttc_set_next_event
	 set_mode:       ttc_set_mode
	 event_handler:  tick_handle_oneshot_broadcast
	 retries:        0
	
	tick_broadcast_mask: 00000000
	tick_broadcast_oneshot_mask: 00000000

Quoted 16 lines · S?ren Brinkmann

Quoted 15 lines · Stephen Boyd

Quoted 14 lines · S?ren Brinkmann

Quoted 13 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

That did it! Booted straight into the system.

Good news :)

Quoted 17 lines

The broadcast device is
the TTC instead of GT, now.

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: ttc_clockevent
	 max_delta_ns:   1207932479
	 min_delta_ns:   18432
	 mult:           233015
	 shift:          32
	 mode:           1
	 next_event:     9223372036854775807 nsecs
	 set_next_event: ttc_set_next_event
	 set_mode:       ttc_set_mode
	 event_handler:  tick_handle_oneshot_broadcast
	 retries:        0

Quoted 2 lines

	tick_broadcast_mask: 00000000
	tick_broadcast_oneshot_mask: 00000000

At the first glance, the timer broadcast usage is not set, right ? Can you try with the cpuidle flag even if it is not needed ?

Quoted 48 lines · Daniel Lezcano

Quoted 16 lines · S?ren Brinkmann

Quoted 15 lines · Stephen Boyd

Quoted 14 lines · S?ren Brinkmann

Quoted 13 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

That did it! Booted straight into the system.

Good news :)

Quoted 19 lines

The broadcast device is
the TTC instead of GT, now.

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: ttc_clockevent
	 max_delta_ns:   1207932479
	 min_delta_ns:   18432
	 mult:           233015
	 shift:          32
	 mode:           1
	 next_event:     9223372036854775807 nsecs
	 set_next_event: ttc_set_next_event
	 set_mode:       ttc_set_mode
	 event_handler:  tick_handle_oneshot_broadcast
	 retries:        0
	
	tick_broadcast_mask: 00000000
	tick_broadcast_oneshot_mask: 00000000

Quoted 2 lines

At the first glance, the timer broadcast usage is not set, right ? Can you try with the cpuidle flag even if it is not needed ?

It's actually present. I have a clean 3.11-rc3 and the only changes are my patch to enable the GT and Stephen's fix.
The cpuidle stats show both idle states being used.

Quoted 52 lines · S?ren Brinkmann

Quoted 50 lines · Daniel Lezcano

Quoted 16 lines · S?ren Brinkmann

Quoted 15 lines · Stephen Boyd

Quoted 14 lines · S?ren Brinkmann

Quoted 13 lines · Stephen Boyd

Quoted 5 lines · Daniel Lezcano

yes, but at least the broadcast mechanism should send an IPI to cpu0 to wake it up, no ? As Stephen stated this kind of configuration should has never been tested before so the tick broadcast code is not handling this case properly IMHO.

If you have a per-cpu tick device that isn't suffering from FEAT_C3_STOP why wouldn't you use that for the tick versus a per-cpu tick device that has FEAT_C3_STOP? It sounds like there is a bug in the preference logic or you should boost the rating of the arm global timer above the twd. Does this patch help? It should make the arm global timer the tick device and whatever the cadence timer you have into the broadcast device.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

I finally got to test your patch. Unfortunately, it makes the system hang even earlier:

Sorry it had a bug depending on the registration order. Can you try this one (tabs are probably spaces, sorry)? I will go read through this thread to see if we already covered the registration order.

That did it! Booted straight into the system.

Good news :)

Quoted 19 lines

The broadcast device is
the TTC instead of GT, now.

	Tick Device: mode:     1
	Broadcast device
	Clock Event Device: ttc_clockevent
	 max_delta_ns:   1207932479
	 min_delta_ns:   18432
	 mult:           233015
	 shift:          32
	 mode:           1
	 next_event:     9223372036854775807 nsecs
	 set_next_event: ttc_set_next_event
	 set_mode:       ttc_set_mode
	 event_handler:  tick_handle_oneshot_broadcast
	 retries:        0
	
	tick_broadcast_mask: 00000000
	tick_broadcast_oneshot_mask: 00000000

At the first glance, the timer broadcast usage is not set, right ? Can you try with the cpuidle flag even if it is not needed ?

Quoted 3 lines

It's actually present. I have a clean 3.11-rc3 and the only changes are my patch to enable the GT and Stephen's fix.
The cpuidle stats show both idle states being used.

Ah, right. The tick_broadcast_mask is not set because the arm global timer has not the CLOCK_EVT_FEAT_C3STOP feature flag set.

Quoted 6 lines · Daniel Lezcano

Quoted 3 lines · S?ren Brinkmann

It's actually present. I have a clean 3.11-rc3 and the only changes are my patch to enable the GT and Stephen's fix.
The cpuidle stats show both idle states being used.

Ah, right. The tick_broadcast_mask is not set because the arm global timer has not the CLOCK_EVT_FEAT_C3STOP feature flag set.

Just to check in. Do you want any additional testing done? Or can I expect Stephens fix to get merged, so Zynq can use the GT?

Quoted 5 lines · S?ren Brinkmann

Quoted 4 lines · Daniel Lezcano

Quoted 3 lines · S?ren Brinkmann

It's actually present. I have a clean 3.11-rc3 and the only changes are my patch to enable the GT and Stephen's fix.
The cpuidle stats show both idle states being used.

Quoted 5 lines

Quoted 2 lines

Ah, right. The tick_broadcast_mask is not set because the arm global timer has not the CLOCK_EVT_FEAT_C3STOP feature flag set.

Just to check in. Do you want any additional testing done? Or can I expect Stephens fix to get merged, so Zynq can use the GT?

I was curious, can you use just the first hunk of the patch that applied to tick-broadcast.c to fix the problem? I think the answer is yes.

Hi Stephen,

Quoted 8 lines · Stephen Boyd

Quoted 7 lines · S??ren Brinkmann

Quoted 6 lines · Daniel Lezcano

Quoted 3 lines · S??ren Brinkmann

It's actually present. I have a clean 3.11-rc3 and the only changes are my patch to enable the GT and Stephen's fix.
The cpuidle stats show both idle states being used.

Ah, right. The tick_broadcast_mask is not set because the arm global timer has not the CLOCK_EVT_FEAT_C3STOP feature flag set.

Quoted 6 lines

Quoted 3 lines

Just to check in. Do you want any additional testing done? Or can I expect Stephens fix to get merged, so Zynq can use the GT?

I was curious, can you use just the first hunk of the patch that applied to tick-broadcast.c to fix the problem? I think the answer is yes.

Yes, that seems to be enough.

Quoted 18 lines · S??ren Brinkmann

Hi Stephen,

Quoted 14 lines · Stephen Boyd

Quoted 10 lines · S??ren Brinkmann

Quoted 6 lines · Daniel Lezcano

Quoted 3 lines · S??ren Brinkmann

It's actually present. I have a clean 3.11-rc3 and the only changes are my patch to enable the GT and Stephen's fix.
The cpuidle stats show both idle states being used.

Ah, right. The tick_broadcast_mask is not set because the arm global timer has not the CLOCK_EVT_FEAT_C3STOP feature flag set.

Just to check in. Do you want any additional testing done? Or can I expect Stephens fix to get merged, so Zynq can use the GT?

I was curious, can you use just the first hunk of the patch that applied to tick-broadcast.c to fix the problem? I think the answer is yes.

Quoted 2 lines

Yes, that seems to be enough.

Great thank you. I will split the patch into two pieces. That way we can discuss the merit of always using a timer that doesn't suffer from FEAT_C3_STOP over a timer that does.

On some ARM systems there are two sets of per-cpu timers: the TWD timers and the global timers. The TWD timers are rated at 350 and the global timers are rated at 300 but the TWD timers suffer from FEAT_C3_STOP while the arm global timers don't. The tick device selection logic doesn't consider the FEAT_C3_STOP flag so we'll always end up with the TWD as the tick device although we don't want that.

Extend the preference logic to take the FEAT_C3_STOP flag into account and always prefer tick devices that don't suffer from FEAT_C3_STOP even if their rating is lower than tick devices that do suffer from FEAT_C3_STOP. This will remove the need to do any broadcasting on such ARM systems.

Signed-off-by Stephen Boyd <[email protected]>

---
 kernel/time/tick-common.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

kernel/time/tick-common.c +1 -0

index 64522ec..3ae437d 100644
@@ -244,12 +244,22 @@ static bool tick_check_preferred(struct clock_event_device *curdev,
 			return false;
 	}
 
+	if (!curdev)

Quoted 36 lines · Stephen Boyd

On some ARM systems there are two sets of per-cpu timers: the TWD timers and the global timers. The TWD timers are rated at 350 and the global timers are rated at 300 but the TWD timers suffer from FEAT_C3_STOP while the arm global timers don't. The tick device selection logic doesn't consider the FEAT_C3_STOP flag so we'll always end up with the TWD as the tick device although we don't want that.

Extend the preference logic to take the FEAT_C3_STOP flag into account and always prefer tick devices that don't suffer from FEAT_C3_STOP even if their rating is lower than tick devices that do suffer from FEAT_C3_STOP. This will remove the need to do any broadcasting on such ARM systems.

Signed-off-by Stephen Boyd <[email protected]>

---
 kernel/time/tick-common.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

kernel/time/tick-common.c +9 -0

index 64522ec..3ae437d 100644
@@ -244,12 +244,22 @@ static bool tick_check_preferred(struct clock_event_device *curdev,
 			return false;
 	}
 
+	if (!curdev)
+		return true;
+
+	/* Always prefer a tick device that doesn't suffer from FEAT_C3STOP */
+	if (!(newdev->features & CLOCK_EVT_FEAT_C3STOP) &&
+			(curdev->features & CLOCK_EVT_FEAT_C3STOP))
+		return true;
+	if ((newdev->features & CLOCK_EVT_FEAT_C3STOP) &&
+			!(curdev->features & CLOCK_EVT_FEAT_C3STOP))

kernel/time/tick-common.c static bool tick_check_preferred(struct clock_event_device *curdev, · line 257

Quoted 2 lines

kernel/time/tick-common.c +2 -0

+		return false;
+

I don't think preferring the clock-event which doesn't suffer from FEAT_C3STOP is a good idea if the quality of the time source is not same. Generally the global timers are slow and far away from CPU(cycle cost). So as long as we don't get impacted because of low power states, the tick should run out of local timers which are faster access as well as high resolution.

Quoted 37 lines · Santosh Shilimkar

Quoted 36 lines · Stephen Boyd

On some ARM systems there are two sets of per-cpu timers: the TWD timers and the global timers. The TWD timers are rated at 350 and the global timers are rated at 300 but the TWD timers suffer from FEAT_C3_STOP while the arm global timers don't. The tick device selection logic doesn't consider the FEAT_C3_STOP flag so we'll always end up with the TWD as the tick device although we don't want that.

Extend the preference logic to take the FEAT_C3_STOP flag into account and always prefer tick devices that don't suffer from FEAT_C3_STOP even if their rating is lower than tick devices that do suffer from FEAT_C3_STOP. This will remove the need to do any broadcasting on such ARM systems.

Signed-off-by Stephen Boyd <[email protected]>

---
 kernel/time/tick-common.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

kernel/time/tick-common.c +9 -0

index 64522ec..3ae437d 100644
@@ -244,12 +244,22 @@ static bool tick_check_preferred(struct clock_event_device *curdev,
 			return false;
 	}
 
+	if (!curdev)
+		return true;
+
+	/* Always prefer a tick device that doesn't suffer from FEAT_C3STOP */
+	if (!(newdev->features & CLOCK_EVT_FEAT_C3STOP) &&
+			(curdev->features & CLOCK_EVT_FEAT_C3STOP))
+		return true;
+	if ((newdev->features & CLOCK_EVT_FEAT_C3STOP) &&
+			!(curdev->features & CLOCK_EVT_FEAT_C3STOP))

Quoted 9 lines

kernel/time/tick-common.c static bool tick_check_preferred(struct clock_event_device *curdev, · line 257

Quoted 2 lines

kernel/time/tick-common.c +2 -0

+		return false;
+

I don't think preferring the clock-event which doesn't suffer from FEAT_C3STOP is a good idea if the quality of the time source is not same. Generally the global timers are slow and far away from CPU(cycle cost). So as long as we don't get impacted because of low power states, the tick should run out of local timers which are faster access as well as high resolution.