arm: dts: exynos5: Remove multi core timer

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Hi Chirantan,

Quoted 5 lines · Chirantan Ekbote

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Huh? I've always thought MCT is a completely separate hardware block outside of ARM cores, while architected timers are embedded inside the CPU block in which the ARM cores reside. Could you elaborate on this?

Tomasz,

Quoted 9 lines · Tomasz Figa

Hi Chirantan,

Quoted 5 lines · Chirantan Ekbote

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Quoted 3 lines

Huh? I've always thought MCT is a completely separate hardware block outside of ARM cores, while architected timers are embedded inside the CPU block in which the ARM cores reside. Could you elaborate on this?

Yup. Our thoughts exactly.

...but it appears not to be the case. Chirantan demonstrated this in U-Boot just to prove that it's not some strange kernel interaction in
<https://chromium-review.googlesource.com/200035>. I took his patch and tweaked it a little more myself in
<https://chromium-review.googlesource.com/200098>.

Specifically:
* If you stop the MCT, the arch timer stops
* If you reset the MCT, the arch timer resets
* If you start the MCT again, the arch timer starts again
* If you read the MCT and the arch timer, they give the same value.

This is apparently the answer to my question at
<http://www.spinics.net/lists/linux-samsung-soc/msg29085.html>.
Specifically Chirantan found that the big jump in time happened when MCT reset to 0. That made the arch timer code think that there was a wraparound and jump forward in time a lot.

Please confirm if you have a system that has MCT and arch timer in front of you.

Quoted 38 lines · Doug Anderson

Tomasz,

Quoted 9 lines · Tomasz Figa

Hi Chirantan,

Quoted 5 lines · Chirantan Ekbote

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Quoted 3 lines

Huh? I've always thought MCT is a completely separate hardware block outside of ARM cores, while architected timers are embedded inside the CPU block in which the ARM cores reside. Could you elaborate on this?

Yup. Our thoughts exactly.

...but it appears not to be the case. Chirantan demonstrated this in U-Boot just to prove that it's not some strange kernel interaction in
<https://chromium-review.googlesource.com/200035>. I took his patch and tweaked it a little more myself in
<https://chromium-review.googlesource.com/200098>.

Specifically:
* If you stop the MCT, the arch timer stops
* If you reset the MCT, the arch timer resets
* If you start the MCT again, the arch timer starts again
* If you read the MCT and the arch timer, they give the same value.

This is apparently the answer to my question at
<http://www.spinics.net/lists/linux-samsung-soc/msg29085.html>.
Specifically Chirantan found that the big jump in time happened when MCT reset to 0. That made the arch timer code think that there was a wraparound and jump forward in time a lot.

Quoted 1 line

Please confirm if you have a system that has MCT and arch timer in front of you.

Wow, this is so strange that I just can't believe it, but if you proved it using such detailed test then I don't have any reasons to object anymore.

One more thing, though, is whether the architected timer "interface" can wake the system from lighter power states, such as AFTR or LPA. Could you check this?

Tomasz,

Quoted 39 lines · Tomasz Figa

Quoted 38 lines · Doug Anderson

Tomasz,

Quoted 9 lines · Tomasz Figa

Hi Chirantan,

Quoted 5 lines · Chirantan Ekbote

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Quoted 3 lines

Huh? I've always thought MCT is a completely separate hardware block outside of ARM cores, while architected timers are embedded inside the CPU block in which the ARM cores reside. Could you elaborate on this?

Yup. Our thoughts exactly.

...but it appears not to be the case. Chirantan demonstrated this in U-Boot just to prove that it's not some strange kernel interaction in
<https://chromium-review.googlesource.com/200035>. I took his patch and tweaked it a little more myself in
<https://chromium-review.googlesource.com/200098>.

Specifically:
* If you stop the MCT, the arch timer stops
* If you reset the MCT, the arch timer resets
* If you start the MCT again, the arch timer starts again
* If you read the MCT and the arch timer, they give the same value.

This is apparently the answer to my question at
<http://www.spinics.net/lists/linux-samsung-soc/msg29085.html>.
Specifically Chirantan found that the big jump in time happened when MCT reset to 0. That made the arch timer code think that there was a wraparound and jump forward in time a lot.

Quoted 8 lines

Quoted 1 line

Please confirm if you have a system that has MCT and arch timer in front of you.

Wow, this is so strange that I just can't believe it, but if you proved it using such detailed test then I don't have any reasons to object anymore.

One more thing, though, is whether the architected timer "interface" can wake the system from lighter power states, such as AFTR or LPA. Could you check this?

I've never used AFTR or LPA and so can't really comment on it. Hopefully someone on the list can?

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

* it's faster to access.
* it is accessible from userspace for really fast access.

* it implements the bits needed for udelay() to use it.

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

Quoted 1 line

* it implements the bits needed for udelay() to use it.

Hmm, do you know what bits are those? On Exynos4 MCT is the only option and it would be nice to let udelay() use it.

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Hi Tomasz,

Quoted 25 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

Quoted 4 lines

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

- Chirantan

Quoted 33 lines · Chirantan Ekbote

Hi Tomasz,

Quoted 29 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 2 lines

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

When I profiled gettimeofday(), the standard systems call version took about 2.5x longer than through a vDSO interface.

Quoted 35 lines · David Riley

Quoted 33 lines · Chirantan Ekbote

Hi Tomasz,

Quoted 29 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 5 lines

Quoted 2 lines

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

When I profiled gettimeofday(), the standard systems call version took about 2.5x longer than through a vDSO interface.

Sounds like we should invent a new kind of jokes, starting with "When I profiled gettimeofday()". ;)

Just kidding.

The raw improvement looks quite good, but what I'm more concerned about is whether this has any significant effect on real use cases. As Doug mentioned, Chrome makes a lot of calls to gettimeofday(), but he also mentioned that this is for profiling purposes. Is performance of gettimeofday() really that crucial in this case? Are there any other use cases when this improvement is significant?

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Quoted 45 lines · Tomasz Figa

Quoted 35 lines · David Riley

Quoted 33 lines · Chirantan Ekbote

Hi Tomasz,

Quoted 29 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 5 lines

Quoted 2 lines

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

When I profiled gettimeofday(), the standard systems call version took about 2.5x longer than through a vDSO interface.

Sounds like we should invent a new kind of jokes, starting with "When I profiled gettimeofday()". ;)

Quoted 8 lines

Just kidding.

The raw improvement looks quite good, but what I'm more concerned about is whether this has any significant effect on real use cases. As Doug mentioned, Chrome makes a lot of calls to gettimeofday(), but he also mentioned that this is for profiling purposes. Is performance of gettimeofday() really that crucial in this case? Are there any other use cases when this improvement is significant?

In general, yes. gettimeofday() is normally the prime candidate for vDSO implementaiton (see Nathan Lynch's patch set in the last couple of months for enabling this). Speeding up gettimeofday() does have real benefit for some workloads.

Quoted 5 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Yeah, this will definitely need to be tested. Do the low power states work on exynos5 upstream such that they can be tested? The snow chromebook has a u-boot bug that makes AFTR not work, so it's not a suitable platform to test on, unfortunately.

Quoted 45 lines · Tomasz Figa

Quoted 35 lines · David Riley

Quoted 33 lines · Chirantan Ekbote

Hi Tomasz,

Quoted 29 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 5 lines

Quoted 2 lines

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

When I profiled gettimeofday(), the standard systems call version took about 2.5x longer than through a vDSO interface.

Sounds like we should invent a new kind of jokes, starting with "When I profiled gettimeofday()". ;)

Quoted 8 lines

Just kidding.

The raw improvement looks quite good, but what I'm more concerned about is whether this has any significant effect on real use cases. As Doug mentioned, Chrome makes a lot of calls to gettimeofday(), but he also mentioned that this is for profiling purposes. Is performance of gettimeofday() really that crucial in this case? Are there any other use cases when this improvement is significant?

In general, yes. gettimeofday() is normally the prime candidate for vDSO implementaiton (see Nathan Lynch's patch set in the last couple of months for enabling this). Speeding up gettimeofday() does have real benefit for some workloads.

Quoted 6 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Yeah, this will definitely need to be tested. Do the low power states work on exynos5 upstream such that they can be tested? The snow chromebook has a u-boot bug that makes AFTR not work, so it's not a suitable platform to test on, unfortunately.

And if we need MCT for low power states, we'll need MCT to co-exist with arch timers. Maybe by checking for the presence of those dt nodes
+ CONFIG_ARM_ARCH_TIMER, or somesuch. But let's discuss that when we know more.

Quoted 35 lines · David Riley

Quoted 33 lines · Chirantan Ekbote

Hi Tomasz,

Quoted 29 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 5 lines

Quoted 2 lines

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

When I profiled gettimeofday(), the standard systems call version took about 2.5x longer than through a vDSO interface.

Sounds like we should invent a new kind of jokes, starting with "When I profiled gettimeofday()". ;)

Just kidding.

The raw improvement looks quite good, but what I'm more concerned about is whether this has any significant effect on real use cases. As Doug mentioned, Chrome makes a lot of calls to gettimeofday(), but he also mentioned that this is for profiling purposes. Is performance of gettimeofday() really that crucial in this case? Are there any other use cases when this improvement is significant?

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Tomasz,

Quoted 45 lines · Tomasz Figa

Quoted 35 lines · David Riley

Quoted 33 lines · Chirantan Ekbote

Hi Tomasz,

Quoted 29 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 5 lines

Quoted 2 lines

When I profiled gettimeofday() calls, they were about 50 - 60% faster with the arch timers compared to the mct.

When I profiled gettimeofday(), the standard systems call version took about 2.5x longer than through a vDSO interface.

Sounds like we should invent a new kind of jokes, starting with "When I profiled gettimeofday()". ;)

Quoted 8 lines

Just kidding.

The raw improvement looks quite good, but what I'm more concerned about is whether this has any significant effect on real use cases. As Doug mentioned, Chrome makes a lot of calls to gettimeofday(), but he also mentioned that this is for profiling purposes. Is performance of gettimeofday() really that crucial in this case? Are there any other use cases when this improvement is significant?

I guess I should restate. Chrome is always profiling to some extent. I think that the Javascript engine, for instance, uses gettimeofday() to figure out how much time it's spending in various places.

Sonny just sent me some recent benchmarks using perf. On this particular benchmark it shows 1.85% of the time was spent in exynos_frc_read. If we can half that then we'll get a ~1% speedup in the system, which is nothing to sneeze at. If getting rid of the system call overead makes this several times faster again, then we roughly eliminate the overhead totally.

Quoted 7 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

-Doug

Quoted 6 lines

Quoted 5 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Quoted 6 lines

Quoted 6 lines

Quoted 5 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Quoted 28 lines · Chirantan Ekbote

Quoted 10 lines

Quoted 10 lines

Quoted 8 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes
looking at the code, which is always good. However if they don't support
low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Quoted 7 lines

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Hi all,

Even though I've heard something about the behavior of mct and arch timer...but I couldn't finish the talk to h/w guys yet. I need to talk again in next week then I could provide some useful information. Sorry for late and can you please wait a minute before deciding whatever.

Kukjin,

Quoted 36 lines · Chirantan Ekbote

Quoted 28 lines

Quoted 10 lines

Quoted 10 lines

Quoted 8 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes
looking at the code, which is always good. However if they don't support
low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Quoted 7 lines

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Quoted 4 lines

Hi all,

Even though I've heard something about the behavior of mct and arch timer...but I couldn't finish the talk to h/w guys yet. I need to talk

again

Quoted 2 lines

in next week then I could provide some useful information. Sorry for late and can you please wait a minute before deciding whatever.

I think we could wait a few days. Note however, that Chirantan's latest proposal keeps all existing functionality (can use both MCT and arch timers). It merely removes the unnecessary bit of the MCT init code setting the timer. No functionality is affected by that.

Quoted 3 lines · Doug Anderson

Kukjin,

Hi Doug,

Quoted 41 lines · Kukjin Kim

Quoted 36 lines · Chirantan Ekbote

Quoted 28 lines

Quoted 10 lines

Quoted 10 lines

Quoted 8 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes
looking at the code, which is always good. However if they don't support
low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Quoted 7 lines

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Quoted 4 lines

Hi all,

Even though I've heard something about the behavior of mct and arch timer...but I couldn't finish the talk to h/w guys yet. I need to talk

Quoted 9 lines

again

Quoted 2 lines

in next week then I could provide some useful information. Sorry for late and can you please wait a minute before deciding whatever.

I think we could wait a few days. Note however, that Chirantan's latest proposal keeps all existing functionality (can use both MCT and arch timers). It merely removes the unnecessary bit of the MCT init code setting the timer. No functionality is affected by that.

Let me explain the behavior of MCT and arch timer.

Basically the two blocks are connected and the arch timer uses the count value from MCT for reference on exynos5250 so following in this mail thread is expected and it's true.

* If you read the MCT and the arch timer, they give the same value.

And as you know, usually the access to arch timer is faster than MCT because of APB bus access...but using MCT has some benefits sometimes it depends on use case of power management though. BTW, since exynos5260, exynos5420 and exynos5800 doesn't support arch timer, we have been using MCT on exynos5 SoCs.

Quoted 3 lines · Doug Anderson

Kukjin,

Hi Doug,

Quoted 51 lines

Quoted 39 lines · Kukjin Kim

Quoted 30 lines · Chirantan Ekbote

Quoted 12 lines

Quoted 12 lines

Quoted 8 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

I think low power states aren't in mainline (right?).

Quoted 7 lines

Quoted 7 lines

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Quoted 8 lines

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/.
Effectively the visible behavior will not change. Would either of these options work?

Quoted 5 lines

Hi all,

Even though I've heard something about the behavior of mct and arch timer...but I couldn't finish the talk to h/w guys yet. I need to talk

again

Quoted 3 lines

in next week then I could provide some useful information. Sorry for late and can you please wait a minute before deciding whatever.

Quoted 6 lines

I think we could wait a few days. Note however, that Chirantan's latest proposal keeps all existing functionality (can use both MCT and arch timers). It merely removes the unnecessary bit of the MCT init code setting the timer. No functionality is affected by that.

Let me explain the behavior of MCT and arch timer.

Basically the two blocks are connected and the arch timer uses the count value from MCT for reference on exynos5250 so following in this mail thread is expected and it's true.

* If you read the MCT and the arch timer, they give the same value.

And as you know, usually the access to arch timer is faster than MCT because of APB bus access...but using MCT has some benefits sometimes it depends on use case of power management though.

BTW, since
exynos5260, exynos5420 and exynos5800 doesn't support arch timer, we have been using MCT on exynos5 SoCs.

Quoted 3 lines · Kukjin Kim

BTW, since
exynos5260, exynos5420 and exynos5800 doesn't support arch timer, we have been using MCT on exynos5 SoCs.

Hmm, I thought arch timer was a core feature of ARM Cortex A15 (and A7) cores. Also you mention Exynos5420, while Chirantan's patch removing MCT node from exynos5420.dtsi, would suggest that it worked for him fine. (I assume it was tested.)

Quoted 11 lines · Tomasz Figa

Quoted 3 lines · Kukjin Kim

BTW, since
exynos5260, exynos5420 and exynos5800 doesn't support arch timer, we have been using MCT on exynos5 SoCs.

Hmm, I thought arch timer was a core feature of ARM Cortex A15 (and A7) cores. Also you mention Exynos5420, while Chirantan's patch removing MCT node from exynos5420.dtsi, would suggest that it worked for him fine. (I assume it was tested.)

I was able to boot both 5420 and 5800 with just the arch timers on our 3.8 based franken-kernel. The upstream kernel doesn't boot on those systems with or without the mct so I wasn't able to test there. Although looking at the upstream device tree now I see that there is no entry for the arch timer in exynos5420.dtsi. Maybe it should be added in?

Kukjin,

Quoted 13 lines · Chirantan Ekbote

Quoted 11 lines · Tomasz Figa

Quoted 3 lines · Kukjin Kim

BTW, since
exynos5260, exynos5420 and exynos5800 doesn't support arch timer, we have been using MCT on exynos5 SoCs.

Hmm, I thought arch timer was a core feature of ARM Cortex A15 (and A7) cores. Also you mention Exynos5420, while Chirantan's patch removing MCT node from exynos5420.dtsi, would suggest that it worked for him fine. (I assume it was tested.)

Quoted 6 lines

I was able to boot both 5420 and 5800 with just the arch timers on our 3.8 based franken-kernel. The upstream kernel doesn't boot on those systems with or without the mct so I wasn't able to test there. Although looking at the upstream device tree now I see that there is no entry for the arch timer in exynos5420.dtsi. Maybe it should be added in?

A gentle reminder to respond here about the status of MCT on 5420 and 5800.

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Quoted 15 lines · Chirantan Ekbote

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Quoted 9 lines

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Yeah, actually we don't need to reset the count value after suspend/resume. So, how about following? I think, it should be fine to you.

drivers/clocksource/exynos_mct.c +0 -5

index 8d64200..d24db6f 100644
@@ -157,12 +157,15 @@ static void exynos4_mct_frc_start(u32 hi, u32 lo)
 {
        u32 reg;
 
-       exynos4_mct_write(lo, EXYNOS4_MCT_G_CNT_L);
-       exynos4_mct_write(hi, EXYNOS4_MCT_G_CNT_U);
-
        reg = __raw_readl(reg_base + EXYNOS4_MCT_G_TCON);
-       reg |= MCT_G_TCON_START;
-       exynos4_mct_write(reg, EXYNOS4_MCT_G_TCON);

Quoted 8 lines

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Hmm...I cannot open the webpage :(

Quoted 15 lines · Chirantan Ekbote

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Anyway, I'm by no means opposed to switching to arch timers. They provide a well designed, generic interface and drivers shared by multiple platforms, which means more code sharing and possibly more eyes looking at the code, which is always good. However if they don't support low power states correctly, we can't just remove MCT.

Quoted 8 lines

Quoted 8 lines

I think low power states aren't in mainline (right?).

One solution that might work could be to leave the device tree entry alone but change the MCT init code to simply act as a no-op if it sees an arch timer is in the device tree and enabled. Then when/if someone got the low power states enabled we could just change source code rather than dts files.

Quoted 9 lines

Doug and I were talking about this and we think we may have a way to have the mct and arch timers co-exist. The main issue is that the mct (and therefore arch timer) gets cleared once during boot and every time we do a suspend / resume. This happens in exynos4_mct_frc_start() but it's not immediately clear to us why the counter needs to be reset at all. If we remove the lines that clear the counter then there is no longer an issue with having both the mct and the arch timers on at the same time.

Yeah, actually we don't need to reset the count value after suspend/resume. So, how about following? I think, it should be fine to you.

drivers/clocksource/exynos_mct.c +0 -3

index 8d64200..d24db6f 100644
@@ -157,12 +157,15 @@ static void exynos4_mct_frc_start(u32 hi, u32 lo)
 {
        u32 reg;
 
-       exynos4_mct_write(lo, EXYNOS4_MCT_G_CNT_L);
-       exynos4_mct_write(hi, EXYNOS4_MCT_G_CNT_U);
-
        reg = __raw_readl(reg_base + EXYNOS4_MCT_G_TCON);

Quoted 7 lines

Alternately, if there is some code that depends on the mct being reset we could store an offset instead of clearing the counter and then subtract that offset every time something reads it. Doug has a patch that does this at
https://chromium-review.googlesource.com/#/c/200298/. Effectively the visible behavior will not change. Would either of these options work?

Hmm...I cannot open the webpage :(

Yeah, actually we don't need to reset the count value after suspend/resume. So, how about following? I think, it should be fine to you.

drivers/clocksource/exynos_mct.c +1 -5

index 8d64200..d24db6f 100644
@@ -157,12 +157,15 @@ static void exynos4_mct_frc_start(u32 hi, u32 lo)
 {
        u32 reg;
 
-       exynos4_mct_write(lo, EXYNOS4_MCT_G_CNT_L);
-       exynos4_mct_write(hi, EXYNOS4_MCT_G_CNT_U);
-
        reg = __raw_readl(reg_base + EXYNOS4_MCT_G_TCON);
-       reg |= MCT_G_TCON_START;
-       exynos4_mct_write(reg, EXYNOS4_MCT_G_TCON);
+

Hi Kukjin,

Quoted 26 lines

Yeah, actually we don't need to reset the count value after suspend/resume. So, how about following? I think, it should be fine to you.

drivers/clocksource/exynos_mct.c +8 -5

index 8d64200..d24db6f 100644
@@ -157,12 +157,15 @@ static void exynos4_mct_frc_start(u32 hi, u32 lo)
 {
        u32 reg;
 
-       exynos4_mct_write(lo, EXYNOS4_MCT_G_CNT_L);
-       exynos4_mct_write(hi, EXYNOS4_MCT_G_CNT_U);
-
        reg = __raw_readl(reg_base + EXYNOS4_MCT_G_TCON);
-       reg |= MCT_G_TCON_START;
-       exynos4_mct_write(reg, EXYNOS4_MCT_G_TCON);
+
+       if (!(reg & MCT_G_TCON_START)) {
+               exynos4_mct_write(lo, EXYNOS4_MCT_G_CNT_L);
+               exynos4_mct_write(hi, EXYNOS4_MCT_G_CNT_U);
+
+               reg |= MCT_G_TCON_START;
+               exynos4_mct_write(reg, EXYNOS4_MCT_G_TCON);
+       }
 }

Quoted 2 lines

As Doug mentioned, this seems more complicated than necessary since the kernel doesn't care about the initial value of the mct counter at all. Is there some reason from a hardware standpoint that the counter needs to be cleared? If not, I would rather just delete the two offending lines. I am sending a patch that does this instead.

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 4 lines

Quoted 1 line

* it implements the bits needed for udelay() to use it.

Hmm, do you know what bits are those? On Exynos4 MCT is the only option and it would be nice to let udelay() use it.

Look for register_current_timer_delay(). It seems like we could do this for MCT, but I've never done the investigation because we were always planning to move to arch timers. ;)

Tomasz,

Quoted 30 lines · Doug Anderson

Tomasz,

Quoted 5 lines · Tomasz Figa

Quoted 4 lines

NOTE: if for some reason we need to keep the MCT around, we're definitely going to need to account for the fact that tweaking it affects the arch timer. ...and having the arch timer is really nice since:

Quoted 7 lines

[Let me reorder the points below to make it easier to comment:]

Quoted 2 lines

* it's faster to access.
* it is accessible from userspace for really fast access.

Do you have some data on whether it is a significant difference, especially considering real use cases?

I know that Chrome makes _a lot_ of calls to gettimeofday() for profiling purposes, enough that it showed up on benchmarks. In fact, we made a change to the MCT to make accesses faster and there's a small mention of the benchmarking that was done at:

https://chromium-review.googlesource.com/#/c/32190/

...that change probably should be sent upstream, actually.

I'll let Chirantan comment on how much faster arch timers were. ...and I think David Riley (also CCed now) may be able to comment on the benefits of userspace timers.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

* it implements the bits needed for udelay() to use it.

Hmm, do you know what bits are those? On Exynos4 MCT is the only option and it would be nice to let udelay() use it.

Look for register_current_timer_delay(). It seems like we could do this for MCT, but I've never done the investigation because we were always planning to move to arch timers. ;)

If you're planning to add support for MCT as a source for udelay, let me know. It sounds like there's a chance that we won't be able to use the ARCH timers on 5420 so we may be interested in these patches as well.

Also note that it appears that MCT upstream is also not used as a scheduler clock. Perhaps you would want those patches, too? I think Chirantan said that it will cause problems on systems that have both MCT and arch timers though, since the system didn't like two scheduler clocks to be registered (?).

In summary, we've got the following MCT patches proposed to go upstream:

1. MCT scheduler clock: <http://crosreview.com/56363> and
<http://crosreview.com/56364>
2. Speed MCT access: <http://crosreview.com/56365>. I wonder if we could also speed it up further with a 64-bit read.
3. Use MCT for udelay: yet to be written.

...does someone want to claim the task of sending those things up?

Oh, actually it looks like (93bfb76 clocksource: exynos_mct: register sched_clock callback) in linuxnext adds a partial version of the first patch but isn't as complete as what's in our tree (it's missing the KConfig change we have locally as well as the notrace so it probably breaks ftrace?). Adding Vincent.

-Doug

Quoted 8 lines

In summary, we've got the following MCT patches proposed to go upstream:

1. MCT scheduler clock: <http://crosreview.com/56363> and
<http://crosreview.com/56364>
2. Speed MCT access: <http://crosreview.com/56365>. I wonder if we could also speed it up further with a 64-bit read.
3. Use MCT for udelay: yet to be written.

Quoted 8 lines

...does someone want to claim the task of sending those things up?

Oh, actually it looks like (93bfb76 clocksource: exynos_mct: register sched_clock callback) in linuxnext adds a partial version of the first patch but isn't as complete as what's in our tree (it's missing the KConfig change we have locally as well as the notrace so it probably breaks ftrace?). Adding Vincent.

Hi Doug,

Thanks for adding me in the loop.

The only difference i see are:
-HAVE_SCHED_CLOCK which is no more needed
-and the use of 32bit vs 64bit in the for-next
-notrace is present in the for-next AFAICT

Tomasz,

Quoted 9 lines · Tomasz Figa

Hi Chirantan,

Quoted 5 lines · Chirantan Ekbote

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Quoted 3 lines

Huh? I've always thought MCT is a completely separate hardware block outside of ARM cores, while architected timers are embedded inside the CPU block in which the ARM cores reside. Could you elaborate on this?

Yup. Our thoughts exactly.

...but it appears not to be the case. Chirantan demonstrated this in U-Boot just to prove that it's not some strange kernel interaction in
<https://chromium-review.googlesource.com/200035>. I took his patch and tweaked it a little more myself in
<https://chromium-review.googlesource.com/200098>.

Specifically:
* If you stop the MCT, the arch timer stops
* If you reset the MCT, the arch timer resets
* If you start the MCT again, the arch timer starts again
* If you read the MCT and the arch timer, they give the same value.

This is apparently the answer to my question at
<http://www.spinics.net/lists/linux-samsung-soc/msg29085.html>.
Specifically Chirantan found that the big jump in time happened when MCT reset to 0. That made the arch timer code think that there was a wraparound and jump forward in time a lot.

Please confirm if you have a system that has MCT and arch timer in front of you.

Quoted 39 lines · Doug Anderson

Tomasz,

Quoted 9 lines · Tomasz Figa

Hi Chirantan,

Quoted 5 lines · Chirantan Ekbote

The multi core timer and the ARM architected timer are two different interfaces to the same underlying hardware timer. This causes some strange timing issues when they are both enabled at the same time so remove the mct from the device tree and keep only the architected timer.

Quoted 3 lines

Huh? I've always thought MCT is a completely separate hardware block outside of ARM cores, while architected timers are embedded inside the CPU block in which the ARM cores reside. Could you elaborate on this?

Yup. Our thoughts exactly.

...but it appears not to be the case. Chirantan demonstrated this in U-Boot just to prove that it's not some strange kernel interaction in
<https://chromium-review.googlesource.com/200035>. I took his patch and tweaked it a little more myself in
<https://chromium-review.googlesource.com/200098>.

Specifically:
* If you stop the MCT, the arch timer stops
* If you reset the MCT, the arch timer resets
* If you start the MCT again, the arch timer starts again
* If you read the MCT and the arch timer, they give the same value.

This is apparently the answer to my question at
<http://www.spinics.net/lists/linux-samsung-soc/msg29085.html>.
Specifically Chirantan found that the big jump in time happened when MCT reset to 0. That made the arch timer code think that there was a wraparound and jump forward in time a lot.

Quoted 3 lines

Please confirm if you have a system that has MCT and arch timer in front of you.

Hi all,

I need to talk to hardware guy to clarify the issue then I'll let you know.