HRTimer causing rtctest to fail

I'm facing an odd problem with v4.16-rc2 (also reproducible on v4.15 final) where rtctest fails sometimes which PIE coming too late with frequencies >= 1024 Hz.

I've modified rtctest.c a bit so that it continues running even after first failure just so I could get a glimpse of how many times it fails.

Here are the results:

Error 1024Hz i 17/20 diff 1089/976

Error 2048Hz i 2/20 diff 538/488
Error 2048Hz i 9/20 diff 558/488
Error 2048Hz i 16/20 diff 560/488

Error 4096Hz i 2/20 diff 305/244
Error 4096Hz i 7/20 diff 288/244
Error 4096Hz i 9/20 diff 285/244
Error 4096Hz i 11/20 diff 310/244
Error 4096Hz i 16/20 diff 284/244
Error 4096Hz i 20/20 diff 317/244

I added a few trace_printk() calls around rtc_dev_read() (and some other functions) and captured the time for entry and exit of that function. I noticed that HRTimer fires way too late and also way too early sometimes. On the order of 100+ us.

For example with iterations 17 of 1024 Hz (seen above) and 18 (also of 1024 Hz) I captured the following time stamps (all in uS):

|    Start |      End | Expected End | Diff |
|----------+----------+--------------+------|
| 35900626 | 35901711 |     35901603 | -108 | (too late)
| 35901826 | 35902628 |     35902803 |  174 | (too early)

I can't come up with a proper explanation as to why HRTimer is firing at such wildly odd extremes. On most of the measurements, HRTimer executes within 5 uS of scheduled time and I can't convince myself that scheduling latency would get so high all of a sudden. Specially considering I'm testing with a 4 core machine and there's nothing else running on it. It's mostly idle.

Quoted 23 lines · Felipe Balbi

I'm facing an odd problem with v4.16-rc2 (also reproducible on v4.15 final) where rtctest fails sometimes which PIE coming too late with frequencies >= 1024 Hz.

I've modified rtctest.c a bit so that it continues running even after first failure just so I could get a glimpse of how many times it fails.

Here are the results:

Error 1024Hz i 17/20 diff 1089/976

Error 2048Hz i 2/20 diff 538/488
Error 2048Hz i 9/20 diff 558/488
Error 2048Hz i 16/20 diff 560/488

Error 4096Hz i 2/20 diff 305/244
Error 4096Hz i 7/20 diff 288/244
Error 4096Hz i 9/20 diff 285/244
Error 4096Hz i 11/20 diff 310/244
Error 4096Hz i 16/20 diff 284/244
Error 4096Hz i 20/20 diff 317/244

Quoted 4 lines

I added a few trace_printk() calls around rtc_dev_read() (and some other functions) and captured the time for entry and exit of that function. I noticed that HRTimer fires way too late and also way too early sometimes. On the order of 100+ us.

Power management, deep idle states?

Quoted 7 lines

For example with iterations 17 of 1024 Hz (seen above) and 18 (also of 1024 Hz) I captured the following time stamps (all in uS):

|    Start |      End | Expected End | Diff |
|----------+----------+--------------+------|
| 35900626 | 35901711 |     35901603 | -108 | (too late)
| 35901826 | 35902628 |     35902803 |  174 | (too early)

Too early? hrtimers never fire too early. The measurement in that test is bogus:

                        gettimeofday(&start, NULL);
                        /* This blocks */
                        retval = read(fd, &data, sizeof(unsigned long));
                        if (retval == -1) {
                                perror("read");
                                exit(errno);
                        }
                        gettimeofday(&end, NULL);

It's measuring relative time between going to sleep and wakeup, so if an event comes late, then the next one will claim to be early which is nonsense because periodic mode runs on an absolute time line.

		1	2       3       4	5	6	7
Expected:	|	|	|	|	|	|	|

Real: | | | | | | |

^ late event
So 2->3 would give you (period + latency) and 3->4 (period - latency) ....

And if the task gets scheduled out before the taking the start time stamp then this is also giving bogus results.

What the test really should figure out is the deviation of the absolute time line. That's hard because the RTC code starts the hrtimer with:

     hrtimer_start(&rtc->pie_timer, period, HRTIMER_MODE_REL);

which means relative to 'now'. So there is no good way for user space to determine the expected timeline.

Quoted 6 lines

I can't come up with a proper explanation as to why HRTimer is firing at such wildly odd extremes. On most of the measurements, HRTimer executes within 5 uS of scheduled time and I can't convince myself that scheduling latency would get so high all of a sudden. Specially considering I'm testing with a 4 core machine and there's nothing else running on it. It's mostly idle.

Enable the hrtimer and scheduling tracepoints. That should give you a hint what's going on.

Quoted 2 lines · Thomas Gleixner

Enable the hrtimer and scheduling tracepoints. That should give you a hint what's going on.

Thanks, that does give me a lot more information. So here you go:

         rtctest-1348  [003] d..2   313.766141: hrtimer_start: hrtimer=00000000667ce595 function=rtc_pie_update_irq expires=313573983010 softexpires=313573983010 mode=REL
          <idle>-0     [003] d.h1   313.767189: hrtimer_expire_entry: hrtimer=00000000667ce595 function=rtc_pie_update_irq now=313574053764

We still have a 70754 nS deviation. After changing to absolute time, the deviation remains:

          <idle>-0     [000] dNh2    29.303251: hrtimer_start: hrtimer=000000006858b496 function=rtc_pie_update_irq expires=28765551360 softexpires=28765551360 mode=ABS
          <idle>-0     [000] d.h1    29.303565: hrtimer_expire_entry: hrtimer=000000006858b496 function=rtc_pie_update_irq now=28765621916

Quoted 6 lines · Felipe Balbi

Quoted 2 lines · Thomas Gleixner

Enable the hrtimer and scheduling tracepoints. That should give you a hint what's going on.

Thanks, that does give me a lot more information. So here you go:

Quoted 8 lines

         rtctest-1348  [003] d..2   313.766141: hrtimer_start: hrtimer=00000000667ce595 function=rtc_pie_update_irq expires=313573983010 softexpires=313573983010 mode=REL
          <idle>-0     [003] d.h1   313.767189: hrtimer_expire_entry: hrtimer=00000000667ce595 function=rtc_pie_update_irq now=313574053764

We still have a 70754 nS deviation. After changing to absolute time, the deviation remains:

          <idle>-0     [000] dNh2    29.303251: hrtimer_start: hrtimer=000000006858b496 function=rtc_pie_update_irq expires=28765551360 softexpires=28765551360 mode=ABS
          <idle>-0     [000] d.h1    29.303565: hrtimer_expire_entry: hrtimer=000000006858b496 function=rtc_pie_update_irq now=28765621916

Changing REL/ABS in the kernel does not make a difference because periodic mode just forwards by period so even if the first timer is started with REL it results in a absolute timeline. What I meant is the user space measurement as it cannot figure out when the first event was supposed to happen so it's hard to calculate latency information.

The interesting information is that the timer fires late and the system is idle. Now the question is in which idle state did the machine go?

Wake up from deeper C-states can be slow. On my laptop the wakeup latencies are:

POLL 0
C1 2
C1E 10
C3 33
C6 133
C7S 166
C8 300
C9 600
C10 2600

All numbers in micro seconds! What happens if you load the system or restrict C-States?

Quoted 2 lines · Thomas Gleixner

All numbers in micro seconds! What happens if you load the system or restrict C-States?

added intel_idle.max_cstate=0 to cmdline. Still see some failures, albeit a lot more rare, for example, caught one on iteration 8 of 8192Hz:

8192Hz: 1 2 3 4 5 6 7
PIE delta error: 0.000135 should be close to 0.000122

That is, however, only 13 uS late. Is that within the limits of "we don't quite care"?

BTW, it took some effort to get that one failure. I'll leave rtctest running in a long loop and try to figure out approximately how frequently it fails, but it seems like it's within the realm of "don't care", would you agree?

Quoted 11 lines · Felipe Balbi

Quoted 2 lines · Thomas Gleixner

All numbers in micro seconds! What happens if you load the system or restrict C-States?

added intel_idle.max_cstate=0 to cmdline. Still see some failures, albeit a lot more rare, for example, caught one on iteration 8 of 8192Hz:

8192Hz: 1 2 3 4 5 6 7
PIE delta error: 0.000135 should be close to 0.000122

Quoted 2 lines

That is, however, only 13 uS late. Is that within the limits of "we don't quite care"?

I think so. 13us (depending on the machine) can easily happen if multiple interrupts/timers happen at the same time or if something keeps interrupts disabled for a few microseconds which is certainly possible.

What kind of system is this?

Quoted 4 lines

BTW, it took some effort to get that one failure. I'll leave rtctest running in a long loop and try to figure out approximately how frequently it fails, but it seems like it's within the realm of "don't care", would you agree?

Yes.