PREEMPT_RT_FULL breaks NO_HZ_FULL (full dynticks)

Normally CONFIG_NO_HZ_FULL (full dynticks) feature blocks timer interrupts when a single task is running in an isolated CPU.

This behavior changes when CONFIG_PREEMPT_RT_FULL is also enabled. I notice timer ticks are blocked the very first time after boot and a single task is running. However, when the task is stopped and run for the second time, the timer ticks are not blocked anymore.

Disabling CONFIG_PREEMPT_RT_FULL brings back the expected behavior of CONFIG_NO_HZ_FULL even if the task is stopped and rerun many times, as long as it is the single task running.

Following are the system and kernel configurations used in my tests:

Kernel version: 4.18.0-rc8, branch linux-4.18.y-rt Set CONFIG_NO_HZ_FULL=y

CPU isolaion is done using following kernel parameters:
isolcpus=nohz,domain,1-3 nohz_full=1-3 rcu_nocbs=1-3 irqaffinity=0

Workload is run in CPU 3, with SCHED_FIFO priority 99 as follows:
taskset -c 3 chrt -f 99 ./jitter

Give RT process 100% time for testing purpose:
echo -1 > /proc/sys/kernel/sched_rt_runtime_us

Following are the list of processes in CPU 3 in the 3 cases:

Case #1 with CONFIG_PREEMPT_RT_FULL=n
S [cpuhp/3]
S [migration/3]
S [rcuc/3]
S [ksoftirqd/3]
I [kworker/3:0-mm_]
I [kworker/3:0H]
R [kworker/3:1-mm_]
R ./jitter

Case #2 with CONFIG_PREEMPT_RT_FULL=y (First run after boot) S [cpuhp/3]
S [migration/3]
S [posixcputmr/3]
S [rcuc/3]
S [ktimersoftd/3]
S [ksoftirqd/3]
I [kworker/3:0-mm_]
I [kworker/3:0H]
R [irq/125-nvme0q4]
R [kworker/3:1-mm_]
R ./jitter

Case #3 with CONFIG_PREEMPT_RT_FULL=y (Second run after boot) S [cpuhp/3]
S [migration/3]
S [posixcputmr/3]
S [rcuc/3]
R [ktimersoftd/3]
S [ksoftirqd/3]
I [kworker/3:0-mm_]
I [kworker/3:0H]
S [irq/125-nvme0q4]
R [kworker/3:1-mm_]
R ./jitter

In Case #3, /proc/interupts show timer interrupts occuring on CPU 3 while it is stopped in the other cases. ktimersoftd is in runnable state in Case #3

Case #1 with PREEMPT_RT_FULL disabled has fewer processes and looks clean.

We need both PREEMPT_RT_FULL and NO_HZ_FULL for use cases where a single RT task in a core can run with bare metal performance without timer interrupts.

Is this a known issue and is it being looked at by anyone?

If it is an issue, I would be glad to help in any way to get these 2 very important features compatible with each other.

Thanks,
Ramesh

irq/125 shouldn't be there, right?

can you trace down who or what is arming the timer on CPU3?

now that I know of. Do you happen to know if this is a regression compared to v4.14-RT?

So if the ktimersoftd runs and you see the interrupt counter incrementing for CPU3 then it would be interesting to figure out why there is an armed timer on the second invocation (and none on the first one).

Sebastian

Yes. Also posixcputmr and ktimersoftd are not seen if PREEMPT_RT_FULL is not enabled. They don't seem to be running when the timer interrupts occur. But they being there by itself indicates something different is happening.
Sched RT_Prio Cpu_Time

S [posixcputmr/3]      FF      99 	00:00:00
R [ktimersoftd/3]      FF       1 	00:00:00
S [irq/125-nvme0q4]    FF      50 	00:00:00
R ./jitter             FF      99 	00:01:53

Ok, I will take a look.

I see the issue in 4.14.63 RT as well. There are slight differences in behavior due to changes that went in 4.17, but the main issue is seen there also.

In 4.18.5 RT, the issue does not always happen in the second invocation. Sometimes it works as expected, but the issue will show up after a few tries.

Looks like there are 2 issues when PREEMPT_RT_FULL is enabled
1. Some additional processes are pinned to isolated cores.
2. Timer is armed even though only a single high priority task is running.

both threads are created on RT and should be idle unless used.

okay, so it is not a regression. Thanks for confirming.

as explained, above that is expected.

something must have armed that timer. The clock_nanosleep() API won't use those new threads but for instance timer_create() / timer_settime() will.

Sebastian

[..]

Did we actually ever solve the "ktimersoftd uselessly woken on every tick" problem?

Julia

I think so, yes. According to the report the problem does not happen the first invocation. However the "ktimersoftd uselessly woken on every tick" problem occurred "always".

Sebastian