interrupt latency while resuming.

I am trying to address an issue of not handling a wakeup interrupt quick enough while resuming. It is an edge triggered interrupt with the IRQ_WAKEUP flag set. The interrupt controller implements lazy disabling of interrupts, IOW does not have a disable callback in the irq_chip.

So while going in to supend that interrupt is marked IRQ_DISABLED in dpm_suspend_noirq().

On resume handle_edge_trigger is run right after arch_suspend_enable_irqs(). It finds the interrupt marked IRQ_DISABLED and it sets the IRQ_PENDING flag and does not call the handler.

As the resume path unrolls, non boot cpus are enabled, dpm_resume_noirq() is run. At that time it finds the IRQ_PENDING flag is set on this interrupt and the interrupt handler is run.

The problem is, this is very late for the interrupt to be run. Possibly because enable_nonboot_cpu takes a while or the resume_noirq callbacks take a long time etc.

I tried using IRQF_NO_SUSPEND for that interrupt and it seems the interrupt is handled as soon as arch_suspend_enable_irqs() is run.
However I suspect that with this the system will fail to abort suspend - The interrupt could trigger between dpm_suspend_noirq() and arch_suspend_disable_irqs() and since it wont be marked IRQF_PENDING the system goes to suspend never to be woken up again. (the check to abort suspend because of a pending interrupt is done in check_wakeup_irqs() in sysdev_suspend). I dont think IRQF_NO_SUSPEND was designed for wakeup interrupts. Please correct me if I am missing something here.

A solution that comes to mind is enabling such interrupts right before arch_suspend_enable_irqs() is run. In some more detail, mark these interrupts as IRQF_LOW_SUSPEND_LATENCY in their irq_desc->status and enable such interrupts before doing arch_suspend_enable_irqs(). That way when arch_suspend_enable_irqs() happens, the handler is run immediately. We skip enabling such interrupts in resume_device_irqs() to avoid enabling them twice.

Will appreciate any other suggestions towards fixing the delay.

Abhijeet

Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

The reason it's after non-boot CPUs are enabled is that the affinity settings may require it to be run on one of these CPUs, so we need to ensure that those CPUs are up.

We know it takes something like 200ms to run the lpg calibration, and this is responsible for the majority of the time when bringing a CPU online.

One solution is to sort out whether we need to re-run the lpj calibration each time we resume a CPU. At the moment, we're just using the boot CPU loops_per_jiffy for every udelay(), assuming that all CPUs are running at the same clock rate - so we _could_ skip the calibration as its never used.

Or we could fix things so that it is used (which also sorts udelay for asymetrically clocked secondary CPUs) but then we always have to re-run the calibration, or we have to stipulate that secondary CPUs come up at the same clock rate as they went down. I suspect there's systems out there where the latter is not true at present, so it's not something we can impose - which means we're stuck having to do 200ms of recalibration per CPU.

Another solution is to check whether we can run the handler before other CPUs are online - it may mean we're breaking IRQ affinity, but any non-boot affinity has already been broken by taking the other CPUs offline.

Agree. I think we are breaking affinity anyways by not setting the interrupt targets to pre disable_nonboot_cpus().

We can say that for interrupts marked IRQF_LOW_SUSPEND_LATENCY we may not be honoring the affinity. They *will* run on the boot cpu, even if their affinity is set to others upon resume. Driver authors should excercise caution wrt to affinity for such interrupts. For all the other interrupts we can restore the targets to pre disable_nonboot_cpu state. But this is a different problem.

Do you agree?

I will test the proposal of IRQF_LOW_SUSPEND_LATENCY and push out a patch soon.

Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

We're actually using a timer based udelay() (which uses calibrate_delay_direct() instead of the more expensive delay loop) and thus lpj is calculated in about 50ms. You raise an interesting point though since we don't need to recalibrate as our timesource is constant. It would be great to skip the recalibration in such a situation and speed up onlining CPUs.

Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.