Skip to contentSkip to conversation

iio: hrtimer-trigger: Mark hrtimer to expire in hard interrupt context

PATCH Superseded Lars-Peter Clausen 9 participants 20 messages Aug 13, 2020 – Oct 10, 2020 IIO

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 iio: hrtimer-trigger: Mark hrtimer to expire in hard interrupt context

On PREEMPT_RT enabled kernels unmarked hrtimers are moved into soft interrupt expiry mode by default.

The IIO hrtimer-trigger needs to run in hard interrupt context since it will end up calling generic_handle_irq() which has the requirement to run in hard interrupt context.

Explicitly specify that the timer needs to run in hard interrupt context by using the HRTIMER_MODE_REL_HARD flag.

Fixes f5c2f0215e36 ("hrtimer: Move unmarked hrtimers to soft interrupt expiry on RT")
---
 drivers/iio/trigger/iio-trig-hrtimer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iio/trigger/iio-trig-hrtimer.c +1 -1
index f59bf8d58586..b11ca915fcb2 100644
@@ -132,7 +132,7 @@ static struct iio_sw_trigger *iio_trig_hrtimer_probe(const char *name)
 	trig_info->swt.trigger->ops = &iio_hrtimer_trigger_ops;
 	trig_info->swt.trigger->dev.groups = iio_hrtimer_attr_groups;
 
-	hrtimer_init(&trig_info->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&trig_info->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
 	trig_info->timer.function = iio_hrtimer_trig_handler;
 
 	trig_info->sampling_frequency = HRTIMER_DEFAULT_SAMPLING_FREQUENCY;
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/2 iio: sysfs-trigger: Mark irq_work to expire in hardirq context

Mark the IIO sysfs-trigger irq_work with IRQ_WORK_HARD_IRQ to ensure that it is always executed in hard interrupt context, even with PREEMPT_RT=y.

The IIO sysfs-trigger irq_work needs to run in hard interrupt context since it will end up calling generic_handle_irq() which has the requirement to run in hard interrupt context.

Note that the IRQ_WORK_HARD_IRQ flag, while it exists, does not seem to do anything in the mainline kernel yet. It does have an effect in the RT patchset though and presumably this is sooner or later going to be added to mainline as well.

---
 drivers/iio/trigger/iio-trig-sysfs.c | 1 +
 1 file changed, 1 insertion(+)
drivers/iio/trigger/iio-trig-sysfs.c +1 -0
index e09e58072872..10a3fd29362b 100644
@@ -161,6 +161,7 @@ static int iio_sysfs_trigger_probe(int id)
 	iio_trigger_set_drvdata(t->trig, t);
 
 	init_irq_work(&t->work, iio_sysfs_trigger_work);
+	atomic_set(&t->work.flags, IRQ_WORK_HARD_IRQ);
 
 	ret = iio_trigger_register(t->trig);
 	if (ret)
Signature · 1 line
2.20.1
#2 lore raw
In reply to Lars-Peter Clausen
Quoted 9 lines · Lars-Peter Clausen

On PREEMPT_RT enabled kernels unmarked hrtimers are moved into soft interrupt expiry mode by default.

The IIO hrtimer-trigger needs to run in hard interrupt context since it will end up calling generic_handle_irq() which has the requirement to run in hard interrupt context.

Explicitly specify that the timer needs to run in hard interrupt context by using the HRTIMER_MODE_REL_HARD flag.

No, I don't think that this is good. It basically renders threaded-irqs in context of IIO useless. This also requires that the IRQ-handler in question runs with IRQs disabled / uses raw_spinlock_t which is in not good idea either.

Has this change (including the second patch in thread) been tested on RT in terms of locking and latency?

Quoted 12 lines
Fixes f5c2f0215e36 ("hrtimer: Move unmarked hrtimers to soft interrupt expiry on RT")
---
 drivers/iio/trigger/iio-trig-hrtimer.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iio/trigger/iio-trig-hrtimer.c +0 -0
index f59bf8d58586..b11ca915fcb2 100644

drivers/iio/trigger/iio-trig-hrtimer.c static struct iio_sw_trigger *iio_trig_hrtimer_probe(const char *name) · line 138

Quoted 8 lines
drivers/iio/trigger/iio-trig-hrtimer.c +1 -1
@@ -132,7 +132,7 @@ static struct iio_sw_trigger *iio_trig_hrtimer_probe(const char *name)
 	trig_info->swt.trigger->ops = &iio_hrtimer_trigger_ops;
 	trig_info->swt.trigger->dev.groups = iio_hrtimer_attr_groups;
 
-	hrtimer_init(&trig_info->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+	hrtimer_init(&trig_info->timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD);
 	trig_info->timer.function = iio_hrtimer_trig_handler;
 
 	trig_info->sampling_frequency = HRTIMER_DEFAULT_SAMPLING_FREQUENCY;

Sebastian

#3 lore raw
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 7 lines · Lars-Peter Clausen

On PREEMPT_RT enabled kernels unmarked hrtimers are moved into soft interrupt expiry mode by default.

The IIO hrtimer-trigger needs to run in hard interrupt context since it will end up calling generic_handle_irq() which has the requirement to run in hard interrupt context.

Quoted 6 lines
Quoted 2 lines

Explicitly specify that the timer needs to run in hard interrupt context by using the HRTIMER_MODE_REL_HARD flag.

No, I don't think that this is good. It basically renders threaded-irqs in context of IIO useless. This also requires that the IRQ-handler in question runs with IRQs disabled / uses raw_spinlock_t which is in not good idea either.

It should not affect the IRQ handlers of individual drivers. The hrtimer triggers acts like an IRQ chip and will call generic_handle_irq() to multiplex the interrupt handling onto all consumers. As far as I understand it there is a requirement that generic_handle_irq() is called in hard irq context, even with PREEMT_RT=y.

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

Quoted 3 lines

Has this change (including the second patch in thread) been tested on RT in terms of locking and latency?

It has not been tested in terms of latency. But like I said if you are running with forced IRQ threads the effect should be minimal.

Without this patch there is an correctness issue when PREEMT_RT=y since generic_handle_irq() runs with interrupts on which breaks its internal assumptions.

#4 lore raw
Quoted 6 lines · Lars-Peter Clausen

It should not affect the IRQ handlers of individual drivers. The hrtimer triggers acts like an IRQ chip and will call generic_handle_irq() to multiplex the interrupt handling onto all consumers. As far as I understand it there is a requirement that generic_handle_irq() is called in hard irq context, even with PREEMT_RT=y.

That is correct.

Quoted 3 lines

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

So if it is really just the wakeup of the IRQ-thread then it should be okay.
One thing: iio_trigger_poll() may invoke iio_trigger_notify_done(). This would invoke trig->ops->try_reenable callback if available. I grepped and found
- bma180_trig_try_reen()
It appears to perform i2c_smbus_read_byte_data() and smbus sounds sleeping. I don't know if it attempts to acquire any spinlock_t but it will be wrong on RT.

- bmc150_accel_trig_try_reen()
This one has mutex_lock() which is wrong even on !RT in this context (unless the previous `if' saves us).

- mxc4005_trigger_try_reen()
This one uses regmap_write(). regmap internally uses a lock and the config does not disable / provide a lock. This means regmap_lock_mutex() is used (or regmap_lock_spinlock() in case of bus->fast_io but I doubt it with i2c).

Am I looking somehow wrong at this or did just nobody try the combination of one of the three drivers here together with the hrtimer trigger?

Quoted 10 lines
Quoted 3 lines

Has this change (including the second patch in thread) been tested on RT in terms of locking and latency?

It has not been tested in terms of latency. But like I said if you are running with forced IRQ threads the effect should be minimal.

Without this patch there is an correctness issue when PREEMT_RT=y since generic_handle_irq() runs with interrupts on which breaks its internal assumptions.

I'm trying to understand the scope of the change. As I said above, if it is just wakeup of the thread, then it is fine. I have memory of people running iio drivers (or triggers) in hardirq-context for $reason and try avoid something like this.

Sebastian

#5 lore raw
Quoted 5 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Lars-Peter Clausen

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

Quoted 9 lines

So if it is really just the wakeup of the IRQ-thread then it should be okay.
One thing: iio_trigger_poll() may invoke iio_trigger_notify_done(). This would invoke trig->ops->try_reenable callback if available. I grepped and found
- bma180_trig_try_reen()
It appears to perform i2c_smbus_read_byte_data() and smbus sounds sleeping. I don't know if it attempts to acquire any spinlock_t but it will be wrong on RT.

It's wrong even on !RT. i2c reads cannot be invoked from hard interrupt context.

#6 lore raw
Quoted 16 lines · On Thu, 13 Aug 2020 14:19:57 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 5 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Lars-Peter Clausen

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

Quoted 9 lines

So if it is really just the wakeup of the IRQ-thread then it should be okay.
One thing: iio_trigger_poll() may invoke iio_trigger_notify_done(). This would invoke trig->ops->try_reenable callback if available. I grepped and found
- bma180_trig_try_reen()
It appears to perform i2c_smbus_read_byte_data() and smbus sounds sleeping. I don't know if it attempts to acquire any spinlock_t but it will be wrong on RT.

Quoted 3 lines

It's wrong even on !RT. i2c reads cannot be invoked from hard interrupt context.

We would hit this (and resulting warnings) all the time if it actually happened, so my suspicion is that it doesn't.

I think the path doesn't actually exist although it looks at first glance like it does.

The interrupt can only be enabled if there is someone using the trigger. Thus usecount will be non zero and for at least one element trig->subirq[i].enabled == true

So we will decrement trig->usecount in the call to iio_trigger_notify_done but never reach 0 thus the call to trig->ops->try_reenable never happens in the hard interrupt context.

It does happen later when which ever driver we triggered finishes the threaded part of it's handler and calls iio_trigger_notify_done, but that is fine.

Assuming people agree with my analysis it would be good to make it explicit that we cannot hit the problem path.

Perhaps call a new iio_trigger_notify_no_needed() that simply does the decrement without test, or does it with test and spits out a warning if we hit 0.

Jonathan

#7 lore raw
Quoted 25 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 13 Aug 2020 14:19:57 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 4 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Lars-Peter Clausen

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

Quoted 9 lines

So if it is really just the wakeup of the IRQ-thread then it should be okay.
One thing: iio_trigger_poll() may invoke iio_trigger_notify_done(). This would invoke trig->ops->try_reenable callback if available.
I grepped and found
- bma180_trig_try_reen()
It appears to perform i2c_smbus_read_byte_data() and smbus sounds sleeping. I don't know if it attempts to acquire any spinlock_t but it will be wrong on RT.

Quoted 3 lines

It's wrong even on !RT. i2c reads cannot be invoked from hard interrupt context.

We would hit this (and resulting warnings) all the time if it actually happened, so my suspicion is that it doesn't.

I think the path doesn't actually exist although it looks at first glance like it does.

Quoted 7 lines

The interrupt can only be enabled if there is someone using the trigger. Thus usecount will be non zero and for at least one element trig->subirq[i].enabled == true

So we will decrement trig->usecount in the call to iio_trigger_notify_done but never reach 0 thus the call to trig->ops->try_reenable never happens in the hard interrupt context.

I think there is a race condition here. If a consumer is disabled concurrently with iio_trigger_poll() there is a chance that enabled is false for all consumers.

The odds of this happening are very low, but there is nothing that prevents it.

Quoted 11 lines

It does happen later when which ever driver we triggered finishes the threaded part of it's handler and calls iio_trigger_notify_done, but that is fine.

Assuming people agree with my analysis it would be good to make it explicit that we cannot hit the problem path.

Perhaps call a new iio_trigger_notify_no_needed() that simply does the decrement without test, or does it with test and spits out a warning if we hit 0.

I think we need to re-think the whole try_reenable() functionality. Looking at it I think there are more issues here.

For example lets say we call iio_trigger_notify_done() from the threaded handler and try_reenable() returns true. We'd now call iio_trigger_poll() from the threaded context, which is wrong.

There is also the issue that iio_trigger_poll() effectively can end up calling itself recursively indefinitely via the iio_trigger_notify_done() chain, which might not be the best thing in hard IRQ context.

#8 lore raw
Quoted 34 lines · On Fri, 14 Aug 2020 07:24:52 +0200 Lars-Peter Clausen <[email protected]> wrote:
Quoted 25 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 13 Aug 2020 14:19:57 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 4 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Lars-Peter Clausen

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

Quoted 9 lines

So if it is really just the wakeup of the IRQ-thread then it should be okay.
One thing: iio_trigger_poll() may invoke iio_trigger_notify_done(). This would invoke trig->ops->try_reenable callback if available.
I grepped and found
- bma180_trig_try_reen()
It appears to perform i2c_smbus_read_byte_data() and smbus sounds sleeping. I don't know if it attempts to acquire any spinlock_t but it will be wrong on RT.

Quoted 3 lines

It's wrong even on !RT. i2c reads cannot be invoked from hard interrupt context.

We would hit this (and resulting warnings) all the time if it actually happened, so my suspicion is that it doesn't.

I think the path doesn't actually exist although it looks at first glance like it does.

Quoted 7 lines

The interrupt can only be enabled if there is someone using the trigger. Thus usecount will be non zero and for at least one element trig->subirq[i].enabled == true

So we will decrement trig->usecount in the call to iio_trigger_notify_done but never reach 0 thus the call to trig->ops->try_reenable never happens in the hard interrupt context.

Quoted 6 lines

I think there is a race condition here. If a consumer is disabled concurrently with iio_trigger_poll() there is a chance that enabled is false for all consumers.

The odds of this happening are very low, but there is nothing that prevents it.

A consumer disable should have previously resulted in the trigger stopping (or do we have he ordering wrong there?) Might be worth checking that, but see below, as I think we can stop it having harmful effects anyway.

Quoted 13 lines
Quoted 6 lines

It does happen later when which ever driver we triggered finishes the threaded part of it's handler and calls iio_trigger_notify_done, but that is fine.

Quoted 6 lines

Assuming people agree with my analysis it would be good to make it explicit that we cannot hit the problem path.

Perhaps call a new iio_trigger_notify_no_needed() that simply does the decrement without test, or does it with test and spits out a warning if we hit 0.

Quoted 6 lines

I think we need to re-think the whole try_reenable() functionality. Looking at it I think there are more issues here.

For example lets say we call iio_trigger_notify_done() from the threaded handler and try_reenable() returns true. We'd now call iio_trigger_poll() from the threaded context, which is wrong.

The history behind that functionality was some early devices which were using level interrupts tied to edge interrupts. One of our earliest drivers had to
do a dance with checking a gpio to avoid stalling. That went away a long
time ago when we basically concluded that platform was too broken to support interrupts on.

Probably would have helped if try_reenable returned a bool as then people wouldn't have returned error codes via it which is completely wrong as it stands (though handy given what I suggest below)

From a quick look.
bma180, bmc150, kxcjk1013, mxc4005m bmg180, kmx61, bmc150 all only fail if an error occurs in an bus access. That's a bug.

at91 never returns anything other than 0.

So I think we can probably rip the iio_trigger_poll call out entirely and replace it with an error message, or push an error message into the drivers and drop the check of the value of try_reenable.

Quoted 6 lines

There is also the issue that iio_trigger_poll() effectively can end up calling itself recursively indefinitely via the iio_trigger_notify_done() chain, which might not be the best thing in hard IRQ context.

:) Indeed unwise. I think the only driver that could in theory do this that we had a long time back had a counter in the try_reen to give up if it happened more than a few times.

Jonathan

maintainer
#9 lore raw
Quoted 95 lines · On Fri, 14 Aug 2020 11:30:08 +0100 Jonathan Cameron…
Quoted 34 lines · On Fri, 14 Aug 2020 07:24:52 +0200 Lars-Peter Clausen <[email protected]> wrote:
Quoted 25 lines · Jonathan Cameron
Quoted 14 lines · On Thu, 13 Aug 2020 14:19:57 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 4 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Lars-Peter Clausen

If you are running with forced IRQ threads the only thing that will then happen in the actual hard IRQ context is the launching of the IRQ threads. Th e IRQ handler of the device driver will run in a threaded IRQ.

Quoted 9 lines

So if it is really just the wakeup of the IRQ-thread then it should be okay.
One thing: iio_trigger_poll() may invoke iio_trigger_notify_done(). This would invoke trig->ops->try_reenable callback if available.
I grepped and found
- bma180_trig_try_reen()
It appears to perform i2c_smbus_read_byte_data() and smbus sounds sleeping. I don't know if it attempts to acquire any spinlock_t but it will be wrong on RT.

Quoted 3 lines

It's wrong even on !RT. i2c reads cannot be invoked from hard interrupt context.

We would hit this (and resulting warnings) all the time if it actually happened, so my suspicion is that it doesn't.

I think the path doesn't actually exist although it looks at first glance like it does.

Quoted 7 lines

The interrupt can only be enabled if there is someone using the trigger. Thus usecount will be non zero and for at least one element trig->subirq[i].enabled == true

So we will decrement trig->usecount in the call to iio_trigger_notify_done but never reach 0 thus the call to trig->ops->try_reenable never happens in the hard interrupt context.

Quoted 6 lines

I think there is a race condition here. If a consumer is disabled concurrently with iio_trigger_poll() there is a chance that enabled is false for all consumers.

The odds of this happening are very low, but there is nothing that prevents it.

A consumer disable should have previously resulted in the trigger stopping (or do we have he ordering wrong there?) Might be worth checking that, but see below, as I think we can stop it having harmful effects anyway.

Quoted 13 lines
Quoted 6 lines

It does happen later when which ever driver we triggered finishes the threaded part of it's handler and calls iio_trigger_notify_done, but that is fine.

Quoted 6 lines

Assuming people agree with my analysis it would be good to make it explicit that we cannot hit the problem path.

Perhaps call a new iio_trigger_notify_no_needed() that simply does the decrement without test, or does it with test and spits out a warning if we hit 0.

Quoted 6 lines

I think we need to re-think the whole try_reenable() functionality. Looking at it I think there are more issues here.

For example lets say we call iio_trigger_notify_done() from the threaded handler and try_reenable() returns true. We'd now call iio_trigger_poll() from the threaded context, which is wrong.

The history behind that functionality was some early devices which were using level interrupts tied to edge interrupts. One of our earliest drivers had to
do a dance with checking a gpio to avoid stalling. That went away a long
time ago when we basically concluded that platform was too broken to support interrupts on.

Probably would have helped if try_reenable returned a bool as then people wouldn't have returned error codes via it which is completely wrong as it stands (though handy given what I suggest below)

From a quick look.
bma180, bmc150, kxcjk1013, mxc4005m bmg180, kmx61, bmc150 all only fail if an error occurs in an bus access. That's a bug.

at91 never returns anything other than 0.

So I think we can probably rip the iio_trigger_poll call out entirely and replace it with an error message, or push an error message into the drivers and drop the check of the value of try_reenable.

Quoted 6 lines

There is also the issue that iio_trigger_poll() effectively can end up calling itself recursively indefinitely via the iio_trigger_notify_done() chain, which might not be the best thing in hard IRQ context.

Quoted 4 lines

:) Indeed unwise. I think the only driver that could in theory do this that we had a long time back had a counter in the try_reen to give up if it happened more than a few times.

A quick update on this one. I've sent out a fix for the try_reenable() patch. Short version is that we no longer let it fail as no one correctly used that path anyway anymore.

https://lore.kernel.org/linux-iio/[email protected]/T/#u

My understanding is that resolves the open question on this series. I'd like to get both issues resolved, so if people have a chance to look that would be great.

Thanks,

Jonathan

Quoted 3 lines

Jonathan

#10 lore raw

Tested together with

- iio: Fix: Do not poll the driver again if try_reenable() callback returns non 0. and
- iio:trigger: rename try_reenable() to reenable() plus return void

on latest mainline (without PREEMPT_RT). The original WARN_ONCE() in kernel/irq/handle.c:159 was not raised anymore. But even without the current patches, this warning is not shown (as this problem only applies to -RT).

Currently I haven't ported a RT kernel > 5.4 for my board, so I cannot check with current RT. On 5.4. there the patches seem not to work fully as kernel/timer/hrtimer.c is not up to date enough.

Sorry for being not very helpful...

Best regards
Christian

Quoted 16 lines · Jonathan Cameron

From: Jonathan Cameron <[email protected]>

As we no longer support a try again if we cannot reenable the trigger
rename the function to reflect this. Also we don't do anything with
the value returned so stop it returning anything. For the few drivers that didn't already print an error message in this patch, add such a print.

Cc Linus Walleij <[email protected]>
Cc Srinivas Pandruvada <[email protected]>
Cc Christian Oder <[email protected]>
Cc Eugen Hristev <[email protected]>
Cc Nishant Malpani <[email protected]>
Cc Daniel Baluta <[email protected]>

---

#11 lore raw
Quoted 10 lines · On Mon, 21 Sep 2020 09:17:26 +0200 Christian Eggers <[email protected]> wrote:

Tested together with

- iio: Fix: Do not poll the driver again if try_reenable() callback returns non 0. and
- iio:trigger: rename try_reenable() to reenable() plus return void

on latest mainline (without PREEMPT_RT). The original WARN_ONCE() in kernel/irq/handle.c:159 was not raised anymore. But even without the current patches, this warning is not shown (as this problem only applies to -RT).

Quoted 5 lines

Currently I haven't ported a RT kernel > 5.4 for my board, so I cannot check with current RT. On 5.4. there the patches seem not to work fully as kernel/timer/hrtimer.c is not up to date enough.

Sorry for being not very helpful...

Thanks for at least trying!

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

Jonathan

Quoted 24 lines

Best regards
Christian

Quoted 16 lines · Jonathan Cameron

From: Jonathan Cameron <[email protected]>

As we no longer support a try again if we cannot reenable the trigger
rename the function to reflect this. Also we don't do anything with
the value returned so stop it returning anything. For the few drivers that didn't already print an error message in this patch, add such a print.

Cc Linus Walleij <[email protected]>
Cc Srinivas Pandruvada <[email protected]>
Cc Christian Oder <[email protected]>
Cc Eugen Hristev <[email protected]>
Cc Nishant Malpani <[email protected]>
Cc Daniel Baluta <[email protected]>

---

#12 lore raw
Quoted 3 lines · Jonathan Cameron

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

I've been told that it only performs IRQ-thread wake-ups in hard-IRQ context. This is fine then.

Looking at the other series where ->try_renable() got renamed. It still looks like bmc150_accel_trig_try_reen() may acquire a mutex. Is it still the case or do I miss something essential?

Quoted 1 line

Jonathan

Sebastian

#13 lore raw
Quoted 11 lines · On Mon, 21 Sep 2020 14:27:28 +0200 Sebastian Andrzej Siewior…
Quoted 3 lines · Jonathan Cameron

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

I've been told that it only performs IRQ-thread wake-ups in hard-IRQ context. This is fine then.

Looking at the other series where ->try_renable() got renamed. It still looks like bmc150_accel_trig_try_reen() may acquire a mutex. Is it still the case or do I miss something essential?

True. We could safely drop the mutex there as it's not doing anything useful, but it can sleep anyway as it's doing a bus write.

So question is whether we can actually hit that path. I think the reality is no (almost - see below), even though it looks like it from a high level.

The path would be that we enter iio_trigger_poll() in interrupt context. That will call generic_handle_irq() to trigger individual devices that are using this trigger.
It will also call iio_trigger_notify_done() to decrement the counter for spare outputs of the irq_chip().

It doesn't actually matter if the problem iio_trigger_notify_done()
(the one that results in a count of 0 and hence reenable()) occurs
as a result of generic_handle_irq() or the direct call of iio_trigger_notify_done() either way it can only happen if we have any drivers calling iio_trigger_notify_done() in interrupt context

Someone who is better at coccinelle than me could probably automate checking this.

Given this is always called after reading the data we should be safe for any device that requires sleeping. So that just leaves a few SoC ADCs to audit. Thankfully most of them don't implement triggered buffers. Of the ones that do only one doesn't call it from a threaded interrupt handler.

drivers/iio/adc/vf610-adc.c

However, there looks to be a lot more wrong in there than just this. So normally for a device with a data ready signal like this we would hook up as follows.

Data ready #1 -> IRQ chip (trigger) -> Read sensor #1 + iio_trigger_notify_done()

                                    ->  Read sensor #2 + iio_trigger_notify_done()

(note that the read etc is normally in a thread - all we do in interrupt context is usually to
grab a timestamp if that makes sense for a given sensor).

This driver does both of.
Data ready -> Read data from itself and call iio_trigger_notify_done()
IRQ chip for a different trigger -> Take a timestamp and never call iio_trigger_notify_done()
or read any data for that matter.

Which won't do what we want at all.

Andy, if I have a go at fixing this are you able to test the result?
I think the simplest is probably to introduce a trigger to tie the two halves together. We can set it as the default trigger so things should keep on working for existing users.

For more general case, we should probably have two functions.

iio_trigger_notify_done() which is only called from places we can sleep. iio_trigger_notify_done_no_action() which only decrements the counter (or given this is only called inside industrialio-trigger.c could just replace
with atomic_dec(&trig->use_count)).

That change is about avoiding confusion rather than fixing a bug and I think tangential to both of the currently changes.

Thanks Sebastian. You are certainly finding some rats holes in my code :)

Jonathan

Quoted 4 lines
Quoted 2 lines

Jonathan

Sebastian

#14 lore raw

From: Jonathan Cameron <[email protected]>

Quoted 40 lines
Quoted 9 lines · On Mon, 21 Sep 2020 14:27:28 +0200 Sebastian Andrzej Siewior…
Quoted 4 lines · Jonathan Cameron

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

I've been told that it only performs IRQ-thread wake-ups in hard-IRQ context. This is fine then.

drivers/iio/adc/vf610-adc.c

However, there looks to be a lot more wrong in there than just this. So normally for a device with a data ready signal like this we would hook up as follows.

Data ready #1 -> IRQ chip (trigger) -> Read sensor #1 + iio_trigger_notify_done()

                                    ->  Read sensor #2 +

iio_trigger_notify_done()

(note that the read etc is normally in a thread - all we do in interrupt context is usually to grab a timestamp if that makes sense for a given sensor).

This driver does both of.
Data ready -> Read data from itself and call iio_trigger_notify_done() IRQ chip for a different trigger -> Take a timestamp and never call iio_trigger_notify_done()
or read any data for that matter.

Which won't do what we want at all.

Andy, if I have a go at fixing this are you able to test the result? I think the simplest is probably to introduce a trigger to tie the two halves together.
We can set it as the default trigger so things should keep on working for existing users.

Quoted 7 lines

For more general case, we should probably have two functions.

iio_trigger_notify_done() which is only called from places we can sleep. iio_trigger_notify_done_no_action() which only decrements the counter (or given this is only called inside industrialio-trigger.c could just replace with atomic_dec(&trig->use_count)).

Sanchayan, can you help to verify the fixes that Jonathan will send out ?

#15 lore raw
Quoted 43 lines · Andy Duan

From: Jonathan Cameron <[email protected]>

Quoted 42 lines
Quoted 9 lines · On Mon, 21 Sep 2020 14:27:28 +0200 Sebastian Andrzej Siewior…
Quoted 4 lines · Jonathan Cameron

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

I've been told that it only performs IRQ-thread wake-ups in hard-IRQ context. This is fine then.

drivers/iio/adc/vf610-adc.c

However, there looks to be a lot more wrong in there than just this. So normally for a device with a data ready signal like this we would hook up as follows.

Data ready #1 -> IRQ chip (trigger) -> Read sensor #1 + iio_trigger_notify_done()

                                    ->  Read sensor #2 +

iio_trigger_notify_done()

(note that the read etc is normally in a thread - all we do in interrupt context is usually to grab a timestamp if that makes sense for a given sensor).

This driver does both of.
Data ready -> Read data from itself and call iio_trigger_notify_done() IRQ chip for a different trigger -> Take a timestamp and never call iio_trigger_notify_done()
or read any data for that matter.

Which won't do what we want at all.

Andy, if I have a go at fixing this are you able to test the result? I think the simplest is probably to introduce a trigger to tie the two halves together.
We can set it as the default trigger so things should keep on working for existing users.

For more general case, we should probably have two functions.

Quoted 8 lines
Quoted 5 lines

iio_trigger_notify_done() which is only called from places we can sleep. iio_trigger_notify_done_no_action() which only decrements the counter (or given this is only called inside industrialio-trigger.c could just replace with atomic_dec(&trig->use_count)).

Sanchayan, can you help to verify the fixes that Jonathan will send out ?

Sorry for the delay in reply. Unfortunately can't as I do not access to the hardware having left Toradex.

CCed Stefan Agner who might be able to help.

@Stefan

Hello Stefan :), may be you can help here?

#16 lore raw
Quoted 53 lines · Sanchayan Maity
Quoted 43 lines · Andy Duan

From: Jonathan Cameron <[email protected]>

Quoted 42 lines
Quoted 9 lines · On Mon, 21 Sep 2020 14:27:28 +0200 Sebastian Andrzej Siewior…
Quoted 4 lines · Jonathan Cameron

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

I've been told that it only performs IRQ-thread wake-ups in hard-IRQ context. This is fine then.

drivers/iio/adc/vf610-adc.c

However, there looks to be a lot more wrong in there than just this. So normally for a device with a data ready signal like this we would hook up as follows.

Data ready #1 -> IRQ chip (trigger) -> Read sensor #1 + iio_trigger_notify_done()

                                    ->  Read sensor #2 +

iio_trigger_notify_done()

(note that the read etc is normally in a thread - all we do in interrupt context is usually to grab a timestamp if that makes sense for a given sensor).

This driver does both of.
Data ready -> Read data from itself and call iio_trigger_notify_done() IRQ chip for a different trigger -> Take a timestamp and never call iio_trigger_notify_done()
or read any data for that matter.

Which won't do what we want at all.

Andy, if I have a go at fixing this are you able to test the result? I think the simplest is probably to introduce a trigger to tie the two halves together.
We can set it as the default trigger so things should keep on working for existing users.

For more general case, we should probably have two functions.

Quoted 8 lines
Quoted 5 lines

iio_trigger_notify_done() which is only called from places we can sleep. iio_trigger_notify_done_no_action() which only decrements the counter (or given this is only called inside industrialio-trigger.c could just replace with atomic_dec(&trig->use_count)).

Sanchayan, can you help to verify the fixes that Jonathan will send out ?

Quoted 8 lines

Sorry for the delay in reply. Unfortunately can't as I do not access to the hardware having left Toradex.

CCed Stefan Agner who might be able to help.

@Stefan

Hello Stefan :), may be you can help here?

Hi all,

I do have access to the hardware and can run a test if required. I guess I would just need to test if ADC sampling still work with something like tools/iio/iio_generic_buffer.c?

--
Stefan

maintainer
#17 lore raw
Quoted 61 lines · On Thu, 24 Sep 2020 10:54:39 +0200 Stefan Agner <[email protected]> wrote:
Quoted 60 lines · Sanchayan Maity
Quoted 43 lines · Andy Duan

From: Jonathan Cameron <[email protected]>

Quoted 42 lines
Quoted 9 lines · On Mon, 21 Sep 2020 14:27:28 +0200 Sebastian Andrzej Siewior…
Quoted 4 lines · Jonathan Cameron

So looking at this the other way, are there any significant risks associated with this change? If not I'm tempted to queue them up and we have the rcX time to fix anything we've missed (just like every other patch!)

I've been told that it only performs IRQ-thread wake-ups in hard-IRQ context. This is fine then.

drivers/iio/adc/vf610-adc.c

However, there looks to be a lot more wrong in there than just this. So normally for a device with a data ready signal like this we would hook up as follows.

Data ready #1 -> IRQ chip (trigger) -> Read sensor #1 + iio_trigger_notify_done()

                                    ->  Read sensor #2 +  

iio_trigger_notify_done()

(note that the read etc is normally in a thread - all we do in interrupt context is usually to grab a timestamp if that makes sense for a given sensor).

This driver does both of.
Data ready -> Read data from itself and call iio_trigger_notify_done() IRQ chip for a different trigger -> Take a timestamp and never call iio_trigger_notify_done()
or read any data for that matter.

Which won't do what we want at all.

Andy, if I have a go at fixing this are you able to test the result? I think the simplest is probably to introduce a trigger to tie the two halves together.
We can set it as the default trigger so things should keep on working for existing users.

For more general case, we should probably have two functions.

Quoted 8 lines
Quoted 5 lines

iio_trigger_notify_done() which is only called from places we can sleep. iio_trigger_notify_done_no_action() which only decrements the counter (or given this is only called inside industrialio-trigger.c could just replace with atomic_dec(&trig->use_count)).

Sanchayan, can you help to verify the fixes that Jonathan will send out ?

Sorry for the delay in reply. Unfortunately can't as I do not access to the hardware having left Toradex.

CCed Stefan Agner who might be able to help.

@Stefan

Quoted 7 lines
Quoted 1 line

Hello Stefan :), may be you can help here?

Hi all,

I do have access to the hardware and can run a test if required. I guess I would just need to test if ADC sampling still work with something like tools/iio/iio_generic_buffer.c?

Yup. I haven't written the fix yet though. Will make sure to cc you!

Thanks,

Jonathan

Quoted 3 lines

--
Stefan

#18 lore raw
In reply to Jonathan Cameron

Hi Jonathan,

Quoted 14 lines · Jonathan Cameron

On Mon, 21 Sep 2020 09:17:26 +0200

Quoted 11 lines · Christian Eggers

Tested together with

- iio: Fix: Do not poll the driver again if try_reenable() callback returns non 0. and
- iio:trigger: rename try_reenable() to reenable() plus return void

on latest mainline (without PREEMPT_RT). The original WARN_ONCE() in kernel/irq/handle.c:159 was not raised anymore. But even without the current patches, this warning is not shown (as this problem only applies to -RT).

Quoted 7 lines
Quoted 5 lines

Currently I haven't ported a RT kernel > 5.4 for my board, so I cannot check with current RT. On 5.4. there the patches seem not to work fully as kernel/timer/hrtimer.c is not up to date enough.

Sorry for being not very helpful...

Thanks for at least trying!

I've just ported my BSP to v5.9-rc7-rt10. It looks like your patch misses one additional change in iio_trig_hrtimer_set_state():

- hrtimer_start(&trig_info->timer, trig_info->period,
- HRTIMER_MODE_REL);
+ hrtimer_start(&trig_info->timer, trig_info->period,
+ HRTIMER_MODE_REL_HARD);

Without this, WARN_ON_ONCE() in kernel/time/hrtimer.c:1133 will be hit:

WARN_ON_ONCE(!(mode & HRTIMER_MODE_HARD) ^ !timer->is_hard);

So the mode HRTIMER_MODE_REL_HARD is required for hrtimer_init() (will be stored in timer->is_hard) and for hrtimer_start().

Best regards
Christian

maintainer
#19 lore raw
Quoted 1 line · On Fri, 2 Oct 2020 16:10:09 +0200 Christian Eggers <[email protected]> wrote:

Hi Jonathan,

Lars-Peter's patch! But I'm sure he'll do an updated version to deal with the point you make below.

@Lars no great hurry as this isn't going to make the merge window anyway so we have a couple of weeks. Will have to hit after rc1 and get backported as relevant.

Thanks,

Jonathan

Quoted 54 lines · Jonathan Cameron
Quoted 14 lines

On Mon, 21 Sep 2020 09:17:26 +0200

Quoted 11 lines · Christian Eggers

Tested together with

- iio: Fix: Do not poll the driver again if try_reenable() callback returns non 0. and
- iio:trigger: rename try_reenable() to reenable() plus return void

on latest mainline (without PREEMPT_RT). The original WARN_ONCE() in kernel/irq/handle.c:159 was not raised anymore. But even without the current patches, this warning is not shown (as this problem only applies to -RT).

Quoted 7 lines
Quoted 5 lines

Currently I haven't ported a RT kernel > 5.4 for my board, so I cannot check with current RT. On 5.4. there the patches seem not to work fully as kernel/timer/hrtimer.c is not up to date enough.

Sorry for being not very helpful...

Thanks for at least trying!

I've just ported my BSP to v5.9-rc7-rt10. It looks like your patch misses one additional change in iio_trig_hrtimer_set_state():

(quoted diff) +2 -2
-               hrtimer_start(&trig_info->timer, trig_info->period,
-                             HRTIMER_MODE_REL);
+               hrtimer_start(&trig_info->timer, trig_info->period,
+                             HRTIMER_MODE_REL_HARD);

Without this, WARN_ON_ONCE() in kernel/time/hrtimer.c:1133 will be hit:

WARN_ON_ONCE(!(mode & HRTIMER_MODE_HARD) ^ !timer->is_hard);

So the mode HRTIMER_MODE_REL_HARD is required for hrtimer_init() (will be stored in timer->is_hard) and for hrtimer_start().

Best regards
Christian

________________________________
[http://assets.arri.com/media/sign/2020-04-03-E-mail-signature-Stellar2_V1.jpg] <https://microsites.arri.com/stellar/>

Get all the latest information from www.arri.com<https://www.arri.com/>, Facebook<https://www.facebook.com/TeamARRI>, Twitter<https://twitter.com/ARRIChannel>, Instagram<https://instagram.com/arri> and YouTube<https://www.youtube.com/user/ARRIChannel>.

Arnold & Richter Cine Technik GmbH & Co. Betriebs KG
Sitz: München - Registergericht: Amtsgericht München - Handelsregisternummer: HRA 57918 Persönlich haftender Gesellschafter: Arnold & Richter Cine Technik GmbH
Sitz: München - Registergericht: Amtsgericht München - Handelsregisternummer: HRB 54477 Geschäftsführer: Dr. Michael Neuhäuser; Stephan Schenk; Walter Trauninger; Markus Zeiler