amba: Retry adding deferred devices at late_initcall
PATCH 1/1 amba: Retry adding deferred devices at late_initcall
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.
The list_empty() check now happens outside the mutex, but the mutex wasn't necessary in the first place.
This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the apb_pclk has a number of probe dependencies (vexpress-sysregs, vexpress-config).
--- drivers/amba/bus.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/amba/bus.c +11 -3
@@ -505,7 +505,7 @@ static DECLARE_DELAYED_WORK(deferred_retry_work, amba_deferred_retry_func);#define DEFERRED_DEVICE_TIMEOUT (msecs_to_jiffies(5 * 1000)) -static void amba_deferred_retry_func (struct work_struct *dummy) +static int amba_deferred_retry (void ) {struct deferred_device *ddev, *tmp;
@@ -521,11 +521,19 @@ static void amba_deferred_retry_func(struct work_struct *dummy)kfree (ddev); } +mutex_unlock (&deferred_devices_lock); + +return 0 ; +} +late_initcall (amba_deferred_retry); + +static void amba_deferred_retry_func (struct work_struct *dummy) +{ +amba_deferred_retry (); +if (!list_empty (&deferred_devices))schedule_delayed_work (&deferred_retry_work,DEFERRED_DEVICE_TIMEOUT ); - -mutex_unlock (&deferred_devices_lock); }/**
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 12 lines · Rob Herring
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during
late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.The
list_empty()check now happens outside the mutex, but the mutex wasn't necessary in the first place.
Quoted 8 lines
This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the
apb_pclkhas a number of probe dependencies (vexpress-sysregs, vexpress-config).Cc John Stultz <[email protected]>Cc Saravana Kannan <[email protected]>Cc Linus Walleij <[email protected]>Cc Sudeep Holla <[email protected]>
The change makes sense to me,
This also fixed the issue I reported @[1] is fixed with this patch, so:
--
Regards,
Sudeep
[1] https://lore.kernel.org/linux-arm-kernel/20200423133342.GA10628@bogus/
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 16 lines · Rob Herring
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during
late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.The
list_empty()check now happens outside the mutex, but the mutex wasn't necessary in the first place.This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the
apb_pclkhas a number of probe dependencies (vexpress-sysregs, vexpress-config).
Quoted 8 lines
Cc John Stultz <[email protected]>Cc Saravana Kannan <[email protected]>Cc Linus Walleij <[email protected]>Cc Sudeep Holla <[email protected]>Cc Nicolas Saenz Julienne <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Russell King <[email protected]>Signed-off-by Rob Herring <[email protected]>
Makes sense to me, and the same approach is found in the generic code in drivers/base/dd.c so
The timer-based re-probe was added by Marek Szyprowski in commit a41980f2a3eb33ed7a2636e83498b47e95ceb05b do deal with power domains. I guess it mimics dd.c deferred probe at this point?
There are a bit of other differences that have piled up, should we take a quick look at dd.c so there is not something else we're missing? I see some PM code for example.
Yours,
Linus Walleij
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Linus,
Quoted 29 lines · Linus Walleij
Quoted 16 lines · Rob Herring
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during
late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.The
list_empty()check now happens outside the mutex, but the mutex wasn't necessary in the first place.This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the
apb_pclkhas a number of probe dependencies (vexpress-sysregs, vexpress-config).Quoted 8 lines
Cc John Stultz <[email protected]>Cc Saravana Kannan <[email protected]>Cc Linus Walleij <[email protected]>Cc Sudeep Holla <[email protected]>Cc Nicolas Saenz Julienne <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Russell King <[email protected]>Signed-off-by Rob Herring <[email protected]>Makes sense to me, and the same approach is found in the generic code in
drivers/base/dd.csoReviewed-by Linus Walleij <[email protected]>
Quoted 8 lines
The timer-based re-probe was added by Marek Szyprowski in commit
a41980f2a3eb33ed7a2636e83498b47e95ceb05bdo deal with power domains. I guess it mimicsdd.cdeferred probe at this point?There are a bit of other differences that have piled up, should we take a quick look at
dd.cso there is not something else we're missing? I see some PM code for example.
Well, late initcall based approach is what earlier version of my patch did:
https://lkml.org/lkml/2016/4/12/414
but then it has been requested to solve the issue 'properly':
https://lkml.org/lkml/2016/4/12/455
https://lkml.org/lkml/2016/4/14/875
For me it is fine to get back to late initcall based solution, though.
Best regards
Signature · 8 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 46 lines · Marek Szyprowski
Hi Linus,
Quoted 29 lines · Linus Walleij
Quoted 16 lines · Rob Herring
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during
late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.The
list_empty()check now happens outside the mutex, but the mutex wasn't necessary in the first place.This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the
apb_pclkhas a number of probe dependencies (vexpress-sysregs, vexpress-config).Quoted 8 lines
Cc John Stultz <[email protected]>Cc Saravana Kannan <[email protected]>Cc Linus Walleij <[email protected]>Cc Sudeep Holla <[email protected]>Cc Nicolas Saenz Julienne <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Russell King <[email protected]>Signed-off-by Rob Herring <[email protected]>Makes sense to me, and the same approach is found in the generic code in
drivers/base/dd.csoReviewed-by Linus Walleij <[email protected]>Quoted 8 lines
The timer-based re-probe was added by Marek Szyprowski in commit
a41980f2a3eb33ed7a2636e83498b47e95ceb05bdo deal with power domains. I guess it mimicsdd.cdeferred probe at this point?There are a bit of other differences that have piled up, should we take a quick look at
dd.cso there is not something else we're missing? I see some PM code for example.Well, late initcall based approach is what earlier version of my patch did:
Quoted 8 lines
but then it has been requested to solve the issue 'properly':
https://lkml.org/lkml/2016/4/12/455
https://lkml.org/lkml/2016/4/14/875
For me it is fine to get back to late initcall based solution, though.
I haven't really dealt with a platform with AMBA devices and am not too familiar with it, so apologies in advance if any of my suggestions are silly.
This whole "don't add a device before the clocks/resources needed to read the PID/CID" seems like something that can be solved by having a dummy device that "probes" when those resources are available. And during its probe, it adds the real amba device. That should avoid all the reinvention of deferred probing that amba/bus.c seems to be attempting.
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the fw_devlink feature to optimize this even more when fw_devlink=on.
Just my 2 cents.
-Saravana
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 62 lines · Saravana Kannan
Quoted 46 lines · Marek Szyprowski
Hi Linus,
Quoted 29 lines · Linus Walleij
Quoted 16 lines · Rob Herring
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during
late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.The
list_empty()check now happens outside the mutex, but the mutex wasn't necessary in the first place.This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the
apb_pclkhas a number of probe dependencies (vexpress-sysregs, vexpress-config).Quoted 8 lines
Cc John Stultz <[email protected]>Cc Saravana Kannan <[email protected]>Cc Linus Walleij <[email protected]>Cc Sudeep Holla <[email protected]>Cc Nicolas Saenz Julienne <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Russell King <[email protected]>Signed-off-by Rob Herring <[email protected]>Makes sense to me, and the same approach is found in the generic code in
drivers/base/dd.csoReviewed-by Linus Walleij <[email protected]>Quoted 8 lines
The timer-based re-probe was added by Marek Szyprowski in commit
a41980f2a3eb33ed7a2636e83498b47e95ceb05bdo deal with power domains. I guess it mimicsdd.cdeferred probe at this point?There are a bit of other differences that have piled up, should we take a quick look at
dd.cso there is not something else we're missing? I see some PM code for example.Well, late initcall based approach is what earlier version of my patch did:
Quoted 8 lines
but then it has been requested to solve the issue 'properly':
https://lkml.org/lkml/2016/4/12/455
https://lkml.org/lkml/2016/4/14/875
For me it is fine to get back to late initcall based solution, though.
I haven't really dealt with a platform with AMBA devices and am not too familiar with it, so apologies in advance if any of my suggestions are silly.
Quoted 6 lines
This whole "don't add a device before the clocks/resources needed to read the PID/CID" seems like something that can be solved by having a dummy device that "probes" when those resources are available. And during its probe, it adds the real amba device. That should avoid all the reinvention of deferred probing that
amba/bus.cseems to be attempting.
Sounds like a clever idea.
In principle we should then be able to rely on the regular defered probe mechanism, just that it's the dummy device that is being defered probed (if we fail to read PID/CID).
Quoted 7 lines
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the
fw_devlinkfeature to optimize this even more when fw_devlink=on.Just my 2 cents.
Someone should try to implement this to see if it fits well.
Kind regards
Uffe
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 64 lines · Ulf Hansson
Quoted 62 lines · Saravana Kannan
Quoted 46 lines · Marek Szyprowski
Hi Linus,
Quoted 29 lines · Linus Walleij
Quoted 16 lines · Rob Herring
If amba bus devices defer when adding, the amba bus code simply retries adding the devices every 5 seconds. This doesn't work well as it completely unsynchronized with starting the init process which can happen in less than 5 secs. Add a retry during
late_initcall. If the amba devices are added, then deferred probe takes over. If the dependencies have not probed at this point, then there's no improvement over previous behavior. To completely solve this, we'd need to retry after every successful probe as deferred probe does.The
list_empty()check now happens outside the mutex, but the mutex wasn't necessary in the first place.This needed to use deferred probe instead of fragile initcall ordering on 32-bit VExpress systems where the
apb_pclkhas a number of probe dependencies (vexpress-sysregs, vexpress-config).Quoted 8 lines
Cc John Stultz <[email protected]>Cc Saravana Kannan <[email protected]>Cc Linus Walleij <[email protected]>Cc Sudeep Holla <[email protected]>Cc Nicolas Saenz Julienne <[email protected]>Cc Geert Uytterhoeven <[email protected]>Cc Russell King <[email protected]>Signed-off-by Rob Herring <[email protected]>Makes sense to me, and the same approach is found in the generic code in
drivers/base/dd.csoReviewed-by Linus Walleij <[email protected]>Quoted 8 lines
The timer-based re-probe was added by Marek Szyprowski in commit
a41980f2a3eb33ed7a2636e83498b47e95ceb05bdo deal with power domains. I guess it mimicsdd.cdeferred probe at this point?There are a bit of other differences that have piled up, should we take a quick look at
dd.cso there is not something else we're missing? I see some PM code for example.Well, late initcall based approach is what earlier version of my patch did:
Quoted 8 lines
but then it has been requested to solve the issue 'properly':
https://lkml.org/lkml/2016/4/12/455
https://lkml.org/lkml/2016/4/14/875
For me it is fine to get back to late initcall based solution, though.
I haven't really dealt with a platform with AMBA devices and am not too familiar with it, so apologies in advance if any of my suggestions are silly.
Quoted 8 lines
Quoted 6 lines
This whole "don't add a device before the clocks/resources needed to read the PID/CID" seems like something that can be solved by having a dummy device that "probes" when those resources are available. And during its probe, it adds the real amba device. That should avoid all the reinvention of deferred probing that
amba/bus.cseems to be attempting.Sounds like a clever idea.
Thanks
Quoted 5 lines
In principle we should then be able to rely on the regular defered probe mechanism, just that it's the dummy device that is being defered probed (if we fail to read PID/CID).
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the
fw_devlinkfeature to optimize this even more when fw_devlink=on.Just my 2 cents.
Someone should try to implement this to see if it fits well.
I don't mind taking a stab at this if people are actually okay with this approach and will test and merge it if it works. I have no platform to test this. I'll wait to hear what others think before I jump on this.
-Saravana
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Saravana,
Quoted 9 lines · Saravana Kannan
Quoted 8 lines · Ulf Hansson
In principle we should then be able to rely on the regular defered probe mechanism, just that it's the dummy device that is being defered probed (if we fail to read PID/CID).
Quoted 5 lines
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the
fw_devlinkfeature to optimize this even more when fw_devlink=on.
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Just my 2 cents.
Someone should try to implement this to see if it fits well.
I don't mind taking a stab at this if people are actually okay with this approach and will test and merge it if it works. I have no platform to test this. I'll wait to hear what others think before I jump on this.
The time I've prepared my patch I've also considered something like that, but I gave up because timer or notifier based approach was much simpler. If you have some time to implement your idea I would be happy to test it.
Best regards
Signature · 8 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 15 lines · Marek Szyprowski
Hi Saravana,
Quoted 11 lines · Saravana Kannan
Quoted 10 lines · Ulf Hansson
In principle we should then be able to rely on the regular defered probe mechanism, just that it's the dummy device that is being defered probed (if we fail to read PID/CID).
Quoted 6 lines
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the
fw_devlinkfeature to optimize this even more when fw_devlink=on.Just my 2 cents.
Someone should try to implement this to see if it fits well.
Quoted 8 lines
Quoted 4 lines
I don't mind taking a stab at this if people are actually okay with this approach and will test and merge it if it works. I have no platform to test this. I'll wait to hear what others think before I jump on this.
The time I've prepared my patch I've also considered something like that, but I gave up because timer or notifier based approach was much simpler.
Maybe I'll reach the same conclusion. We'll see.
Quoted 2 lines
If you have some time to implement your idea I would be happy to test it.
Thanks. I'll take a stab at it then. Btw, does this need to support the non-DT/machine file way of populating device/getting resources too? Are or all supported configurations DT based?
-Saravana
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 27 lines · Saravana Kannan
Quoted 15 lines · Marek Szyprowski
Hi Saravana,
Quoted 11 lines · Saravana Kannan
Quoted 10 lines · Ulf Hansson
In principle we should then be able to rely on the regular defered probe mechanism, just that it's the dummy device that is being defered probed (if we fail to read PID/CID).
Quoted 6 lines
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the
fw_devlinkfeature to optimize this even more when fw_devlink=on.Just my 2 cents.
Someone should try to implement this to see if it fits well.
Quoted 8 lines
Quoted 4 lines
I don't mind taking a stab at this if people are actually okay with this approach and will test and merge it if it works. I have no platform to test this. I'll wait to hear what others think before I jump on this.
The time I've prepared my patch I've also considered something like that, but I gave up because timer or notifier based approach was much simpler.
Quoted 8 lines
Maybe I'll reach the same conclusion. We'll see.
Quoted 2 lines
If you have some time to implement your idea I would be happy to test it.
Thanks. I'll take a stab at it then. Btw, does this need to support the non-DT/machine file way of populating device/getting resources too? Are or all supported configurations DT based?
Not sure. There may still be a few cases of non-DT. If everything is DT, we could probably just get rid of AMBA bus and make everything platform drivers. We have the compatible strings to match on already (we only use 'arm,primecell' currently).
I'm not really a fan of creating a dummy platform device. There's other cases of needing to power on or enable devices before probe. SDIO devices are one case. That's how we ended up with the mmc-pwrseq binding. I really think we need some bus/driver hook between device add and probe. Or maybe more precisely it is just between add and the add uevent.
Rob
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 34 lines · Rob Herring
Quoted 32 lines · Saravana Kannan
Quoted 15 lines · Marek Szyprowski
Hi Saravana,
Quoted 11 lines · Saravana Kannan
Quoted 10 lines · Ulf Hansson
In principle we should then be able to rely on the regular defered probe mechanism, just that it's the dummy device that is being defered probed (if we fail to read PID/CID).
Quoted 6 lines
Any reason to not do something like that? I'd think that should clean up a whole lot of this code. Also, if we are primarily dealing with AMBA devices created from DT, then we might even be able to massage the
fw_devlinkfeature to optimize this even more when fw_devlink=on.Just my 2 cents.
Someone should try to implement this to see if it fits well.
Quoted 8 lines
Quoted 4 lines
I don't mind taking a stab at this if people are actually okay with this approach and will test and merge it if it works. I have no platform to test this. I'll wait to hear what others think before I jump on this.
The time I've prepared my patch I've also considered something like that, but I gave up because timer or notifier based approach was much simpler.
Maybe I'll reach the same conclusion. We'll see.
Quoted 2 lines
If you have some time to implement your idea I would be happy to test it.
Quoted 8 lines
Quoted 3 lines
Thanks. I'll take a stab at it then. Btw, does this need to support the non-DT/machine file way of populating device/getting resources too? Are or all supported configurations DT based?
Not sure. There may still be a few cases of non-DT. If everything is DT, we could probably just get rid of AMBA bus and make everything platform drivers. We have the compatible strings to match on already (we only use 'arm,primecell' currently).
Ok
Quoted 1 line
I'm not really a fan of creating a dummy platform device.
I'm not saying it's awesome, but I think it would make amba code a bit cleaner and more importantly easier to maintain? It'll get all future deferred probe fixes automatically instead of having to replicate it here? Don't have a strong opinion, just thinking out aloud.
Quoted 4 lines
There's
other cases of needing to power on or enable devices before probe. SDIO devices are one case. That's how we ended up with the mmc-pwrseq binding.
But how many of them have this weird userspace dependency where you can't tell userspace about the device (too soon) but also can't probe without userspace? At least the mmc-pwrseq doesn't seem to have those issues.
Quoted 3 lines
I really think we need some bus/driver hook between device add and probe. Or maybe more precisely it is just between add and the add uevent.
That feels kinda odd to me though "Hey, add this device, but don't tell userspace". Might be better to keep the "complexity" away in the places that have odd requirements?
-Saravana
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 6 lines · Marek Szyprowski
[Me]
Quoted 3 lines
There are a bit of other differences that have piled up, should we take a quick look at
dd.cso there is not something else we're missing? I see some PM code for example.Well, late initcall based approach is what earlier version of my patch did:
I see, thanks for the context.
Quoted 1 line
For me it is fine to get back to late initcall based solution, though.
The idea here is to do both/and initcall and timer deferred probe, not either/or.
And the dd.c core also does both/and.
And then it does some more things, so I was thinking we may be missing out on some of them?
Yours,
Linus Walleij
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 8 lines · Linus Walleij
Quoted 5 lines · Marek Szyprowski
[Me]
Quoted 3 lines
There are a bit of other differences that have piled up, should we take a quick look at
dd.cso there is not something else we're missing? I see some PM code for example.
Quoted 8 lines
Quoted 1 line
Well, late initcall based approach is what earlier version of my patch did:
I see, thanks for the context.
Quoted 1 line
For me it is fine to get back to late initcall based solution, though.
The idea here is to do both/and initcall and timer deferred probe, not either/or.
Yes.
Quoted 1 line
And the
dd.ccore also does both/and.
Not really. Deferred probe delays retrying until late_initcall and then somewhat intelligently retries after successful probes rather than time based.
Quoted 2 lines
And then it does some more things, so I was thinking we may be missing out on some of them?
You mean like timeout (which is getting moved back to 0 as timeout causes problems with NFS root)? Remember this is deferred add, not deferred probe. Once the amba devices are added, the deferred probe in dd.c takes over.
Rob
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel