dd: Invoke one probe retry cycle after every initcall level

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Signed-off-by Vikram Mulukutla <[email protected]>

Signed-off-by Rishabh Bhatnagar <[email protected]>

---
 drivers/base/dd.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

drivers/base/dd.c +27 -7

index 1435d72..e6a6821 100644
@@ -224,23 +224,44 @@ void device_unblock_probing(void)
        driver_deferred_probe_trigger();
 }
 
+static void enable_trigger_defer_cycle(void)
+{
+       driver_deferred_probe_enable = true;
+       driver_deferred_probe_trigger();
+       /*
+        * Sort as many dependencies as possible before the next initcall
+        * level
+        */
+       flush_work(&deferred_probe_work);
+}
+
 /**
  * deferred_probe_initcall() - Enable probing of deferred devices
  *
  * We don't want to get in the way when the bulk of drivers are getting probed.
  * Instead, this initcall makes sure that deferred probing is delayed until
- * late_initcall time.
+ * all the registered initcall functions at a particular level are completed.
+ * This function is invoked at every *_initcall_sync level.
  */
 static int deferred_probe_initcall(void)
 {
-       driver_deferred_probe_enable = true;
-       driver_deferred_probe_trigger();
-       /* Sort as many dependencies as possible before exiting initcalls */
-       flush_work(&deferred_probe_work);
+       enable_trigger_defer_cycle();
+       driver_deferred_probe_enable = false;
+       return 0;
+}
+arch_initcall_sync(deferred_probe_initcall);
+subsys_initcall_sync(deferred_probe_initcall);
+fs_initcall_sync(deferred_probe_initcall);
+device_initcall_sync(deferred_probe_initcall);
+
+static int deferred_probe_enable_fn(void)
+{
+       /* Enable deferred probing for all time */
+       enable_trigger_defer_cycle();
        initcalls_done = true;
        return 0;
 }
-late_initcall(deferred_probe_initcall);
+late_initcall(deferred_probe_enable_fn);
 
 /**
  * device_is_bound() - Check if device is bound to a driver
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Quoted 7 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Can you please say something about the actual use case this is expected to address?

Quoted 70 lines

Signed-off-by Vikram Mulukutla <[email protected]>

Signed-off-by Rishabh Bhatnagar <[email protected]>

---
 drivers/base/dd.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

drivers/base/dd.c +6 -0

index 1435d72..e6a6821 100644
@@ -224,23 +224,44 @@ void device_unblock_probing(void)
        driver_deferred_probe_trigger();
 }
 
+static void enable_trigger_defer_cycle(void)
+{
+       driver_deferred_probe_enable = true;
+       driver_deferred_probe_trigger();
+       /*
+        * Sort as many dependencies as possible before the next 

initcall

drivers/base/dd.c +5 -0

+        * level
+        */
+       flush_work(&deferred_probe_work);
+}
+
 /**
  * deferred_probe_initcall() - Enable probing of deferred devices
  *
  * We don't want to get in the way when the bulk of drivers are 

getting probed.
* Instead, this initcall makes sure that deferred probing is delayed
until
- * late_initcall time.
+ * all the registered initcall functions at a particular level are completed.
+ * This function is invoked at every *_initcall_sync level. */
static int deferred_probe_initcall(void)
{

drivers/base/dd.c +0 -3

-       driver_deferred_probe_enable = true;
-       driver_deferred_probe_trigger();
-       /* Sort as many dependencies as possible before exiting 

initcalls */

drivers/base/dd.c +14 -3

-       flush_work(&deferred_probe_work);
+       enable_trigger_defer_cycle();
+       driver_deferred_probe_enable = false;
+       return 0;
+}
+arch_initcall_sync(deferred_probe_initcall);
+subsys_initcall_sync(deferred_probe_initcall);
+fs_initcall_sync(deferred_probe_initcall);
+device_initcall_sync(deferred_probe_initcall);
+
+static int deferred_probe_enable_fn(void)
+{
+       /* Enable deferred probing for all time */
+       enable_trigger_defer_cycle();
        initcalls_done = true;
        return 0;
 }
-late_initcall(deferred_probe_initcall);
+late_initcall(deferred_probe_enable_fn);
 
 /**
  * device_is_bound() - Check if device is bound to a driver
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Quoted 13 lines · Rafael J. Wysocki

Quoted 8 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of dependencies not being met the deferral mechanism defers the probes for a later time, even though the dependencies might be met earlier. With this change display can be brought up much earlier.

Quoted 14 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 13 lines · Rafael J. Wysocki

Quoted 8 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 8 lines

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of dependencies not being met the deferral mechanism defers the probes for a later time, even though the dependencies might be met earlier. With this change display can be brought up much earlier.

OK

What runlevel brings up the display after the change?

Thanks,
Rafael

Quoted 31 lines · Rafael J. Wysocki

Quoted 22 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 16 lines · Rafael J. Wysocki

Quoted 9 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

Quoted 5 lines

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of dependencies not being met the deferral mechanism defers the probes for a later time, even though the dependencies might be met earlier. With this change display can be brought up much earlier.

Quoted 6 lines

OK

What runlevel brings up the display after the change?

Thanks,
Rafael

After the change the display can come up after device_initcall level itself.
The above mentioned 3 devices are probed at 0.503253, 0.505210 and 0.523264 secs.
Only the first device is probed successfully. With the current deferral mechanism the devices get probed again after late_initcall at 9.19 and 9.35 secs. So display can only come up after 9.35 secs. With this change the devices are re-probed successfully at 0.60 and 0.613 secs. Therefore display can come just after 0.613 secs.

This change helps in overall android bootup as well.

From: RAFAEL J. WYSOCKI <[email protected]>
Date: Wed, Aug 1, 2018 at 2:21 PM
Subject: Re: [PATCH] dd: Invoke one probe retry cycle after every initcall level
To: Rishabh Bhatnagar <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>, Greg Kroah-Hartman <[email protected]>, Linux Kernel Mailing List <[email protected]>, [email protected], [email protected], Vikram Mulukutla <[email protected]>

Quoted 21 lines · On Wed, Aug 1, 2018 at 11:18 PM, <[email protected]> wrote:

Quoted 20 lines · Rafael J. Wysocki

Quoted 18 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 16 lines · Rafael J. Wysocki

Quoted 9 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of

dependencies

Quoted 1 line

Quoted 1 line

Quoted 1 line

not being met the deferral mechanism defers the probes for a later

time,

Quoted 1 line

Quoted 1 line

Quoted 1 line

even though the dependencies might be met earlier. With this

change

Quoted 5 lines

Quoted 5 lines

Quoted 1 line

display can be brought up much earlier.

OK

Quoted 7 lines

Quoted 4 lines

What runlevel brings up the display after the change?

Thanks,
Rafael

After the change the display can come up after device_initcall level itself.
The above mentioned 3 devices are probed at 0.503253, 0.505210 and

0.523264

Quoted 6 lines

secs.
Only the first device is probed successfully. With the current deferral mechanism the devices get probed again after late_initcall at 9.19 and 9.35 secs. So display can only come up after 9.35 secs. With this change the devices are re-probed successfully at 0.60 and 0.613 secs. Therefore display can come just after 0.613 secs.

OK, so why do you touch the initcall levels earlier than device_?

Quoted 2 lines

This change helps in overall android bootup as well.

How exactly?

Quoted 82 lines

From: RAFAEL J. WYSOCKI <[email protected]>
Date: Wed, Aug 1, 2018 at 2:21 PM
Subject: Re: [PATCH] dd: Invoke one probe retry cycle after every initcall level
To: Rishabh Bhatnagar <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>, Greg Kroah-Hartman <[email protected]>, Linux Kernel Mailing List <[email protected]>, [email protected], [email protected], Vikram Mulukutla <[email protected]>

Quoted 27 lines · On Wed, Aug 1, 2018 at 11:18 PM, <[email protected]> wrote:

Quoted 25 lines · Rafael J. Wysocki

Quoted 22 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 19 lines · Rafael J. Wysocki

Quoted 10 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of

dependencies

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

not being met the deferral mechanism defers the probes for a later

time,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

even though the dependencies might be met earlier. With this

change

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

display can be brought up much earlier.

OK

Quoted 8 lines

Quoted 4 lines

What runlevel brings up the display after the change?

Thanks,
Rafael

After the change the display can come up after device_initcall level itself.
The above mentioned 3 devices are probed at 0.503253, 0.505210 and

0.523264

Quoted 7 lines

secs.
Only the first device is probed successfully. With the current deferral mechanism the devices get probed again after late_initcall at 9.19 and 9.35 secs. So display can only come up after 9.35 secs. With this change the devices are re-probed successfully at 0.60 and 0.613 secs. Therefore display can come just after 0.613 secs.

Quoted 1 line

OK, so why do you touch the initcall levels earlier than device_?

1) re-probe probing devices in the active list on every level help to
avoid circular dependency pending list.
2) There are so many devices which gets deferred in earlier init call
levels, so we wanted to reprobe them at every successive init call level.

Quoted 5 lines

Quoted 2 lines

This change helps in overall android bootup as well.

How exactly?

We have seen less no of re-probes at late_init and most of the driver's dependency met earlier than late_init call level. It helped display and couple of other drivers by executing the re probe work at every init level.

Quoted 82 lines · Sodagudi Prasad

Quoted 82 lines

From: RAFAEL J. WYSOCKI <[email protected]>
Date: Wed, Aug 1, 2018 at 2:21 PM
Subject: Re: [PATCH] dd: Invoke one probe retry cycle after every initcall level
To: Rishabh Bhatnagar <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>, Greg Kroah-Hartman <[email protected]>, Linux Kernel Mailing List <[email protected]>, [email protected], [email protected], Vikram Mulukutla <[email protected]>

Quoted 27 lines · On Wed, Aug 1, 2018 at 11:18 PM, <[email protected]> wrote:

Quoted 25 lines · Rafael J. Wysocki

Quoted 22 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 19 lines · Rafael J. Wysocki

Quoted 10 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of

dependencies

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

not being met the deferral mechanism defers the probes for a later

time,

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

even though the dependencies might be met earlier. With this

change

Quoted 7 lines

Quoted 7 lines

Quoted 2 lines

display can be brought up much earlier.

OK

Quoted 8 lines

Quoted 4 lines

What runlevel brings up the display after the change?

Thanks,
Rafael

After the change the display can come up after device_initcall level itself.
The above mentioned 3 devices are probed at 0.503253, 0.505210 and

0.523264

Quoted 7 lines

secs.
Only the first device is probed successfully. With the current deferral mechanism the devices get probed again after late_initcall at 9.19 and 9.35 secs. So display can only come up after 9.35 secs. With this change the devices are re-probed successfully at 0.60 and 0.613 secs. Therefore display can come just after 0.613 secs.

Quoted 6 lines

Quoted 1 line

OK, so why do you touch the initcall levels earlier than device_?

1) re-probe probing devices in the active list on every level help to
avoid circular dependency pending list.
2) There are so many devices which gets deferred in earlier init call
levels, so we wanted to reprobe them at every successive init call level.

Do you have specific examples of devices for which that helps?

Quoted 6 lines

Quoted 5 lines

Quoted 2 lines

This change helps in overall android bootup as well.

How exactly?

Quoted 3 lines

We have seen less no of re-probes at late_init and most of the driver's dependency met earlier than late_init call level. It helped display and couple of other drivers by executing the re probe work at every init level.

So I can believe that walking the deferred list on device_initcall and maybe on device_initcall_sync may help, but I'm not quite convinced that it matters for the earlier initcall levels.

Quoted 131 lines · Rafael J. Wysocki

Quoted 104 lines · Sodagudi Prasad

Quoted 101 lines

From: RAFAEL J. WYSOCKI <[email protected]>
Date: Wed, Aug 1, 2018 at 2:21 PM
Subject: Re: [PATCH] dd: Invoke one probe retry cycle after every initcall level
To: Rishabh Bhatnagar <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>, Greg Kroah-Hartman <[email protected]>, Linux Kernel Mailing List <[email protected]>, [email protected], [email protected], Vikram Mulukutla <[email protected]>

Quoted 33 lines · On Wed, Aug 1, 2018 at 11:18 PM, <[email protected]> wrote:

Quoted 30 lines · Rafael J. Wysocki

Quoted 26 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 22 lines · Rafael J. Wysocki

Quoted 11 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of

dependencies

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

not being met the deferral mechanism defers the probes for a later

time,

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

even though the dependencies might be met earlier. With this

change

Quoted 11 lines

Quoted 11 lines

Quoted 3 lines

display can be brought up much earlier.

OK

What runlevel brings up the display after the change?

Quoted 7 lines

Quoted 2 lines

Thanks,
Rafael

After the change the display can come up after device_initcall level itself.
The above mentioned 3 devices are probed at 0.503253, 0.505210 and

0.523264

Quoted 8 lines

secs.
Only the first device is probed successfully. With the current deferral mechanism the devices get probed again after late_initcall at 9.19 and 9.35 secs. So display can only come up after 9.35 secs. With this change the devices are re-probed successfully at 0.60 and 0.613 secs. Therefore display can come just after 0.613 secs.

Quoted 1 line

OK, so why do you touch the initcall levels earlier than device_?

Quoted 6 lines

1) re-probe probing devices in the active list on every level help
to
avoid circular dependency pending list.
2) There are so many devices which gets deferred in earlier init
call
levels, so we wanted to reprobe them at every successive init call level.

Do you have specific examples of devices for which that helps?

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

This change helps in overall android bootup as well.

Quoted 7 lines

Quoted 1 line

How exactly?

We have seen less no of re-probes at late_init and most of the driver's dependency met earlier than late_init call level. It helped display and couple of other drivers by executing the re probe work at every init level.

Quoted 3 lines

So I can believe that walking the deferred list on device_initcall and maybe on device_initcall_sync may help, but I'm not quite convinced that it matters for the earlier initcall levels.

Many of our drivers are dependent on the regulator and bus driver. Both the regulator and bus driver are probed in the subsys_initcall level. Now the probe of bus driver requires regulator to be working. If the probe of
bus driver happens before regulator, then bus driver's probe will be deferred and all other device's probes which depend on bus driver will also be deferred.
The impact of this problem is reduced if we have this patch.

Quoted 137 lines · On Fri, Aug 10, 2018 at 12:30 AM, <[email protected]> wrote:

Quoted 131 lines · Rafael J. Wysocki

Quoted 104 lines · Sodagudi Prasad

Quoted 101 lines

From: RAFAEL J. WYSOCKI <[email protected]>
Date: Wed, Aug 1, 2018 at 2:21 PM
Subject: Re: [PATCH] dd: Invoke one probe retry cycle after every initcall level
To: Rishabh Bhatnagar <[email protected]>
Cc: "Rafael J. Wysocki" <[email protected]>, Greg Kroah-Hartman <[email protected]>, Linux Kernel Mailing List <[email protected]>, [email protected], [email protected], Vikram Mulukutla <[email protected]>

Quoted 33 lines · On Wed, Aug 1, 2018 at 11:18 PM, <[email protected]> wrote:

Quoted 30 lines · Rafael J. Wysocki

Quoted 26 lines · On Mon, Jul 23, 2018 at 10:22 PM, <[email protected]> wrote:

Quoted 22 lines · Rafael J. Wysocki

Quoted 11 lines · Rishabh Bhatnagar

Drivers that are registered at an initcall level may have to wait until late_init before the probe deferral mechanism can retry their probe functions. It is possible that their dependencies were resolved much earlier, in some cases even before the next initcall level. Invoke one probe retry cycle at every _sync initcall level, allowing these drivers to be probed earlier.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

Can you please say something about the actual use case this is expected to address?

We have a display driver that depends 3 other devices to be probed so that it can bring-up the display. Because of

dependencies

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

not being met the deferral mechanism defers the probes for a later

time,

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

even though the dependencies might be met earlier. With this

change

Quoted 11 lines

Quoted 11 lines

Quoted 3 lines

display can be brought up much earlier.

OK

What runlevel brings up the display after the change?

Quoted 7 lines

Quoted 2 lines

Thanks,
Rafael

After the change the display can come up after device_initcall level itself.
The above mentioned 3 devices are probed at 0.503253, 0.505210 and

0.523264

Quoted 8 lines

secs.
Only the first device is probed successfully. With the current deferral mechanism the devices get probed again after late_initcall at 9.19 and 9.35 secs. So display can only come up after 9.35 secs. With this change the devices are re-probed successfully at 0.60 and 0.613 secs. Therefore display can come just after 0.613 secs.

Quoted 1 line

OK, so why do you touch the initcall levels earlier than device_?

Quoted 6 lines

1) re-probe probing devices in the active list on every level help
to
avoid circular dependency pending list.
2) There are so many devices which gets deferred in earlier init
call
levels, so we wanted to reprobe them at every successive init call level.

Do you have specific examples of devices for which that helps?

Quoted 5 lines

Quoted 5 lines

Quoted 2 lines

This change helps in overall android bootup as well.

Quoted 7 lines

Quoted 1 line

How exactly?

We have seen less no of re-probes at late_init and most of the driver's dependency met earlier than late_init call level. It helped display and couple of other drivers by executing the re probe work at every init level.

Quoted 3 lines

So I can believe that walking the deferred list on device_initcall and maybe on device_initcall_sync may help, but I'm not quite convinced that it matters for the earlier initcall levels.

Quoted 8 lines

Many of our drivers are dependent on the regulator and bus driver. Both the regulator and bus driver are probed in the subsys_initcall level. Now the probe of bus driver requires regulator to be working. If the probe of
bus driver happens before regulator, then bus driver's probe will be deferred and all other device's probes which depend on bus driver will also be deferred.
The impact of this problem is reduced if we have this patch.

Fair enough, but this information should be there in the changelog of your patch.

And why do you do that for arch_initcall_sync()?