Drivers: hv: vmbus: Deferring per cpu tasks
Subsystems (2)
PATCH 1/1 Drivers: hv: vmbus: Deferring per cpu tasks
Currently on a very large system with 1780 CPUs, hv_acpi_init takes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.
Defer these tasks so that each CPU executes hv_acpi_init in parallel to take full advantage of multiple CPUs.
This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)
drivers/hv/vmbus_drv.c +30 -3
@@ -1306,6 +1306,13 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)return IRQ_HANDLED ; } +static void vmbus_percpu_work (struct work_struct *work) +{ +unsigned int cpu =smp_processor_id (); + +hv_synic_init (cpu); +} +/* * vmbus_bus_init -Main vmbus driver initialization routine. *
@@ -1316,7 +1323,8 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)*/ static int vmbus_bus_init (void ) { -int ret; +int ret, cpu; +struct work_struct __percpu *works; ret =hv_init ();if (ret !=0 ) {
@@ -1355,12 +1363,31 @@ static int vmbus_bus_init(void)if (ret) goto err_alloc; + works = alloc_percpu(struct work_struct); +if (!works) { + ret = -ENOMEM ; + goto err_alloc; + } +/* * Initialize the per-cpu interrupt state and stimer state. * Then connect to the host. */ - ret =cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , - hv_synic_init, hv_synic_cleanup); +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu); + +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu) +flush_work (per_cpu_ptr (works, cpu)); + + ret =__cpuhp_setup_state_cpuslocked (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , false, + hv_synic_init, hv_synic_cleanup, false); +cpus_read_unlock (); +free_percpu (works);if (ret <0 ) goto err_alloc; hyperv_cpuhp_online = ret;
Signature · 1 line
2.43.0
Quoted 6 lines · Saurabh Sengar
Currently on a very large system with 1780 CPUs,
hv_acpi_inittakes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.Defer these tasks so that each CPU executes
hv_acpi_initin parallel to take full advantage of multiple CPUs.
I think you mean hv_synic_init() here, not hv_acpi_init()?
Quoted 9 lines
This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
Signed-off-by Saurabh Sengar <[email protected]>--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)
LGTM otherwise.
Quoted 9 lines · Nuno Das Neves
Quoted 6 lines · Saurabh Sengar
Currently on a very large system with 1780 CPUs,
hv_acpi_inittakes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.Defer these tasks so that each CPU executes
hv_acpi_initin parallel to take full advantage of multiple CPUs.I think you mean
hv_synic_init()here, nothv_acpi_init()?
Thanks, will send v2 correcting this.
- Saurabh
Quoted 14 lines
Quoted 10 lines
This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
Signed-off-by Saurabh Sengar <[email protected]>--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)LGTM otherwise.
Reviewed-by Nuno Das Neves <[email protected]>
Dexuan Cui
maintainerQuoted 3 lines
From: Saurabh Singh Sengar <[email protected]>
Sent: Thursday, July 25, 2024 8:35 AM
Subject: Re: [PATCH] Drivers: hv: vmbus: Deferring per cpu tasks
Without the patch, I think the current CPU uses IPIs to let the other CPUs, one by one, run the function calls, and synchronously waits for the function calls to finish.
IMO the patch is not "Deferring per cpu tasks". "Defer" means "let it happen later". Here it schedules work items to different CPUs, and the work items immediately start to run on these CPUs.
I would suggest a more accurate subject:
Drivers: hv: vmbus: Run hv_synic_init() concurrently
Quoted 4 lines
(quoted diff) +0
-1- ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,"hyperv/vmbus:online",
drivers/hv/vmbus_drv.c +1
-1- hv_synic_init, hv_synic_cleanup); +cpus_read_lock ();
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1386
Quoted 10 lines
drivers/hv/vmbus_drv.c +10
-0+for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu); + +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu) +flush_work (per_cpu_ptr (works, cpu)); +
Can you please add a comment to explain we need this for CPU online/offline'ing:
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1389
Quoted 5 lines
(quoted diff) +1
-0+ ret = __cpuhp_setup_state_cpuslocked(CPUHP_AP_ONLINE_DYN,"hyperv/vmbus:online", false,
(quoted diff) +1
-0+ hv_synic_init, hv_synic_cleanup,false);
drivers/hv/vmbus_drv.c +1
-0+cpus_read_unlock ();
Add an empty line here to make it slightly more readable? :-)
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1392
Quoted 3 lines
drivers/hv/vmbus_drv.c +1
-0+free_percpu (works);if (ret <0 ) goto err_alloc;
Thanks,
Dexuan
Quoted 8 lines · Dexuan Cui
Quoted 3 lines
From: Saurabh Singh Sengar <[email protected]>
Sent: Thursday, July 25, 2024 8:35 AM
Subject: Re: [PATCH] Drivers: hv: vmbus: Deferring per cpu tasksWithout the patch, I think the current CPU uses IPIs to let the other CPUs, one by one, run the function calls, and synchronously waits for the function calls to finish.
Quoted 6 lines
IMO the patch is not "Deferring per cpu tasks". "Defer" means "let it happen later". Here it schedules work items to different CPUs, and the work items immediately start to run on these CPUs.
I would suggest a more accurate subject:
Drivers: hv: vmbus: Runhv_synic_init()concurrently
Agree, this explains the change better.
Quoted 17 lines
Quoted 5 lines
(quoted diff) +0
-1- ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,"hyperv/vmbus:online",
drivers/hv/vmbus_drv.c +1
-1- hv_synic_init, hv_synic_cleanup); +cpus_read_lock ();drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1386
Quoted 10 lines
drivers/hv/vmbus_drv.c +10
-0+for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu); + +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu) +flush_work (per_cpu_ptr (works, cpu)); +Can you please add a comment to explain we need this for CPU online/offline'ing:
ok
Quoted 7 lines
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1389
Quoted 5 lines
(quoted diff) +1
-0+ ret = __cpuhp_setup_state_cpuslocked(CPUHP_AP_ONLINE_DYN,"hyperv/vmbus:online", false,
(quoted diff) +1
-0+ hv_synic_init, hv_synic_cleanup,false);
drivers/hv/vmbus_drv.c +1
-0+cpus_read_unlock ();Add an empty line here to make it slightly more readable? :-)
My personal preference was to have empty line as well here, but then I looked the other places in this file where we used cpus_read_unlock, hence I maintained that style consistent.
Please let me know if you have strong opinion about this empty line, I can add in V2.
- Saurabh
Quoted 8 lines
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1392
Quoted 3 lines
drivers/hv/vmbus_drv.c +1
-0+free_percpu (works);if (ret <0 ) goto err_alloc;Thanks,
Dexuan
Dexuan Cui
maintainerQuoted 16 lines
From: Saurabh Singh Sengar <[email protected]>
Sent: Thursday, July 25, 2024 10:27 PM
[...]Quoted 7 lines
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1389
Quoted 5 lines
(quoted diff) +1
-0+ ret = __cpuhp_setup_state_cpuslocked(CPUHP_AP_ONLINE_DYN,"hyperv/vmbus:online", false,
(quoted diff) +1
-0+ hv_synic_init, hv_synic_cleanup,false);
drivers/hv/vmbus_drv.c +1
-0+cpus_read_unlock ();Add an empty line here to make it slightly more readable? :-)
My personal preference was to have empty line as well here, but then I looked the
other places in this file where we usedcpus_read_unlock, hence I maintained that style consistent.
Quoted 4 lines
Please let me know if you have strong opinion about this empty line, I can add in V2.
- Saurabh
I have no strong opinion here :-)
Quoted 8 lines · Dexuan Cui
Quoted 3 lines
From: Saurabh Singh Sengar <[email protected]>
Sent: Thursday, July 25, 2024 8:35 AM
Subject: Re: [PATCH] Drivers: hv: vmbus: Deferring per cpu tasksWithout the patch, I think the current CPU uses IPIs to let the other CPUs, one by one, run the function calls, and synchronously waits for the function calls to finish.
Quoted 7 lines
IMO the patch is not "Deferring per cpu tasks". "Defer" means "let it happen later". Here it schedules work items to different CPUs, and the work items immediately start to run on these CPUs.
I would suggest a more accurate subject:
Drivers: hv: vmbus: Runhv_synic_init()concurrently
It would be great to call out the "why" as well in the title, something like:
Drivers: hv: vmbus: Speed-up booting by running hv_synic_init() concurrently
Regards,
Srivatsa
Microsoft Linux Systems Group
Quoted 13 lines · Srivatsa S. Bhat
Quoted 12 lines · Dexuan Cui
Quoted 3 lines
From: Saurabh Singh Sengar <[email protected]>
Sent: Thursday, July 25, 2024 8:35 AM
Subject: Re: [PATCH] Drivers: hv: vmbus: Deferring per cpu tasksWithout the patch, I think the current CPU uses IPIs to let the other CPUs, one by one, run the function calls, and synchronously waits for the function calls to finish.
IMO the patch is not "Deferring per cpu tasks". "Defer" means "let it happen later". Here it schedules work items to different CPUs, and the work items immediately start to run on these CPUs.
Quoted 8 lines
Quoted 3 lines
I would suggest a more accurate subject:
Drivers: hv: vmbus: Runhv_synic_init()concurrentlyIt would be great to call out the "why" as well in the title, something like:
Drivers: hv: vmbus: Speed-up booting by running
hv_synic_init()concurrently
Thanks, I can also rephrase it like below if sounds fine to you:
Drivers: hv: vmbus: Optimize boot time by concurrent execution of hv_synic_init()
- Saurabh
Quoted 4 lines
Regards,
Srivatsa
Microsoft Linux Systems Group
Quoted 18 lines · Saurabh Singh Sengar
Quoted 17 lines · Srivatsa S. Bhat
Quoted 15 lines · Dexuan Cui
Quoted 3 lines
From: Saurabh Singh Sengar <[email protected]>
Sent: Thursday, July 25, 2024 8:35 AM
Subject: Re: [PATCH] Drivers: hv: vmbus: Deferring per cpu tasksWithout the patch, I think the current CPU uses IPIs to let the other CPUs, one by one, run the function calls, and synchronously waits for the function calls to finish.
IMO the patch is not "Deferring per cpu tasks". "Defer" means "let it happen later". Here it schedules work items to different CPUs, and the work items immediately start to run on these CPUs.
I would suggest a more accurate subject:
Drivers: hv: vmbus: Runhv_synic_init()concurrently
Quoted 9 lines
Quoted 4 lines
It would be great to call out the "why" as well in the title, something like:
Drivers: hv: vmbus: Speed-up booting by running
hv_synic_init()concurrentlyThanks, I can also rephrase it like below if sounds fine to you:
Drivers: hv: vmbus: Optimize boot time by concurrent execution of
hv_synic_init()
Sure, that sounds good too.
Regards,
Srivatsa
Microsoft Linux Systems Group
From: Saurabh Sengar <[email protected]> Sent: Wednesday, July 24, 2024 10:26 PM
Quoted 63 lines
Currently on a very large system with 1780 CPUs,
hv_acpi_inittakes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.Defer these tasks so that each CPU executes
hv_acpi_initin parallel to take full advantage of multiple CPUs.This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
Signed-off-by Saurabh Sengar <[email protected]>--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)drivers/hv/vmbus_drv.c +18
-3@@ -1306,6 +1306,13 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)return IRQ_HANDLED ; } +static void vmbus_percpu_work (struct work_struct *work) +{ +unsigned int cpu =smp_processor_id (); + +hv_synic_init (cpu); +} +/* * vmbus_bus_init -Main vmbus driver initialization routine. * @@ -1316,7 +1323,8 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)*/ static int vmbus_bus_init (void ) { -int ret; +int ret, cpu; +struct work_struct __percpu *works; ret =hv_init ();if (ret !=0 ) {@@ -1355,12 +1363,31 @@ static int vmbus_bus_init(void)if (ret) goto err_alloc; + works = alloc_percpu(struct work_struct); +if (!works) { + ret = -ENOMEM ; + goto err_alloc; + } +/* * Initialize the per-cpu interrupt state and stimer state. * Then connect to the host. */ - ret =cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , - hv_synic_init, hv_synic_cleanup); +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu);
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1388
Quoted 10 lines
drivers/hv/vmbus_drv.c +10
-0+ +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu) +flush_work (per_cpu_ptr (works, cpu)); + + ret =__cpuhp_setup_state_cpuslocked (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , false, + hv_synic_init, hv_synic_cleanup, false);
I'd suggest using cpuhp_setup_state_nocalls_cpuslocked(). It appears to be the interface intended for users outside the cpuhotplug code, whereas __cpuhp_setup_state_cpuslocked() should be private to the cpuhotplug code.
Michael
Quoted 8 lines
drivers/hv/vmbus_drv.c +2
-1+cpus_read_unlock (); +free_percpu (works);if (ret <0 ) goto err_alloc; hyperv_cpuhp_online = ret; --2.43.0
Quoted 70 lines · Michael Kelley
From: Saurabh Sengar <[email protected]> Sent: Wednesday, July 24, 2024 10:26 PM
Quoted 69 lines
Currently on a very large system with 1780 CPUs,
hv_acpi_inittakes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.Defer these tasks so that each CPU executes
hv_acpi_initin parallel to take full advantage of multiple CPUs.This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
Signed-off-by Saurabh Sengar <[email protected]>--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)drivers/hv/vmbus_drv.c +24
-3@@ -1306,6 +1306,13 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)return IRQ_HANDLED ; } +static void vmbus_percpu_work (struct work_struct *work) +{ +unsigned int cpu =smp_processor_id (); + +hv_synic_init (cpu); +} +/* * vmbus_bus_init -Main vmbus driver initialization routine. * @@ -1316,7 +1323,8 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)*/ static int vmbus_bus_init (void ) { -int ret; +int ret, cpu; +struct work_struct __percpu *works; ret =hv_init ();if (ret !=0 ) {@@ -1355,12 +1363,31 @@ static int vmbus_bus_init(void)if (ret) goto err_alloc; + works = alloc_percpu(struct work_struct); +if (!works) { + ret = -ENOMEM ; + goto err_alloc; + } +/* * Initialize the per-cpu interrupt state and stimer state. * Then connect to the host. */ - ret =cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , - hv_synic_init, hv_synic_cleanup); +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu); + +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu)
Quoted 9 lines
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1388
Quoted 4 lines
drivers/hv/vmbus_drv.c +4
-0+flush_work (per_cpu_ptr (works, cpu)); + + ret =__cpuhp_setup_state_cpuslocked (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , false, + hv_synic_init, hv_synic_cleanup, false);I'd suggest using
cpuhp_setup_state_nocalls_cpuslocked(). It appears to be the interface intended for users outside the cpuhotplug code, whereas__cpuhp_setup_state_cpuslocked()should be private to the cpuhotplug code.
Thanks for your review.
The function cpuhp_setup_state_nocalls_cpuslocked() is commonly used across the kernel drivers hence it was a first choice for me as well. However, it includes a cpus_read_lock that we already introduced separately in above code. To avoid recursive locking, I opted for __cpuhp_setup_state_cpuslocked.
One might argue that unlocking and then calling cpuhp_setup_state_nocalls_cpuslocked could be a solution, but I am concerned about potential race conditions, as CPUs could come online during this interval and in such case synic initialization for those CPUs would be missed.
- Saurabh
From: Saurabh Singh Sengar <[email protected]> Sent: Sunday, July 28, 2024 2:18 AM
Quoted 83 lines · Michael Kelley
Quoted 1 line
From: Saurabh Sengar <[email protected]> Sent: Wednesday, July 24,
2024 10:26 PM
Quoted 69 lines
Quoted 69 lines
Currently on a very large system with 1780 CPUs,
hv_acpi_inittakes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.Defer these tasks so that each CPU executes
hv_acpi_initin parallel to take full advantage of multiple CPUs.This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
Signed-off-by Saurabh Sengar <[email protected]>--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)drivers/hv/vmbus_drv.c +24
-3@@ -1306,6 +1306,13 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)return IRQ_HANDLED ; } +static void vmbus_percpu_work (struct work_struct *work) +{ +unsigned int cpu =smp_processor_id (); + +hv_synic_init (cpu); +} +/* * vmbus_bus_init -Main vmbus driver initialization routine. * @@ -1316,7 +1323,8 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)*/ static int vmbus_bus_init (void ) { -int ret; +int ret, cpu; +struct work_struct __percpu *works; ret =hv_init ();if (ret !=0 ) {@@ -1355,12 +1363,31 @@ static int vmbus_bus_init(void)if (ret) goto err_alloc; + works = alloc_percpu(struct work_struct); +if (!works) { + ret = -ENOMEM ; + goto err_alloc; + } +/* * Initialize the per-cpu interrupt state and stimer state. * Then connect to the host. */ - ret =cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , - hv_synic_init, hv_synic_cleanup); +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu); + +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu)Quoted 9 lines
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1388
Quoted 4 lines
drivers/hv/vmbus_drv.c +4
-0+flush_work (per_cpu_ptr (works, cpu)); + + ret =__cpuhp_setup_state_cpuslocked (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , false, + hv_synic_init, hv_synic_cleanup, false);I'd suggest using
cpuhp_setup_state_nocalls_cpuslocked(). It appears to be the interface intended for users outside the cpuhotplug code, whereas__cpuhp_setup_state_cpuslocked()should be private to the cpuhotplug code.
Quoted 6 lines
Thanks for your review.
The function
cpuhp_setup_state_nocalls_cpuslocked()is commonly used across the kernel drivers hence it was a first choice for me as well. However, it includes acpus_read_lockthat we already introduced separately in above code. To avoid recursive locking, I opted for__cpuhp_setup_state_cpuslocked.
cpuhp_setup_state_nocalls() includes the cpus_read_lock() as you describe. But cpuhp_setup_state_nocalls_cpuslocked() explicitly assumes that the cpus_read_lock() is already held, so is suitable for use in this case. There are several variants with the _cpuslocked suffix, which indicates that the caller is responsible for the cpus_read_lock().
Michael
Quoted 7 lines
One might argue that unlocking and then calling
cpuhp_setup_state_nocalls_cpuslockedcould be a solution, but I am concerned about potential race conditions, as CPUs could come online during this interval and in such case synic initialization for those CPUs would be missed.- Saurabh
Quoted 91 lines · Michael Kelley
From: Saurabh Singh Sengar <[email protected]> Sent: Sunday, July 28, 2024 2:18 AM
Quoted 83 lines · Michael Kelley
Quoted 1 line
From: Saurabh Sengar <[email protected]> Sent: Wednesday, July 24,
2024 10:26 PM
Quoted 69 lines
Quoted 69 lines
Currently on a very large system with 1780 CPUs,
hv_acpi_inittakes around 3 seconds to complete for all the CPUs. This is because of sequential synic initialization for each CPU.Defer these tasks so that each CPU executes
hv_acpi_initin parallel to take full advantage of multiple CPUs.This solution saves around 2 seconds of boot time on a 1780 CPU system, that around 66% improvement in the existing logic.
Signed-off-by Saurabh Sengar <[email protected]>--- drivers/hv/vmbus_drv.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-)drivers/hv/vmbus_drv.c +24
-3@@ -1306,6 +1306,13 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)return IRQ_HANDLED ; } +static void vmbus_percpu_work (struct work_struct *work) +{ +unsigned int cpu =smp_processor_id (); + +hv_synic_init (cpu); +} +/* * vmbus_bus_init -Main vmbus driver initialization routine. * @@ -1316,7 +1323,8 @@ static irqreturn_t vmbus_percpu_isr(int irq, void *dev_id)*/ static int vmbus_bus_init (void ) { -int ret; +int ret, cpu; +struct work_struct __percpu *works; ret =hv_init ();if (ret !=0 ) {@@ -1355,12 +1363,31 @@ static int vmbus_bus_init(void)if (ret) goto err_alloc; + works = alloc_percpu(struct work_struct); +if (!works) { + ret = -ENOMEM ; + goto err_alloc; + } +/* * Initialize the per-cpu interrupt state and stimer state. * Then connect to the host. */ - ret =cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , - hv_synic_init, hv_synic_cleanup); +cpus_read_lock (); +for_each_online_cpu (cpu) { +struct work_struct *work =per_cpu_ptr (works, cpu); + +INIT_WORK (work, vmbus_percpu_work); +schedule_work_on (cpu, work); + } + +for_each_online_cpu (cpu)Quoted 9 lines
drivers/hv/vmbus_drv.c static int vmbus_bus_init(void) · line 1388
Quoted 4 lines
drivers/hv/vmbus_drv.c +4
-0+flush_work (per_cpu_ptr (works, cpu)); + + ret =__cpuhp_setup_state_cpuslocked (CPUHP_AP_ONLINE_DYN ,"hyperv/vmbus:online" , false, + hv_synic_init, hv_synic_cleanup, false);I'd suggest using
cpuhp_setup_state_nocalls_cpuslocked(). It appears to be the interface intended for users outside the cpuhotplug code, whereas__cpuhp_setup_state_cpuslocked()should be private to the cpuhotplug code.Quoted 6 lines
Thanks for your review.
The function
cpuhp_setup_state_nocalls_cpuslocked()is commonly used across the kernel drivers hence it was a first choice for me as well. However, it includes acpus_read_lockthat we already introduced separately in above code. To avoid recursive locking, I opted for__cpuhp_setup_state_cpuslocked.
Quoted 6 lines
cpuhp_setup_state_nocalls()includes thecpus_read_lock()as you describe. Butcpuhp_setup_state_nocalls_cpuslocked()explicitly assumes that thecpus_read_lock()is already held, so is suitable for use in this case. There are several variants with the_cpuslockedsuffix, which indicates that the caller is responsible for thecpus_read_lock().
Thank you for the clarification. I will fix this up in v2.
- Saurabh