eal: initialize alarms early
PATCH 1/1 eal: initialize alarms early
We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.
--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
lib/librte_eal/linux/eal/eal.c +6 -6
@@ -1005,6 +1005,12 @@ rte_eal_init(int argc, char **argv)return -1 ; } +if (rte_eal_alarm_init () <0 ) { +rte_eal_init_alert ("Cannot init interrupt-handling thread" ); +/* rte_eal_alarm_init sets rte_errno on failure. */ +return -1 ; + } +/* Put mp channel init before bus scan so that we can init the vdev * bus through mp channel in the secondary process before the bus scan. */
@@ -1125,12 +1131,6 @@ rte_eal_init(int argc, char **argv)return -1 ; } -if (rte_eal_alarm_init () <0 ) { -rte_eal_init_alert ("Cannot init interrupt-handling thread" ); -/* rte_eal_alarm_init sets rte_errno on failure. */ -return -1 ; - } -if (rte_eal_timer_init () <0 ) {rte_eal_init_alert ("Cannot init HPET or TSC timers" ); rte_errno =ENOTSUP ;
Signature · 1 line
2.17.1
26/03/2019 19:43, Darek Stojaczyk:
Quoted 7 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.
Quoted 5 lines
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.
Quoted 9 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
You probably need to update the FreeBSD version too.
Quoted 24 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, March 27, 2019 7:12 PM
To: Stojaczyk, Dariusz <[email protected]>
Cc: [email protected]; Zhang, Qi Z <[email protected]>; Burakov, Anatoly <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] eal: initialize alarms early26/03/2019 19:43, Darek Stojaczyk:
Quoted 7 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.Quoted 5 lines
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.
Quoted 3 lines
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.
Hah, ok - I'll check again the possible outcomes of this.
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)You probably need to update the FreeBSD version too.
Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.
D.
27/03/2019 21:33, Stojaczyk, Dariusz:
Quoted 35 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 14 lines
26/03/2019 19:43, Darek Stojaczyk:
Quoted 7 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.Quoted 5 lines
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.Quoted 3 lines
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.Hah, ok - I'll check again the possible outcomes of this.
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>Quoted 7 lines
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)You probably need to update the FreeBSD version too.
Quoted 6 lines
Oh, that I cannot do. First of all, in bsd code I don't see
rte_mp_dev_hotplug_init()called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.
The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.
Quoted 43 lines · Thomas Monjalon
27/03/2019 21:33, Stojaczyk, Dariusz:
Quoted 35 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 14 lines
26/03/2019 19:43, Darek Stojaczyk:
Quoted 7 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.Quoted 5 lines
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.Quoted 3 lines
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.Hah, ok - I'll check again the possible outcomes of this.
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>Quoted 7 lines
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)You probably need to update the FreeBSD version too.
Quoted 6 lines
Oh, that I cannot do. First of all, in bsd code I don't see
rte_mp_dev_hotplug_init()called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.
Quoted 3 lines
The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.
I don't think we support IPC or hotplug on BSD, so I don't think this patch is relevant on the BSD side.
/Bruce
28/03/2019 11:43, Bruce Richardson:
Quoted 44 lines · Thomas Monjalon
Quoted 43 lines
27/03/2019 21:33, Stojaczyk, Dariusz:
Quoted 41 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 14 lines
26/03/2019 19:43, Darek Stojaczyk:
Quoted 7 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.Quoted 5 lines
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.Quoted 3 lines
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.Hah, ok - I'll check again the possible outcomes of this.
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>Quoted 7 lines
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)You probably need to update the FreeBSD version too.
Oh, that I cannot do. First of all, in bsd code I don't see
rte_mp_dev_hotplug_init()called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.
Quoted 5 lines
Quoted 3 lines
The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.I don't think we support IPC or hotplug on BSD, so I don't think this patch is relevant on the BSD side.
Yes, but then, the initialization order is different on Linux and BSD. I'm thinking about consistency and maintenance ease.
Quoted 45 lines · Thomas Monjalon
28/03/2019 11:43, Bruce Richardson:
Quoted 44 lines · Thomas Monjalon
Quoted 43 lines
27/03/2019 21:33, Stojaczyk, Dariusz:
Quoted 41 lines
From: Thomas Monjalon [mailto:[email protected]]
Quoted 14 lines
26/03/2019 19:43, Darek Stojaczyk:
Quoted 7 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.Quoted 5 lines
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.Quoted 3 lines
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.Hah, ok - I'll check again the possible outcomes of this.
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>Quoted 7 lines
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)You probably need to update the FreeBSD version too.
Oh, that I cannot do. First of all, in bsd code I don't see
rte_mp_dev_hotplug_init()called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.I don't think we support IPC or hotplug on BSD, so I don't think this patch is relevant on the BSD side.
Yes, but then, the initialization order is different on Linux and BSD. I'm thinking about consistency and maintenance ease.
From my cursory inspection, nothing should be broken on FreeBSD as a result of moving alarm API init earlier.
Signature · 2 lines
Thanks, Anatoly
Quoted 34 lines
-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Wednesday, March 27, 2019 9:34 PM
To: Thomas Monjalon <[email protected]>
Cc: [email protected]; Zhang, Qi Z <[email protected]>; Burakov, Anatoly <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [PATCH] eal: initialize alarms earlyQuoted 22 lines
-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, March 27, 2019 7:12 PM
To: Stojaczyk, Dariusz <[email protected]>
Cc: [email protected]; Zhang, Qi Z <[email protected]>; Burakov, Anatoly <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] eal: initialize alarms early26/03/2019 19:43, Darek Stojaczyk:
Quoted 12 lines
We currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.
Quoted 6 lines
Quoted 3 lines
I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.Hah, ok - I'll check again the possible outcomes of this.
Done, I can't see any case I could break.
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>Quoted 7 lines
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>--- lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)You probably need to update the FreeBSD version too.
Quoted 6 lines
Oh, that I cannot do. First of all, in bsd code I don't see
rte_mp_dev_hotplug_init()called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.
Basing on Anatoly's feedback, I pushed a v2 with bsd changes included as well.
Thanks!
D.
Quoted 2 lines
D.
PATCH v2 1/1 eal: initialize alarms early
In reply to Darek StojaczykOn linux, we currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.
To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.
We update the bsd code as well to keep the initialization order the same in both eal implementations.
---
v2:
- updated the bsd code as well (Thomas)
lib/librte_eal/freebsd/eal/eal.c | 12 ++++++------ lib/librte_eal/linux/eal/eal.c | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-)
lib/librte_eal/freebsd/eal/eal.c +6 -6
@@ -662,6 +662,12 @@ rte_eal_init(int argc, char **argv)return -1 ; } +if (rte_eal_alarm_init () <0 ) { +rte_eal_init_alert ("Cannot init interrupt-handling thread" ); +/* rte_eal_alarm_init sets rte_errno on failure. */ +return -1 ; + } +/* Put mp channel init before bus scan so that we can init the vdev * bus through mp channel in the secondary process before the bus scan. */
@@ -751,12 +757,6 @@ rte_eal_init(int argc, char **argv)return -1 ; } -if (rte_eal_alarm_init () <0 ) { -rte_eal_init_alert ("Cannot init interrupt-handling thread" ); -/* rte_eal_alarm_init sets rte_errno on failure. */ -return -1 ; - } -if (rte_eal_timer_init () <0 ) {rte_eal_init_alert ("Cannot init HPET or TSC timers" ); rte_errno =ENOTSUP ;
lib/librte_eal/linux/eal/eal.c +6 -6
@@ -1005,6 +1005,12 @@ rte_eal_init(int argc, char **argv)return -1 ; } +if (rte_eal_alarm_init () <0 ) { +rte_eal_init_alert ("Cannot init interrupt-handling thread" ); +/* rte_eal_alarm_init sets rte_errno on failure. */ +return -1 ; + } +/* Put mp channel init before bus scan so that we can init the vdev * bus through mp channel in the secondary process before the bus scan. */
@@ -1125,12 +1131,6 @@ rte_eal_init(int argc, char **argv)return -1 ; } -if (rte_eal_alarm_init () <0 ) { -rte_eal_init_alert ("Cannot init interrupt-handling thread" ); -/* rte_eal_alarm_init sets rte_errno on failure. */ -return -1 ; - } -if (rte_eal_timer_init () <0 ) {rte_eal_init_alert ("Cannot init HPET or TSC timers" ); rte_errno =ENOTSUP ;
Signature · 1 line
2.17.1
Thomas Monjalon
Applied01/04/2019 16:18, Darek Stojaczyk:
Quoted 19 lines
On linux, we currently initialize
rte_alarmsafter starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move
rte_eal_alarm_init()right afterrte_eal_intr_init()as it makes some sense to keep those two close to each other.We update the bsd code as well to keep the initialization order the same in both eal implementations.
Fixes 244d5130719c ("eal: enable hotplug on multi-process")Cc Qi Zhang <[email protected]>Cc Anatoly Burakov <[email protected]>
Quoted 4 lines
Signed-off-by Darek Stojaczyk <[email protected]>---
v2:
- updated the bsd code as well (Thomas)
Applied, thanks