Skip to contentSkip to conversation

eal: initialize alarms early

PATCH Superseded Darek Stojaczyk 4 participants 10 messages Mar 26, 2019 – Apr 2, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 eal: initialize alarms early

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
lib/librte_eal/linux/eal/eal.c +6 -6
index 13f401684..75ed0cf10 100644
@@ -1005,6 +1005,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	if (rte_eal_alarm_init() < 0) {
+		rte_eal_init_alert("Cannot init interrupt-handling thread");
+		/* rte_eal_alarm_init sets rte_errno on failure. */
+		return -1;
+	}
+
 	/* Put mp channel init before bus scan so that we can init the vdev
 	 * bus through mp channel in the secondary process before the bus scan.
 	 */
@@ -1125,12 +1131,6 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
-	if (rte_eal_alarm_init() < 0) {
-		rte_eal_init_alert("Cannot init interrupt-handling thread");
-		/* rte_eal_alarm_init sets rte_errno on failure. */
-		return -1;
-	}
-
 	if (rte_eal_timer_init() < 0) {
 		rte_eal_init_alert("Cannot init HPET or TSC timers");
 		rte_errno = ENOTSUP;
Signature · 1 line
2.17.1
#1 lore raw

26/03/2019 19:43, Darek Stojaczyk:

Quoted 7 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

Quoted 5 lines

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Quoted 9 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

#2 lore raw
Quoted 24 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, March 27, 2019 7:12 PM
To: Stojaczyk, Dariusz <[email protected]>
Cc: [email protected]; Zhang, Qi Z <[email protected]>; Burakov, Anatoly <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] eal: initialize alarms early

26/03/2019 19:43, Darek Stojaczyk:

Quoted 7 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

Quoted 5 lines

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Quoted 3 lines

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Hah, ok - I'll check again the possible outcomes of this.

Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.

D.

#3 lore raw

27/03/2019 21:33, Stojaczyk, Dariusz:

Quoted 35 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 14 lines

26/03/2019 19:43, Darek Stojaczyk:

Quoted 7 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

Quoted 5 lines

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Quoted 3 lines

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Hah, ok - I'll check again the possible outcomes of this.

Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

Quoted 6 lines

Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.

The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.

#4 lore raw
Quoted 43 lines · Thomas Monjalon

27/03/2019 21:33, Stojaczyk, Dariusz:

Quoted 35 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 14 lines

26/03/2019 19:43, Darek Stojaczyk:

Quoted 7 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

Quoted 5 lines

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Quoted 3 lines

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Hah, ok - I'll check again the possible outcomes of this.

Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

Quoted 6 lines

Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.

Quoted 3 lines

The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.

I don't think we support IPC or hotplug on BSD, so I don't think this patch is relevant on the BSD side.

/Bruce

#5 lore raw

28/03/2019 11:43, Bruce Richardson:

Quoted 44 lines · Thomas Monjalon
Quoted 43 lines

27/03/2019 21:33, Stojaczyk, Dariusz:

Quoted 41 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 14 lines

26/03/2019 19:43, Darek Stojaczyk:

Quoted 7 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

Quoted 5 lines

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Quoted 3 lines

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Hah, ok - I'll check again the possible outcomes of this.

Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.

Quoted 5 lines
Quoted 3 lines

The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.

I don't think we support IPC or hotplug on BSD, so I don't think this patch is relevant on the BSD side.

Yes, but then, the initialization order is different on Linux and BSD. I'm thinking about consistency and maintenance ease.

#6 lore raw
Quoted 45 lines · Thomas Monjalon

28/03/2019 11:43, Bruce Richardson:

Quoted 44 lines · Thomas Monjalon
Quoted 43 lines

27/03/2019 21:33, Stojaczyk, Dariusz:

Quoted 41 lines

From: Thomas Monjalon [mailto:[email protected]]

Quoted 14 lines

26/03/2019 19:43, Darek Stojaczyk:

Quoted 7 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

Quoted 5 lines

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Quoted 3 lines

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Hah, ok - I'll check again the possible outcomes of this.

Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
---
  lib/librte_eal/linux/eal/eal.c | 12 ++++++------
  1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

The problem is the consistency between OSes.
May you ask help here? Bruce is maintaining the FreeBSD side.

I don't think we support IPC or hotplug on BSD, so I don't think this patch is relevant on the BSD side.

Yes, but then, the initialization order is different on Linux and BSD. I'm thinking about consistency and maintenance ease.

From my cursory inspection, nothing should be broken on FreeBSD as a result of moving alarm API init earlier.

Signature · 2 lines
Thanks,
Anatoly
#7 lore raw
In reply to Stojaczyk, Dariusz
Quoted 34 lines

-----Original Message-----
From: Stojaczyk, Dariusz
Sent: Wednesday, March 27, 2019 9:34 PM
To: Thomas Monjalon <[email protected]>
Cc: [email protected]; Zhang, Qi Z <[email protected]>; Burakov, Anatoly <[email protected]>; [email protected]
Subject: RE: [dpdk-dev] [PATCH] eal: initialize alarms early

Quoted 22 lines

-----Original Message-----
From: Thomas Monjalon [mailto:[email protected]]
Sent: Wednesday, March 27, 2019 7:12 PM
To: Stojaczyk, Dariusz <[email protected]>
Cc: [email protected]; Zhang, Qi Z <[email protected]>; Burakov, Anatoly <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] eal: initialize alarms early

26/03/2019 19:43, Darek Stojaczyk:

Quoted 12 lines

We currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

Quoted 6 lines
Quoted 3 lines

I wonder which regression it will bring :)
The experience shows that we cannot touch this function without introducing a regression. Please check twice.

Hah, ok - I'll check again the possible outcomes of this.

Done, I can't see any case I could break.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 7 lines
Quoted 4 lines
---
 lib/librte_eal/linux/eal/eal.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

You probably need to update the FreeBSD version too.

Quoted 6 lines

Oh, that I cannot do. First of all, in bsd code I don't see rte_mp_dev_hotplug_init() called anywhere, as if bsd did not listen for IPC hotplug messages at all and hence did not have any data race in this area. Second, I would be afraid to touch any bsd code as I'm not running any bsd system.

Basing on Anatoly's feedback, I pushed a v2 with bsd changes included as well.

Thanks!
D.

Quoted 2 lines

D.

#8 lore raw

PATCH v2 1/1 eal: initialize alarms early

In reply to Darek Stojaczyk

On linux, we currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

We update the bsd code as well to keep the initialization order the same in both eal implementations.

Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>

---
v2:
- updated the bsd code as well (Thomas)

 lib/librte_eal/freebsd/eal/eal.c | 12 ++++++------
 lib/librte_eal/linux/eal/eal.c   | 12 ++++++------
 2 files changed, 12 insertions(+), 12 deletions(-)
lib/librte_eal/freebsd/eal/eal.c +6 -6
index 4e86b10b1..790c6afa7 100644
@@ -662,6 +662,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	if (rte_eal_alarm_init() < 0) {
+		rte_eal_init_alert("Cannot init interrupt-handling thread");
+		/* rte_eal_alarm_init sets rte_errno on failure. */
+		return -1;
+	}
+
 	/* Put mp channel init before bus scan so that we can init the vdev
 	 * bus through mp channel in the secondary process before the bus scan.
 	 */
@@ -751,12 +757,6 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
-	if (rte_eal_alarm_init() < 0) {
-		rte_eal_init_alert("Cannot init interrupt-handling thread");
-		/* rte_eal_alarm_init sets rte_errno on failure. */
-		return -1;
-	}
-
 	if (rte_eal_timer_init() < 0) {
 		rte_eal_init_alert("Cannot init HPET or TSC timers");
 		rte_errno = ENOTSUP;
lib/librte_eal/linux/eal/eal.c +6 -6
index 13f401684..75ed0cf10 100644
@@ -1005,6 +1005,12 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
+	if (rte_eal_alarm_init() < 0) {
+		rte_eal_init_alert("Cannot init interrupt-handling thread");
+		/* rte_eal_alarm_init sets rte_errno on failure. */
+		return -1;
+	}
+
 	/* Put mp channel init before bus scan so that we can init the vdev
 	 * bus through mp channel in the secondary process before the bus scan.
 	 */
@@ -1125,12 +1131,6 @@ rte_eal_init(int argc, char **argv)
 		return -1;
 	}
 
-	if (rte_eal_alarm_init() < 0) {
-		rte_eal_init_alert("Cannot init interrupt-handling thread");
-		/* rte_eal_alarm_init sets rte_errno on failure. */
-		return -1;
-	}
-
 	if (rte_eal_timer_init() < 0) {
 		rte_eal_init_alert("Cannot init HPET or TSC timers");
 		rte_errno = ENOTSUP;
Signature · 1 line
2.17.1
#9 lore raw

01/04/2019 16:18, Darek Stojaczyk:

Quoted 19 lines

On linux, we currently initialize rte_alarms after starting to listen for IPC hotplug requests, which gives us a data race window. Upon receiving such hotplug request we always try to set an alarm and this obviously doesn't work if the alarms weren't initialized yet.

To fix it, we initialize alarms before starting to listen for IPC hotplug messages. Specifically, we move rte_eal_alarm_init() right after rte_eal_intr_init() as it makes some sense to keep those two close to each other.

We update the bsd code as well to keep the initialization order the same in both eal implementations.

Fixes 244d5130719c ("eal: enable hotplug on multi-process")
Cc Qi Zhang <[email protected]>
Cc Anatoly Burakov <[email protected]>
Quoted 4 lines

---
v2:
- updated the bsd code as well (Thomas)

Applied, thanks