Skip to contentSkip to conversation

iavf: Fix deadlock in initialization

PATCH In review Ivan Vecera 2 participants 2 messages Aug 8, 2022 – Aug 12, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 iavf: Fix deadlock in initialization

Fix deadlock that occurs when iavf interface is a part of failover configuration.

1. Mutex crit_lock is taken at the beginning of iavf_watchdog_task()
2. Function iavf_init_config_adapter() is called when adapter state is __IAVF_INIT_CONFIG_ADAPTER
3. iavf_init_config_adapter() calls register_netdevice() that emits NETDEV_REGISTER event
4. Notifier function failover_event() then calls net_failover_slave_register() that calls dev_open()
5. dev_open() calls iavf_open() that tries to take crit_lock in end-less loop

Stack trace:
...

[  790.251876]  usleep_range_state+0x5b/0x80
[  790.252547]  iavf_open+0x37/0x1d0 [iavf]
[  790.253139]  __dev_open+0xcd/0x160
[  790.253699]  dev_open+0x47/0x90
[  790.254323]  net_failover_slave_register+0x122/0x220 [net_failover]
[  790.255213]  failover_slave_register.part.7+0xd2/0x180 [failover]
[  790.256050]  failover_event+0x122/0x1ab [failover]
[  790.256821]  notifier_call_chain+0x47/0x70
[  790.257510]  register_netdevice+0x20f/0x550
[  790.258263]  iavf_watchdog_task+0x7c8/0xea0 [iavf]
[  790.259009]  process_one_work+0x1a7/0x360
[  790.259705]  worker_thread+0x30/0x390

To fix the situation we should check the current adapter state after first unsuccessful mutex_trylock() and return with -EBUSY if it is __IAVF_INIT_CONFIG_ADAPTER.

Fixes 226d528512cf ("iavf: fix locking of critical sections")
---
 drivers/net/ethernet/intel/iavf/iavf_main.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +10 -1
index 45d097a164ad..f9dcaadc7ea0 100644
@@ -4085,8 +4085,17 @@ static int iavf_open(struct net_device *netdev)
 		return -EIO;
 	}
 
-	while (!mutex_trylock(&adapter->crit_lock))
+	while (!mutex_trylock(&adapter->crit_lock)) {
+		/* If we are in __IAVF_INIT_CONFIG_ADAPTER state the crit_lock
+		 * is already taken and iavf_open is called from an upper
+		 * device's notifier reacting on NETDEV_REGISTER event.
+		 * We have to leave here to avoid dead lock.
+		 */
+		if (adapter->state == __IAVF_INIT_CONFIG_ADAPTER)
+			return -EBUSY;
+
 		usleep_range(500, 1000);
+	}
 
 	if (adapter->state != __IAVF_DOWN) {
 		err = -EBUSY;
Signature · 1 line
2.35.1
Tested-by on v1 1/1
#1 lore raw

drivers/net/ethernet/intel/iavf/iavf_main.c static int iavf_open(struct net_device *netdev)

Quoted 56 lines

-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Ivan Vecera
Sent: Monday, August 8, 2022 7:59 PM
To: [email protected]
Cc: moderated list:INTEL ETHERNET DRIVERS <intel-wired- [email protected]>; Brandeburg, Jesse <[email protected]>; open list <[email protected]>; Stefan Assmann <[email protected]>; Eric Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo Abeni <[email protected]>; David S. Miller <[email protected]>
Subject: [Intel-wired-lan] [PATCH net] iavf: Fix deadlock in initialization

Fix deadlock that occurs when iavf interface is a part of failover configuration.

1. Mutex crit_lock is taken at the beginning of iavf_watchdog_task() 2. Function iavf_init_config_adapter() is called when adapter state is __IAVF_INIT_CONFIG_ADAPTER
3. iavf_init_config_adapter() calls register_netdevice() that emits NETDEV_REGISTER event
4. Notifier function failover_event() then calls net_failover_slave_register() that calls dev_open() 5. dev_open() calls
iavf_open() that tries to take crit_lock in
end-less loop

Stack trace:
...

[  790.251876]  usleep_range_state+0x5b/0x80 [  790.252547]
iavf_open+0x37/0x1d0 [iavf] [  790.253139]  __dev_open+0xcd/0x160 [
790.253699]  dev_open+0x47/0x90 [  790.254323]
net_failover_slave_register+0x122/0x220 [net_failover] [  790.255213]
failover_slave_register.part.7+0xd2/0x180 [failover] [  790.256050]
failover_event+0x122/0x1ab [failover] [  790.256821]
notifier_call_chain+0x47/0x70 [  790.257510]
register_netdevice+0x20f/0x550 [  790.258263]
iavf_watchdog_task+0x7c8/0xea0 [iavf] [  790.259009]
process_one_work+0x1a7/0x360 [  790.259705]  worker_thread+0x30/0x390

To fix the situation we should check the current adapter state after first unsuccessful mutex_trylock() and return with -EBUSY if it is __IAVF_INIT_CONFIG_ADAPTER.

Fixes 226d528512cf ("iavf: fix locking of critical sections")
---
 drivers/net/ethernet/intel/iavf/iavf_main.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/net/ethernet/intel/iavf/iavf_main.c +0 -0
index 45d097a164ad..f9dcaadc7ea0 100644
@@ -4085,8 +4085,17 @@ static int iavf_open(struct net_device *netdev)

_______________________________________________
Intel-wired-lan mailing list
[email protected]
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan