Skip to contentSkip to conversation

pull request for net: batman-adv 2023-06-07

PATCH Applied Simon Wunderlich 7 participants 8 messages Jun 7, 2023 – Jun 8, 2023

Subsystems (3)
lore
Share & tools
lore

Simon Wunderlich

author maintainer
#0 lore raw

cover pull request for net: batman-adv 2023-06-07

Hi David, hi Jakub,

here is a bugfix for batman-adv which we would like to have integrated into net.

Please pull or let me know of any problem!

Thank you,
Simon

The following changes since commit 44c026a73be8038f03dbdeef028b642880cf1511:

Linux 6.4-rc3 (2023-05-21 14:05:48 -0700)

are available in the Git repository at:

git://git.open-mesh.org/linux-merge.git tags/batadv-net-pullrequest-20230607

for you to fetch changes up to abac3ac97fe8734b620e7322a116450d7f90aa43:

batman-adv: Broken sync while rescheduling delayed work (2023-05-26 23:14:49 +0200)

----------------------------------------------------------------
Here is a batman-adv bugfix:

- fix a broken sync while rescheduling delayed work, by Vladislav Efanov

----------------------------------------------------------------
Vladislav Efanov (1):
batman-adv: Broken sync while rescheduling delayed work

 net/batman-adv/distributed-arp-table.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
author maintainer
#1 lore raw

PATCH 1/1 batman-adv: Broken sync while rescheduling delayed work

From: Vladislav Efanov <[email protected]>

Syzkaller got a lot of crashes like:

KASAN: use-after-free Write in *_timers*

All of these crashes point to the same memory area:

The buggy address belongs to the object at ffff88801f870000
 which belongs to the cache kmalloc-8k of size 8192

The buggy address is located 5320 bytes inside of
8192-byte region [ffff88801f870000, ffff88801f872000)

This area belongs to :

        batadv_priv->batadv_priv_dat->delayed_work->timer_list

The reason for these issues is the lack of synchronization. Delayed work (batadv_dat_purge) schedules new timer/work while the device is being deleted. As the result new timer/delayed work is set after cancel_delayed_work_sync() was called. So after the device is freed the timer list contains pointer to already freed memory.

Found by Linux Verification Center (linuxtesting.org) with syzkaller.

Fixes 2f1dfbe18507 ("batman-adv: Distributed ARP Table - implement local storage")
---
 net/batman-adv/distributed-arp-table.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/batman-adv/distributed-arp-table.c +1 -1
index 6968e55eb971..28a939d56090 100644
@@ -101,7 +101,6 @@ static void batadv_dat_purge(struct work_struct *work);
  */
 static void batadv_dat_start_timer(struct batadv_priv *bat_priv)
 {
-	INIT_DELAYED_WORK(&bat_priv->dat.work, batadv_dat_purge);
 	queue_delayed_work(batadv_event_workqueue, &bat_priv->dat.work,
 			   msecs_to_jiffies(10000));
 }
@@ -819,6 +818,7 @@ int batadv_dat_init(struct batadv_priv *bat_priv)
 	if (!bat_priv->dat.hash)
 		return -ENOMEM;
 
+	INIT_DELAYED_WORK(&bat_priv->dat.work, batadv_dat_purge);
 	batadv_dat_start_timer(bat_priv);
 
 	batadv_tvlv_handler_register(bat_priv, batadv_dat_tvlv_ogm_handler_v1,
Signature · 1 line
2.30.2
#2 lore raw

Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

This patch was applied to netdev/net.git (main) by Simon Wunderlich <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Simon Wunderlich <[email protected]>:

Quoted 8 lines · On Wed, 7 Jun 2023 17:55:15 +0200 you wrote:

From: Vladislav Efanov <[email protected]>

Syzkaller got a lot of crashes like:

KASAN: use-after-free Write in *_timers*

All of these crashes point to the same memory area:

[...]

Here is the summary with links:
- [1/1] batman-adv: Broken sync while rescheduling delayed work

    https://git.kernel.org/netdev/net/c/abac3ac97fe8

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
maintainer
#3 lore raw

Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

In reply to Simon Wunderlich
Quoted 5 lines · On Wed, 7 Jun 2023 17:55:15 +0200 Simon Wunderlich wrote:

The reason for these issues is the lack of synchronization. Delayed work (batadv_dat_purge) schedules new timer/work while the device is being deleted. As the result new timer/delayed work is set after cancel_delayed_work_sync() was called. So after the device is freed the timer list contains pointer to already freed memory.

I guess this is better than status quo but is the fix really complete? We're still not preventing the timer / work from getting scheduled and staying alive after the netdev has been freed, right?

#4 lore raw

RE: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

Quoted 19 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Wednesday, June 7, 2023 10:01 PM
To: Simon Wunderlich <[email protected]>
Cc: [email protected]; [email protected]; [email protected] mesh.org; Vladislav Efanov <[email protected]>; [email protected]; Sven Eckelmann <[email protected]>
Subject: Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

Quoted 5 lines · On Wed, 7 Jun 2023 17:55:15 +0200 Simon Wunderlich wrote:

The reason for these issues is the lack of synchronization. Delayed work (batadv_dat_purge) schedules new timer/work while the device is being deleted. As the result new timer/delayed work is set after cancel_delayed_work_sync() was called. So after the device is freed the timer list contains pointer to already freed memory.

Quoted 3 lines

I guess this is better than status quo but is the fix really complete? We're still not preventing the timer / work from getting scheduled and staying alive after the netdev has been freed, right?

Yea, I would expect some synchronization mechanism to ensure that after cancel_delayed_work_sync() you can't queue the work again.

I know for timers there is recently timer_shutdown_sync() which can be used to guarantee a timer can't re-arm at all, and its intended for some situations where there is a cyclic dependency...

Thanks,
Jake

#5 lore raw

Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

As far as I found the synchronization is provided by delayed work subsystem. It is based on the WORK_STRUCT_PENDING_BIT in work->data field.

The cancel_delayed_work_sync() atomically sets this bit and queue_delayed_work() checks it before scheduling new delayed work.

The problem is caused by the INIT_DELAYED_WORK() call inside batadv_dat_start_timer(). This call happens before the queue_delayed_work() call and clears this bit.

Best regards,

Vlad

Quoted 26 lines · Jacob E
Quoted 12 lines

-----Original Message-----
From: Jakub Kicinski <[email protected]>
Sent: Wednesday, June 7, 2023 10:01 PM
To: Simon Wunderlich <[email protected]>
Cc: [email protected]; [email protected]; [email protected] mesh.org; Vladislav Efanov <[email protected]>; [email protected]; Sven Eckelmann <[email protected]>
Subject: Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

On Wed, 7 Jun 2023 17:55:15 +0200 Simon Wunderlich wrote:

Quoted 8 lines
Quoted 5 lines

The reason for these issues is the lack of synchronization. Delayed work (batadv_dat_purge) schedules new timer/work while the device is being deleted. As the result new timer/delayed work is set after cancel_delayed_work_sync() was called. So after the device is freed the timer list contains pointer to already freed memory.

I guess this is better than status quo but is the fix really complete? We're still not preventing the timer / work from getting scheduled and staying alive after the netdev has been freed, right?

Yea, I would expect some synchronization mechanism to ensure that after cancel_delayed_work_sync() you can't queue the work again.

I know for timers there is recently timer_shutdown_sync() which can be used to guarantee a timer can't re-arm at all, and its intended for some situations where there is a cyclic dependency...

Thanks,
Jake

maintainer
#6 lore raw

Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski
Quoted 5 lines · On Wed, 7 Jun 2023 17:55:15 +0200 Simon Wunderlich wrote:

The reason for these issues is the lack of synchronization. Delayed work (batadv_dat_purge) schedules new timer/work while the device is being deleted. As the result new timer/delayed work is set after cancel_delayed_work_sync() was called. So after the device is freed the timer list contains pointer to already freed memory.

I guess this is better than status quo but is the fix really complete? We're still not preventing the timer / work from getting scheduled and staying alive after the netdev has been freed, right?

I *think* this specific use case does not expose such problem, as the delayed work is (AFAICS) scheduled only at device creation time and by the work itself, it should never be re-scheduled after cancel_delayed_work_sync()

Cheers,

Paolo

maintainer
#7 lore raw

Re: [PATCH 1/1] batman-adv: Broken sync while rescheduling delayed work

On Thursday, 8 June 2023 11:27:31 CEST Paolo Abeni wrote:
[...]

Quoted 7 lines
Quoted 2 lines

We're still not preventing the timer / work from getting scheduled and staying alive after the netdev has been freed, right?

I *think* this specific use case does not expose such problem, as the delayed work is (AFAICS) scheduled only at device creation time and by the work itself, it should never be re-scheduled after cancel_delayed_work_sync()

Correct.

* batadv_dat_start_timer is the only thing scheduling it
* batadv_dat_start_timer is called by:

- batadv_dat_purge (the worker rearming itself)
- batadv_dat_init (when the interface is created)

Kind regards,
Sven