Skip to contentSkip to conversation

btrfs: Fix UAF during concurrent mount and device scan

PATCH In review Nikolay Borisov 5 participants 7 messages Jan 9, 2020 – Jan 13, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs: Fix UAF during concurrent mount and device scan

Johannes' recent device close cleanup patches uncovered an UAF which was triggered during device rescan, in device_list_add. In particular, device_list_add was seeing a struct btrfs_device with a stale fs_info member but valid bdev one. This caused btrfs_info_in_rcu to crash inside btrfs_printk when the latter was dereferencing device->fs_info. Kasan report looks like:

==================================================================

BUG: KASAN: use-after-free in device_list_add+0xf60/0x10e0 [btrfs]
BTRFS info (device dm-0): has skinny extents
Read of size 8 at addr ffff88810c70a348 by task systemd-udevd/10461

CPU: 3 PID: 10461 Comm: systemd-udevd Tainted: G        W    L    5.5.0-rc5-default #842
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
Call Trace:
 dump_stack+0x75/0xa0
 ? device_list_add+0xf60/0x10e0 [btrfs]
 print_address_description.constprop.8+0x92/0x3c2
 ? device_list_add+0xf60/0x10e0 [btrfs]
 ? device_list_add+0xf60/0x10e0 [btrfs]
 __kasan_report.cold.10+0x1a/0x3b
 ? iput+0x40/0x600
 ? device_list_add+0xf60/0x10e0 [btrfs]
 kasan_report+0xe/0x20
 device_list_add+0xf60/0x10e0 [btrfs]
 ? btrfs_alloc_device+0x680/0x680 [btrfs]
 ? blkdev_get+0x166/0x220
 ? btrfs_scan_one_device+0x38f/0x530 [btrfs]
 btrfs_scan_one_device+0x38f/0x530 [btrfs]
 ? device_list_add+0x10e0/0x10e0 [btrfs]
 ? __mutex_lock_slowpath+0x10/0x10
 ? _copy_from_user+0x70/0xa0
 btrfs_control_ioctl+0xb8/0x210 [btrfs]
 do_vfs_ioctl+0x190/0xf10
 ? may_open_dev+0xc0/0xc0
 ? ___slab_alloc+0x4af/0x4f0
 ? compat_ioctl_preallocate+0x1b0/0x1b0
 ? __fsnotify_inode_delete+0x20/0x20
 ? __kasan_slab_free+0x1da/0x270
 ? do_sys_open+0x16b/0x350
 ? kmem_cache_free+0xab/0x330
 ksys_ioctl+0x3a/0x70
 __x64_sys_ioctl+0x6f/0xb0
 do_syscall_64+0x8c/0x3a0
 ? prepare_exit_to_usermode+0x1d/0x1a0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f3bca9caf47
RSP: 002b:00007ffd418e45b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3bca9caf47
RDX: 00007ffd418e45d0 RSI: 0000000090009427 RDI: 0000000000000007
RBP: 00007ffd418e45d0 R08: 302d6d642f766564 R09: 0000000000000003
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000007
R13: 0000000000000000 R14: 000055c30563ab70 R15: 000055c305636870

Allocated by task 10425:
 save_stack+0x19/0x80
 __kasan_kmalloc.constprop.8+0xc1/0xd0
 kvmalloc_node+0x40/0x70
 btrfs_mount_root+0xc6/0xee0 [btrfs]
 legacy_get_tree+0xed/0x1d0
 vfs_get_tree+0x7d/0x230
 fc_mount+0xf/0x80
 vfs_kern_mount.part.44+0x5c/0x80
 btrfs_mount+0x220/0x1310 [btrfs]
 legacy_get_tree+0xed/0x1d0
 vfs_get_tree+0x7d/0x230
 do_mount+0xfbf/0x1560
 __x64_sys_mount+0x162/0x1b0
 do_syscall_64+0x8c/0x3a0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9

Freed by task 10454:
 save_stack+0x19/0x80
 __kasan_slab_free+0x1c5/0x270
 kfree+0xc9/0x2a0
 deactivate_locked_super+0x74/0xc0
 deactivate_super+0x123/0x140
 cleanup_mnt+0x204/0x450
 task_work_run+0x104/0x180
 exit_to_usermode_loop+0xfa/0x120
 do_syscall_64+0x2d7/0x3a0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9

Memory state around the buggy address:
 ffff88810c70a200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88810c70a280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff88810c70a300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                              ^
 ffff88810c70a380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88810c70a400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================

Instrumenting the code proved the theory:

mount-1231 [002] .... 46.498558: btrfs_init_devices_late: Updating fs_info for ffff888114e14400: old = 0000000000000000 new=ffff888110f2a000

umount-1271  [001] ....    46.562981: close_fs_devices.part.34: set bdev to NULL for ffff888114e14400 (STALE FS_DEVICES), fs_info= ffff888110f2a000
umount-1271  [001] ....    46.563317: btrfs_kill_super: Freeing fs_info: ffff888110f2a000
mount-1280  [001] ....    46.586577: open_fs_devices: Setting valid bdev to ffff888114e14400 (fs_info = ffff888110f2a000
systemd-udevd-1282  [003] ....    46.595672: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1282  [003] ....    46.603702: device_list_add: device: dm-0
systemd-udevd-1282  [003] ....    46.611201: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1282  [003] ....    46.611204: device_list_add: device: dm-0
systemd-udevd-1287  [003] .N..    46.646134: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1287  [003] .N..    46.646138: device_list_add: device: dm-0
systemd-udevd-1287  [003] ....    46.653470: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1287  [003] ....    46.653473: device_list_add: device: dm-0

mount-1280 [001] .... 46.696126: btrfs_init_devices_late: Updating fs_info for ffff888114e14400: old = ffff888110f2a000 new=ffff888111662000

This log shows how an fs is being unmounted which causes device close routine to be invoked. It sets bdev to NULL but doesn't reset fs_info. Afterwards the fs_info itself is freed from btrfs_kill_super at the same time the device is still anchored at fs_devices list. Subsequently a mount is triggered which sets btrfs_fs_device::bdev to a valid value, yet btrfs_fs_device::fs_info is still stale/freed. Before btrfs_fill_super is called and re-initializes btrfs_fs_device::fs_info a concurrent device scan is triggered, it finds the device with its ->bdev pointer set to valid value and eventually calls btrfs_info_in_rcu in device_list_add which causes the crash.

Simply setting btrfs_fs_device::fs_info to NULL prevents the crash but doesn't fix the race. In fact the race cannot be solved because device scan is asynchronous in its nature so it makes no sense to try and synchronize it with pending mounts.

Fixes cc1824fcd334 ("btrfs: reset device back to allocation state when removing")

---

With this fix I can no longer get generic/085 to crash/generate the KASAN warning, even with an mdelay added in btrfs_mount_root which triggered the issue reliably.

 fs/btrfs/volumes.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/volumes.c +1 -0
index 65e78e59d5c4..ad8944cc4dd1 100644
@@ -1086,6 +1086,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 
 	atomic_set(&device->dev_stats_ccnt, 0);
 	extent_io_tree_release(&device->alloc_state);
+	device->fs_info = NULL;
 
 	/* Verify the device is back in a pristine state  */
 	ASSERT(!test_bit(BTRFS_DEV_STATE_FLUSH_SENT, &device->dev_state));

--
2.17.1

#1 lore raw
Quoted 11 lines · Nikolay Borisov

This log shows how an fs is being unmounted which causes device close routine to be invoked. It sets bdev to NULL but doesn't reset fs_info. Afterwards the fs_info itself is freed from btrfs_kill_super at the same time the device is still anchored at fs_devices list. Subsequently a mount is triggered which sets btrfs_fs_device::bdev to a valid value, yet btrfs_fs_device::fs_info is still stale/freed. Before btrfs_fill_super is called and re-initializes btrfs_fs_device::fs_info a concurrent device scan is triggered, it finds the device with its ->bdev pointer set to valid value and eventually calls btrfs_info_in_rcu in device_list_add which causes the crash.

Quoted 4 lines

Simply setting btrfs_fs_device::fs_info to NULL prevents the crash but doesn't fix the race. In fact the race cannot be solved because device scan is asynchronous in its nature so it makes no sense to try and synchronize it with pending mounts.

We've had bugs when mount and scan raced, I don't see why you think this cannot be fixed and synchronized in this case. At minimum I'd think that the device_list_mutex should be enough as it's the one thing that excludes mount and scan for the new and removed members of the device lists etc.

Quoted 1 line
Fixes cc1824fcd334 ("btrfs: reset device back to allocation state when removing")

That's still it misc-next so I'd rather fold it into the patch than to have this split fix.

Quoted 23 lines

---

With this fix I can no longer get generic/085 to crash/generate the KASAN warning, even with an mdelay added in btrfs_mount_root which triggered the issue reliably.

 fs/btrfs/volumes.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/volumes.c +1 -1
index 65e78e59d5c4..ad8944cc4dd1 100644
@@ -1086,6 +1086,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 
 	atomic_set(&device->dev_stats_ccnt, 0);
 	extent_io_tree_release(&device->alloc_state);
+	device->fs_info = NULL;
 
 	/* Verify the device is back in a pristine state  */
 	ASSERT(!test_bit(BTRFS_DEV_STATE_FLUSH_SENT, &device->dev_state));
--

2.17.1

#2 lore raw
Quoted 17 lines · David Sterba
Quoted 11 lines · Nikolay Borisov

This log shows how an fs is being unmounted which causes device close routine to be invoked. It sets bdev to NULL but doesn't reset fs_info. Afterwards the fs_info itself is freed from btrfs_kill_super at the same time the device is still anchored at fs_devices list. Subsequently a mount is triggered which sets btrfs_fs_device::bdev to a valid value, yet btrfs_fs_device::fs_info is still stale/freed. Before btrfs_fill_super is called and re-initializes btrfs_fs_device::fs_info a concurrent device scan is triggered, it finds the device with its ->bdev pointer set to valid value and eventually calls btrfs_info_in_rcu in device_list_add which causes the crash.

Quoted 4 lines

Simply setting btrfs_fs_device::fs_info to NULL prevents the crash but doesn't fix the race. In fact the race cannot be solved because device scan is asynchronous in its nature so it makes no sense to try and synchronize it with pending mounts.

Quoted 5 lines

We've had bugs when mount and scan raced, I don't see why you think this cannot be fixed and synchronized in this case. At minimum I'd think that the device_list_mutex should be enough as it's the one thing that excludes mount and scan for the new and removed members of the device lists etc.

Yes, but this means mount will also need to take device_list_mutex for the duration of the whole mount operation.

Quoted 29 lines
Quoted 2 lines
Fixes cc1824fcd334 ("btrfs: reset device back to allocation state when removing")

That's still it misc-next so I'd rather fold it into the patch than to have this split fix.

Quoted 23 lines

---

With this fix I can no longer get generic/085 to crash/generate the KASAN warning, even with an mdelay added in btrfs_mount_root which triggered the issue reliably.

 fs/btrfs/volumes.c | 1 +
 1 file changed, 1 insertion(+)
fs/btrfs/volumes.c +1 -1
index 65e78e59d5c4..ad8944cc4dd1 100644
@@ -1086,6 +1086,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 
 	atomic_set(&device->dev_stats_ccnt, 0);
 	extent_io_tree_release(&device->alloc_state);
+	device->fs_info = NULL;
 
 	/* Verify the device is back in a pristine state  */
 	ASSERT(!test_bit(BTRFS_DEV_STATE_FLUSH_SENT, &device->dev_state));
--

2.17.1

#3 lore raw
In reply to David Sterba
Quoted 23 lines · David Sterba
Quoted 11 lines · Nikolay Borisov

This log shows how an fs is being unmounted which causes device close routine to be invoked. It sets bdev to NULL but doesn't reset fs_info. Afterwards the fs_info itself is freed from btrfs_kill_super at the same time the device is still anchored at fs_devices list. Subsequently a mount is triggered which sets btrfs_fs_device::bdev to a valid value, yet btrfs_fs_device::fs_info is still stale/freed. Before btrfs_fill_super is called and re-initializes btrfs_fs_device::fs_info a concurrent device scan is triggered, it finds the device with its ->bdev pointer set to valid value and eventually calls btrfs_info_in_rcu in device_list_add which causes the crash.

Quoted 4 lines

Simply setting btrfs_fs_device::fs_info to NULL prevents the crash but doesn't fix the race. In fact the race cannot be solved because device scan is asynchronous in its nature so it makes no sense to try and synchronize it with pending mounts.

We've had bugs when mount and scan raced, I don't see why you think this cannot be fixed and synchronized in this case. At minimum I'd think that the device_list_mutex should be enough as it's the one thing that excludes mount and scan for the new and removed members of the device lists etc.

Quoted 4 lines
Quoted 1 line
Fixes cc1824fcd334 ("btrfs: reset device back to allocation state when removing")

That's still it misc-next so I'd rather fold it into the patch than to have this split fix.

Now folded to "btrfs: reset device back to allocation state when removing" and pushed.

#6 lore raw
In reply to Nikolay Borisov
Quoted 8 lines · Nikolay Borisov

Johannes' recent device close cleanup patches uncovered an UAF which was

triggered during device rescan, in device_list_add. In particular, device_list_add was seeing a struct btrfs_device with a stale fs_info member

but valid bdev one.

Strange. But a closed device shall have bdev NULL may be it was just closing, not yet completely closed?

----------
static void btrfs_close_one_device(struct btrfs_device *device) ::

        btrfs_close_bdev(device);
        if (device->bdev) {
                fs_devices->open_devices--;
                device->bdev = NULL;   <----
        }
        clear_bit(BTRFS_DEV_STATE_WRITEABLE, &device->dev_state);

----------

So to me it looks like the thread which called device scan (systemd) is racing with some other thread, do you see the any contending threads?

Quoted 3 lines

This caused btrfs_info_in_rcu to
crash inside btrfs_printk when the latter was dereferencing device->fs_info.

Do you mean when dereferencing device::fs_info::sb ?
A null device::fs_info we won't crash it will just print unknown.

------------

                printk("%sBTRFS %s (device %s): %pV\n", lvl, type,
                        fs_info ? fs_info->sb->s_id : "<unknown>", &vaf);

-------------

Quoted 101 lines

Kasan report looks like:

==================================================================

BUG: KASAN: use-after-free in device_list_add+0xf60/0x10e0 [btrfs]
BTRFS info (device dm-0): has skinny extents
Read of size 8 at addr ffff88810c70a348 by task systemd-udevd/10461

CPU: 3 PID: 10461 Comm: systemd-udevd Tainted: G        W    L    5.5.0-rc5-default #842
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
Call Trace:
  dump_stack+0x75/0xa0
  ? device_list_add+0xf60/0x10e0 [btrfs]
  print_address_description.constprop.8+0x92/0x3c2
  ? device_list_add+0xf60/0x10e0 [btrfs]
  ? device_list_add+0xf60/0x10e0 [btrfs]
  __kasan_report.cold.10+0x1a/0x3b
  ? iput+0x40/0x600
  ? device_list_add+0xf60/0x10e0 [btrfs]
  kasan_report+0xe/0x20
  device_list_add+0xf60/0x10e0 [btrfs]
  ? btrfs_alloc_device+0x680/0x680 [btrfs]
  ? blkdev_get+0x166/0x220
  ? btrfs_scan_one_device+0x38f/0x530 [btrfs]
  btrfs_scan_one_device+0x38f/0x530 [btrfs]
  ? device_list_add+0x10e0/0x10e0 [btrfs]
  ? __mutex_lock_slowpath+0x10/0x10
  ? _copy_from_user+0x70/0xa0
  btrfs_control_ioctl+0xb8/0x210 [btrfs]
  do_vfs_ioctl+0x190/0xf10
  ? may_open_dev+0xc0/0xc0
  ? ___slab_alloc+0x4af/0x4f0
  ? compat_ioctl_preallocate+0x1b0/0x1b0
  ? __fsnotify_inode_delete+0x20/0x20
  ? __kasan_slab_free+0x1da/0x270
  ? do_sys_open+0x16b/0x350
  ? kmem_cache_free+0xab/0x330
  ksys_ioctl+0x3a/0x70
  __x64_sys_ioctl+0x6f/0xb0
  do_syscall_64+0x8c/0x3a0
  ? prepare_exit_to_usermode+0x1d/0x1a0
  entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f3bca9caf47
RSP: 002b:00007ffd418e45b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3bca9caf47
RDX: 00007ffd418e45d0 RSI: 0000000090009427 RDI: 0000000000000007
RBP: 00007ffd418e45d0 R08: 302d6d642f766564 R09: 0000000000000003
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000007
R13: 0000000000000000 R14: 000055c30563ab70 R15: 000055c305636870

Allocated by task 10425:
  save_stack+0x19/0x80
  __kasan_kmalloc.constprop.8+0xc1/0xd0
  kvmalloc_node+0x40/0x70
  btrfs_mount_root+0xc6/0xee0 [btrfs]
  legacy_get_tree+0xed/0x1d0
  vfs_get_tree+0x7d/0x230
  fc_mount+0xf/0x80
  vfs_kern_mount.part.44+0x5c/0x80
  btrfs_mount+0x220/0x1310 [btrfs]
  legacy_get_tree+0xed/0x1d0
  vfs_get_tree+0x7d/0x230
  do_mount+0xfbf/0x1560
  __x64_sys_mount+0x162/0x1b0
  do_syscall_64+0x8c/0x3a0
  entry_SYSCALL_64_after_hwframe+0x44/0xa9

Freed by task 10454:
  save_stack+0x19/0x80
  __kasan_slab_free+0x1c5/0x270
  kfree+0xc9/0x2a0
  deactivate_locked_super+0x74/0xc0
  deactivate_super+0x123/0x140
  cleanup_mnt+0x204/0x450
  task_work_run+0x104/0x180
  exit_to_usermode_loop+0xfa/0x120
  do_syscall_64+0x2d7/0x3a0
  entry_SYSCALL_64_after_hwframe+0x44/0xa9

Memory state around the buggy address:
  ffff88810c70a200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  ffff88810c70a280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Quoted 1 line

ffff88810c70a300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb

                                               ^

ffff88810c70a380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88810c70a400: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================

Instrumenting the code proved the theory:

mount-1231 [002] .... 46.498558: btrfs_init_devices_late: Updating fs_info for ffff888114e14400: old = 0000000000000000 new=ffff888110f2a000

umount-1271  [001] ....    46.562981: close_fs_devices.part.34: set bdev to NULL for ffff888114e14400 (STALE FS_DEVICES), fs_info= ffff888110f2a000
umount-1271  [001] ....    46.563317: btrfs_kill_super: Freeing fs_info: ffff888110f2a000
mount-1280  [001] ....    46.586577: open_fs_devices: Setting valid bdev to ffff888114e14400 (fs_info = ffff888110f2a000
systemd-udevd-1282  [003] ....    46.595672: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1282  [003] ....    46.603702: device_list_add: device: dm-0
systemd-udevd-1282  [003] ....    46.611201: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1282  [003] ....    46.611204: device_list_add: device: dm-0
systemd-udevd-1287  [003] .N..    46.646134: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1287  [003] .N..    46.646138: device_list_add: device: dm-0
systemd-udevd-1287  [003] ....    46.653470: device_list_add: found device with bdev: ffff888114e14400 fs_info: ffff888110f2a000
systemd-udevd-1287  [003] ....    46.653473: device_list_add: device: dm-0
Quoted 7 lines

mount-1280 [001] .... 46.696126: btrfs_init_devices_late: Updating fs_info for ffff888114e14400: old = ffff888110f2a000 new=ffff888111662000

This log shows how an fs is being unmounted which causes device close routine to be invoked. It sets bdev to NULL but doesn't reset fs_info. Afterwards the fs_info itself is freed from btrfs_kill_super at the same time the device is still anchored at fs_devices list. Subsequently a mount is triggered which sets btrfs_fs_device::bdev to a valid value, yet

                                 ^^^

device::bdev.

Quoted 6 lines

btrfs_fs_device::fs_info is still stale/freed. Before btrfs_fill_super is called and re-initializes btrfs_fs_device::fs_info a concurrent device scan is triggered, it finds the device with its ->bdev pointer set to valid value and eventually calls btrfs_info_in_rcu in device_list_add which causes the crash.

Quoted 6 lines

Simply setting btrfs_fs_device::fs_info to NULL prevents the crash but doesn't fix the race. In fact the race cannot be solved because device scan is asynchronous in its nature so it makes no sense to try and synchronize it with pending mounts.

Fixes cc1824fcd334 ("btrfs: reset device back to allocation state when removing")

On the basics of that its fundamentally wrong to use the fs_info in device_list_add(), I have sent a fix. Which will indirectly fix this UAF, unfortunately. And as of now I am not convinced if this is related to cc1824fcd334. Few analysis items are missing IMO.

Thanks for the report.

Anand

Quoted 24 lines

---

With this fix I can no longer get generic/085 to crash/generate the KASAN warning, even with an mdelay added in btrfs_mount_root which triggered the issue reliably.

  fs/btrfs/volumes.c | 1 +
  1 file changed, 1 insertion(+)
fs/btrfs/volumes.c +1 -1
index 65e78e59d5c4..ad8944cc4dd1 100644
@@ -1086,6 +1086,7 @@ static void btrfs_close_one_device(struct btrfs_device *device)
 
  	atomic_set(&device->dev_stats_ccnt, 0);
  	extent_io_tree_release(&device->alloc_state);
+	device->fs_info = NULL;
 
  	/* Verify the device is back in a pristine state  */
  	ASSERT(!test_bit(BTRFS_DEV_STATE_FLUSH_SENT, &device->dev_state));
--

2.17.1