Skip to contentSkip to conversation

discard storm on idle since v6.1-rc8-59-g63a7cb130718 discard=async

6.2 regression Sergei Trofimovich 3 participants 8 messages Mar 1, 2023 – Apr 21, 2023

Subsystems (2)
lore
Share & tools
lore

Sergei Trofimovich

author Reviewed-by Boris Burkov
#0 lore raw

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with perf: led flashes align with output of:

    # perf ftrace -a -T 'nvme_setup*' | cat
    kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

iotop shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

  $ git bisect good
  63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
  commit 63a7cb13071842966c1ce931edacbc23573aada5
  Author: David Sterba <[email protected]>
  Date:   Tue Jul 26 20:54:10 2022 +0200

btrfs: auto enable discard=async when possible

There's a request to automatically enable async discard for capable devices. We can do that, the async mode is designed to wait for larger freed extents and is not intrusive, with limits to iops, kbps or latency.

The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

The automatic selection is done if there's at least one discard capable device in the filesystem (not capable devices are skipped). Mounting with any other discard option will honor that option, notably mounting with nodiscard will keep it disabled.

   fs/btrfs/ctree.h   |  1 +
   fs/btrfs/disk-io.c | 14 ++++++++++++++
   fs/btrfs/super.c   |  2 ++
   fs/btrfs/volumes.c |  3 +++
   fs/btrfs/volumes.h |  2 ++
   5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But glancing at it this bit looks slightly off:

+ if (bdev_max_discard_sectors(bdev))
+ fs_devices->discardable = true;

Is it expected that there is no = false assignment?

This is the list of btrfs filesystems I have:

  $ cat /proc/mounts | fgrep btrfs
  /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
  /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
  # skipped bind mounts

The device is:

  $ lspci | fgrep -i Solid
  01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)

Can you help me debug the source of discards storm?

Is it an expected discard storm?

Is it problematic for SSD life span?

Thank you!

Signature · 1 line
  Sergei
#1 lore raw

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel regressions; the text you find below is based on a few templates paragraphs you might have encountered already in similar form. See link in footer if these mails annoy you.]

Quoted 9 lines · Sergei Trofimovich

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

Thanks for the report. To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot:

#regzbot ^introduced 63a7cb13071842
#regzbot title btrfs: DISCARD storm towards NVME device be it idle or not #regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply and tell me -- ideally while also telling regzbot about it, as explained by the page listed in the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing to the report (the parent of this mail). See page linked in footer for details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Quoted 81 lines

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with perf: led flashes align with output of:

    # perf ftrace -a -T 'nvme_setup*' | cat
    kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
    kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
    kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

iotop shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

  $ git bisect good
  63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
  commit 63a7cb13071842966c1ce931edacbc23573aada5
  Author: David Sterba <[email protected]>
  Date:   Tue Jul 26 20:54:10 2022 +0200

btrfs: auto enable discard=async when possible

There's a request to automatically enable async discard for capable devices. We can do that, the async mode is designed to wait for larger freed extents and is not intrusive, with limits to iops, kbps or latency.

The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

The automatic selection is done if there's at least one discard capable device in the filesystem (not capable devices are skipped). Mounting with any other discard option will honor that option, notably mounting with nodiscard will keep it disabled.

   fs/btrfs/ctree.h   |  1 +
   fs/btrfs/disk-io.c | 14 ++++++++++++++
   fs/btrfs/super.c   |  2 ++
   fs/btrfs/volumes.c |  3 +++
   fs/btrfs/volumes.h |  2 ++
   5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But glancing at it this bit looks slightly off:

+ if (bdev_max_discard_sectors(bdev))
+ fs_devices->discardable = true;

Is it expected that there is no = false assignment?

This is the list of btrfs filesystems I have:

  $ cat /proc/mounts | fgrep btrfs
  /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
  /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
  # skipped bind mounts

The device is:

  $ lspci | fgrep -i Solid
  01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)

Can you help me debug the source of discards storm?

Is it an expected discard storm?

Is it problematic for SSD life span?

Thank you!

#2 lore raw
Quoted 11 lines · Linux regression tracking #adding (Thorsten Leemhuis)
Quoted 9 lines · Sergei Trofimovich

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

Quoted 7 lines

Thanks for the report. To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot:

#regzbot ^introduced 63a7cb13071842
#regzbot title btrfs: DISCARD storm towards NVME device be it idle or not #regzbot ignore-activity

#regzbot monitor:
https://lore.kernel.org/all/CAHmG9huwQcQXvy3HS0OP9bKFxwUa3aQj9MXZCr74emn0U+efqQ@mail.gmail.com/
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

#3 lore raw
In reply to Linux regression tracking #adding (Thorsten Leemhuis)

[TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.]

Quoted 10 lines · Linux regression tracking #adding (Thorsten Leemhuis)
Quoted 8 lines · Sergei Trofimovich

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

Quoted 7 lines

Thanks for the report. To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot:

#regzbot ^introduced 63a7cb13071842
#regzbot title btrfs: DISCARD storm towards NVME device be it idle or not #regzbot ignore-activity

FWIW, the fix is in next now, regzbot just missed it due to a url encoding bug:

#regzbot monitor:
https://lore.kernel.org/linux-btrfs/[email protected]/T/#m21c1856ccc8fd24ccb6fbad7d9b54f0bae115d25
#regzbot fix: 2e55571fddf
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

#4 lore raw
In reply to Sergei Trofimovich
Quoted 81 lines · Sergei Trofimovich

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with perf: led flashes align with output of:

     # perf ftrace -a -T 'nvme_setup*' | cat
     kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

iotop shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

   $ git bisect good
   63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
   commit 63a7cb13071842966c1ce931edacbc23573aada5
   Author: David Sterba <[email protected]>
   Date:   Tue Jul 26 20:54:10 2022 +0200

btrfs: auto enable discard=async when possible

There's a request to automatically enable async discard for capable devices. We can do that, the async mode is designed to wait for larger freed extents and is not intrusive, with limits to iops, kbps or latency.

The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

The automatic selection is done if there's at least one discard capable device in the filesystem (not capable devices are skipped). Mounting with any other discard option will honor that option, notably mounting with nodiscard will keep it disabled.

fs/btrfs/ctree.h | 1 +
fs/btrfs/disk-io.c | 14 ++++++++++++++
fs/btrfs/super.c | 2 ++
fs/btrfs/volumes.c | 3 +++
fs/btrfs/volumes.h | 2 ++
5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But glancing at it this bit looks slightly off:

+ if (bdev_max_discard_sectors(bdev))
+ fs_devices->discardable = true;

Is it expected that there is no = false assignment?

This is the list of btrfs filesystems I have:

   $ cat /proc/mounts | fgrep btrfs
   /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
   /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
   # skipped bind mounts
Quoted 4 lines

The device is:

   $ lspci | fgrep -i Solid
   01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)

It is a SSD device with NVME interface, that needs regular discard. Why not try tune io intensity using

/sys/fs/btrfs/<uuid>/discard

options?

Maybe not all discardable sectors are not issued at once. It is a good idea to try with a fresh mkfs (which runs discard at mkfs) to see if discard is being issued even if there are no fs activities.

Thanks, Anand

#5 lore raw
Quoted 91 lines · On Thu, 2 Mar 2023 17:12:27 +0800 Anand Jain <[email protected]> wrote:
Quoted 81 lines · Sergei Trofimovich

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with perf: led flashes align with output of:

     # perf ftrace -a -T 'nvme_setup*' | cat
     kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

iotop shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

   $ git bisect good
   63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
   commit 63a7cb13071842966c1ce931edacbc23573aada5
   Author: David Sterba <[email protected]>
   Date:   Tue Jul 26 20:54:10 2022 +0200

btrfs: auto enable discard=async when possible

There's a request to automatically enable async discard for capable devices. We can do that, the async mode is designed to wait for larger freed extents and is not intrusive, with limits to iops, kbps or latency.

The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

The automatic selection is done if there's at least one discard capable device in the filesystem (not capable devices are skipped). Mounting with any other discard option will honor that option, notably mounting with nodiscard will keep it disabled.

fs/btrfs/ctree.h | 1 +
fs/btrfs/disk-io.c | 14 ++++++++++++++
fs/btrfs/super.c | 2 ++
fs/btrfs/volumes.c | 3 +++
fs/btrfs/volumes.h | 2 ++
5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But glancing at it this bit looks slightly off:

+ if (bdev_max_discard_sectors(bdev))
+ fs_devices->discardable = true;

Is it expected that there is no = false assignment?

This is the list of btrfs filesystems I have:

   $ cat /proc/mounts | fgrep btrfs
   /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
   /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
   # skipped bind mounts
Quoted 4 lines

The device is:

   $ lspci | fgrep -i Solid
   01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)  

It is a SSD device with NVME interface, that needs regular discard. Why not try tune io intensity using

Quoted 7 lines

/sys/fs/btrfs/<uuid>/discard

options?

Maybe not all discardable sectors are not issued at once. It is a good idea to try with a fresh mkfs (which runs discard at mkfs) to see if discard is being issued even if there are no fs activities.

Ah, thank you Anand! I poked a bit more in perf ftrace and I think I see a "slow" pass through the discard backlog:

/sys/fs/btrfs/<UUID>/discard$ cat iops_limit
10

Twice a minute I get a short burst of file creates/deletes that produces a bit of free space in many block groups. That enqueues hundreds of work items.

    $ sudo perf ftrace -a -T 'btrfs_discard_workfn' -T 'btrfs_issue_discard' -T 'btrfs_discard_queue_work'
     btrfs-transacti-407     [011]  42800.424027: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [011]  42800.424070: btrfs_discard_queue_work <-__btrfs_add_free_space
     ...
     btrfs-transacti-407     [011]  42800.425053: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [011]  42800.425055: btrfs_discard_queue_work <-__btrfs_add_free_space

193 entries of btrfs_discard_queue_work.
It took 1ms to enqueue all of the work into the workqueue.

     kworker/u64:1-2379115 [000]  42800.487010: btrfs_discard_workfn <-process_one_work
     kworker/u64:1-2379115 [000]  42800.487028: btrfs_issue_discard <-btrfs_discard_extent
     kworker/u64:1-2379115 [005]  42800.594010: btrfs_discard_workfn <-process_one_work
     kworker/u64:1-2379115 [005]  42800.594031: btrfs_issue_discard <-btrfs_discard_extent
     ...
     kworker/u64:15-2396822 [007]  42830.441487: btrfs_discard_workfn <-process_one_work
     kworker/u64:15-2396822 [007]  42830.441502: btrfs_issue_discard <-btrfs_discard_extent
     kworker/u64:15-2396822 [000]  42830.546497: btrfs_discard_workfn <-process_one_work
     kworker/u64:15-2396822 [000]  42830.546524: btrfs_issue_discard <-btrfs_discard_extent

286 pairs of btrfs_discard_workfn / btrfs_issue_discard. Each pair takes 10ms to process, which seems to match iops_limit=10. That means I can get about 300 discards per second max.

     btrfs-transacti-407     [002]  42830.634216: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [002]  42830.634228: btrfs_discard_queue_work <-__btrfs_add_free_space
     ...

Next transaction started 30 seconds later, which is a default commit interval.

My file system is of 512GB size. My guess I get about one discard entry per block group on each

Does my system keeps up with scheduled discard backlog? Can I peek at workqueue size?

Is iops_limit=10 a reasonable default for discard=async? It feels like for larger file systems it will not be enough even for this idle state.

Signature · 1 line
  Sergei
#6 lore raw

Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.

I added this to the tracking, but it seems nothing happened for nearly two weeks.

Sergei, did you find a workaround that works for you? Or is this something that other people might run into as well and thus better should be fixed?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

Quoted 152 lines · Sergei Trofimovich
Quoted 91 lines · On Thu, 2 Mar 2023 17:12:27 +0800 Anand Jain <[email protected]> wrote:
Quoted 81 lines · Sergei Trofimovich

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with perf: led flashes align with output of:

     # perf ftrace -a -T 'nvme_setup*' | cat
     kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

iotop shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

   $ git bisect good
   63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
   commit 63a7cb13071842966c1ce931edacbc23573aada5
   Author: David Sterba <[email protected]>
   Date:   Tue Jul 26 20:54:10 2022 +0200

btrfs: auto enable discard=async when possible

There's a request to automatically enable async discard for capable devices. We can do that, the async mode is designed to wait for larger freed extents and is not intrusive, with limits to iops, kbps or latency.

The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

The automatic selection is done if there's at least one discard capable device in the filesystem (not capable devices are skipped). Mounting with any other discard option will honor that option, notably mounting with nodiscard will keep it disabled.

fs/btrfs/ctree.h | 1 +
fs/btrfs/disk-io.c | 14 ++++++++++++++
fs/btrfs/super.c | 2 ++
fs/btrfs/volumes.c | 3 +++
fs/btrfs/volumes.h | 2 ++
5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But glancing at it this bit looks slightly off:

+ if (bdev_max_discard_sectors(bdev))
+ fs_devices->discardable = true;

Is it expected that there is no = false assignment?

This is the list of btrfs filesystems I have:

   $ cat /proc/mounts | fgrep btrfs
   /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
   /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
   # skipped bind mounts
Quoted 4 lines

The device is:

   $ lspci | fgrep -i Solid
   01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)  

It is a SSD device with NVME interface, that needs regular discard. Why not try tune io intensity using

Quoted 7 lines

/sys/fs/btrfs/<uuid>/discard

options?

Maybe not all discardable sectors are not issued at once. It is a good idea to try with a fresh mkfs (which runs discard at mkfs) to see if discard is being issued even if there are no fs activities.

Ah, thank you Anand! I poked a bit more in perf ftrace and I think I see a "slow" pass through the discard backlog:

/sys/fs/btrfs/<UUID>/discard$ cat iops_limit
10

Twice a minute I get a short burst of file creates/deletes that produces a bit of free space in many block groups. That enqueues hundreds of work items.

    $ sudo perf ftrace -a -T 'btrfs_discard_workfn' -T 'btrfs_issue_discard' -T 'btrfs_discard_queue_work'
     btrfs-transacti-407     [011]  42800.424027: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [011]  42800.424070: btrfs_discard_queue_work <-__btrfs_add_free_space
     ...
     btrfs-transacti-407     [011]  42800.425053: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [011]  42800.425055: btrfs_discard_queue_work <-__btrfs_add_free_space

193 entries of btrfs_discard_queue_work.
It took 1ms to enqueue all of the work into the workqueue.

     kworker/u64:1-2379115 [000]  42800.487010: btrfs_discard_workfn <-process_one_work
     kworker/u64:1-2379115 [000]  42800.487028: btrfs_issue_discard <-btrfs_discard_extent
     kworker/u64:1-2379115 [005]  42800.594010: btrfs_discard_workfn <-process_one_work
     kworker/u64:1-2379115 [005]  42800.594031: btrfs_issue_discard <-btrfs_discard_extent
     ...
     kworker/u64:15-2396822 [007]  42830.441487: btrfs_discard_workfn <-process_one_work
     kworker/u64:15-2396822 [007]  42830.441502: btrfs_issue_discard <-btrfs_discard_extent
     kworker/u64:15-2396822 [000]  42830.546497: btrfs_discard_workfn <-process_one_work
     kworker/u64:15-2396822 [000]  42830.546524: btrfs_issue_discard <-btrfs_discard_extent

286 pairs of btrfs_discard_workfn / btrfs_issue_discard. Each pair takes 10ms to process, which seems to match iops_limit=10. That means I can get about 300 discards per second max.

     btrfs-transacti-407     [002]  42830.634216: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [002]  42830.634228: btrfs_discard_queue_work <-__btrfs_add_free_space
     ...

Next transaction started 30 seconds later, which is a default commit interval.

My file system is of 512GB size. My guess I get about one discard entry per block group on each

Does my system keeps up with scheduled discard backlog? Can I peek at workqueue size?

Is iops_limit=10 a reasonable default for discard=async? It feels like for larger file systems it will not be enough even for this idle state.

#7 lore raw
Quoted 9 lines · On Wed, 15 Mar 2023 12:44:34 +0100 "Linux regression tracking (Thorsten…

Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone.

I added this to the tracking, but it seems nothing happened for nearly two weeks.

Sergei, did you find a workaround that works for you? Or is this something that other people might run into as well and thus better should be fixed?

I used the workaround of cranking up IOPS of discard from 10 to 1000:

    # echo 1000 > /sys//fs/btrfs/<UUID>/discard/iops_limit

But I am not sure if it's a safe or a reasonable fix. I would prefer someone from btrfs to comment if it's an expected kernel's behaviour.

Quoted 161 lines

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

Quoted 152 lines · Sergei Trofimovich
Quoted 91 lines · On Thu, 2 Mar 2023 17:12:27 +0800 Anand Jain <[email protected]> wrote:
Quoted 81 lines · Sergei Trofimovich

Hi btrfs maintainers!

Tl;DR:

After 63a7cb13071842 "btrfs: auto enable discard=async when possible" I see constant DISCARD storm towards my NVME device be it idle or not.

No storm: v6.1 and older
Has storm: v6.2 and newer

More words:

After upgrade from 6.1 to 6.2 I noticed that Disk led on my desktop started flashing incessantly regardless of present or absent workload.

I think I confirmed the storm with perf: led flashes align with output of:

     # perf ftrace -a -T 'nvme_setup*' | cat
     kworker/6:1H-298     [006]   2569.645201: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.645205: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.749198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.749202: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.853204: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.853209: nvme_setup_discard <-nvme_setup_cmd
     kworker/6:1H-298     [006]   2569.958198: nvme_setup_cmd <-nvme_queue_rq
     kworker/6:1H-298     [006]   2569.958202: nvme_setup_discard <-nvme_setup_cmd

iotop shows no read/write IO at all (expected).

I was able to bisect it down to this commit:

   $ git bisect good
   63a7cb13071842966c1ce931edacbc23573aada5 is the first bad commit
   commit 63a7cb13071842966c1ce931edacbc23573aada5
   Author: David Sterba <[email protected]>
   Date:   Tue Jul 26 20:54:10 2022 +0200

btrfs: auto enable discard=async when possible

There's a request to automatically enable async discard for capable devices. We can do that, the async mode is designed to wait for larger freed extents and is not intrusive, with limits to iops, kbps or latency.

The status and tunables will be exported in /sys/fs/btrfs/FSID/discard .

The automatic selection is done if there's at least one discard capable device in the filesystem (not capable devices are skipped). Mounting with any other discard option will honor that option, notably mounting with nodiscard will keep it disabled.

fs/btrfs/ctree.h | 1 +
fs/btrfs/disk-io.c | 14 ++++++++++++++
fs/btrfs/super.c | 2 ++
fs/btrfs/volumes.c | 3 +++
fs/btrfs/volumes.h | 2 ++
5 files changed, 22 insertions(+)

Is this storm a known issue? I did not dig too much into the patch. But glancing at it this bit looks slightly off:

+ if (bdev_max_discard_sectors(bdev))
+ fs_devices->discardable = true;

Is it expected that there is no = false assignment?

This is the list of btrfs filesystems I have:

   $ cat /proc/mounts | fgrep btrfs
   /dev/nvme0n1p3 / btrfs rw,noatime,compress=zstd:3,ssd,space_cache,subvolid=848,subvol=/nixos 0 0
   /dev/sda3 /mnt/archive btrfs rw,noatime,compress=zstd:3,space_cache,subvolid=5,subvol=/ 0 0
   # skipped bind mounts
Quoted 4 lines

The device is:

   $ lspci | fgrep -i Solid
   01:00.0 Non-Volatile memory controller: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive (rev 03)    

It is a SSD device with NVME interface, that needs regular discard. Why not try tune io intensity using

Quoted 7 lines

/sys/fs/btrfs/<uuid>/discard

options?

Maybe not all discardable sectors are not issued at once. It is a good idea to try with a fresh mkfs (which runs discard at mkfs) to see if discard is being issued even if there are no fs activities.

Ah, thank you Anand! I poked a bit more in perf ftrace and I think I see a "slow" pass through the discard backlog:

/sys/fs/btrfs/<UUID>/discard$ cat iops_limit
10

Twice a minute I get a short burst of file creates/deletes that produces a bit of free space in many block groups. That enqueues hundreds of work items.

    $ sudo perf ftrace -a -T 'btrfs_discard_workfn' -T 'btrfs_issue_discard' -T 'btrfs_discard_queue_work'
     btrfs-transacti-407     [011]  42800.424027: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [011]  42800.424070: btrfs_discard_queue_work <-__btrfs_add_free_space
     ...
     btrfs-transacti-407     [011]  42800.425053: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [011]  42800.425055: btrfs_discard_queue_work <-__btrfs_add_free_space

193 entries of btrfs_discard_queue_work.
It took 1ms to enqueue all of the work into the workqueue.

     kworker/u64:1-2379115 [000]  42800.487010: btrfs_discard_workfn <-process_one_work
     kworker/u64:1-2379115 [000]  42800.487028: btrfs_issue_discard <-btrfs_discard_extent
     kworker/u64:1-2379115 [005]  42800.594010: btrfs_discard_workfn <-process_one_work
     kworker/u64:1-2379115 [005]  42800.594031: btrfs_issue_discard <-btrfs_discard_extent
     ...
     kworker/u64:15-2396822 [007]  42830.441487: btrfs_discard_workfn <-process_one_work
     kworker/u64:15-2396822 [007]  42830.441502: btrfs_issue_discard <-btrfs_discard_extent
     kworker/u64:15-2396822 [000]  42830.546497: btrfs_discard_workfn <-process_one_work
     kworker/u64:15-2396822 [000]  42830.546524: btrfs_issue_discard <-btrfs_discard_extent

286 pairs of btrfs_discard_workfn / btrfs_issue_discard. Each pair takes 10ms to process, which seems to match iops_limit=10. That means I can get about 300 discards per second max.

     btrfs-transacti-407     [002]  42830.634216: btrfs_discard_queue_work <-__btrfs_add_free_space
     btrfs-transacti-407     [002]  42830.634228: btrfs_discard_queue_work <-__btrfs_add_free_space
     ...

Next transaction started 30 seconds later, which is a default commit interval.

My file system is of 512GB size. My guess I get about one discard entry per block group on each

Does my system keeps up with scheduled discard backlog? Can I peek at workqueue size?

Is iops_limit=10 a reasonable default for discard=async? It feels like for larger file systems it will not be enough even for this idle state.

Signature · 1 line
  Sergei