BTRFS free space handling still needs more work: Hangs again

It currently is here:

merkaba:~> btrfs balance status /home
Balance on '/home' is running
32 out of about 164 chunks balanced (53 considered), 80% left

merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=142.10GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.33GiB GlobalReserve, single: total=512.00MiB, used=254.31MiB

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599

�ޙ����&�)ߡ�a�����G���h��j:+v���w��٥

And I wonder about:

Quoted 4 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599

Quoted 2 lines

�ޙ����&�)ߡ�a�����G���h��j:+v���w��٥

These random chars are not supposed to be there: I better run scrub straight after this balance.

Am Freitag, 26. Dezember 2014, 15:20:42 schrieben Sie:

Quoted 7 lines

And I wonder about:

Quoted 5 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599

84C7N�����r��y����b�X��ǧv�^�)޺{.n�+����{�n�߲)����w*jg��������ݢj/

Quoted 5 lines

Quoted 2 lines

�ޙ����&�)ߡ�a�����G���h��j:+v���w��٥

These random chars are not supposed to be there: I better run scrub straight after this balance.

Okay, thats not me I think. scrub didn´t report any errors and when I look in kmail send folder I don´t see these random chars as well, so it seems some server on the wire added the garbage.

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and compress=lzo:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a
Total devices 2 FS bytes used 144.41GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only run the VM behind a firewall).

Dec 26 16:17:57 merkaba kernel: [ 8102.029438] mce:
[Hardware Error]: Machine check events logged
Dec 26 16:20:27 merkaba kernel: [ 8252.054015] mce:

[Hardware Error]: Machine check events logged

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Second: At a time you feel like it, here is a little rant, but also a bug report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and compress=lzo:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a
Total devices 2 FS bytes used 144.41GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only run the VM behind a firewall).

And thus I try the balance dance again:

merkaba:~> btrfs balance start -dusage=5 -musage=5 /home
ERROR: error during balancing '/home' - No space left on device

There may be more info in syslog - try dmesg | tail merkaba:~#1> btrfs balance start -dusage=5 -musage=5 /home
ERROR: error during balancing '/home' - No space left on device There may be more info in syslog - try dmesg | tail merkaba:~#1> btrfs balance start -dusage=5 /home

.... losts deleted for brevity ....

So I am rebalancing everything basically, without need I bet, so causing more churn to SSDs than is needed.

Otherwise alternative would be to make BTRFS larger I bet.

Well this is still not what I would consider stable. So I will still

recommend: If you want to use BTRFS on a server and estimate 25 GiB of usage, make drive at least 50GiB big or even 100GiB to be on the safe side. Like I recommended for SLES 11 SP 2/3 BTRFS deployments – but hey, there say meanwhile "don´t" as in "just don´t use it at all and use SLES 12 instead, cause BTRFS with 3.0 kernel with a ton of snapper snapshots is really not asking for anything even near to production or enterprise reliability" (if you need proof, I think I still have a snapshot of a SLES 11 SP3 VM that broke over night due to me having installed an LDAP server for preparing some training slides). Even 3.12 kernel seems daring regarding BTRFS, unless SUSE actively backports fixes.

In kernel log the failed attempts look like this:

[  209.783437] BTRFS info (device dm-3): relocating block group 501238202368 flags 17
[  210.116416] BTRFS info (device dm-3): relocating block group 501238202368 flags 17

My expection for a *stable* and *production quality* filesystem would be:

I never ever get hangs with one kworker running on 100% of one Sandybridge core *for minutes* in a production filesystem and thats about it.

Especially for a filesystem that claims to still have a good amount of free space:

merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  160G  146G   25G  86% /home

(yeah, these don´t add up, I account this to compression, but hey, who knows)

In kernel log I have things like this, but some earlier time and these I have not yet perceived as hangs:

Dec 23 23:33:26 merkaba kernel: [23040.621678] ------------[ cut here ]------------
Dec 23 23:33:26 merkaba kernel: [23040.621792] WARNING: CPU: 3 PID: 308 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empt
y+0x2d/0x2f [btrfs]()
Dec 23 23:33:26 merkaba kernel: [23040.621796] Modules linked in: mmc_block ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c snd

_usb_audio snd_usbmidi_lib snd_rawmidi snd_seq_device hid_generic hid_pl ff_memless usbhid hid nls_utf8 nls_cp437 vfat fat uas usb_stor age bnep bluetooth binfmt_misc cpufreq_userspace cpufreq_stats pci_stub cpufreq_powersave vboxpci(O) cpufreq_conservative vboxnetadp(O)
vboxnetflt(O) vboxdrv(O) ext4 crc16 mbcache jbd2 intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32
_pclmul ghash_clmulni_intel snd_hda_codec_hdmi iwldvm aesni_intel snd_hda_codec_conexant mac80211 aes_x86_64 snd_hda_codec_generic lrw gf128mul glue_helper ablk_helper cryptd psmouse snd_hda_intel serio_raw iwlwifi pcspkr lpc_ich snd_hda_controller i2c_i801 mfd_core snd _hda_codec snd_hwdep cfg80211 snd_pcm snd_timer shpchp thinkpad_acpi nvram snd soundcore rfkill battery ac tpm_tis tpm processor evdev joydev sbs sbshc coretemp hdaps(O) tp_smapi(O) thinkpad_ec(O) loop firewire_sbp2 fuse ecryptfs autofs4 md_mod btrfs xor raid6_pq microc ode dm_mirror dm_region_hash dm_log dm_mod sg sr_mod sd_mod cdrom crc32c_intel ahci firewire_ohci libahci sata_sil24 e1000e libata ptp sdhci_pci ehci_pci sdhci firewire_core ehci_hcd crc_itu_t pps_core mmc_core scsi_mod usbcore usb_common thermal

Dec 23 23:33:26 merkaba kernel: [23040.621978] CPU: 3 PID: 308 Comm: btrfs-transacti Tainted: G        W  O   3.18.0-tp520 #14
Dec 23 23:33:26 merkaba kernel: [23040.621982] Hardware name: LENOVO 42433WG/42433WG, BIOS 8AET63WW (1.43 ) 05/08/2013
Dec 23 23:33:26 merkaba kernel: [23040.621985]  0000000000000009 ffff8804044c7d88 ffffffff814a516e 0000000080000000
Dec 23 23:33:26 merkaba kernel: [23040.621992]  0000000000000000 ffff8804044c7dc8 ffffffff8103f83e ffff8804044c7db8
Dec 23 23:33:26 merkaba kernel: [23040.621999]  ffffffffc04bd5a1 ffff880037590800 ffff8800a599c320 0000000000000000
Dec 23 23:33:26 merkaba kernel: [23040.622006] Call Trace:
Dec 23 23:33:26 merkaba kernel: [23040.622026]  [<ffffffff814a516e>] dump_stack+0x4f/0x7c
Dec 23 23:33:26 merkaba kernel: [23040.622034]  [<ffffffff8103f83e>] warn_slowpath_common+0x7c/0x96
Dec 23 23:33:26 merkaba kernel: [23040.622104]  [<ffffffffc04bd5a1>] ? btrfs_assert_delayed_root_empty+0x2d/0x2f [btrfs]
Dec 23 23:33:26 merkaba kernel: [23040.622111]  [<ffffffff8103f8ec>] warn_slowpath_null+0x15/0x17
Dec 23 23:33:26 merkaba kernel: [23040.622164]  [<ffffffffc04bd5a1>] btrfs_assert_delayed_root_empty+0x2d/0x2f [btrfs]
Dec 23 23:33:26 merkaba kernel: [23040.622211]  [<ffffffffc047a830>] btrfs_commit_transaction+0x394/0x8bc [btrfs]
Dec 23 23:33:26 merkaba kernel: [23040.622254]  [<ffffffffc0476dd5>] transaction_kthread+0xf9/0x1af [btrfs]

Dec 23 23:33:26 merkaba kernel: [23040.622295]  [<ffffffffc0476cdc>] ? btrfs_cleanup_transaction+0x43a/0x43a [btrfs]
Dec 23 23:33:26 merkaba kernel: [23040.622305]  [<ffffffff8105697c>] kthread+0xb2/0xba
Dec 23 23:33:26 merkaba kernel: [23040.622312]  [<ffffffff814a0000>] ? dcbnl_newmsg+0x14/0xa8
Dec 23 23:33:26 merkaba kernel: [23040.622317]  [<ffffffff810568ca>] ? __kthread_parkme+0x62/0x62
Dec 23 23:33:26 merkaba kernel: [23040.622324]  [<ffffffff814a9f6c>] ret_from_fork+0x7c/0xb0
Dec 23 23:33:26 merkaba kernel: [23040.622329]  [<ffffffff810568ca>] ? __kthread_parkme+0x62/0x62
Dec 23 23:33:26 merkaba kernel: [23040.622334] ---[ end trace 90db5b1c7067cf1d ]---
Dec 23 23:33:56 merkaba kernel: [23070.671999] ------------[ cut here ]------------

Dec 23 23:33:56 merkaba kernel: [23070.671999] ------------[ cut here ]------------
Dec 23 23:33:56 merkaba kernel: [23070.672064] WARNING: CPU: 3 PID: 308 at fs/btrfs/delayed-inode.c:1410 btrfs_assert_delayed_root_empt
y+0x2d/0x2f [btrfs]()
Dec 23 23:33:56 merkaba kernel: [23070.672067] Modules linked in: mmc_block ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c snd_usb_audio snd_usbmidi_lib snd_rawmidi snd_seq_device hid_generic hid_pl ff_memless usbhid hid nls_utf8 nls_cp437 vfat fat uas usb_storage bnep bluetooth binfmt_misc cpufreq_userspace cpufreq_stats pci_stub cpufreq_powersave vboxpci(O) cpufreq_conservative vboxnetadp(O) vboxnetflt(O) vboxdrv(O) ext4 crc16 mbcache jbd2 intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi iwldvm aesni_intel snd_hda_codec_conexant mac80211 aes_x86_64 snd_hda_codec_generic lrw gf128mul glue_helper ablk_helper cryptd psmouse snd_hda_intel serio_raw iwlwifi pcspkr lpc_ich snd_hda_controller i2c_i801 mfd_core snd_hda_codec snd_hwdep cfg80211 snd_pcm snd_timer shpchp thinkpad_acpi nvram snd soundcore rfkill battery ac tpm_tis tpm processor evdev joydev sbs sbshc coretemp hdaps(O) tp_smapi(O) think

Dec 23 23:33:56 merkaba kernel: [23070.672193] CPU: 3 PID: 308 Comm: btrfs-transacti Tainted: G        W  O   3.18.0-tp520 #14
Dec 23 23:33:56 merkaba kernel: [23070.672196] Hardware name: LENOVO 42433WG/42433WG, BIOS 8AET63WW (1.43 ) 05/08/2013
Dec 23 23:33:56 merkaba kernel: [23070.672200]  0000000000000009 ffff8804044c7d88 ffffffff814a516e 0000000080000000
Dec 23 23:33:56 merkaba kernel: [23070.672205]  0000000000000000 ffff8804044c7dc8 ffffffff8103f83e ffff8804044c7db8
Dec 23 23:33:56 merkaba kernel: [23070.672209]  ffffffffc04bd5a1 ffff880037590800 ffff8802cd6e50a0 0000000000000000
Dec 23 23:33:56 merkaba kernel: [23070.672214] Call Trace:
Dec 23 23:33:56 merkaba kernel: [23070.672222]  [<ffffffff814a516e>] dump_stack+0x4f/0x7c
Dec 23 23:33:56 merkaba kernel: [23070.672229]  [<ffffffff8103f83e>] warn_slowpath_common+0x7c/0x96
Dec 23 23:33:56 merkaba kernel: [23070.672264]  [<ffffffffc04bd5a1>] ? btrfs_assert_delayed_root_empty+0x2d/0x2f [btrfs]
Dec 23 23:33:56 merkaba kernel: [23070.672270]  [<ffffffff8103f8ec>] warn_slowpath_null+0x15/0x17
Dec 23 23:33:56 merkaba kernel: [23070.672301]  [<ffffffffc04bd5a1>] btrfs_assert_delayed_root_empty+0x2d/0x2f [btrfs]
Dec 23 23:33:56 merkaba kernel: [23070.672330]  [<ffffffffc047a830>] btrfs_commit_transaction+0x394/0x8bc [btrfs]
Dec 23 23:33:56 merkaba kernel: [23070.672357]  [<ffffffffc0476dd5>] transaction_kthread+0xf9/0x1af [btrfs]
Dec 23 23:33:56 merkaba kernel: [23070.672383]  [<ffffffffc0476cdc>] ? btrfs_cleanup_transaction+0x43a/0x43a [btrfs]
Dec 23 23:33:56 merkaba kernel: [23070.672389]  [<ffffffff8105697c>] kthread+0xb2/0xba
Dec 23 23:33:56 merkaba kernel: [23070.672395]  [<ffffffff814a0000>] ? dcbnl_newmsg+0x14/0xa8
Dec 23 23:33:56 merkaba kernel: [23070.672399]  [<ffffffff810568ca>] ? __kthread_parkme+0x62/0x62
Dec 23 23:33:56 merkaba kernel: [23070.672405]  [<ffffffff814a9f6c>] ret_from_fork+0x7c/0xb0
Dec 23 23:33:56 merkaba kernel: [23070.672409]  [<ffffffff810568ca>] ? __kthread_parkme+0x62/0x62
Dec 23 23:33:56 merkaba kernel: [23070.672412] ---[ end trace 90db5b1c7067cf1e ]---
Dec 23 23:34:26 merkaba kernel: [23100.709530] ------------[ cut here ]------------

The recent hangings today are not in the log, I was upset enough to forcefully switch of the machine. Tax returns are not my all time favorite, but tax returns with hanging filesystems is no fun at all.

I will upgrade to 3.19 with 3.19-rc2.

Lets see what this balance will do.

It currently is here:

merkaba:~> btrfs balance status /home
Balance on '/home' is running
32 out of about 164 chunks balanced (53 considered), 80% left

merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=142.10GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.33GiB GlobalReserve, single: total=512.00MiB, used=254.31MiB

So for once, we are told not to balance needlessly, but then in order for stable operation I need to balance nonetheless?

Well lets see how it will improve things. Last time it did. Considerably. BTRFS only had these hang problems with 3.15 and 3.16 if trees allocated all remaining space. So I expect it to downsize these trees are to there is some device space being freed to allocatable again.

Next I will also defrag the Windows VM image just as an additional safety net.

Okay, doing something else now as the BTRFS will sort things out hopefully.

Ciao,

ITEM: An SSD plus a good fast controller and default system virtual memory and disk scheduler activities can completely bog a system down. You can get into a mode where the system begins doing synchronous writes of vast expanses of dirty cache. The SSD is so fast that there is effectively zero "wait for IO time" and the IO subsystem is effectively locked or just plain busy.

Look at /proc/sys/vm/dirty_background_ratio which is probably set to 10% of system ram.

You may need/want to change this number to something closer to 4. That's not a hard suggestion. Some reading and analysis will be needed to find the best possible tuning for an advanced system.

Quoted 11 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Second: At a time you feel like it, here is a little rant, but also a bug report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 8 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

Quoted 2 lines

merkaba:~> btrfs balance start -dusage=5 -musage=5 /home
ERROR: error during balancing '/home' - No space left on device

ITEM: Running out of space during a balance is not running out of space for files. BTRFS has two layers of allocation. That is, there are two levels of abstraction where "no space" can occur.

The first level of allocation is the "making more BTRFS structures out

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 51 lines · Martin Steigerwald

Quoted 11 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Second: At a time you feel like it, here is a little rant, but also a bug report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 8 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

This may be a bug, but this is what I see.

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Quoted 58 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 56 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

Quoted 2 lines

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 7 lines

This may be a bug, but this is what I see.

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space
usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 59 lines · Martin Steigerwald

Quoted 58 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 56 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

          Total devices 2 FS bytes used 144.41GiB
          devid    1 size 160.00GiB used 160.00GiB path
          /dev/mapper/msata-home
          devid    2 size 160.00GiB used 160.00GiB path
          /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

Quoted 6 lines

Quoted 2 lines

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Ok, let me rephrase that: Then the space *reserved* for the trees occupies all space on the device. Or okay, when that I see in btrfs fi df as "total" in summary occupies what I see as "size" in btrfs fi sh, i.e. when "used" equals space in "btrfs fi sh"

What happened here is this:

I tried

https://blogs.oracle.com/virtualbox/entry/how_to_compact_your_virtual

in order to regain some space from the Windows XP VDI file. I just wanted to get around upsizing the BTRFS again.

And on the defragementation step in Windows it first ran fast. For about 46-47% there, during that fast phase btrfs fi df showed that BTRFS was quickly reserving the remaining free device space for data trees (not metadata).

Only after a while after it did so, it got slow again, basically the Windows defragmentation process stopped at 46-47% altogether and then after a while even the desktop locked due to processes being blocked in I/O.

I decided to forget about this downsizing of the Virtualbox VDI file, it will extend again on next Windows work and it is already 18 GB of its maximum 20GB, so… I dislike the approach anyway, and don´t even understand why the defragmentation step would be necessary as I think Virtualbox can poke holes into the file for any space not allocated inside the VM, whether it is defragmented or not.

Quoted 5 lines

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

The *only* person? The compression lockups with 3.15 and 3.16, quite some people saw them, I thought. For me also these lockups only happened with all space on device allocated.

And these seem to be gone. In regular use it doesn´t lockup totally hard. But in the a processes writes a lot into one big no-cowed file case, it seems it can still get into a lockup, but this time one where a kworker thread consumes 100% of CPU for minutes.

Quoted 6 lines

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Okay, I thought it is the space BTRFS reserves for a tree or well the *chunks* the tree manages. I am aware of that it isn´t already *used* space, its just *reserved*

Quoted 8 lines

Quoted 7 lines

This may be a bug, but this is what I see.

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Quoted 6 lines

Duncan's assertion is correct in its detail. Looking at your space
usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

It was Robert writing this I think.

Well I do not like to balance the FS, but I see the result, I see that it helps here. And thats about it.

My theory from watching the Windows XP defragmentation case is this:

- For writing into the file BTRFS needs to actually allocate and use free space in the current tree allocation, or, as we seem to misunderstood from the words we use, it needs to fit data in

Data, RAID1: total=144.98GiB, used=140.94GiB

between 144,98 GiB and 140,94 GiB given that total space of this tree, or if its not a tree, but the chunks in that the tree manages, in these chunks can *not* be extended anymore.

System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=5.00GiB, used=3.24GiB

- What I see now is as long as it can be extended, BTRFS on this workload *happily* does so. *Quickly*. Up to the amount of the free, unreserved space of the device. And *even* if in my eyes there is a big enough difference between total and used in btrfs fi df.

- Then as all the device space is *reserved*, BTRFS needs to fit the allocation within the *existing* chunks instead of reserving a new one and fill the empty one. And I think this is where it gets problems.

I extended both devices of /home by 10 GiB now and I was able to comlete some balance steps with these results.

Original after my last partly failed balance attempts:

Label: 'home' uuid: […]
Total devices 2 FS bytes used 144.20GiB
devid 1 size 170.00GiB used 159.01GiB path /dev/mapper/msata-home
devid 2 size 170.00GiB used 159.01GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=153.98GiB, used=140.95GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.25GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Then balancing, but not all of them:

merkaba:~#1> btrfs balance start -dusage=70 /home Done, had to relocate 9 out of 162 chunks merkaba:~> btrfs fi df /home
Data, RAID1: total=146.98GiB, used=140.95GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.25GiB GlobalReserve, single: total=512.00MiB, used=0.00B merkaba:~> btrfs balance start -dusage=80 /home Done, had to relocate 9 out of 155 chunks merkaba:~> btrfs fi df /home
Data, RAID1: total=144.98GiB, used=140.94GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.24GiB GlobalReserve, single: total=512.00MiB, used=0.00B merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 144.19GiB
devid 1 size 170.00GiB used 150.01GiB path /dev/mapper/msata-home
devid 2 size 170.00GiB used 150.01GiB path /dev/mapper/sata-home

Btrfs v3.17

This is a situation where I do not see any slowdowns with BTRFS.

As far as I understand the balance commands I used I told BTRFS the following:

- go and balance all chunks that has 70% or less used
- go and balance all chunks that have 80% or less used

I rarely see any chunks that have 60% or less used and get something like this if I try:

merkaba:~> btrfs balance start -dusage=60 /home
Done, had to relocate 0 out of 153 chunks

Now my idea is this: BTRFS will need to satisfy the allocations it need to do for writing heavily into a cow´ed file from the already reserved space. Yet if I have lots of chunks that are filled between 60-70% it needs to spread the allocations in the 40-30% of the chunk that are not yet used.

My theory is this: If BTRFS needs to do this *heavily*, it at some time gets problems while doing so. Apparently it seems *easier* to just reserve a new chunk and fill the fresh chunk then. Otherwise I don´t know why BTRFS is doing it like this. It prefers to reserve free device space on this defragmentation inside VM then.

And these issues may be due to an inefficient implementation or bug.

Now if no one else if ever having this, this may be a speciality with my filesystem and heck I can recreate it from scratch if need be. Yet I would prefer to find out what is happening here.

Quoted 6 lines

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Thanks, I think this seems to be a way to go.

Actually the logging should be safe I´d say, cause it wents into a different BTRFS. The BTRFS for /, which is also a RAID 1 and which didn´t show this behavior yet, although it has also all space reserved since quite some time:

merkaba:~> btrfs fi sh /
Label: 'debian' uuid: […]
Total devices 2 FS bytes used 17.79GiB
devid 1 size 30.00GiB used 30.00GiB path /dev/mapper/sata-debian
devid 2 size 30.00GiB used 30.00GiB path /dev/mapper/msata-debian

Btrfs v3.17
merkaba:~> btrfs fi df /
Data, RAID1: total=27.99GiB, used=17.21GiB System, RAID1: total=8.00MiB, used=16.00KiB Metadata, RAID1: total=2.00GiB, used=596.12MiB GlobalReserve, single: total=208.00MiB, used=0.00B

*Unlike* if one BTRFS locks up the other will also lock up, logging should be safe.

Actually I got the last task hung messages as I posted them here. So I may just try to reproduce this and trigger

echo "t" > /proc/sysrq-trigger

this gives

[32459.707323] systemd-journald[314]: /dev/kmsg buffer overrun, some messages

lost.

but I bet rsyslog will capture it just nice. I may even disable journald to reduce writes to / during reproducing the bug.

Ciao,

Quoted 11 lines

My theory from watching the Windows XP defragmentation case is this:

- For writing into the file BTRFS needs to actually allocate and use free space in the current tree allocation, or, as we seem to misunderstood from the words we use, it needs to fit data in

Data, RAID1: total=144.98GiB, used=140.94GiB

between 144,98 GiB and 140,94 GiB given that total space of this tree, or if its not a tree, but the chunks in that the tree manages, in these chunks can *not* be extended anymore.

If your file was actually COW (and you have _not_ been taking snapshots) then there is no extenting to be had. But if you are using snapper (which I believe you mentioned previously) then the snapshots cause a write boundary and a layer of copying. Frequently taking snapshots of a COW file is self defeating. If you are going to take snapshots then you might as well turn copy on write back on and, for the love of pete, stop defragging things.

Am Samstag, 27. Dezember 2014, 03:52:56 schrieb Robert White:

Quoted 12 lines

Quoted 11 lines

My theory from watching the Windows XP defragmentation case is this:

- For writing into the file BTRFS needs to actually allocate and use free space in the current tree allocation, or, as we seem to misunderstood from the words we use, it needs to fit data in

Data, RAID1: total=144.98GiB, used=140.94GiB

between 144,98 GiB and 140,94 GiB given that total space of this tree, or if its not a tree, but the chunks in that the tree manages, in these chunks can *not* be extended anymore.

Quoted 7 lines

If your file was actually COW (and you have _not_ been taking snapshots) then there is no extenting to be had. But if you are using snapper (which I believe you mentioned previously) then the snapshots cause a write boundary and a layer of copying. Frequently taking snapshots of a COW file is self defeating. If you are going to take snapshots then you might as well turn copy on write back on and, for the love of pete, stop defragging things.

I don´t use any snapshots on the filesystems. None, zero, zilch, nada.

And as I understand it copy on write means: It has to write the new write requests to somewhere else. For this it needs to allocate space. Either withing existing chunks or in a newly allocated one.

So for COW when writing to a file it will always need to allocate new space (although it can forget about the old space afterwards unless there isn´t a snapshot holding it)

Anyway, I got it reproduced. And am about to write a lengthy mail about.

It can easily be reproduced without even using Virtualbox, just by a nice simple fio job.

Quoted 1 line

Anyway, I got it reproduced. And am about to write a lengthy mail about.

Have fun with that lengthy email, but the devs already know about the data waste profile of the system. They just don't have a good solution yet.

Practical use cases involving _not_ defragging and _not_ packing files, or disabling COW and using raw image formats for VM disk storage are, meanwhile, also well understood.

It can easily be reproduced without even using Virtualbox, just by a nice simple fio job.

Quoted 2 lines

It can easily be reproduced without even using Virtualbox, just by a nice simple fio job.

TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file...

#!/bin/bash

# not tested, so correct any syntax errors
typeset -i counter
for ((counter=250;counter>0;counter--)); do
  dd if=/dev/urandom of=/some/file bs=4k count=$counter
done
exit

Each pass over /some/file is 4k shorter than the previous one, but none of the extents can be deallocated. File will be 1MiB in size and usage will be something like 125.5MiB (if I've done the math correctly). larger values of counter will result in exponentially larger amounts of waste.

Am Samstag, 27. Dezember 2014, 06:00:48 schrieb Robert White:

Quoted 19 lines · Martin Steigerwald

Quoted 3 lines

It can easily be reproduced without even using Virtualbox, just by a nice
simple fio job.

TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file...

#!/bin/bash

# not tested, so correct any syntax errors
typeset -i counter
for ((counter=250;counter>0;counter--)); do

dd if=/dev/urandom of=/some/file bs=4k count=$counter

done
exit

Quoted 5 lines

Each pass over /some/file is 4k shorter than the previous one, but none of the extents can be deallocated. File will be 1MiB in size and usage will be something like 125.5MiB (if I've done the math correctly). larger values of counter will result in exponentially larger amounts of waste.

Robert, I experienced this hang issues even before the defragmenting case. It happened while just installed a 400 MiB tax returns application to it (that is no joke, it is that big).

It happens while just using the VM.

Yes, I recommend not to use BTRFS for any VM image or any larger database on rotating storage for exactly that COW semantics.

But on SSD?

Its busy looping a CPU core and while the flash is basically idling.

I refuse to believe that this is by design.

I do think there is a *bug*.

Either acknowledge it and try to fix it, or say its by design *without even looking at it closely enough to be sure that it is not a bug* and limit your own possibilities by it.

I´d rather see it treated as a bug for now.

Come on, 254 IOPS on a filesystem with still 17 GiB of free space while randomly writing to a 4 GiB file.

People do these kind of things. Ditch that defrag Windows XP VM case, I had performance issue even before by just installing things to it. Databases, VMs, emulators. And heck even while just *creating* the file with fio as I shown.

Am Samstag, 27. Dezember 2014, 15:14:05 schrieb Martin Steigerwald:

Quoted 49 lines

Am Samstag, 27. Dezember 2014, 06:00:48 schrieb Robert White:

Quoted 19 lines · Martin Steigerwald

Quoted 3 lines

It can easily be reproduced without even using Virtualbox, just by a nice
simple fio job.

TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file...

#!/bin/bash

# not tested, so correct any syntax errors
typeset -i counter
for ((counter=250;counter>0;counter--)); do

dd if=/dev/urandom of=/some/file bs=4k count=$counter

done
exit

Quoted 5 lines

Each pass over /some/file is 4k shorter than the previous one, but none of the extents can be deallocated. File will be 1MiB in size and usage will be something like 125.5MiB (if I've done the math correctly). larger values of counter will result in exponentially larger amounts of waste.

Robert, I experienced this hang issues even before the defragmenting case. It happened while just installed a 400 MiB tax returns application to it (that is no joke, it is that big).

It happens while just using the VM.

Yes, I recommend not to use BTRFS for any VM image or any larger database on rotating storage for exactly that COW semantics.

But on SSD?

Its busy looping a CPU core and while the flash is basically idling.

I refuse to believe that this is by design.

I do think there is a *bug*.

Either acknowledge it and try to fix it, or say its by design *without even looking at it closely enough to be sure that it is not a bug* and limit your own possibilities by it.

I´d rather see it treated as a bug for now.

Quoted 7 lines

Come on, 254 IOPS on a filesystem with still 17 GiB of free space while randomly writing to a 4 GiB file.

People do these kind of things. Ditch that defrag Windows XP VM case, I had performance issue even before by just installing things to it. Databases, VMs, emulators. And heck even while just *creating* the file with fio as I shown.

Add to these use cases things like this:

martin@merkaba:~/.local/share/akonadi/db_data/akonadi> ls -lSh | head -5 insgesamt 2,2G

(quoted diff) +0 -4

-rw-rw---- 1 martin martin 1,7G Dez 27 15:17 parttable.ibd
-rw-rw---- 1 martin martin 488M Dez 27 15:17 pimitemtable.ibd
-rw-rw---- 1 martin martin  23M Dez 27 15:17 pimitemflagrelation.ibd
-rw-rw---- 1 martin martin 240K Dez 27 15:17 collectiontable.ibd

Or this:

martin@merkaba:~/.local/share/baloo> du -sch * | sort -rh

9,2G    insgesamt
8,0G    email
1,2G    file
51M     emailContacts
408K    contacts
76K     notes
16K     calendars

martin@merkaba:~/.local/share/baloo> ls -lSh email | head -5 insgesamt 8,0G

These will not be as bad as the fio test case, but still these files are written into. They are updated in place.

And thats running on every Plasma desktop by default. And on GNOME desktops there is similar stuff.

I haven´t seen this spike out a kworker yet tough, so maybe the workload is light enough not to trigger it that easily.

But yes, if you open a file and scribble all over it when your disk is full to within the same order of magnitude as the size of the file you are scribbling on, you will get into a condition where the _application_ will aggressively retry the IO. Particularly if that application is a "test program" or a virtual machine doing asynchronous IO.

That's what those sorts of systems do when they crash against a limit in the underlying system.

So yea... out of space plus agressive writer equals spinning CPU

Before you can assign blame you need to strace your application to see what call its making over and over again to see if its just being stupid.

From how you write I get the impression that you think everyone else

Quoted 1 line

From how you write I get the impression that you think everyone else

beside you is just silly and dumb. Please stop this assumption. I may not always get terms right, and I may make a mistake as with the wrong df figure. But I also highly dislike to feel treated like someone who doesn´t know a thing.

Quoted 6 lines · Martin Steigerwald

Quoted 1 line

From how you write I get the impression that you think everyone else

beside you is just silly and dumb. Please stop this assumption. I may not always get terms right, and I may make a mistake as with the wrong df figure. But I also highly dislike to feel treated like someone who doesn´t
know a thing.

Nope. I'm a systems theorist and I demand/require variable isolation.

Not a question of "silly" or "dumb" but a question of "speaking with sufficient precision and clarity".

For instance you speak of "having an impression" and then decide I've made an assumption.

I define my position. Explain my terms. Give my examples.

I also risk being utterly wrong because sometimes being completely wrong gets others to cut away misconceptions and assumptions.

It annoys some people, but it gets results.

Quoted 21 lines · Robert White

Quoted 6 lines · Martin Steigerwald

Quoted 1 line

From how you write I get the impression that you think everyone else

beside you is just silly and dumb. Please stop this assumption. I may not always get terms right, and I may make a mistake as with the wrong df figure. But I also highly dislike to feel treated like someone who doesn´t
know a thing.

Nope. I'm a systems theorist and I demand/require variable isolation.

Not a question of "silly" or "dumb" but a question of "speaking with sufficient precision and clarity".

For instance you speak of "having an impression" and then decide I've made an assumption.

I define my position. Explain my terms. Give my examples.

I also risk being utterly wrong because sometimes being completely wrong gets others to cut away misconceptions and assumptions.

Quoted 1 line

It annoys some people, but it gets results.

Can you please stop this bullshit posturing nonsense? It accomlishes nothing -- if you're right your other posts will stand for themselves and show that you are indeed "the shit" when it comes to these matters, but this post (so far, didn't read further) accomplishes nothing other than (possibly) convincing everyone that you're a pompous/self-important ass.

Quoted 22 lines · Robert White

Quoted 21 lines · Martin Steigerwald

Quoted 6 lines

Quoted 1 line

From how you write I get the impression that you think everyone else

beside you is just silly and dumb. Please stop this assumption. I may not always get terms right, and I may make a mistake as with the wrong df figure. But I also highly dislike to feel treated like someone who doesn´t
know a thing.

Nope. I'm a systems theorist and I demand/require variable isolation.

Not a question of "silly" or "dumb" but a question of "speaking with sufficient precision and clarity".

For instance you speak of "having an impression" and then decide I've made an assumption.

I define my position. Explain my terms. Give my examples.

I also risk being utterly wrong because sometimes being completely wrong gets others to cut away misconceptions and assumptions.

Quoted 8 lines

Quoted 1 line

It annoys some people, but it gets results.

Can you please stop this bullshit posturing nonsense? It accomlishes nothing -- if you're right your other posts will stand for themselves and show that you are indeed "the shit" when it comes to these matters, but this post (so far, didn't read further) accomplishes nothing other than (possibly) convincing everyone that you're a pompous/self-important ass.

Really? "accomplishes nothing"?

24 hours ago:

the complaining party was talking about

- Windows XP
- Tax software
- Virtual box
- vdi files
- defragging
- balancing
- "data trees"
- system hanging

And the responding party was saying

"you are the only person reporting this as a regular occurrence" with the implication that the report was a duplicate or at least might not get much immediate attention.

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

That's not "accomplishing nothing", thats called engaging in diagnostics instead of dismissing a complaint, and sticking out the diagnostic process until everyone is on the same page.

I never dismissed Martin. I never disbelieved him. I went through his elements one at a time with examples of what I was taking away from him and why they didn't match expectations and experimental evidence. We adjusted our positions and communications.

So you can call it "bullshit posturing nonsense" but I see "taking less than a day to get to the bottom of a bug report that might not have gotten significant attention."

BTW, I found that the Oracle blog didn´t work at all for me. I completed a cycle of defrag, sdelete -c and VBoxManage compact, [...] and it apparently did *nothing* to reduce the size of the file.

Quoted 4 lines

BTW, I found that the Oracle blog didn´t work at all for me. I completed a cycle of defrag, sdelete -c and VBoxManage compact, [...] and it apparently did *nothing* to reduce the size of the file.

They've changed the argument to -z; sdelete -z.

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White:

Quoted 5 lines

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

I didn´t yet provide such a test case.

At the moment I can only reproduce this kworker thread using a CPU for minutes case with my /home filesystem.

A mininmal test case for me would be to be able to reproduce it with a fresh BTRFS filesystem. But yet with my testcase with the fresh BTRFS I get 4800 instead of 270 IOPS.

Quoted 8 lines

Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White:

Quoted 5 lines

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

I didn´t yet provide such a test case.

My bad.

Quoted 8 lines

At the moment I can only reproduce this kworker thread using a CPU for minutes case with my /home filesystem.

A mininmal test case for me would be to be able to reproduce it with a fresh BTRFS filesystem. But yet with my testcase with the fresh BTRFS I get 4800 instead of 270 IOPS.

A version of the test case to demonstrate absolutely system-clogging loads is pretty easy to construct.

Make a raid1 filesystem.
Balance it once to make sure the seed filesystem is fully integrated.

Create a bunch of small files that are at least 4K in size, but are randomly sized. Fill the entire filesystem with them.

BASH Script:
typeset -i counter=0
while
dd if=/dev/urandom of=/mnt/Work/$((++counter)) bs=$((4096 + $RANDOM))
count=1 2>/dev/null
do
echo $counter >/dev/null #basically a noop
done

The while will exit when the dd encounters a full filesystem.

Then delete ~10% of the files with
rm *0

Run the while loop again, then delete a different 10% with "rm *1".

Then again with rm *2, etc...

Do this a few times and with each iteration the CPU usage gets worse and worse. You'll easily get system-wide stalls on all IO tasks lasting ten or more seconds.

I don't have enough spare storage to do this directly, so I used loopback devices. First I did it with the loopback files in COW mode. Then I did it again with the files in NOCOW mode. (the COW files got thick with overwrite real fast. 8-)

So anyway...

After I got through all ten digits on the rm (that is removing *0, then refilling, then *1 etc...) I figured the FS image was nicely fragmented.

At that point it was very easy to spike the kworker to 100% CPU with

dd if=/dev/urandom of=/mnt/Work/scratch bs=40k

The DD wold read 40k (a cpu spike for /dev/urandom processing) then it would write the 40k and the kworker would peg 100% on one CPU and stay there for a while. Then it would be back to the /dev/urandom spike.

So this laptop has been carefully detuned to prevent certain kinds of stalls (particularly the moveablecore= reservation, as previously mentioned, to prevent non-responsiveness of the UI) and I had to go through /dev/loop so that had a smoothing effect... but yep, there were clear kworker spikes that _did_ stop the IO path (the system monitor ap, for instance, could not get I/O statistics for ten and fifteen second intervals and would stop logging/scrolling).

Progressively larger block sizes on the write path made things progressively worse...

dd if=/dev/urandom of=/mnt/Work/scratch bs=160k

And overwriting the file by just invoking DD again, was worse still (presumably from the juggling act) before resulting in a net out-of-space condition.

Switching from /dev/urandom to /dev/zero for writing the large file made things worse still -- probably since there were no respites for the kworker to catch up etc.

ASIDE: Playing with /proc/sys/vm/dirty_{background_,}ratio had lots of interesting and difficult to quantify effects on user-space applications. Cutting in half (5 and 10 instead of 10 and 20 respectively) seemed to give some relief, but going further got harmful quickly. Diverging numbers was odd too. But it seemed a little brittle to play with these numbers.

SUPER FREAKY THING...

Every time I removed and recreated "scratch" I would get _radically_ different results for how much I could write into that remaining space and how long it took to do so. In theory I am reusing the exact same storage again and again. I'm not doing compression (the underlying filessytem behind the loop devices have compression but that would be disabled by the +C attribute). It's not enough space coming-and-going to cause data extents to be reclaimed or displaced by metadata. And the filessytem is otherwise completely unused.

But check it out...

Gust Work # rm scratch
Gust Work # dd if=/dev/zero of=/mnt/Work/scratch bs=160k count=1700 1700+0 records in
1700+0 records out
278528000 bytes (279 MB) copied, 1.4952 s, 186 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/zero of=/mnt/Work/scratch bs=160k count=1700 1700+0 records in
1700+0 records out
278528000 bytes (279 MB) copied, 292.135 s, 953 kB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/zero of=/mnt/Work/scratch bs=160k count=1700
dd: error writing ‘/mnt/Work/scratch’: No space left on device 93+0 records in
92+0 records out
15073280 bytes (15 MB) copied, 0.0453977 s, 332 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k count=1700
dd: error writing ‘/mnt/Work/scratch’: No space left on device 1090+0 records in
1089+0 records out
178421760 bytes (178 MB) copied, 115.991 s, 1.5 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k count=1700
dd: error writing ‘/mnt/Work/scratch’: No space left on device 332+0 records in
331+0 records out
54231040 bytes (54 MB) copied, 30.1589 s, 1.8 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k count=1700
dd: error writing ‘/mnt/Work/scratch’: No space left on device 622+0 records in
621+0 records out
101744640 bytes (102 MB) copied, 37.4813 s, 2.7 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k count=1700 1700+0 records in
1700+0 records out
278528000 bytes (279 MB) copied, 121.863 s, 2.3 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k count=1700 1700+0 records in
1700+0 records out
278528000 bytes (279 MB) copied, 24.2909 s, 11.5 MB/s
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k
dd: error writing ‘/mnt/Work/scratch’: No space left on device 1709+0 records in
1708+0 records out
279838720 bytes (280 MB) copied, 139.538 s, 2.0 MB/s
Gust Work # rm scratch
Gust Work # dd if=/dev/urandom of=/mnt/Work/scratch bs=160k
dd: error writing ‘/mnt/Work/scratch’: No space left on device 1424+0 records in
1423+0 records out
233144320 bytes (233 MB) copied, 102.257 s, 2.3 MB/s
Gust Work #

(and so on)

So...

Repeatable: yes.
Problematic: yes.

Am Sonntag, 28. Dezember 2014, 06:52:41 schrieb Robert White:

Quoted 44 lines · Martin Steigerwald

Quoted 8 lines

Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White:

Quoted 5 lines

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

I didn´t yet provide such a test case.

My bad.

Quoted 8 lines

At the moment I can only reproduce this kworker thread using a CPU for minutes case with my /home filesystem.

A mininmal test case for me would be to be able to reproduce it with a fresh BTRFS filesystem. But yet with my testcase with the fresh BTRFS I get 4800 instead of 270 IOPS.

A version of the test case to demonstrate absolutely system-clogging loads is pretty easy to construct.

Make a raid1 filesystem.
Balance it once to make sure the seed filesystem is fully integrated.

Create a bunch of small files that are at least 4K in size, but are randomly sized. Fill the entire filesystem with them.

BASH Script:
typeset -i counter=0
while
dd if=/dev/urandom of=/mnt/Work/$((++counter)) bs=$((4096 + $RANDOM))
count=1 2>/dev/null
do
echo $counter >/dev/null #basically a noop
done

The while will exit when the dd encounters a full filesystem.

Then delete ~10% of the files with
rm *0

Quoted 7 lines

Run the while loop again, then delete a different 10% with "rm *1".

Then again with rm *2, etc...

Do this a few times and with each iteration the CPU usage gets worse and worse. You'll easily get system-wide stalls on all IO tasks lasting ten or more seconds.

Thanks Robert. Thats wonderful.

I wondered about such a test case already and thought about reproducing it just with fallocate calls instead to reduce the amount of actual writes done. I.e. just do some silly fallocate, truncating, write just some parts with dd seek and remove things again kind of workload.

Feel free to add your testcase to the bug report:

[Bug 90401] New: btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

Cause anything that helps a BTRFS developer to reproduce will make it easier to find and fix the root cause of it.

I think I will try with this little critter:

merkaba:/mnt/btrfsraid1> cat freespracefragment.sh #!/bin/bash

TESTDIR="./test"
mkdir -p "$TESTDIR"

typeset -i counter=0
while true; do
fallocate -l $((4096 + $RANDOM)) "$TESTDIR/$((++counter))" echo $counter >/dev/null #basically a noop
done

It takes a time, the script itself is using only a few percent of one core there, while busying out the SSDs more heavily than I thought it would do. But well I see up to 12000 writes per 10 seconds – thats not that much, still it busies one SSD for 80%:

ATOP - merkaba 2014/12/28 16:40:57 ----------- 10s elapsed

PRC | sys    1.50s | user   3.47s | #proc    367  | #trun      1 | #tslpi   649 | #tslpu     0 | #zombie    0 | clones   839  |              | no  procacct |
CPU | sys      30% | user     38% | irq       1%  | idle    293% | wait     37% |              | steal     0% | guest     0%  | curf 1.63GHz | curscal  50% |
cpu | sys       7% | user     11% | irq       1%  | idle     75% | cpu000 w  6% |              | steal     0% | guest     0%  | curf 1.25GHz | curscal  39% |
cpu | sys       8% | user     11% | irq       0%  | idle     76% | cpu002 w  4% |              | steal     0% | guest     0%  | curf 1.55GHz | curscal  48% |
cpu | sys       7% | user      9% | irq       0%  | idle     71% | cpu001 w 13% |              | steal     0% | guest     0%  | curf 1.75GHz | curscal  54% |
cpu | sys       8% | user      7% | irq       0%  | idle     71% | cpu003 w 14% |              | steal     0% | guest     0%  | curf 1.96GHz | curscal  61% |
CPL | avg1    1.69 | avg5    1.30 | avg15   0.94  |              |              | csw    68387 | intr   36928 |               |              | numcpu     4 |
MEM | tot    15.5G | free    3.1G | cache   8.8G  | buff    4.2M | slab    1.0G | shmem 210.3M | shrss  79.1M | vmbal   0.0M  | hptot   0.0M | hpuse   0.0M |
SWP | tot    12.0G | free   11.5G |               |              |              |              |              |               | vmcom   4.9G | vmlim  19.7G |
LVM | a-btrfsraid1 | busy     80% | read       0  | write  11873 | KiB/r      0 | KiB/w      3 | MBr/s   0.00 | MBw/s   4.31  | avq     1.11 | avio 0.67 ms |
LVM | a-btrfsraid1 | busy      5% | read       0  | write  11873 | KiB/r      0 | KiB/w      3 | MBr/s   0.00 | MBw/s   4.31  | avq     2.45 | avio 0.04 ms |
LVM |   msata-home | busy      3% | read       0  | write    175 | KiB/r      0 | KiB/w      3 | MBr/s   0.00 | MBw/s   0.06  | avq     1.71 | avio 1.43 ms |
LVM | msata-debian | busy      0% | read       0  | write     10 | KiB/r      0 | KiB/w      8 | MBr/s   0.00 | MBw/s   0.01  | avq     1.15 | avio 3.40 ms |
LVM |    sata-home | busy      0% | read       0  | write    175 | KiB/r      0 | KiB/w      3 | MBr/s   0.00 | MBw/s   0.06  | avq     1.71 | avio 0.04 ms |
LVM |  sata-debian | busy      0% | read       0  | write     10 | KiB/r      0 | KiB/w      8 | MBr/s   0.00 | MBw/s   0.01  | avq     1.00 | avio 0.10 ms |
DSK |          sdb | busy     80% | read       0  | write  11880 | KiB/r      0 | KiB/w      3 | MBr/s   0.00 | MBw/s   4.38  | avq     1.11 | avio 0.67 ms |
DSK |          sda | busy      5% | read       0  | write  12069 | KiB/r      0 | KiB/w      3 | MBr/s   0.00 | MBw/s   4.38  | avq     2.51 | avio 0.04 ms |
NET | transport    | tcpi      26 | tcpo      26  | udpi       0 | udpo       0 | tcpao      2 | tcppo      1 | tcprs      0  | tcpie      0 | udpie      0 |
NET | network      | ipi       26 | ipo       26  | ipfrw      0 | deliv     26 |              |              |               | icmpi      0 | icmpo      0 |
NET | eth0      0% | pcki      10 | pcko      10  | si    5 Kbps | so    1 Kbps | coll       0 | erri       0 | erro       0  | drpi       0 | drpo       0 |
NET | lo      ---- | pcki      16 | pcko      16  | si    2 Kbps | so    2 Kbps | coll       0 | erri       0 | erro       0  | drpi       0 | drpo       0 |

  PID     TID    RUID        EUID         THR    SYSCPU    USRCPU     VGROW     RGROW    RDDSK     WRDSK    ST    EXC    S    CPUNR     CPU    CMD        1/4
 9169       -    martin      martin        14     0.22s     1.53s        0K        0K       0K        4K    --      -    S        1     18%    amarok
 1488       -    root        root           1     0.34s     0.27s      220K        0K       0K        0K    --      -    S        2      6%    Xorg
 6816       -    martin      martin         7     0.05s     0.44s        0K        0K       0K        0K    --      -    S        1      5%    kmail
24390       -    root        root           1     0.20s     0.25s       24K       24K       0K    40800K    --      -    S        0      5%    freespracefrag
 3268       -    martin      martin         3     0.08s     0.34s        0K        0K       0K       24K    --      -    S        0      4%    kwin

But only with a low amount of writes:

merkaba:/mnt/btrfsraid1> vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0 538424 3326248   4304 9202576    6   11  1968  4029  273  207 15 10 72  3  0
 1  0 538424 3325244   4304 9202836    0    0     0  6456 3498 7635 11  8 72 10  0
 0  0 538424 3325168   4304 9202932    0    0     0  9032 3719 6764  9  9 74  9  0
 0  0 538424 3334508   4304 9202932    0    0     0  8936 3548 6035  7  8 76  9  0
 0  0 538424 3334144   4304 9202876    0    0     0  9008 3335 5635  7  7 76 10  0
 0  0 538424 3332724   4304 9202728    0    0     0 11240 3555 5699  7  8 76 10  0
 2  0 538424 3333328   4304 9202876    0    0     0  9080 3724 6542  8  8 75  9  0
 0  0 538424 3333328   4304 9202876    0    0     0  6968 2951 5015  7  7 76 10  0
 0  1 538424 3332832   4304 9202584    0    0     0  9160 3663 6772  8  8 76  9  0

Still it busies one of both SSDs for about 80%:

iostat -xz 1

avg-cpu: %user %nice %system %iowait %steal %idle

           7,04    0,00    7,04    9,80    0,00   76,13

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sda               0,00     0,00    0,00 1220,00     0,00  4556,00     7,47     0,12    0,10    0,00    0,10   0,04   5,10
sdb               0,00    10,00    0,00 1210,00     0,00  4556,00     7,53     0,85    0,70    0,00    0,70   0,66  79,90
dm-2              0,00     0,00    0,00    4,00     0,00    36,00    18,00     0,02    5,00    0,00    5,00   4,25   1,70
dm-5              0,00     0,00    0,00    4,00     0,00    36,00    18,00     0,00    0,25    0,00    0,25   0,25   0,10
dm-6              0,00     0,00    0,00 1216,00     0,00  4520,00     7,43     0,12    0,10    0,00    0,10   0,04   5,00
dm-7              0,00     0,00    0,00 1216,00     0,00  4520,00     7,43     0,84    0,69    0,00    0,69   0,66  79,70

avg-cpu: %user %nice %system %iowait %steal %idle

           6,55    0,00    7,81    9,32    0,00   76,32

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sda               0,00     0,00    0,00 1203,00     0,00  4472,00     7,43     0,09    0,07    0,00    0,07   0,03   3,80
sdb               0,00     0,00    0,00 1203,00     0,00  4472,00     7,43     0,79    0,66    0,00    0,66   0,64  77,10
dm-6              0,00     0,00    0,00 1203,00     0,00  4472,00     7,43     0,09    0,07    0,00    0,07   0,03   4,00
dm-7              0,00     0,00    0,00 1203,00     0,00  4472,00     7,43     0,79    0,66    0,00    0,66   0,64  77,10

avg-cpu: %user %nice %system %iowait %steal %idle

           7,79    0,00    7,79    9,30    0,00   75,13

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

sda               0,00     0,00    0,00 1202,00     0,00  4468,00     7,43     0,09    0,07    0,00    0,07   0,04   4,70
sdb               0,00     0,00    4,00 1202,00  2048,00  4468,00    10,81     0,86    0,71    4,75    0,70   0,65  78,10
dm-1              0,00     0,00    4,00    0,00  2048,00     0,00  1024,00     0,02    4,75    4,75    0,00   2,00   0,80
dm-6              0,00     0,00    0,00 1202,00     0,00  4468,00     7,43     0,08    0,07    0,00    0,07   0,04   4,60
dm-7              0,00     0,00    0,00 1202,00     0,00  4468,00     7,43     0,84    0,70    0,00    0,70   0,65  77,80

But yet, neither I hit full CPU usage nor full SSD usage (just 80%), so this is yet another interesting case.

Am Sonntag, 28. Dezember 2014, 06:52:41 schrieb Robert White:

Quoted 44 lines · Martin Steigerwald

Quoted 8 lines

Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White:

Quoted 5 lines

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

I didn´t yet provide such a test case.

My bad.

Quoted 8 lines

At the moment I can only reproduce this kworker thread using a CPU for minutes case with my /home filesystem.

A mininmal test case for me would be to be able to reproduce it with a fresh BTRFS filesystem. But yet with my testcase with the fresh BTRFS I get 4800 instead of 270 IOPS.

A version of the test case to demonstrate absolutely system-clogging loads is pretty easy to construct.

Make a raid1 filesystem.
Balance it once to make sure the seed filesystem is fully integrated.

Create a bunch of small files that are at least 4K in size, but are randomly sized. Fill the entire filesystem with them.

BASH Script:
typeset -i counter=0
while
dd if=/dev/urandom of=/mnt/Work/$((++counter)) bs=$((4096 + $RANDOM))
count=1 2>/dev/null
do
echo $counter >/dev/null #basically a noop
done

The while will exit when the dd encounters a full filesystem.

Then delete ~10% of the files with
rm *0

Quoted 7 lines

Run the while loop again, then delete a different 10% with "rm *1".

Then again with rm *2, etc...

Do this a few times and with each iteration the CPU usage gets worse and worse. You'll easily get system-wide stalls on all IO tasks lasting ten or more seconds.

Thanks Robert. Thats wonderful.

I wondered about such a test case already and thought about reproducing it just with fallocate calls instead to reduce the amount of actual writes done. I.e. just do some silly fallocate, truncating, write just some parts with dd seek and remove things again kind of workload.

Feel free to add your testcase to the bug report:

[Bug 90401] New: btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

Cause anything that helps a BTRFS developer to reproduce will make it easier to find and fix the root cause of it.

I think I will try with this little critter:

merkaba:/mnt/btrfsraid1> cat freespracefragment.sh #!/bin/bash

TESTDIR="./test"
mkdir -p "$TESTDIR"

typeset -i counter=0
while true; do
fallocate -l $((4096 + $RANDOM)) "$TESTDIR/$((++counter))" echo $counter >/dev/null #basically a noop
done

I may try with with my test BTRFS. I could even make it 2x20 GiB RAID 1 as well.

Quoted 73 lines

Am Sonntag, 28. Dezember 2014, 06:52:41 schrieb Robert White:

Quoted 44 lines · Martin Steigerwald

Quoted 8 lines

Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White:

Quoted 5 lines

Now:

The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood and supported the bug.

I didn´t yet provide such a test case.

My bad.

Quoted 8 lines

At the moment I can only reproduce this kworker thread using a CPU for minutes case with my /home filesystem.

A mininmal test case for me would be to be able to reproduce it with a fresh BTRFS filesystem. But yet with my testcase with the fresh BTRFS I get 4800 instead of 270 IOPS.

A version of the test case to demonstrate absolutely system-clogging loads is pretty easy to construct.

Make a raid1 filesystem.
Balance it once to make sure the seed filesystem is fully integrated.

Create a bunch of small files that are at least 4K in size, but are randomly sized. Fill the entire filesystem with them.

BASH Script:
typeset -i counter=0
while
dd if=/dev/urandom of=/mnt/Work/$((++counter)) bs=$((4096 + $RANDOM))
count=1 2>/dev/null
do
echo $counter >/dev/null #basically a noop
done

The while will exit when the dd encounters a full filesystem.

Then delete ~10% of the files with
rm *0

Quoted 7 lines

Run the while loop again, then delete a different 10% with "rm *1".

Then again with rm *2, etc...

Do this a few times and with each iteration the CPU usage gets worse and worse. You'll easily get system-wide stalls on all IO tasks lasting ten or more seconds.

Thanks Robert. Thats wonderful.

I wondered about such a test case already and thought about reproducing it just with fallocate calls instead to reduce the amount of actual writes done. I.e. just do some silly fallocate, truncating, write just some parts with dd seek and remove things again kind of workload.

Feel free to add your testcase to the bug report:

[Bug 90401] New: btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

Cause anything that helps a BTRFS developer to reproduce will make it easier to find and fix the root cause of it.

I think I will try with this little critter:

merkaba:/mnt/btrfsraid1> cat freespracefragment.sh #!/bin/bash

Quoted 8 lines

TESTDIR="./test"
mkdir -p "$TESTDIR"

typeset -i counter=0
while true; do
fallocate -l $((4096 + $RANDOM)) "$TESTDIR/$((++counter))" echo $counter >/dev/null #basically a noop
done

If you don't do the remove/delete passes you won't get as much fragmentation...

I also noticed that fallocate would not actually create the files in my toolset, so I had to touch them first. So the theoretical script became

e.g.

typeset -i counter=0
for AA in {0..9}
do
while

     touch ${TESTDIR}/$((++counter)) &&
     fallocate -l $((4096 + $RANDOM)) $TESTDIR/$((counter))
   do
     if ((counter%100 == 0))
     then
       echo $counter
     fi

done
echo "removing ${AA}"
rm ${TESTDIR}/*${AA}
done

Meanwhile, on my test rig using fallocate did _not_ result in final exhaustion of resources. That is btrfs fi df /mnt/Work didn't show significant changes on a near full expanse.

I also never got a failed response back from fallocate, that is the inner loop never terminated. This could be a problem with the system call itself or it could be a problem with the application wrapper.

Nor did I reach the CPU saturation I expected.

e.g.
Gust vm # btrfs fi df /mnt/Work/
Data, RAID1: total=1.72GiB, used=1.66GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=256.00MiB, used=57.84MiB
GlobalReserve, single: total=32.00MiB, used=0.00B

time passes while script running...

Gust vm # btrfs fi df /mnt/Work/
Data, RAID1: total=1.72GiB, used=1.66GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=256.00MiB, used=57.84MiB
GlobalReserve, single: total=32.00MiB, used=0.00B

So there may be some limiting factor or something.

Without the actual writes to the actual file expanse I don't get the stalls.

(I added a _touch_ of instrumentation, it makes the various catostrophy events a little more obvious in context. 8-)

mount /dev/whattever /mnt/Work
typeset -i counter=0
for AA in {0..9}
do
while

     dd if=/dev/urandom of=/mnt/Work/$((++counter)) bs=$((4096 + 
$RANDOM)) count=1 2>/dev/null
   do
     if ((counter%100 == 0))
     then
       echo $counter
       if ((counter%1000 == 0))
       then
         btrfs fi df /mnt/Work
       fi
     fi

done
btrfs fi df /mnt/Work
echo "removing ${AA}"
rm /mnt/Work/*${AA}
btrfs fi df /mnt/Work
done

So you definitely need the writes to really see the stalls.

Quoted 2 lines

I may try with with my test BTRFS. I could even make it 2x20 GiB RAID 1 as well.

I guess I never mentioned it... I am using 4x1GiB NOCOW files through losetup as the basis of a RAID1. No compression (by virtue of the NOCOW files in underlying fs, and not being set in the resulting mount). No encryption. No LVM.

Quoted 79 lines

Am Samstag, 27. Dezember 2014, 15:14:05 schrieb Martin Steigerwald:

Quoted 49 lines

Am Samstag, 27. Dezember 2014, 06:00:48 schrieb Robert White:

Quoted 19 lines · Martin Steigerwald

Quoted 3 lines

It can easily be reproduced without even using Virtualbox, just by a nice
simple fio job.

TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file...

#!/bin/bash

# not tested, so correct any syntax errors
typeset -i counter
for ((counter=250;counter>0;counter--)); do

dd if=/dev/urandom of=/some/file bs=4k count=$counter

done
exit

Quoted 5 lines

Each pass over /some/file is 4k shorter than the previous one, but none of the extents can be deallocated. File will be 1MiB in size and usage will be something like 125.5MiB (if I've done the math correctly). larger values of counter will result in exponentially larger amounts of waste.

Robert, I experienced this hang issues even before the defragmenting case. It happened while just installed a 400 MiB tax returns application to it (that is no joke, it is that big).

It happens while just using the VM.

Yes, I recommend not to use BTRFS for any VM image or any larger database on rotating storage for exactly that COW semantics.

But on SSD?

Its busy looping a CPU core and while the flash is basically idling.

I refuse to believe that this is by design.

I do think there is a *bug*.

Either acknowledge it and try to fix it, or say its by design *without even looking at it closely enough to be sure that it is not a bug* and limit your own possibilities by it.

I´d rather see it treated as a bug for now.

Quoted 7 lines

Come on, 254 IOPS on a filesystem with still 17 GiB of free space while randomly writing to a 4 GiB file.

People do these kind of things. Ditch that defrag Windows XP VM case, I had performance issue even before by just installing things to it. Databases, VMs, emulators. And heck even while just *creating* the file with fio as I shown.

Add to these use cases things like this:

martin@merkaba:~/.local/share/akonadi/db_data/akonadi> ls -lSh | head -5 insgesamt 2,2G

(quoted diff) +0 -4

-rw-rw---- 1 martin martin 1,7G Dez 27 15:17 parttable.ibd
-rw-rw---- 1 martin martin 488M Dez 27 15:17 pimitemtable.ibd
-rw-rw---- 1 martin martin  23M Dez 27 15:17 pimitemflagrelation.ibd
-rw-rw---- 1 martin martin 240K Dez 27 15:17 collectiontable.ibd

Or this:

martin@merkaba:~/.local/share/baloo> du -sch * | sort -rh

9,2G    insgesamt
8,0G    email
1,2G    file
51M     emailContacts
408K    contacts
76K     notes
16K     calendars

Review

Quoted 6 lines

martin@merkaba:~/.local/share/baloo> ls -lSh email | head -5 insgesamt 8,0G

(quoted diff) +0 -4

-rw-r--r-- 1 martin martin 4,0G Dez 27 15:16 postlist.DB
-rw-r--r-- 1 martin martin 3,9G Dez 27 15:16 termlist.DB
-rw-r--r-- 1 martin martin 143M Dez 27 15:16 record.DB
-rw-r--r-- 1 martin martin  63K Dez 27 15:16 postlist.baseA

/usr/bin/du and /usr/bin/df and /bin/ls are all _useless_ for showing the amount of filespace used by a file in BTRFS.

Quoted 3 lines · Martin Steigerwald

It can easily be reproduced without even using Virtualbox, just by a nice simple fio job.

TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file...

#!/bin/bash

# not tested, so correct any syntax errors
typeset -i counter
for ((counter=250;counter>0;counter--)); do
  dd if=/dev/urandom of=/some/file bs=4k count=$counter
done
exit 0

Each pass over /some/file is 4k shorter than the previous one, but none of the extents can be deallocated. File will be 1MiB in size and usage will be something like 125.5MiB (if I've done the math correctly). larger values of counter will result in exponentially larger amounts of waste.

Doing the bad things is very bad...
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Data, RAID1: total=27.99GiB, used=17.21GiB

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 9 lines

Quoted 4 lines

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Quoted 4 lines

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Okay, just about terms.

What I call trees is this:

merkaba:~> btrfs fi df /
Data, RAID1: total=27.99GiB, used=17.21GiB System, RAID1: total=8.00MiB, used=16.00KiB Metadata, RAID1: total=2.00GiB, used=596.12MiB GlobalReserve, single: total=208.00MiB, used=0.00B

For me each one of "Data", "System", "Metadata" and "GlobalReserve" is what I call a "tree".

How would you call it?

I always thought that BTRFS uses a tree structure not only for metadata, but also for data. But I bet strictly spoken thats only to *manage* the chunks it allocates and what I see above is the actual chunk usage.

I.e. to get terms straight, how would you call it? I think my understanding of how BTRFS handles space allocation is quite correct, but I may use a term incorrectly.

I read

Quoted 1 line

Data, RAID1: total=27.99GiB, used=17.21GiB

as:

I reserved 27,99 GiB for data chunks and used 17,21 GiB in these data chunks so far. So I have about 10,5 GiB free in these data chunks at the moment and all is good.

What it doesn´t tell me at all is how the allocated space is distributed onto these chunks. I may be that some chunks are completely empty or not. It may be that each chunk has some space allocated to it but in total there is that amount of free space yet. I.e. it doesn´t tell me anything about the free space fragmentation inside the chunks.

Yet I still hold my theory that in the case of heavily writing to a COW´d file BTRFS seems to prefer to reserve new empty chunks on this /home filesystem of my laptop instead of trying to find free space in existing only partially empty chunks. And the lockup only happens when it tries to do the latter. And no, I think it shouldn´t lockup then. I also think its a bug. I never said differently.

And yes, I only ever had this on my /home so far. Not on / which is also RAID 1 and has all device space reserved for quite some time, not on /daten which only holds large files and is single instead of RAID. Also not on the server, but the server FS has lots of unallocated device space still, or on the 2 TiB eSATA backup HD, also I do get the impression that BTRFS started to get slower there as well at least the rsync based backup script takes quite long meanwhile and I see rsync reading from backup BTRFS and in this case almost fully ultilizing the disk for longer times. But unlike my /home the backup disk has some timely widely distributed snaphots (about 2 week to 1 months intervalls, and about last half year).

Neither /home nor / on the SSD have snapshots at the moment. So this is happening without snapshots.

Ciao,

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Summarized at

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

see below. This is reproducable with fio, no need for Windows XP in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 80 lines · Martin Steigerwald

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

          Total devices 2 FS bytes used 144.41GiB
          devid    1 size 160.00GiB used 160.00GiB path
          /dev/mapper/msata-home
          devid    2 size 160.00GiB used 160.00GiB path
          /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 7 lines

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

Robert's :)

Quoted 12 lines

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

And I got it reproduced. *Perfectly* reproduced, I´d say.

But let me run the whole story:

1) I downsized my /home BTRFS from dual 170 GiB to dual 160 GiB again.

Which gave me:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a
Total devices 2 FS bytes used 144.19GiB
devid 1 size 160.00GiB used 150.01GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 150.01GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=144.98GiB, used=140.95GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.24GiB GlobalReserve, single: total=512.00MiB, used=0.00B

2) I run the Virtualbox machine again and defragmented the NTFS filesystem in the VDI image file. And: It worked *just* fine. Fine as in *fine*. No issues whatsoever.

I got this during the run:

ATOP - merkaba 2014/12/27 12:58:42 ----------- 10s elapsed

PRC |  sys   10.41s |  user   1.08s |  #proc    357  | #trun      4  |  #tslpi   694 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     107% |  user     11% |  irq       0%  | idle    259%  |  wait     23% |  guest     0% |  curf 3.01GHz  | curscal  93%  |
cpu |  sys      29% |  user      3% |  irq       0%  | idle     63%  |  cpu002 w  5% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      27% |  user      3% |  irq       0%  | idle     65%  |  cpu000 w  5% |  guest     0% |  curf 3.03GHz  | curscal  94%  |
cpu |  sys      26% |  user      3% |  irq       0%  | idle     63%  |  cpu003 w  8% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      24% |  user      2% |  irq       0%  | idle     68%  |  cpu001 w  6% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    1.92 |  avg5    1.01 |  avg15   0.56  |               |  csw   501619 |  intr  129279 |                | numcpu     4  |
MEM |  tot    15.5G |  free  610.1M |  cache   9.1G  | buff    0.1M  |  slab    1.0G |  shmem 183.5M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.6G |                |               |               |               |  vmcom   7.1G  | vmlim  19.7G  |
PAG |  scan  219141 |  steal 215577 |  stall    936  |               |               |               |  swin       0  | swout    940  |
LVM |     sata-home |  busy     53% |  read  181413  | write      0  |  KiB/w      0 |  MBr/s  70.86 |  MBw/s   0.00  | avio 0.03 ms  |
LVM |     sata-swap |  busy      2% |  read       0  | write    940  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.37  | avio 0.17 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write      1  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.00  | avio 1.00 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write      1  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.00 ms  |
DSK |           sda |  busy     53% |  read  181413  | write    477  |  KiB/w      7 |  MBr/s  70.86 |  MBw/s   0.37  | avio 0.03 ms  |
DSK |           sdb |  busy      0% |  read       0  | write      1  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.00 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

   PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
  9650      -   martin    martin       22   7.89s    0.65s      0K     128K  705.5M   382.1M  --     -  S       2   87%   VirtualBox
  9911      -   root      root          1   0.69s    0.01s      0K       0K      0K       0K  --     -  S       3    7%   watch
  9598      -   root      root          1   0.38s    0.00s      0K       0K      0K      20K  --     -  S       0    4%   kworker/u8:9
  9892      -   root      root          1   0.36s    0.00s      0K       0K      0K       0K  --     -  S       1    4%   kworker/u8:17
  9428      -   root      root          1   0.30s    0.00s      0K       0K      0K       0K  --     -  R       0    3%   kworker/u8:3
  9589      -   root      root          1   0.23s    0.00s      0K       0K      0K       0K  --     -  S       1    2%   kworker/u8:6
  4746      -   martin    martin        2   0.04s    0.13s      0K     -16K      0K       0K  --     -  R       2    2%   konsole

Every 1,0s: cat /proc/meminfo Sat Dec 27 12:59:23 2014

MemTotal:       16210512 kB
MemFree:          786632 kB
MemAvailable:   10271500 kB
Buffers:              52 kB
Cached:          9564340 kB
SwapCached:        70268 kB
Active:          6847560 kB
Inactive:        5257956 kB
Active(anon):    2016412 kB
Inactive(anon):   703076 kB
Active(file):    4831148 kB
Inactive(file):  4554880 kB
Unevictable:        9068 kB
Mlocked:            9068 kB
SwapTotal:      12582908 kB
SwapFree:       12186680 kB
Dirty:            972324 kB
Writeback:             0 kB
AnonPages:       2526340 kB
Mapped:          2457096 kB
Shmem:            173564 kB
Slab:             918128 kB
SReclaimable:     848816 kB
SUnreclaim:        69312 kB
KernelStack:       11200 kB
PageTables:        64556 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    20688164 kB
Committed_AS:    7438348 kB

I am not seeing more than one GiB of dirty here during regular usage and it is no problem.

And kworker thread CPU usage just fine. So no, the dirty_background_ratio isn´t an issue with this 16 GiB ThinkPad T520. Please note: I do Linux performance analysis and tuning courses for about 7 years or so meanwhile.

I *know* these knobs. I may have used wrong terms regarding BTRFS, and my understanding of BTRFS space allocation probably can be more accurate, but I do think that I am onto something here. This is no rotating disk, it can handle the write burst just fine and I generally do not tune where there is no need to tune. Here there isn´t. And it wouldn´t be much more than a fine tuning.

With slow devices or with rsync over NFS by all means reduce it. But here it simply isn´t an issue as you can see with the low kworker thread CPU usage and the general SSD usage above.

So defragmentation completed just nice, no issue so far.

But I am close to full device space reservation already:

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:02:40 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 151.58GiB
devid 1 size 160.00GiB used 158.01GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 158.01GiB path /dev/mapper/sata-home

I thought I can trigger it again by defragmenting in Windows XP again, but mind you, its defragmented already so it doesn´t to much. I did the sdelete dance just to trigger something and well I saw kworker a bit higher, but not much.

But finally I got to:

So I did, if Virtualbox can write randomly in a file, I can too.

So I did:

martin@merkaba:~> cat ssd-test.fio
[global]
bs=4k
#ioengine=libaio
#iodepth=4
size=4g
#direct=1
runtime=120
filename=ssd.test.file

[seq-write]
rw=write
stonewall

[rand-write]
rw=randwrite
stonewall

And got:

ATOP - merkaba 2014/12/27 13:41:02 ----------- 10s elapsed

PRC |  sys   10.14s |  user   0.38s |  #proc    332  | #trun      2  |  #tslpi   548 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     102% |  user      4% |  irq       0%  | idle    295%  |  wait      0% |  guest     0% |  curf 3.10GHz  | curscal  96%  |
cpu |  sys      76% |  user      0% |  irq       0%  | idle     24%  |  cpu001 w  0% |  guest     0% |  curf 3.20GHz  | curscal  99%  |
cpu |  sys      24% |  user      1% |  irq       0%  | idle     75%  |  cpu000 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu003 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    0.82 |  avg5    0.78 |  avg15   0.99  |               |  csw     6233 |  intr   12023 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.0G |  cache   9.7G  | buff    0.0M  |  slab  333.1M |  shmem 206.6M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
DSK |           sda |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

   PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
18079      -   martin    martin        2   9.99s    0.00s      0K       0K      0K      16K  --     -  R       1  100%   fio
  4746      -   martin    martin        2   0.01s    0.14s      0K       0K      0K       0K  --     -  S       2    2%   konsole
  3291      -   martin    martin        4   0.01s    0.11s      0K       0K      0K       0K  --     -  S       0    1%   plasma-desktop
  1488      -   root      root          1   0.03s    0.04s      0K       0K      0K       0K  --     -  S       0    1%   Xorg

10036 - root root 1 0.04s 0.02s 0K 0K 0K 0K -- - R 2 1% atop

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

But it gets even more visible:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [19.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01m:57s] 0$ zsh 1$ zsh 2$ zsh 3-$ zsh 4$ zsh 5$* zsh

yes, thats 0 IOPS.

0 IOPS and in zero IOPS. For minutes.

And here is why:

ATOP - merkaba 2014/12/27 13:46:52 ----------- 10s elapsed

PRC |  sys   10.77s |  user   0.31s |  #proc    334  | #trun      2  |  #tslpi   548 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     108% |  user      3% |  irq       0%  | idle    286%  |  wait      2% |  guest     0% |  curf 3.08GHz  | curscal  96%  |
cpu |  sys      72% |  user      1% |  irq       0%  | idle     28%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      19% |  user      0% |  irq       0%  | idle     81%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      11% |  user      1% |  irq       0%  | idle     87%  |  cpu003 w  1% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       6% |  user      1% |  irq       0%  | idle     91%  |  cpu002 w  1% |  guest     0% |  curf 3.11GHz  | curscal  97%  |
CPL |  avg1    2.78 |  avg5    1.34 |  avg15   1.12  |               |  csw    50192 |  intr   32379 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.0G |  cache   8.7G  | buff    0.0M  |  slab  332.6M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      5% |  read     160  | write  11177  |  KiB/w      3 |  MBr/s   0.06 |  MBw/s   4.36  | avio 0.05 ms  |
LVM |    msata-home |  busy      4% |  read      28  | write  11177  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   4.36  | avio 0.04 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
DSK |           sda |  busy      5% |  read     160  | write  10200  |  KiB/w      4 |  MBr/s   0.06 |  MBw/s   4.69  | avio 0.05 ms  |
DSK |           sdb |  busy      4% |  read      28  | write  10558  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   4.69  | avio 0.04 ms  |
NET |  transport    |  tcpi      35 |  tcpo      33  | udpi       3  |  udpo       3 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       38 |  ipo       36  | ipfrw      0  |  deliv     38 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      22 |  pcko      20  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

   PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
14973      -   root      root          1   8.92s    0.00s      0K       0K      0K     144K  --     -  S       0   89%   kworker/u8:14
17450      -   root      root          1   0.86s    0.00s      0K       0K      0K      32K  --     -  R       3    9%   kworker/u8:5
   788      -   root      root          1   0.25s    0.00s      0K       0K    128K   18880K  --     -  S       3    3%   btrfs-transact
12254      -   root      root          1   0.14s    0.00s      0K       0K     64K     576K  --     -  S       2    1%   kworker/u8:3
17332      -   root      root          1   0.11s    0.00s      0K       0K    112K    1348K  --     -  S       2    1%   kworker/u8:4

3291 - martin martin 4 0.01s 0.09s 0K 0K 0K 0K -- - S 1 1% plasma-deskto

ATOP - merkaba 2014/12/27 13:47:12 ----------- 10s elapsed

PRC |  sys   10.78s |  user   0.44s |  #proc    334  | #trun      3  |  #tslpi   547 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     106% |  user      4% |  irq       0%  | idle    288%  |  wait      1% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      93% |  user      0% |  irq       0%  | idle      7%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       7% |  user      0% |  irq       0%  | idle     93%  |  cpu003 w  0% |  guest     0% |  curf 3.01GHz  | curscal  94%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     94%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    3.33 |  avg5    1.56 |  avg15   1.20  |               |  csw    38253 |  intr   23104 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.9G |  cache   8.7G  | buff    0.0M  |  slab  336.5M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |    msata-home |  busy      2% |  read       0  | write   2337  |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |     sata-home |  busy      2% |  read      36  | write   2337  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |  msata-debian |  busy      1% |  read       1  | write   1630  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.65  | avio 0.03 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write   1019  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.41  | avio 0.02 ms  |
DSK |           sdb |  busy      2% |  read       1  | write   2545  |  KiB/w      5 |  MBr/s   0.00 |  MBw/s   1.45  | avio 0.07 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2461  |  KiB/w      5 |  MBr/s   0.01 |  MBw/s   1.28  | avio 0.06 ms  |
NET |  transport    |  tcpi      20 |  tcpo      20  | udpi       1  |  udpo       1 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       21 |  ipo       21  | ipfrw      0  |  deliv     21 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki       5 |  pcko       5  | si    0 Kbps  |  so    0 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

   PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
17450      -   root      root          1   9.96s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:5
  4746      -   martin    martin        2   0.06s    0.15s      0K       0K      0K       0K  --     -  S       1    2%   konsole
10508      -   root      root          1   0.13s    0.00s      0K       0K     96K    4048K  --     -  S       1    1%   kworker/u8:18
  1488      -   root      root          1   0.06s    0.06s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
17332      -   root      root          1   0.12s    0.00s      0K       0K     96K     580K  --     -  R       3    1%   kworker/u8:4
17454      -   root      root          1   0.11s    0.00s      0K       0K     32K    4416K  --     -  D       1    1%   kworker/u8:6
17516      -   root      root          1   0.09s    0.00s      0K       0K     16K     136K  --     -  S       3    1%   kworker/u8:7

3268 - martin martin 3 0.02s 0.05s 0K 0K 0K 0K -- - S 1 1% kwin
10036 - root root 1 0.05s 0.02s 0K 0K 0K 0K -- - R 0 1% atop

So BTRFS is basically busy with itself and nothing else. Look at the SSD usage. They are *idling* around. Heck 2400 write accesses in 10 seconds. Thats a joke with SSDs that can do 40000 IOPS (depending on how and what you measure of course, like request size, read, write, iodepth and so).

Its kworker/u8:5 utilizing 100% of one core for minutes.

Its the random write case it seems. Here are values from fio job:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [3.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:06m:26s]
seq-write: (groupid=0, jobs=1): err= 0: pid=19212: Sat Dec 27 13:48:33 2014
write: io=4096.0MB, bw=343683KB/s, iops=85920, runt= 12204msec

     clat (usec): min=3, max=38048, avg=10.52, stdev=205.25
      lat (usec): min=3, max=38048, avg=10.66, stdev=205.43
     clat percentiles (usec):
      |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
      | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
      | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
      | 99.00th=[   14], 99.50th=[   20], 99.90th=[  211], 99.95th=[ 2128],
      | 99.99th=[10304]
     bw (KB  /s): min=164328, max=812984, per=100.00%, avg=345585.75, stdev=201695.20
     lat (usec) : 4=0.18%, 10=95.31%, 20=4.00%, 50=0.18%, 100=0.12%
     lat (usec) : 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
     lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
   cpu          : usr=13.55%, sys=46.89%, ctx=7810, majf=0, minf=6
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
      latency   : target=0, window=0, percentile=100.00%, depth=1

Seems fine.

But:

rand-write: (groupid=1, jobs=1): err= 0: pid=19243: Sat Dec 27 13:48:33 2014
write: io=140336KB, bw=1018.4KB/s, iops=254, runt=137803msec

     clat (usec): min=4, max=21299K, avg=3708.02, stdev=266885.61
      lat (usec): min=4, max=21299K, avg=3708.14, stdev=266885.61
     clat percentiles (usec):
      |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
      | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
      | 70.00th=[    7], 80.00th=[    7], 90.00th=[    9], 95.00th=[   10],
      | 99.00th=[   18], 99.50th=[   19], 99.90th=[   28], 99.95th=[  116],
      | 99.99th=[16711680]
     bw (KB  /s): min=    0, max= 3426, per=100.00%, avg=1030.10, stdev=938.02
     lat (usec) : 10=92.63%, 20=6.89%, 50=0.43%, 100=0.01%, 250=0.02%
     lat (msec) : 250=0.01%, 500=0.01%, >=2000=0.02%
   cpu          : usr=0.06%, sys=1.59%, ctx=28720, majf=0, minf=7
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      issued    : total=r=0/w=35084/d=0, short=r=0/w=0/d=0
      latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

   WRITE: io=4096.0MB, aggrb=343682KB/s, minb=343682KB/s, maxb=343682KB/s, mint=12204msec, maxt=12204msec

Run status group 1 (all jobs):

   WRITE: io=140336KB, aggrb=1018KB/s, minb=1018KB/s, maxb=1018KB/s, mint=137803msec, maxt=137803msec

What? 254 IOPS? With a Dual SSD BTRFS RAID 1?

What?

Ey, *what*?

Repeating with the random write case.

Its a different kworker now, but similar result:

ATOP - merkaba 2014/12/27 13:51:48 ----------- 10s elapsed

PRC |  sys   10.66s |  user   0.25s |  #proc    330  | #trun      2  |  #tslpi   545 |  #tslpu     2 |  #zombie    0  | no  procacct  |
CPU |  sys     105% |  user      3% |  irq       0%  | idle    292%  |  wait      0% |  guest     0% |  curf 3.07GHz  | curscal  95%  |
cpu |  sys      92% |  user      0% |  irq       0%  | idle      8%  |  cpu002 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       8% |  user      0% |  irq       0%  | idle     92%  |  cpu003 w  0% |  guest     0% |  curf 3.09GHz  | curscal  96%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       2% |  user      1% |  irq       0%  | idle     97%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    1.00 |  avg5    1.32 |  avg15   1.23  |               |  csw    34484 |  intr   23182 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.4G |  cache   8.3G  | buff    0.0M  |  slab  334.8M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      1% |  read      36  | write   2502  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   0.98  | avio 0.06 ms  |
LVM |    msata-home |  busy      1% |  read      48  | write   2502  |  KiB/w      4 |  MBr/s   0.02 |  MBw/s   0.98  | avio 0.04 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write      6  |  KiB/w      7 |  MBr/s   0.00 |  MBw/s   0.00  | avio 1.33 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write      6  |  KiB/w      7 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.17 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2494  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   0.98  | avio 0.06 ms  |
DSK |           sdb |  busy      1% |  read      48  | write   2494  |  KiB/w      4 |  MBr/s   0.02 |  MBw/s   0.98  | avio 0.04 ms  |
NET |  transport    |  tcpi      32 |  tcpo      30  | udpi       2  |  udpo       2 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       35 |  ipo       32  | ipfrw      0  |  deliv     35 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      19 |  pcko      16  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

   PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
11746      -   root      root          1  10.00s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:0
12254      -   root      root          1   0.16s    0.00s      0K       0K    112K    1712K  --     -  S       3    2%   kworker/u8:3
17517      -   root      root          1   0.16s    0.00s      0K       0K    144K    1764K  --     -  S       1    2%   kworker/u8:8

And now the graphical environemnt is locked. Continuining on TTY1.

Doing another fio job with tee so I can get output easily.

Wow! I wonder whether this is reproducable with a fresh BTRFS with fio stressing it.

Like a 10 GiB BTRFS with 5 GiB fio test file and just letting it run.

Okay, I let the final fio job complete and include the output here.

Okay, and there we are and I do have sysrq-t figures.

Okay, this is 1,2 MiB xz packed. So I better start a bug report about this and attach it there. Is dislike cloud URLs that may disappear at some time.

Now please finally acknowledge that there is an issue. Maybe I was not using the correct terms at the beginning, but there is a real issue. I do performance things for half a decade at least, I know that there is an issue when I see it.

There we go:

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

Thanks,

On 12/27/2014 05:55 AM, Martin Steigerwald wrote:

Quoted 9 lines

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

No. /usr/bin/df is an _approximation_ in BTRFS because of the limits of the fsstat() function call. The fstat function call was defined in 1990 and "can't understand" the dynamic allocation model used in BTRFS as it assumes fixed geometry for filesystems. You do _not_ have 17G actually available. You need to rely on btrfs fi df and btrfs fi show to figure out how much space you _really_ have.

According to this block you have a RAID1 of ~ 160GB expanse (two 160G disks)

Quoted 5 lines

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path

/dev/mapper/msata-home

Quoted 1 line

         devid    2 size 160.00GiB used 160.00GiB path

/dev/mapper/sata-home

And according to this block you have about 4.49GiB of data space:

Quoted 5 lines

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

154.97
5.00
0.032
+ 0.512

Pretty much as close to 160GiB as you are going to get (those numbers being rounded up in places for "human readability") BTRFS has allocate 100% of the raw storage into typed extents.

A large datafile can only fit in the 154.97-149.58 = 5.39

Trying to allocate that 4GiB file into that 5.39GiB of space becomes an NP-complete (e.g. "very hard") problem if it is very fragmented.

I also don't know what kind of tool you are using, but it might be repeatedly trying and failing to fallocate the file as a single extent or something equally dumb.

Quoted 1 line · Martin Steigerwald

[snip]

Quoted 41 lines

Quoted 9 lines

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

No. /usr/bin/df is an _approximation_ in BTRFS because of the limits of the fsstat() function call. The fstat function call was defined in 1990 and "can't understand" the dynamic allocation model used in BTRFS as it assumes fixed geometry for filesystems. You do _not_ have 17G actually available. You need to rely on btrfs fi df and btrfs fi show to figure out how much space you _really_ have.

According to this block you have a RAID1 of ~ 160GB expanse (two 160G disks)

Quoted 5 lines

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path

/dev/mapper/msata-home

Quoted 1 line

         devid    2 size 160.00GiB used 160.00GiB path

/dev/mapper/sata-home

And according to this block you have about 4.49GiB of data space:

Quoted 5 lines

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

154.97
5.00
0.032
+ 0.512

Quoted 5 lines

Pretty much as close to 160GiB as you are going to get (those numbers being rounded up in places for "human readability") BTRFS has allocate 100% of the raw storage into typed extents.

A large datafile can only fit in the 154.97-149.58 = 5.39

I appreciate that this is something of a minor point in the grand
scheme of things, but I'm afraid I've lost the enthusiasm to engage with the broader (somewhat rambling, possibly-at-cross-purposes) conversation in this thread. However...

Quoted 2 lines

Trying to allocate that 4GiB file into that 5.39GiB of space becomes an NP-complete (e.g. "very hard") problem if it is very fragmented.

This is... badly mistaken, at best. The problem of where to write a
file into a set of free extents is definitely *not* an NP-hard problem. It's a P problem, with an O(n log n) solution, where n is the number of free extents in the free space cache. The simple approach:
fill the first hole with as many bytes as you can, then move on to the next hole. More complex: order the free extents by size first. Both of these are O(n log n) algorithms, given an efficient general-purpose index of free space.

The problem of placing file data isn't a bin-packing problem; it's
not like allocating RAM (where each allocation must be contiguous). The items being placed may be split as much as you like, although minimising the amount of splitting is a goal.

I suspect that the performance problems that Martin is seeing may
indeed be related to free space fragmentation, in that finding and creating all of those tiny extents for a huge file is causing problems. I believe that btrfs isn't alone in this, but it may well be showing the problem to a far greater degree than other FSes. I don't have figures to compare, I'm afraid.

Quoted 3 lines

I also don't know what kind of tool you are using, but it might be repeatedly trying and failing to fallocate the file as a single extent or something equally dumb.

Userspace doesn't as far as I know, get to make that decision. I've
just read the fallocate(2) man page, and it says nothing at all about the contiguity of the extent(s) storage allocated by the call.

Am Samstag, 27. Dezember 2014, 16:26:42 schrieb Hugo Mills:

Quoted 72 lines · Robert White

Quoted 1 line · Martin Steigerwald

[snip]

Quoted 41 lines

Quoted 9 lines

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

No. /usr/bin/df is an _approximation_ in BTRFS because of the limits of the fsstat() function call. The fstat function call was defined in 1990 and "can't understand" the dynamic allocation model used in BTRFS as it assumes fixed geometry for filesystems. You do _not_ have 17G actually available. You need to rely on btrfs fi df and btrfs fi show to figure out how much space you _really_ have.

According to this block you have a RAID1 of ~ 160GB expanse (two 160G disks)

Quoted 5 lines

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path

/dev/mapper/msata-home

Quoted 1 line

         devid    2 size 160.00GiB used 160.00GiB path

/dev/mapper/sata-home

And according to this block you have about 4.49GiB of data space:

Quoted 5 lines

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

154.97
5.00
0.032
+ 0.512

Quoted 5 lines

Pretty much as close to 160GiB as you are going to get (those numbers being rounded up in places for "human readability") BTRFS has allocate 100% of the raw storage into typed extents.

A large datafile can only fit in the 154.97-149.58 = 5.39

I appreciate that this is something of a minor point in the grand
scheme of things, but I'm afraid I've lost the enthusiasm to engage with the broader (somewhat rambling, possibly-at-cross-purposes) conversation in this thread. However...

Quoted 2 lines

Trying to allocate that 4GiB file into that 5.39GiB of space becomes an NP-complete (e.g. "very hard") problem if it is very fragmented.

This is... badly mistaken, at best. The problem of where to write a
file into a set of free extents is definitely *not* an NP-hard problem. It's a P problem, with an O(n log n) solution, where n is the number of free extents in the free space cache. The simple approach:
fill the first hole with as many bytes as you can, then move on to the next hole. More complex: order the free extents by size first. Both of these are O(n log n) algorithms, given an efficient general-purpose index of free space.

The problem of placing file data isn't a bin-packing problem; it's
not like allocating RAM (where each allocation must be contiguous). The items being placed may be split as much as you like, although minimising the amount of splitting is a goal.

Quoted 6 lines

I suspect that the performance problems that Martin is seeing may
indeed be related to free space fragmentation, in that finding and creating all of those tiny extents for a huge file is causing problems. I believe that btrfs isn't alone in this, but it may well be showing the problem to a far greater degree than other FSes. I don't have figures to compare, I'm afraid.

Thats what I wanted to hint at.

I suspect an issue with free space fragmentation and do what I think I see:

btrfs balance minimizes free space in chunk fragmentation.

And that is my whole case on why I think it does help with my /home filesystem.

So while btrfs filesystem defragment may help with defragmenting individual files, possibly at the cost of fragmenting free space at least on filesystem almost full conditions, I think to help with free space fragmentation there are only three options at the moment:

1) reformat and restore via rsync or btrfs send from backup (i.e. file based)

2) make the BTRFS in itself bigger

3) btrfs balance at least chunks, at least those that are not more than 70% or 80% full.

Do you know of any other ways to deal with it?

This is... badly mistaken, at best. The problem of where to write a
file into a set of free extents is definitely *not* an NP-hard problem. It's a P problem, with an O(n log n) solution, where n is the number of free extents in the free space cache. The simple approach:
fill the first hole with as many bytes as you can, then move on to the next hole. More complex: order the free extents by size first. Both of these are O(n log n) algorithms, given an efficient general-purpose index of free space.

The problem of placing file data isn't a bin-packing problem; it's
not like allocating RAM (where each allocation must be contiguous). The items being placed may be split as much as you like, although minimising the amount of splitting is a goal.

I suspect that the performance problems that Martin is seeing may
indeed be related to free space fragmentation, in that finding and creating all of those tiny extents for a huge file is causing problems. I believe that btrfs isn't alone in this, but it may well be showing the problem to a far greater degree than other FSes. I don't have figures to compare, I'm afraid.

Quoted 3 lines

I also don't know what kind of tool you are using, but it might be repeatedly trying and failing to fallocate the file as a single extent or something equally dumb.

Userspace doesn't as far as I know, get to make that decision. I've
just read the fallocate(2) man page, and it says nothing at all about the contiguity of the extent(s) storage allocated by the call.

Hugo.

[snip]

Quoted 8 lines

Quoted 4 lines

I also don't know what kind of tool you are using, but it might be repeatedly trying and failing to fallocate the file as a single extent or something equally dumb.

Userspace doesn't as far as I know, get to make that decision. I've
just read the fallocate(2) man page, and it says nothing at all about the contiguity of the extent(s) storage allocated by the call.

Yep, my bad. But as soon as I saw that "fio" was starting two threads, one doing random read/write and another doing sequential read/write, both on the same file, it set off my "not just creating a file" mindset. Given the delayed write into/through the cache normally done by casual file io, It seemed likely that fio would be doing something more aggressive (like using O_DIRECT or repeated fdatasync() which could get very tit-for-tat).

From the manpage of fio:

Summarized at

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

see below. This is reproducable with fio, no need for Windows XP in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 80 lines · Martin Steigerwald

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 7 lines

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

Robert's :)

Quoted 12 lines

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

And I got it reproduced. *Perfectly* reproduced, I´d say.

But let me run the whole story:

1) I downsized my /home BTRFS from dual 170 GiB to dual 160 GiB again.

But finally I got to:

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

So I did, if Virtualbox can write randomly in a file, I can too.

So I did:

martin@merkaba:~> cat ssd-test.fio
[global]
bs=4k
#ioengine=libaio
#iodepth=4
size=4g
#direct=1
runtime=120
filename=ssd.test.file

[seq-write]
rw=write
stonewall

[rand-write]
rw=randwrite
stonewall

And got:

ATOP - merkaba 2014/12/27 13:41:02 ----------- 10s elapsed

PRC |  sys   10.14s |  user   0.38s |  #proc    332  | #trun      2  |  #tslpi   548 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     102% |  user      4% |  irq       0%  | idle    295%  |  wait      0% |  guest     0% |  curf 3.10GHz  | curscal  96%  |
cpu |  sys      76% |  user      0% |  irq       0%  | idle     24%  |  cpu001 w  0% |  guest     0% |  curf 3.20GHz  | curscal  99%  |
cpu |  sys      24% |  user      1% |  irq       0%  | idle     75%  |  cpu000 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu003 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    0.82 |  avg5    0.78 |  avg15   0.99  |               |  csw     6233 |  intr   12023 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.0G |  cache   9.7G  | buff    0.0M  |  slab  333.1M |  shmem 206.6M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
DSK |           sda |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
18079      -   martin    martin        2   9.99s    0.00s      0K       0K      0K      16K  --     -  R       1  100%   fio
 4746      -   martin    martin        2   0.01s    0.14s      0K       0K      0K       0K  --     -  S       2    2%   konsole
 3291      -   martin    martin        4   0.01s    0.11s      0K       0K      0K       0K  --     -  S       0    1%   plasma-desktop
 1488      -   root      root          1   0.03s    0.04s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
10036      -   root      root          1   0.04s    0.02s      0K       0K      0K       0K  --     -  R       2    1%   atop

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

But it gets even more visible:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [19.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01m:57s] 0$ zsh 1$ zsh 2$ zsh 3-$ zsh 4$ zsh 5$* zsh

yes, thats 0 IOPS.

0 IOPS and in zero IOPS. For minutes.

And here is why:

ATOP - merkaba 2014/12/27 13:46:52 ----------- 10s elapsed

PRC |  sys   10.77s |  user   0.31s |  #proc    334  | #trun      2  |  #tslpi   548 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     108% |  user      3% |  irq       0%  | idle    286%  |  wait      2% |  guest     0% |  curf 3.08GHz  | curscal  96%  |
cpu |  sys      72% |  user      1% |  irq       0%  | idle     28%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      19% |  user      0% |  irq       0%  | idle     81%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      11% |  user      1% |  irq       0%  | idle     87%  |  cpu003 w  1% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       6% |  user      1% |  irq       0%  | idle     91%  |  cpu002 w  1% |  guest     0% |  curf 3.11GHz  | curscal  97%  |
CPL |  avg1    2.78 |  avg5    1.34 |  avg15   1.12  |               |  csw    50192 |  intr   32379 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.0G |  cache   8.7G  | buff    0.0M  |  slab  332.6M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      5% |  read     160  | write  11177  |  KiB/w      3 |  MBr/s   0.06 |  MBw/s   4.36  | avio 0.05 ms  |
LVM |    msata-home |  busy      4% |  read      28  | write  11177  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   4.36  | avio 0.04 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
DSK |           sda |  busy      5% |  read     160  | write  10200  |  KiB/w      4 |  MBr/s   0.06 |  MBw/s   4.69  | avio 0.05 ms  |
DSK |           sdb |  busy      4% |  read      28  | write  10558  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   4.69  | avio 0.04 ms  |
NET |  transport    |  tcpi      35 |  tcpo      33  | udpi       3  |  udpo       3 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       38 |  ipo       36  | ipfrw      0  |  deliv     38 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      22 |  pcko      20  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
14973      -   root      root          1   8.92s    0.00s      0K       0K      0K     144K  --     -  S       0   89%   kworker/u8:14
17450      -   root      root          1   0.86s    0.00s      0K       0K      0K      32K  --     -  R       3    9%   kworker/u8:5
  788      -   root      root          1   0.25s    0.00s      0K       0K    128K   18880K  --     -  S       3    3%   btrfs-transact
12254      -   root      root          1   0.14s    0.00s      0K       0K     64K     576K  --     -  S       2    1%   kworker/u8:3
17332      -   root      root          1   0.11s    0.00s      0K       0K    112K    1348K  --     -  S       2    1%   kworker/u8:4
 3291      -   martin    martin        4   0.01s    0.09s      0K       0K      0K       0K  --     -  S       1    1%   plasma-deskto

ATOP - merkaba 2014/12/27 13:47:12 ----------- 10s elapsed

PRC |  sys   10.78s |  user   0.44s |  #proc    334  | #trun      3  |  #tslpi   547 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     106% |  user      4% |  irq       0%  | idle    288%  |  wait      1% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      93% |  user      0% |  irq       0%  | idle      7%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       7% |  user      0% |  irq       0%  | idle     93%  |  cpu003 w  0% |  guest     0% |  curf 3.01GHz  | curscal  94%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     94%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    3.33 |  avg5    1.56 |  avg15   1.20  |               |  csw    38253 |  intr   23104 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.9G |  cache   8.7G  | buff    0.0M  |  slab  336.5M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |    msata-home |  busy      2% |  read       0  | write   2337  |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |     sata-home |  busy      2% |  read      36  | write   2337  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |  msata-debian |  busy      1% |  read       1  | write   1630  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.65  | avio 0.03 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write   1019  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.41  | avio 0.02 ms  |
DSK |           sdb |  busy      2% |  read       1  | write   2545  |  KiB/w      5 |  MBr/s   0.00 |  MBw/s   1.45  | avio 0.07 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2461  |  KiB/w      5 |  MBr/s   0.01 |  MBw/s   1.28  | avio 0.06 ms  |
NET |  transport    |  tcpi      20 |  tcpo      20  | udpi       1  |  udpo       1 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       21 |  ipo       21  | ipfrw      0  |  deliv     21 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki       5 |  pcko       5  | si    0 Kbps  |  so    0 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
17450      -   root      root          1   9.96s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:5
 4746      -   martin    martin        2   0.06s    0.15s      0K       0K      0K       0K  --     -  S       1    2%   konsole
10508      -   root      root          1   0.13s    0.00s      0K       0K     96K    4048K  --     -  S       1    1%   kworker/u8:18
 1488      -   root      root          1   0.06s    0.06s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
17332      -   root      root          1   0.12s    0.00s      0K       0K     96K     580K  --     -  R       3    1%   kworker/u8:4
17454      -   root      root          1   0.11s    0.00s      0K       0K     32K    4416K  --     -  D       1    1%   kworker/u8:6
17516      -   root      root          1   0.09s    0.00s      0K       0K     16K     136K  --     -  S       3    1%   kworker/u8:7
 3268      -   martin    martin        3   0.02s    0.05s      0K       0K      0K       0K  --     -  S       1    1%   kwin
10036      -   root      root          1   0.05s    0.02s      0K       0K      0K       0K  --     -  R       0    1%   atop

So BTRFS is basically busy with itself and nothing else. Look at the SSD usage. They are *idling* around. Heck 2400 write accesses in 10 seconds. Thats a joke with SSDs that can do 40000 IOPS (depending on how and what you measure of course, like request size, read, write, iodepth and so).

Its kworker/u8:5 utilizing 100% of one core for minutes.

Its the random write case it seems. Here are values from fio job:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [3.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:06m:26s]
seq-write: (groupid=0, jobs=1): err= 0: pid=19212: Sat Dec 27 13:48:33 2014
write: io=4096.0MB, bw=343683KB/s, iops=85920, runt= 12204msec

    clat (usec): min=3, max=38048, avg=10.52, stdev=205.25
     lat (usec): min=3, max=38048, avg=10.66, stdev=205.43
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[  211], 99.95th=[ 2128],
     | 99.99th=[10304]
    bw (KB  /s): min=164328, max=812984, per=100.00%, avg=345585.75, stdev=201695.20
    lat (usec) : 4=0.18%, 10=95.31%, 20=4.00%, 50=0.18%, 100=0.12%
    lat (usec) : 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=13.55%, sys=46.89%, ctx=7810, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Seems fine.

But:

rand-write: (groupid=1, jobs=1): err= 0: pid=19243: Sat Dec 27 13:48:33 2014
write: io=140336KB, bw=1018.4KB/s, iops=254, runt=137803msec

    clat (usec): min=4, max=21299K, avg=3708.02, stdev=266885.61
     lat (usec): min=4, max=21299K, avg=3708.14, stdev=266885.61
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   19], 99.90th=[   28], 99.95th=[  116],
     | 99.99th=[16711680]
    bw (KB  /s): min=    0, max= 3426, per=100.00%, avg=1030.10, stdev=938.02
    lat (usec) : 10=92.63%, 20=6.89%, 50=0.43%, 100=0.01%, 250=0.02%
    lat (msec) : 250=0.01%, 500=0.01%, >=2000=0.02%
  cpu          : usr=0.06%, sys=1.59%, ctx=28720, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=35084/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=343682KB/s, minb=343682KB/s, maxb=343682KB/s, mint=12204msec, maxt=12204msec

Run status group 1 (all jobs):
WRITE: io=140336KB, aggrb=1018KB/s, minb=1018KB/s, maxb=1018KB/s, mint=137803msec, maxt=137803msec

What? 254 IOPS? With a Dual SSD BTRFS RAID 1?

What?

Ey, *what*?

Repeating with the random write case.

Its a different kworker now, but similar result:

ATOP - merkaba 2014/12/27 13:51:48 ----------- 10s elapsed

PRC |  sys   10.66s |  user   0.25s |  #proc    330  | #trun      2  |  #tslpi   545 |  #tslpu     2 |  #zombie    0  | no  procacct  |
CPU |  sys     105% |  user      3% |  irq       0%  | idle    292%  |  wait      0% |  guest     0% |  curf 3.07GHz  | curscal  95%  |
cpu |  sys      92% |  user      0% |  irq       0%  | idle      8%  |  cpu002 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       8% |  user      0% |  irq       0%  | idle     92%  |  cpu003 w  0% |  guest     0% |  curf 3.09GHz  | curscal  96%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       2% |  user      1% |  irq       0%  | idle     97%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    1.00 |  avg5    1.32 |  avg15   1.23  |               |  csw    34484 |  intr   23182 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.4G |  cache   8.3G  | buff    0.0M  |  slab  334.8M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      1% |  read      36  | write   2502  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   0.98  | avio 0.06 ms  |
LVM |    msata-home |  busy      1% |  read      48  | write   2502  |  KiB/w      4 |  MBr/s   0.02 |  MBw/s   0.98  | avio 0.04 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write      6  |  KiB/w      7 |  MBr/s   0.00 |  MBw/s   0.00  | avio 1.33 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write      6  |  KiB/w      7 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.17 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2494  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   0.98  | avio 0.06 ms  |
DSK |           sdb |  busy      1% |  read      48  | write   2494  |  KiB/w      4 |  MBr/s   0.02 |  MBw/s   0.98  | avio 0.04 ms  |
NET |  transport    |  tcpi      32 |  tcpo      30  | udpi       2  |  udpo       2 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       35 |  ipo       32  | ipfrw      0  |  deliv     35 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      19 |  pcko      16  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
11746      -   root      root          1  10.00s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:0
12254      -   root      root          1   0.16s    0.00s      0K       0K    112K    1712K  --     -  S       3    2%   kworker/u8:3  
17517      -   root      root          1   0.16s    0.00s      0K       0K    144K    1764K  --     -  S       1    2%   kworker/u8:8

And now the graphical environemnt is locked. Continuining on TTY1.

Doing another fio job with tee so I can get output easily.

Wow! I wonder whether this is reproducable with a fresh BTRFS with fio stressing it.

Like a 10 GiB BTRFS with 5 GiB fio test file and just letting it run.

Okay, I let the final fio job complete and include the output here.

Okay, and there we are and I do have sysrq-t figures.

Okay, this is 1,2 MiB xz packed. So I better start a bug report about this and attach it there. Is dislike cloud URLs that may disappear at some time.

Now please finally acknowledge that there is an issue. Maybe I was not using the correct terms at the beginning, but there is a real issue. I do performance things for half a decade at least, I know that there is an issue when I see it.

There we go:

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

Am Samstag, 27. Dezember 2014, 14:55:58 schrieb Martin Steigerwald:

Quoted 121 lines

Summarized at

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

see below. This is reproducable with fio, no need for Windows XP in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 80 lines · Martin Steigerwald

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does... 8-)

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 7 lines

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

Robert's :)

Quoted 12 lines

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Quoted 5 lines

And I got it reproduced. *Perfectly* reproduced, I´d say.

But let me run the whole story:

1) I downsized my /home BTRFS from dual 170 GiB to dual 160 GiB again.

[… story of trying to reproduce with Windows XP defragmenting which was unsuccessful as BTRFS still had free device space to allocate new chunks from …]

Quoted 238 lines

But finally I got to:

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

So I did, if Virtualbox can write randomly in a file, I can too.

So I did:

martin@merkaba:~> cat ssd-test.fio
[global]
bs=4k
#ioengine=libaio
#iodepth=4
size=4g
#direct=1
runtime=120
filename=ssd.test.file

[seq-write]
rw=write
stonewall

[rand-write]
rw=randwrite
stonewall

And got:

ATOP - merkaba 2014/12/27 13:41:02 ----------- 10s elapsed

PRC |  sys   10.14s |  user   0.38s |  #proc    332  | #trun      2  |  #tslpi   548 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     102% |  user      4% |  irq       0%  | idle    295%  |  wait      0% |  guest     0% |  curf 3.10GHz  | curscal  96%  |
cpu |  sys      76% |  user      0% |  irq       0%  | idle     24%  |  cpu001 w  0% |  guest     0% |  curf 3.20GHz  | curscal  99%  |
cpu |  sys      24% |  user      1% |  irq       0%  | idle     75%  |  cpu000 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu003 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    0.82 |  avg5    0.78 |  avg15   0.99  |               |  csw     6233 |  intr   12023 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.0G |  cache   9.7G  | buff    0.0M  |  slab  333.1M |  shmem 206.6M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
DSK |           sda |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
18079      -   martin    martin        2   9.99s    0.00s      0K       0K      0K      16K  --     -  R       1  100%   fio
 4746      -   martin    martin        2   0.01s    0.14s      0K       0K      0K       0K  --     -  S       2    2%   konsole
 3291      -   martin    martin        4   0.01s    0.11s      0K       0K      0K       0K  --     -  S       0    1%   plasma-desktop
 1488      -   root      root          1   0.03s    0.04s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
10036      -   root      root          1   0.04s    0.02s      0K       0K      0K       0K  --     -  R       2    1%   atop

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

But it gets even more visible:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [19.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01m:57s] 0$ zsh 1$ zsh 2$ zsh 3-$ zsh 4$ zsh 5$* zsh

yes, thats 0 IOPS.

0 IOPS and in zero IOPS. For minutes.

And here is why:

ATOP - merkaba 2014/12/27 13:46:52 ----------- 10s elapsed

PRC |  sys   10.77s |  user   0.31s |  #proc    334  | #trun      2  |  #tslpi   548 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     108% |  user      3% |  irq       0%  | idle    286%  |  wait      2% |  guest     0% |  curf 3.08GHz  | curscal  96%  |
cpu |  sys      72% |  user      1% |  irq       0%  | idle     28%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      19% |  user      0% |  irq       0%  | idle     81%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      11% |  user      1% |  irq       0%  | idle     87%  |  cpu003 w  1% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       6% |  user      1% |  irq       0%  | idle     91%  |  cpu002 w  1% |  guest     0% |  curf 3.11GHz  | curscal  97%  |
CPL |  avg1    2.78 |  avg5    1.34 |  avg15   1.12  |               |  csw    50192 |  intr   32379 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.0G |  cache   8.7G  | buff    0.0M  |  slab  332.6M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      5% |  read     160  | write  11177  |  KiB/w      3 |  MBr/s   0.06 |  MBw/s   4.36  | avio 0.05 ms  |
LVM |    msata-home |  busy      4% |  read      28  | write  11177  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   4.36  | avio 0.04 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
DSK |           sda |  busy      5% |  read     160  | write  10200  |  KiB/w      4 |  MBr/s   0.06 |  MBw/s   4.69  | avio 0.05 ms  |
DSK |           sdb |  busy      4% |  read      28  | write  10558  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   4.69  | avio 0.04 ms  |
NET |  transport    |  tcpi      35 |  tcpo      33  | udpi       3  |  udpo       3 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       38 |  ipo       36  | ipfrw      0  |  deliv     38 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      22 |  pcko      20  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
14973      -   root      root          1   8.92s    0.00s      0K       0K      0K     144K  --     -  S       0   89%   kworker/u8:14
17450      -   root      root          1   0.86s    0.00s      0K       0K      0K      32K  --     -  R       3    9%   kworker/u8:5
  788      -   root      root          1   0.25s    0.00s      0K       0K    128K   18880K  --     -  S       3    3%   btrfs-transact
12254      -   root      root          1   0.14s    0.00s      0K       0K     64K     576K  --     -  S       2    1%   kworker/u8:3
17332      -   root      root          1   0.11s    0.00s      0K       0K    112K    1348K  --     -  S       2    1%   kworker/u8:4
 3291      -   martin    martin        4   0.01s    0.09s      0K       0K      0K       0K  --     -  S       1    1%   plasma-deskto

ATOP - merkaba 2014/12/27 13:47:12 ----------- 10s elapsed

PRC |  sys   10.78s |  user   0.44s |  #proc    334  | #trun      3  |  #tslpi   547 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     106% |  user      4% |  irq       0%  | idle    288%  |  wait      1% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      93% |  user      0% |  irq       0%  | idle      7%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       7% |  user      0% |  irq       0%  | idle     93%  |  cpu003 w  0% |  guest     0% |  curf 3.01GHz  | curscal  94%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     94%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    3.33 |  avg5    1.56 |  avg15   1.20  |               |  csw    38253 |  intr   23104 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.9G |  cache   8.7G  | buff    0.0M  |  slab  336.5M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |    msata-home |  busy      2% |  read       0  | write   2337  |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |     sata-home |  busy      2% |  read      36  | write   2337  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |  msata-debian |  busy      1% |  read       1  | write   1630  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.65  | avio 0.03 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write   1019  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.41  | avio 0.02 ms  |
DSK |           sdb |  busy      2% |  read       1  | write   2545  |  KiB/w      5 |  MBr/s   0.00 |  MBw/s   1.45  | avio 0.07 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2461  |  KiB/w      5 |  MBr/s   0.01 |  MBw/s   1.28  | avio 0.06 ms  |
NET |  transport    |  tcpi      20 |  tcpo      20  | udpi       1  |  udpo       1 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       21 |  ipo       21  | ipfrw      0  |  deliv     21 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki       5 |  pcko       5  | si    0 Kbps  |  so    0 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
17450      -   root      root          1   9.96s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:5
 4746      -   martin    martin        2   0.06s    0.15s      0K       0K      0K       0K  --     -  S       1    2%   konsole
10508      -   root      root          1   0.13s    0.00s      0K       0K     96K    4048K  --     -  S       1    1%   kworker/u8:18
 1488      -   root      root          1   0.06s    0.06s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
17332      -   root      root          1   0.12s    0.00s      0K       0K     96K     580K  --     -  R       3    1%   kworker/u8:4
17454      -   root      root          1   0.11s    0.00s      0K       0K     32K    4416K  --     -  D       1    1%   kworker/u8:6
17516      -   root      root          1   0.09s    0.00s      0K       0K     16K     136K  --     -  S       3    1%   kworker/u8:7
 3268      -   martin    martin        3   0.02s    0.05s      0K       0K      0K       0K  --     -  S       1    1%   kwin
10036      -   root      root          1   0.05s    0.02s      0K       0K      0K       0K  --     -  R       0    1%   atop

So BTRFS is basically busy with itself and nothing else. Look at the SSD usage. They are *idling* around. Heck 2400 write accesses in 10 seconds. Thats a joke with SSDs that can do 40000 IOPS (depending on how and what you measure of course, like request size, read, write, iodepth and so).

Its kworker/u8:5 utilizing 100% of one core for minutes.

Its the random write case it seems. Here are values from fio job:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [3.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:06m:26s]
seq-write: (groupid=0, jobs=1): err= 0: pid=19212: Sat Dec 27 13:48:33 2014
write: io=4096.0MB, bw=343683KB/s, iops=85920, runt= 12204msec

    clat (usec): min=3, max=38048, avg=10.52, stdev=205.25
     lat (usec): min=3, max=38048, avg=10.66, stdev=205.43
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[  211], 99.95th=[ 2128],
     | 99.99th=[10304]
    bw (KB  /s): min=164328, max=812984, per=100.00%, avg=345585.75, stdev=201695.20
    lat (usec) : 4=0.18%, 10=95.31%, 20=4.00%, 50=0.18%, 100=0.12%
    lat (usec) : 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=13.55%, sys=46.89%, ctx=7810, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Seems fine.

But:

rand-write: (groupid=1, jobs=1): err= 0: pid=19243: Sat Dec 27 13:48:33 2014
write: io=140336KB, bw=1018.4KB/s, iops=254, runt=137803msec

    clat (usec): min=4, max=21299K, avg=3708.02, stdev=266885.61
     lat (usec): min=4, max=21299K, avg=3708.14, stdev=266885.61
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   19], 99.90th=[   28], 99.95th=[  116],
     | 99.99th=[16711680]
    bw (KB  /s): min=    0, max= 3426, per=100.00%, avg=1030.10, stdev=938.02
    lat (usec) : 10=92.63%, 20=6.89%, 50=0.43%, 100=0.01%, 250=0.02%
    lat (msec) : 250=0.01%, 500=0.01%, >=2000=0.02%
  cpu          : usr=0.06%, sys=1.59%, ctx=28720, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=35084/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=343682KB/s, minb=343682KB/s, maxb=343682KB/s, mint=12204msec, maxt=12204msec

Run status group 1 (all jobs):
WRITE: io=140336KB, aggrb=1018KB/s, minb=1018KB/s, maxb=1018KB/s, mint=137803msec, maxt=137803msec

What? 254 IOPS? With a Dual SSD BTRFS RAID 1?

What?

Ey, *what*?

Quoted 4 lines

There we go:

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

I have done more tests.

This is on the same /home after extending it to 170 GiB and balancing it to btrfs balance start -dusage=80

It has plenty of free space free. I updated the bug report and hope it can give an easy enough to comprehend summary. The new tests are in:

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c6

Pasting below for discussion on list. Summary: I easily get 38000 (!) IOPS. It may be an idea to reduce to 160 GiB, but right now this does not work as it says no free space on device when trying to downsize it. I may try with 165 or 162GiB.

So now we have three IOPS figures:

- 256 IOPS in worst case scenario
- 4700 IOPS when trying to reproduce worst case scenario with a fresh and small BTRFS
- 38000 IOPS when /home has unused device space to allocate chunks from

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c8

This is another test.

Am Sonntag, 28. Dezember 2014, 14:00:19 schrieb Martin Steigerwald:

Quoted 370 lines

Am Samstag, 27. Dezember 2014, 14:55:58 schrieb Martin Steigerwald:

Quoted 121 lines

Summarized at

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

see below. This is reproducable with fio, no need for Windows XP in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 80 lines · Martin Steigerwald

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does...

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 7 lines

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

Robert's

Quoted 12 lines

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Quoted 5 lines

And I got it reproduced. *Perfectly* reproduced, I´d say.

But let me run the whole story:

1) I downsized my /home BTRFS from dual 170 GiB to dual 160 GiB again.

[… story of trying to reproduce with Windows XP defragmenting which was unsuccessful as BTRFS still had free device space to allocate new chunks from …]

Quoted 238 lines

But finally I got to:

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

So I did, if Virtualbox can write randomly in a file, I can too.

So I did:

martin@merkaba:~> cat ssd-test.fio
[global]
bs=4k
#ioengine=libaio
#iodepth=4
size=4g
#direct=1
runtime=120
filename=ssd.test.file

[seq-write]
rw=write
stonewall

[rand-write]
rw=randwrite
stonewall

And got:

ATOP - merkaba 2014/12/27 13:41:02 ----------- 10s elapsed

PRC |  sys   10.14s |  user   0.38s |  #proc    332  | #trun      2  |  #tslpi   548 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     102% |  user      4% |  irq       0%  | idle    295%  |  wait      0% |  guest     0% |  curf 3.10GHz  | curscal  96%  |
cpu |  sys      76% |  user      0% |  irq       0%  | idle     24%  |  cpu001 w  0% |  guest     0% |  curf 3.20GHz  | curscal  99%  |
cpu |  sys      24% |  user      1% |  irq       0%  | idle     75%  |  cpu000 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu003 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    0.82 |  avg5    0.78 |  avg15   0.99  |               |  csw     6233 |  intr   12023 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.0G |  cache   9.7G  | buff    0.0M  |  slab  333.1M |  shmem 206.6M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
DSK |           sda |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
18079      -   martin    martin        2   9.99s    0.00s      0K       0K      0K      16K  --     -  R       1  100%   fio
 4746      -   martin    martin        2   0.01s    0.14s      0K       0K      0K       0K  --     -  S       2    2%   konsole
 3291      -   martin    martin        4   0.01s    0.11s      0K       0K      0K       0K  --     -  S       0    1%   plasma-desktop
 1488      -   root      root          1   0.03s    0.04s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
10036      -   root      root          1   0.04s    0.02s      0K       0K      0K       0K  --     -  R       2    1%   atop

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

But it gets even more visible:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [19.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01m:57s] 0$ zsh 1$ zsh 2$ zsh 3-$ zsh 4$ zsh 5$* zsh

yes, thats 0 IOPS.

0 IOPS and in zero IOPS. For minutes.

And here is why:

ATOP - merkaba 2014/12/27 13:46:52 ----------- 10s elapsed

PRC |  sys   10.77s |  user   0.31s |  #proc    334  | #trun      2  |  #tslpi   548 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     108% |  user      3% |  irq       0%  | idle    286%  |  wait      2% |  guest     0% |  curf 3.08GHz  | curscal  96%  |
cpu |  sys      72% |  user      1% |  irq       0%  | idle     28%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      19% |  user      0% |  irq       0%  | idle     81%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      11% |  user      1% |  irq       0%  | idle     87%  |  cpu003 w  1% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       6% |  user      1% |  irq       0%  | idle     91%  |  cpu002 w  1% |  guest     0% |  curf 3.11GHz  | curscal  97%  |
CPL |  avg1    2.78 |  avg5    1.34 |  avg15   1.12  |               |  csw    50192 |  intr   32379 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.0G |  cache   8.7G  | buff    0.0M  |  slab  332.6M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      5% |  read     160  | write  11177  |  KiB/w      3 |  MBr/s   0.06 |  MBw/s   4.36  | avio 0.05 ms  |
LVM |    msata-home |  busy      4% |  read      28  | write  11177  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   4.36  | avio 0.04 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
DSK |           sda |  busy      5% |  read     160  | write  10200  |  KiB/w      4 |  MBr/s   0.06 |  MBw/s   4.69  | avio 0.05 ms  |
DSK |           sdb |  busy      4% |  read      28  | write  10558  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   4.69  | avio 0.04 ms  |
NET |  transport    |  tcpi      35 |  tcpo      33  | udpi       3  |  udpo       3 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       38 |  ipo       36  | ipfrw      0  |  deliv     38 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      22 |  pcko      20  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
14973      -   root      root          1   8.92s    0.00s      0K       0K      0K     144K  --     -  S       0   89%   kworker/u8:14
17450      -   root      root          1   0.86s    0.00s      0K       0K      0K      32K  --     -  R       3    9%   kworker/u8:5
  788      -   root      root          1   0.25s    0.00s      0K       0K    128K   18880K  --     -  S       3    3%   btrfs-transact
12254      -   root      root          1   0.14s    0.00s      0K       0K     64K     576K  --     -  S       2    1%   kworker/u8:3
17332      -   root      root          1   0.11s    0.00s      0K       0K    112K    1348K  --     -  S       2    1%   kworker/u8:4
 3291      -   martin    martin        4   0.01s    0.09s      0K       0K      0K       0K  --     -  S       1    1%   plasma-deskto

ATOP - merkaba 2014/12/27 13:47:12 ----------- 10s elapsed

PRC |  sys   10.78s |  user   0.44s |  #proc    334  | #trun      3  |  #tslpi   547 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     106% |  user      4% |  irq       0%  | idle    288%  |  wait      1% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      93% |  user      0% |  irq       0%  | idle      7%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       7% |  user      0% |  irq       0%  | idle     93%  |  cpu003 w  0% |  guest     0% |  curf 3.01GHz  | curscal  94%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     94%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    3.33 |  avg5    1.56 |  avg15   1.20  |               |  csw    38253 |  intr   23104 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.9G |  cache   8.7G  | buff    0.0M  |  slab  336.5M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |    msata-home |  busy      2% |  read       0  | write   2337  |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |     sata-home |  busy      2% |  read      36  | write   2337  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |  msata-debian |  busy      1% |  read       1  | write   1630  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.65  | avio 0.03 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write   1019  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.41  | avio 0.02 ms  |
DSK |           sdb |  busy      2% |  read       1  | write   2545  |  KiB/w      5 |  MBr/s   0.00 |  MBw/s   1.45  | avio 0.07 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2461  |  KiB/w      5 |  MBr/s   0.01 |  MBw/s   1.28  | avio 0.06 ms  |
NET |  transport    |  tcpi      20 |  tcpo      20  | udpi       1  |  udpo       1 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       21 |  ipo       21  | ipfrw      0  |  deliv     21 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki       5 |  pcko       5  | si    0 Kbps  |  so    0 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
17450      -   root      root          1   9.96s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:5
 4746      -   martin    martin        2   0.06s    0.15s      0K       0K      0K       0K  --     -  S       1    2%   konsole
10508      -   root      root          1   0.13s    0.00s      0K       0K     96K    4048K  --     -  S       1    1%   kworker/u8:18
 1488      -   root      root          1   0.06s    0.06s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
17332      -   root      root          1   0.12s    0.00s      0K       0K     96K     580K  --     -  R       3    1%   kworker/u8:4
17454      -   root      root          1   0.11s    0.00s      0K       0K     32K    4416K  --     -  D       1    1%   kworker/u8:6
17516      -   root      root          1   0.09s    0.00s      0K       0K     16K     136K  --     -  S       3    1%   kworker/u8:7
 3268      -   martin    martin        3   0.02s    0.05s      0K       0K      0K       0K  --     -  S       1    1%   kwin
10036      -   root      root          1   0.05s    0.02s      0K       0K      0K       0K  --     -  R       0    1%   atop

So BTRFS is basically busy with itself and nothing else. Look at the SSD usage. They are *idling* around. Heck 2400 write accesses in 10 seconds. Thats a joke with SSDs that can do 40000 IOPS (depending on how and what you measure of course, like request size, read, write, iodepth and so).

Its kworker/u8:5 utilizing 100% of one core for minutes.

Its the random write case it seems. Here are values from fio job:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [3.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:06m:26s]
seq-write: (groupid=0, jobs=1): err= 0: pid=19212: Sat Dec 27 13:48:33 2014
write: io=4096.0MB, bw=343683KB/s, iops=85920, runt= 12204msec

    clat (usec): min=3, max=38048, avg=10.52, stdev=205.25
     lat (usec): min=3, max=38048, avg=10.66, stdev=205.43
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[  211], 99.95th=[ 2128],
     | 99.99th=[10304]
    bw (KB  /s): min=164328, max=812984, per=100.00%, avg=345585.75, stdev=201695.20
    lat (usec) : 4=0.18%, 10=95.31%, 20=4.00%, 50=0.18%, 100=0.12%
    lat (usec) : 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=13.55%, sys=46.89%, ctx=7810, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Seems fine.

But:

rand-write: (groupid=1, jobs=1): err= 0: pid=19243: Sat Dec 27 13:48:33 2014
write: io=140336KB, bw=1018.4KB/s, iops=254, runt=137803msec

    clat (usec): min=4, max=21299K, avg=3708.02, stdev=266885.61
     lat (usec): min=4, max=21299K, avg=3708.14, stdev=266885.61
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   19], 99.90th=[   28], 99.95th=[  116],
     | 99.99th=[16711680]
    bw (KB  /s): min=    0, max= 3426, per=100.00%, avg=1030.10, stdev=938.02
    lat (usec) : 10=92.63%, 20=6.89%, 50=0.43%, 100=0.01%, 250=0.02%
    lat (msec) : 250=0.01%, 500=0.01%, >=2000=0.02%
  cpu          : usr=0.06%, sys=1.59%, ctx=28720, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=35084/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=343682KB/s, minb=343682KB/s, maxb=343682KB/s, mint=12204msec, maxt=12204msec

Run status group 1 (all jobs):
WRITE: io=140336KB, aggrb=1018KB/s, minb=1018KB/s, maxb=1018KB/s, mint=137803msec, maxt=137803msec

What? 254 IOPS? With a Dual SSD BTRFS RAID 1?

What?

Ey, *what*?

[…]

Quoted 29 lines

Quoted 4 lines

There we go:

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

I have done more tests.

This is on the same /home after extending it to 170 GiB and balancing it to btrfs balance start -dusage=80

It has plenty of free space free. I updated the bug report and hope it can give an easy enough to comprehend summary. The new tests are in:

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c6

Pasting below for discussion on list. Summary: I easily get 38000 (!) IOPS. It may be an idea to reduce to 160 GiB, but right now this does not work as it says no free space on device when trying to downsize it. I may try with 165 or 162GiB.

So now we have three IOPS figures:

- 256 IOPS in worst case scenario
- 4700 IOPS when trying to reproduce worst case scenario with a fresh and small BTRFS
- 38000 IOPS when /home has unused device space to allocate chunks from

Quoted 4 lines

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c8

This is another test.

Okay, and this is the last series of tests for today.

Conclusion:

I cannot manage to get it down to the knees as before, but I come near to it.

Still its 8000 IOPS, instead of 250 IOPS, in an according to btrfs fi sh even *worse* situation than before.

That hints me at the need to look at the free space fragmentation, as in the beginning the problem started appearing with:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 144.41GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Yes, thats 13 GiB of free space *within* the chunks.

So while I can get it down in IOPS by bringing it to a situation where it can not reserve additional data chunks again, I cannot recreate the abysmal 250 IOPS figure by this. Not even with my /home filesystem.

So there is more to it. I think its important to look into free space fragmentation. It seems it needs an *aged* filesystem to recreate. At it seems the balances really helped. As I am not able to recreate the issue to that extent right now.

So this shows my original idea about free device space to allocate from also doesn´t explain it fully. It seems to be something thats going on within the chunks that explains the worst case <300 IOPS, kworker using one core for minutes and desktop locked scenario.

Is there a way to view free space fragmentation in BTRFS?

Am Sonntag, 28. Dezember 2014, 14:40:32 schrieb Martin Steigerwald:

Quoted 445 lines

Am Sonntag, 28. Dezember 2014, 14:00:19 schrieb Martin Steigerwald:

Quoted 370 lines

Am Samstag, 27. Dezember 2014, 14:55:58 schrieb Martin Steigerwald:

Quoted 121 lines

Summarized at

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

see below. This is reproducable with fio, no need for Windows XP in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 80 lines · Martin Steigerwald

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does...

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 7 lines

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

Robert's

Quoted 12 lines

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Quoted 5 lines

And I got it reproduced. *Perfectly* reproduced, I´d say.

But let me run the whole story:

1) I downsized my /home BTRFS from dual 170 GiB to dual 160 GiB again.

[… story of trying to reproduce with Windows XP defragmenting which was unsuccessful as BTRFS still had free device space to allocate new chunks from …]

Quoted 238 lines

But finally I got to:

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

So I did, if Virtualbox can write randomly in a file, I can too.

So I did:

martin@merkaba:~> cat ssd-test.fio
[global]
bs=4k
#ioengine=libaio
#iodepth=4
size=4g
#direct=1
runtime=120
filename=ssd.test.file

[seq-write]
rw=write
stonewall

[rand-write]
rw=randwrite
stonewall

And got:

ATOP - merkaba 2014/12/27 13:41:02 ----------- 10s elapsed

PRC |  sys   10.14s |  user   0.38s |  #proc    332  | #trun      2  |  #tslpi   548 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     102% |  user      4% |  irq       0%  | idle    295%  |  wait      0% |  guest     0% |  curf 3.10GHz  | curscal  96%  |
cpu |  sys      76% |  user      0% |  irq       0%  | idle     24%  |  cpu001 w  0% |  guest     0% |  curf 3.20GHz  | curscal  99%  |
cpu |  sys      24% |  user      1% |  irq       0%  | idle     75%  |  cpu000 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu003 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    0.82 |  avg5    0.78 |  avg15   0.99  |               |  csw     6233 |  intr   12023 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.0G |  cache   9.7G  | buff    0.0M  |  slab  333.1M |  shmem 206.6M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
DSK |           sda |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
18079      -   martin    martin        2   9.99s    0.00s      0K       0K      0K      16K  --     -  R       1  100%   fio
 4746      -   martin    martin        2   0.01s    0.14s      0K       0K      0K       0K  --     -  S       2    2%   konsole
 3291      -   martin    martin        4   0.01s    0.11s      0K       0K      0K       0K  --     -  S       0    1%   plasma-desktop
 1488      -   root      root          1   0.03s    0.04s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
10036      -   root      root          1   0.04s    0.02s      0K       0K      0K       0K  --     -  R       2    1%   atop

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

But it gets even more visible:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [19.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01m:57s] 0$ zsh 1$ zsh 2$ zsh 3-$ zsh 4$ zsh 5$* zsh

yes, thats 0 IOPS.

0 IOPS and in zero IOPS. For minutes.

And here is why:

ATOP - merkaba 2014/12/27 13:46:52 ----------- 10s elapsed

PRC |  sys   10.77s |  user   0.31s |  #proc    334  | #trun      2  |  #tslpi   548 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     108% |  user      3% |  irq       0%  | idle    286%  |  wait      2% |  guest     0% |  curf 3.08GHz  | curscal  96%  |
cpu |  sys      72% |  user      1% |  irq       0%  | idle     28%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      19% |  user      0% |  irq       0%  | idle     81%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      11% |  user      1% |  irq       0%  | idle     87%  |  cpu003 w  1% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       6% |  user      1% |  irq       0%  | idle     91%  |  cpu002 w  1% |  guest     0% |  curf 3.11GHz  | curscal  97%  |
CPL |  avg1    2.78 |  avg5    1.34 |  avg15   1.12  |               |  csw    50192 |  intr   32379 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.0G |  cache   8.7G  | buff    0.0M  |  slab  332.6M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      5% |  read     160  | write  11177  |  KiB/w      3 |  MBr/s   0.06 |  MBw/s   4.36  | avio 0.05 ms  |
LVM |    msata-home |  busy      4% |  read      28  | write  11177  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   4.36  | avio 0.04 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
DSK |           sda |  busy      5% |  read     160  | write  10200  |  KiB/w      4 |  MBr/s   0.06 |  MBw/s   4.69  | avio 0.05 ms  |
DSK |           sdb |  busy      4% |  read      28  | write  10558  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   4.69  | avio 0.04 ms  |
NET |  transport    |  tcpi      35 |  tcpo      33  | udpi       3  |  udpo       3 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       38 |  ipo       36  | ipfrw      0  |  deliv     38 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      22 |  pcko      20  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
14973      -   root      root          1   8.92s    0.00s      0K       0K      0K     144K  --     -  S       0   89%   kworker/u8:14
17450      -   root      root          1   0.86s    0.00s      0K       0K      0K      32K  --     -  R       3    9%   kworker/u8:5
  788      -   root      root          1   0.25s    0.00s      0K       0K    128K   18880K  --     -  S       3    3%   btrfs-transact
12254      -   root      root          1   0.14s    0.00s      0K       0K     64K     576K  --     -  S       2    1%   kworker/u8:3
17332      -   root      root          1   0.11s    0.00s      0K       0K    112K    1348K  --     -  S       2    1%   kworker/u8:4
 3291      -   martin    martin        4   0.01s    0.09s      0K       0K      0K       0K  --     -  S       1    1%   plasma-deskto

ATOP - merkaba 2014/12/27 13:47:12 ----------- 10s elapsed

PRC |  sys   10.78s |  user   0.44s |  #proc    334  | #trun      3  |  #tslpi   547 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     106% |  user      4% |  irq       0%  | idle    288%  |  wait      1% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      93% |  user      0% |  irq       0%  | idle      7%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       7% |  user      0% |  irq       0%  | idle     93%  |  cpu003 w  0% |  guest     0% |  curf 3.01GHz  | curscal  94%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     94%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    3.33 |  avg5    1.56 |  avg15   1.20  |               |  csw    38253 |  intr   23104 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.9G |  cache   8.7G  | buff    0.0M  |  slab  336.5M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |    msata-home |  busy      2% |  read       0  | write   2337  |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |     sata-home |  busy      2% |  read      36  | write   2337  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |  msata-debian |  busy      1% |  read       1  | write   1630  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.65  | avio 0.03 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write   1019  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.41  | avio 0.02 ms  |
DSK |           sdb |  busy      2% |  read       1  | write   2545  |  KiB/w      5 |  MBr/s   0.00 |  MBw/s   1.45  | avio 0.07 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2461  |  KiB/w      5 |  MBr/s   0.01 |  MBw/s   1.28  | avio 0.06 ms  |
NET |  transport    |  tcpi      20 |  tcpo      20  | udpi       1  |  udpo       1 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       21 |  ipo       21  | ipfrw      0  |  deliv     21 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki       5 |  pcko       5  | si    0 Kbps  |  so    0 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
17450      -   root      root          1   9.96s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:5
 4746      -   martin    martin        2   0.06s    0.15s      0K       0K      0K       0K  --     -  S       1    2%   konsole
10508      -   root      root          1   0.13s    0.00s      0K       0K     96K    4048K  --     -  S       1    1%   kworker/u8:18
 1488      -   root      root          1   0.06s    0.06s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
17332      -   root      root          1   0.12s    0.00s      0K       0K     96K     580K  --     -  R       3    1%   kworker/u8:4
17454      -   root      root          1   0.11s    0.00s      0K       0K     32K    4416K  --     -  D       1    1%   kworker/u8:6
17516      -   root      root          1   0.09s    0.00s      0K       0K     16K     136K  --     -  S       3    1%   kworker/u8:7
 3268      -   martin    martin        3   0.02s    0.05s      0K       0K      0K       0K  --     -  S       1    1%   kwin
10036      -   root      root          1   0.05s    0.02s      0K       0K      0K       0K  --     -  R       0    1%   atop

So BTRFS is basically busy with itself and nothing else. Look at the SSD usage. They are *idling* around. Heck 2400 write accesses in 10 seconds. Thats a joke with SSDs that can do 40000 IOPS (depending on how and what you measure of course, like request size, read, write, iodepth and so).

Its kworker/u8:5 utilizing 100% of one core for minutes.

Its the random write case it seems. Here are values from fio job:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [3.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:06m:26s]
seq-write: (groupid=0, jobs=1): err= 0: pid=19212: Sat Dec 27 13:48:33 2014
write: io=4096.0MB, bw=343683KB/s, iops=85920, runt= 12204msec

    clat (usec): min=3, max=38048, avg=10.52, stdev=205.25
     lat (usec): min=3, max=38048, avg=10.66, stdev=205.43
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[  211], 99.95th=[ 2128],
     | 99.99th=[10304]
    bw (KB  /s): min=164328, max=812984, per=100.00%, avg=345585.75, stdev=201695.20
    lat (usec) : 4=0.18%, 10=95.31%, 20=4.00%, 50=0.18%, 100=0.12%
    lat (usec) : 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=13.55%, sys=46.89%, ctx=7810, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Seems fine.

But:

rand-write: (groupid=1, jobs=1): err= 0: pid=19243: Sat Dec 27 13:48:33 2014
write: io=140336KB, bw=1018.4KB/s, iops=254, runt=137803msec

    clat (usec): min=4, max=21299K, avg=3708.02, stdev=266885.61
     lat (usec): min=4, max=21299K, avg=3708.14, stdev=266885.61
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   19], 99.90th=[   28], 99.95th=[  116],
     | 99.99th=[16711680]
    bw (KB  /s): min=    0, max= 3426, per=100.00%, avg=1030.10, stdev=938.02
    lat (usec) : 10=92.63%, 20=6.89%, 50=0.43%, 100=0.01%, 250=0.02%
    lat (msec) : 250=0.01%, 500=0.01%, >=2000=0.02%
  cpu          : usr=0.06%, sys=1.59%, ctx=28720, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=35084/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=343682KB/s, minb=343682KB/s, maxb=343682KB/s, mint=12204msec, maxt=12204msec

Run status group 1 (all jobs):
WRITE: io=140336KB, aggrb=1018KB/s, minb=1018KB/s, maxb=1018KB/s, mint=137803msec, maxt=137803msec

What? 254 IOPS? With a Dual SSD BTRFS RAID 1?

What?

Ey, *what*?

[…]

Quoted 29 lines

Quoted 4 lines

There we go:

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

I have done more tests.

This is on the same /home after extending it to 170 GiB and balancing it to btrfs balance start -dusage=80

It has plenty of free space free. I updated the bug report and hope it can give an easy enough to comprehend summary. The new tests are in:

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c6

Pasting below for discussion on list. Summary: I easily get 38000 (!) IOPS. It may be an idea to reduce to 160 GiB, but right now this does not work as it says no free space on device when trying to downsize it. I may try with 165 or 162GiB.

So now we have three IOPS figures:

- 256 IOPS in worst case scenario
- 4700 IOPS when trying to reproduce worst case scenario with a fresh and small BTRFS
- 38000 IOPS when /home has unused device space to allocate chunks from

Quoted 4 lines

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c8

This is another test.

Okay, and this is the last series of tests for today.

Conclusion:

I cannot manage to get it down to the knees as before, but I come near to it.

Still its 8000 IOPS, instead of 250 IOPS, in an according to btrfs fi sh even *worse* situation than before.

That hints me at the need to look at the free space fragmentation, as in the beginning the problem started appearing with:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 144.41GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Yes, thats 13 GiB of free space *within* the chunks.

So while I can get it down in IOPS by bringing it to a situation where it can not reserve additional data chunks again, I cannot recreate the abysmal 250 IOPS figure by this. Not even with my /home filesystem.

So there is more to it. I think its important to look into free space fragmentation. It seems it needs an *aged* filesystem to recreate. At it seems the balances really helped. As I am not able to recreate the issue to that extent right now.

Quoted 6 lines

So this shows my original idea about free device space to allocate from also doesn´t explain it fully. It seems to be something thats going on within the chunks that explains the worst case <300 IOPS, kworker using one core for minutes and desktop locked scenario.

Is there a way to view free space fragmentation in BTRFS?

So to rephrase that:

From what I perceive the worst case issue happens when

1) BTRFS cannot reserve any new chunks from unused device space anymore.

2) The free space in the existing chunks is highly fragmented.

Only one of those conditions is not sufficient to trigger it.

Thats at least my current idea about it.

Am Sonntag, 28. Dezember 2014, 14:40:32 schrieb Martin Steigerwald:

Quoted 445 lines

Am Sonntag, 28. Dezember 2014, 14:00:19 schrieb Martin Steigerwald:

Quoted 370 lines

Am Samstag, 27. Dezember 2014, 14:55:58 schrieb Martin Steigerwald:

Quoted 121 lines

Summarized at

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

see below. This is reproducable with fio, no need for Windows XP in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem.

Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills:

Quoted 80 lines · Martin Steigerwald

Quoted 55 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 54 lines · Martin Steigerwald

Quoted 4 lines

Hello!

First: Have a merry christmas and enjoy a quiet time in these days.

Quoted 8 lines

Second: At a time you feel like it, here is a little rant, but also a bug
report:

I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, skinny meta data extents – are these a problem? – and

compress=lzo:

(there is no known problem with skinny metadata, it's actually more efficient than the older format. There has been some anecdotes about mixing the skinny and fat metadata but nothing has ever been demonstrated problematic.)

Quoted 9 lines

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: b96c4f72-0523-45ac-a401-f7be73dd624a

         Total devices 2 FS bytes used 144.41GiB
         devid    1 size 160.00GiB used 160.00GiB path
         /dev/mapper/msata-home
         devid    2 size 160.00GiB used 160.00GiB path
         /dev/mapper/sata-home

Quoted 6 lines

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

This filesystem, at the allocation level, is "very full" (see below).

Quoted 10 lines

And I had hangs with BTRFS again. This time as I wanted to install tax return software in Virtualbox´d Windows XP VM (which I use once a year cause I know no tax return software for Linux which would be suitable for
Germany and I frankly don´t care about the end of security cause all surfing and other network access I will do from the Linux box and I only
run the VM behind a firewall).

And thus I try the balance dance again:

ITEM: Balance... it doesn't do what you think it does...

"Balancing" is something you should almost never need to do. It is only for cases of changing geometry (adding disks, switching RAID levels, etc.) of for cases when you've radically changed allocation behaviors (like you decided to remove all your VM's or you've decided to remove a mail spool directory full of thousands of tiny files).

Quoted 5 lines

Quoted 2 lines

People run balance all the time because they think they should. They are _usually_ incorrect in that belief.

I only see the lockups of BTRFS is the trees *occupy* all space on the device.

No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata
space. What's more, balance does *not* balance the metadata trees. The remaining space -- 154.97 GiB -- is unstructured storage for file data, and you have some 13 GiB of that available for use.

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 2 lines

I *never* so far saw it lockup if there is still space BTRFS can allocate from to *extend* a tree.

It's not a tree. It's simply space allocation. It's not even space
*usage* you're talking about here -- it's just allocation (i.e. the FS saying "I'm going to use this piece of disk for this purpose").

Quoted 2 lines

This may be a bug, but this is what I see.

Quoted 7 lines

Quoted 5 lines

And no amount of "you should not balance a BTRFS" will make that perception go away.

See, I see the sun coming out on a morning and you tell me "no, it doesn´t". Simply that is not going to match my perception.

Duncan's assertion is correct in its detail. Looking at your space

Robert's

Quoted 12 lines

usage, I would not suggest that running a balance is something you need to do. Now, since you have these lockups that seem quite repeatable, there's probably a lurking bug in there, but hacking around with balance every time you hit it isn't going to get the problem solved properly.

I think I would suggest the following:

- make sure you have some way of logging your dmesg permanently (use a different filesystem for /var/log, or a serial console, or a netconsole)

Quoted 7 lines

- when the lockup happens, hit Alt-SysRq-t a few times

- send the dmesg output here, or post to bugzilla.kernel.org

That's probably going to give enough information to the developers
to work out where the lockup is happening, and is clearly the way forward here.

Quoted 5 lines

And I got it reproduced. *Perfectly* reproduced, I´d say.

But let me run the whole story:

1) I downsized my /home BTRFS from dual 170 GiB to dual 160 GiB again.

[… story of trying to reproduce with Windows XP defragmenting which was unsuccessful as BTRFS still had free device space to allocate new chunks from …]

Quoted 238 lines

But finally I got to:

merkaba:~> date; btrfs fi sh /home ; btrfs fi df /home Sa 27. Dez 13:26:39 CET 2014
Label: 'home' uuid: [some UUID]
Total devices 2 FS bytes used 152.83GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
Data, RAID1: total=154.97GiB, used=149.58GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.26GiB GlobalReserve, single: total=512.00MiB, used=0.00B

So I did, if Virtualbox can write randomly in a file, I can too.

So I did:

martin@merkaba:~> cat ssd-test.fio
[global]
bs=4k
#ioengine=libaio
#iodepth=4
size=4g
#direct=1
runtime=120
filename=ssd.test.file

[seq-write]
rw=write
stonewall

[rand-write]
rw=randwrite
stonewall

And got:

ATOP - merkaba 2014/12/27 13:41:02 ----------- 10s elapsed

PRC |  sys   10.14s |  user   0.38s |  #proc    332  | #trun      2  |  #tslpi   548 |  #tslpu     0 |  #zombie    0  | no  procacct  |
CPU |  sys     102% |  user      4% |  irq       0%  | idle    295%  |  wait      0% |  guest     0% |  curf 3.10GHz  | curscal  96%  |
cpu |  sys      76% |  user      0% |  irq       0%  | idle     24%  |  cpu001 w  0% |  guest     0% |  curf 3.20GHz  | curscal  99%  |
cpu |  sys      24% |  user      1% |  irq       0%  | idle     75%  |  cpu000 w  0% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       1% |  user      1% |  irq       0%  | idle     98%  |  cpu003 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    0.82 |  avg5    0.78 |  avg15   0.99  |               |  csw     6233 |  intr   12023 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.0G |  cache   9.7G  | buff    0.0M  |  slab  333.1M |  shmem 206.6M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
DSK |           sda |  busy      0% |  read       8  | write      0  |  KiB/w      0 |  MBr/s   0.00 |  MBw/s   0.00  | avio 0.12 ms  |
NET |  transport    |  tcpi      16 |  tcpo      16  | udpi       0  |  udpo       0 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       16 |  ipo       16  | ipfrw      0  |  deliv     16 |               |  icmpi      0  | icmpo      0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/2
18079      -   martin    martin        2   9.99s    0.00s      0K       0K      0K      16K  --     -  R       1  100%   fio
 4746      -   martin    martin        2   0.01s    0.14s      0K       0K      0K       0K  --     -  S       2    2%   konsole
 3291      -   martin    martin        4   0.01s    0.11s      0K       0K      0K       0K  --     -  S       0    1%   plasma-desktop
 1488      -   root      root          1   0.03s    0.04s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
10036      -   root      root          1   0.04s    0.02s      0K       0K      0K       0K  --     -  R       2    1%   atop

while fio was just *laying* out the 4 GiB file. Yes, thats 100% system CPU for 10 seconds while allocatiing a 4 GiB file on a filesystem like:

martin@merkaba:~> LANG=C df -hT /home

Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/msata-home btrfs  170G  156G   17G  91% /home

where a 4 GiB file should easily fit, no? (And this output is with the 4 GiB file. So it was even 4 GiB more free before.)

But it gets even more visible:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [19.3% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01m:57s] 0$ zsh 1$ zsh 2$ zsh 3-$ zsh 4$ zsh 5$* zsh

yes, thats 0 IOPS.

0 IOPS and in zero IOPS. For minutes.

And here is why:

ATOP - merkaba 2014/12/27 13:46:52 ----------- 10s elapsed

PRC |  sys   10.77s |  user   0.31s |  #proc    334  | #trun      2  |  #tslpi   548 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     108% |  user      3% |  irq       0%  | idle    286%  |  wait      2% |  guest     0% |  curf 3.08GHz  | curscal  96%  |
cpu |  sys      72% |  user      1% |  irq       0%  | idle     28%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      19% |  user      0% |  irq       0%  | idle     81%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      11% |  user      1% |  irq       0%  | idle     87%  |  cpu003 w  1% |  guest     0% |  curf 3.19GHz  | curscal  99%  |
cpu |  sys       6% |  user      1% |  irq       0%  | idle     91%  |  cpu002 w  1% |  guest     0% |  curf 3.11GHz  | curscal  97%  |
CPL |  avg1    2.78 |  avg5    1.34 |  avg15   1.12  |               |  csw    50192 |  intr   32379 |                | numcpu     4  |
MEM |  tot    15.5G |  free    5.0G |  cache   8.7G  | buff    0.0M  |  slab  332.6M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |     sata-home |  busy      5% |  read     160  | write  11177  |  KiB/w      3 |  MBr/s   0.06 |  MBw/s   4.36  | avio 0.05 ms  |
LVM |    msata-home |  busy      4% |  read      28  | write  11177  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   4.36  | avio 0.04 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
LVM |  msata-debian |  busy      0% |  read       0  | write    844  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.33  | avio 0.02 ms  |
DSK |           sda |  busy      5% |  read     160  | write  10200  |  KiB/w      4 |  MBr/s   0.06 |  MBw/s   4.69  | avio 0.05 ms  |
DSK |           sdb |  busy      4% |  read      28  | write  10558  |  KiB/w      4 |  MBr/s   0.01 |  MBw/s   4.69  | avio 0.04 ms  |
NET |  transport    |  tcpi      35 |  tcpo      33  | udpi       3  |  udpo       3 |  tcpao      2 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       38 |  ipo       36  | ipfrw      0  |  deliv     38 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki      22 |  pcko      20  | si    9 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
14973      -   root      root          1   8.92s    0.00s      0K       0K      0K     144K  --     -  S       0   89%   kworker/u8:14
17450      -   root      root          1   0.86s    0.00s      0K       0K      0K      32K  --     -  R       3    9%   kworker/u8:5
  788      -   root      root          1   0.25s    0.00s      0K       0K    128K   18880K  --     -  S       3    3%   btrfs-transact
12254      -   root      root          1   0.14s    0.00s      0K       0K     64K     576K  --     -  S       2    1%   kworker/u8:3
17332      -   root      root          1   0.11s    0.00s      0K       0K    112K    1348K  --     -  S       2    1%   kworker/u8:4
 3291      -   martin    martin        4   0.01s    0.09s      0K       0K      0K       0K  --     -  S       1    1%   plasma-deskto

ATOP - merkaba 2014/12/27 13:47:12 ----------- 10s elapsed

PRC |  sys   10.78s |  user   0.44s |  #proc    334  | #trun      3  |  #tslpi   547 |  #tslpu     3 |  #zombie    0  | no  procacct  |
CPU |  sys     106% |  user      4% |  irq       0%  | idle    288%  |  wait      1% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys      93% |  user      0% |  irq       0%  | idle      7%  |  cpu002 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       7% |  user      0% |  irq       0%  | idle     93%  |  cpu003 w  0% |  guest     0% |  curf 3.01GHz  | curscal  94%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     94%  |  cpu000 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
cpu |  sys       3% |  user      2% |  irq       0%  | idle     95%  |  cpu001 w  0% |  guest     0% |  curf 3.00GHz  | curscal  93%  |
CPL |  avg1    3.33 |  avg5    1.56 |  avg15   1.20  |               |  csw    38253 |  intr   23104 |                | numcpu     4  |
MEM |  tot    15.5G |  free    4.9G |  cache   8.7G  | buff    0.0M  |  slab  336.5M |  shmem 207.2M |  vmbal   0.0M  | hptot   0.0M  |
SWP |  tot    12.0G |  free   11.7G |                |               |               |               |  vmcom   3.4G  | vmlim  19.7G  |
LVM |    msata-home |  busy      2% |  read       0  | write   2337  |  KiB/w      3 |  MBr/s   0.00 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |     sata-home |  busy      2% |  read      36  | write   2337  |  KiB/w      3 |  MBr/s   0.01 |  MBw/s   0.91  | avio 0.07 ms  |
LVM |  msata-debian |  busy      1% |  read       1  | write   1630  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.65  | avio 0.03 ms  |
LVM |   sata-debian |  busy      0% |  read       0  | write   1019  |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.41  | avio 0.02 ms  |
DSK |           sdb |  busy      2% |  read       1  | write   2545  |  KiB/w      5 |  MBr/s   0.00 |  MBw/s   1.45  | avio 0.07 ms  |
DSK |           sda |  busy      1% |  read      36  | write   2461  |  KiB/w      5 |  MBr/s   0.01 |  MBw/s   1.28  | avio 0.06 ms  |
NET |  transport    |  tcpi      20 |  tcpo      20  | udpi       1  |  udpo       1 |  tcpao      1 |  tcppo      1  | tcprs      0  |
NET |  network      |  ipi       21 |  ipo       21  | ipfrw      0  |  deliv     21 |               |  icmpi      0  | icmpo      0  |
NET |  eth0      0% |  pcki       5 |  pcko       5  | si    0 Kbps  |  so    0 Kbps |  erri       0 |  erro       0  | drpo       0  |
NET |  lo      ---- |  pcki      16 |  pcko      16  | si    2 Kbps  |  so    2 Kbps |  erri       0 |  erro       0  | drpo       0  |

  PID    TID   RUID      EUID        THR  SYSCPU   USRCPU   VGROW    RGROW   RDDSK    WRDSK  ST   EXC  S   CPUNR   CPU   CMD        1/3
17450      -   root      root          1   9.96s    0.00s      0K       0K      0K       0K  --     -  R       2  100%   kworker/u8:5
 4746      -   martin    martin        2   0.06s    0.15s      0K       0K      0K       0K  --     -  S       1    2%   konsole
10508      -   root      root          1   0.13s    0.00s      0K       0K     96K    4048K  --     -  S       1    1%   kworker/u8:18
 1488      -   root      root          1   0.06s    0.06s      0K       0K      0K       0K  --     -  S       0    1%   Xorg
17332      -   root      root          1   0.12s    0.00s      0K       0K     96K     580K  --     -  R       3    1%   kworker/u8:4
17454      -   root      root          1   0.11s    0.00s      0K       0K     32K    4416K  --     -  D       1    1%   kworker/u8:6
17516      -   root      root          1   0.09s    0.00s      0K       0K     16K     136K  --     -  S       3    1%   kworker/u8:7
 3268      -   martin    martin        3   0.02s    0.05s      0K       0K      0K       0K  --     -  S       1    1%   kwin
10036      -   root      root          1   0.05s    0.02s      0K       0K      0K       0K  --     -  R       0    1%   atop

So BTRFS is basically busy with itself and nothing else. Look at the SSD usage. They are *idling* around. Heck 2400 write accesses in 10 seconds. Thats a joke with SSDs that can do 40000 IOPS (depending on how and what you measure of course, like request size, read, write, iodepth and so).

Its kworker/u8:5 utilizing 100% of one core for minutes.

Its the random write case it seems. Here are values from fio job:

martin@merkaba:~> fio ssd-test.fio
seq-write: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
rand-write: (g=1): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.11
Starting 2 processes
Jobs: 1 (f=1): [_(1),w(1)] [3.6% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:06m:26s]
seq-write: (groupid=0, jobs=1): err= 0: pid=19212: Sat Dec 27 13:48:33 2014
write: io=4096.0MB, bw=343683KB/s, iops=85920, runt= 12204msec

    clat (usec): min=3, max=38048, avg=10.52, stdev=205.25
     lat (usec): min=3, max=38048, avg=10.66, stdev=205.43
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[    9],
     | 99.00th=[   14], 99.50th=[   20], 99.90th=[  211], 99.95th=[ 2128],
     | 99.99th=[10304]
    bw (KB  /s): min=164328, max=812984, per=100.00%, avg=345585.75, stdev=201695.20
    lat (usec) : 4=0.18%, 10=95.31%, 20=4.00%, 50=0.18%, 100=0.12%
    lat (usec) : 250=0.12%, 500=0.02%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.01%, 50=0.01%
  cpu          : usr=13.55%, sys=46.89%, ctx=7810, majf=0, minf=6
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Seems fine.

But:

rand-write: (groupid=1, jobs=1): err= 0: pid=19243: Sat Dec 27 13:48:33 2014
write: io=140336KB, bw=1018.4KB/s, iops=254, runt=137803msec

    clat (usec): min=4, max=21299K, avg=3708.02, stdev=266885.61
     lat (usec): min=4, max=21299K, avg=3708.14, stdev=266885.61
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    6], 60.00th=[    6],
     | 70.00th=[    7], 80.00th=[    7], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   18], 99.50th=[   19], 99.90th=[   28], 99.95th=[  116],
     | 99.99th=[16711680]
    bw (KB  /s): min=    0, max= 3426, per=100.00%, avg=1030.10, stdev=938.02
    lat (usec) : 10=92.63%, 20=6.89%, 50=0.43%, 100=0.01%, 250=0.02%
    lat (msec) : 250=0.01%, 500=0.01%, >=2000=0.02%
  cpu          : usr=0.06%, sys=1.59%, ctx=28720, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=35084/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=343682KB/s, minb=343682KB/s, maxb=343682KB/s, mint=12204msec, maxt=12204msec

Run status group 1 (all jobs):
WRITE: io=140336KB, aggrb=1018KB/s, minb=1018KB/s, maxb=1018KB/s, mint=137803msec, maxt=137803msec

What? 254 IOPS? With a Dual SSD BTRFS RAID 1?

What?

Ey, *what*?

[…]

Quoted 29 lines

Quoted 4 lines

There we go:

Bug 90401 - btrfs kworker thread uses up 100% of a Sandybridge core for minutes on random write into big file
https://bugzilla.kernel.org/show_bug.cgi?id=90401

I have done more tests.

This is on the same /home after extending it to 170 GiB and balancing it to btrfs balance start -dusage=80

It has plenty of free space free. I updated the bug report and hope it can give an easy enough to comprehend summary. The new tests are in:

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c6

Pasting below for discussion on list. Summary: I easily get 38000 (!) IOPS. It may be an idea to reduce to 160 GiB, but right now this does not work as it says no free space on device when trying to downsize it. I may try with 165 or 162GiB.

So now we have three IOPS figures:

- 256 IOPS in worst case scenario
- 4700 IOPS when trying to reproduce worst case scenario with a fresh and small BTRFS
- 38000 IOPS when /home has unused device space to allocate chunks from

Quoted 4 lines

https://bugzilla.kernel.org/show_bug.cgi?id=90401#c8

This is another test.

Okay, and this is the last series of tests for today.

Conclusion:

I cannot manage to get it down to the knees as before, but I come near to it.

Still its 8000 IOPS, instead of 250 IOPS, in an according to btrfs fi sh even *worse* situation than before.

That hints me at the need to look at the free space fragmentation, as in the beginning the problem started appearing with:

merkaba:~> btrfs fi sh /home
Label: 'home' uuid: […]
Total devices 2 FS bytes used 144.41GiB
devid 1 size 160.00GiB used 160.00GiB path /dev/mapper/msata-home
devid 2 size 160.00GiB used 160.00GiB path /dev/mapper/sata-home

Btrfs v3.17
merkaba:~> btrfs fi df /home
Data, RAID1: total=154.97GiB, used=141.12GiB System, RAID1: total=32.00MiB, used=48.00KiB Metadata, RAID1: total=5.00GiB, used=3.29GiB GlobalReserve, single: total=512.00MiB, used=0.00B

Yes, thats 13 GiB of free space *within* the chunks.

So while I can get it down in IOPS by bringing it to a situation where it can not reserve additional data chunks again, I cannot recreate the abysmal 250 IOPS figure by this. Not even with my /home filesystem.

So there is more to it. I think its important to look into free space fragmentation. It seems it needs an *aged* filesystem to recreate. At it seems the balances really helped. As I am not able to recreate the issue to that extent right now.

Quoted 6 lines

So this shows my original idea about free device space to allocate from also doesn´t explain it fully. It seems to be something thats going on within the chunks that explains the worst case <300 IOPS, kworker using one core for minutes and desktop locked scenario.

Is there a way to view free space fragmentation in BTRFS?

So to rephrase that:

From what I perceive the worst case issue happens when

1) BTRFS cannot reserve any new chunks from unused device space anymore.

2) The free space in the existing chunks is highly fragmented.

Only one of those conditions is not sufficient to trigger it.

Thats at least my current idea about it.

Quoted 2 lines · Martin Steigerwald

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 1 line · Martin Steigerwald

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

Quoted 7 lines · Hugo Mills

Quoted 2 lines · Martin Steigerwald

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 1 line · Martin Steigerwald

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

I do see something similar, but there are so many problems going on I have no idea which ones to report, and which ones are my own doing. :-P

I see lots of CPU being burned when all the disk space is allocated to chunks, but there is still lots of space free (multiple GB) inside the chunks.

iotop shows a crapton of disk writes (1-5MB/sec) from one kworker. There are maybe a few kB/sec of writes through the filesystem at the time.

The filesystem where I see this most is on a laptop, so the disk writes also hit the CPU again for encryption. There's so much CPU usage it's worth mentioning twice. :-(

'watch cat /proc/12345/stack' on the active processes shows the kernel fairly often in that new chunk deallocator function whose name escapes me at the moment.

Deleting a bunch of data then running balance helps return to sane CPU usage...for a while (maybe a week?).

It's not technically "locked up" per se, but when a 5KB download takes a minute or more, most users won't wait around to see the difference.

Kernel versions I'm using are 3.17.7 and 3.18.1.

Quoted 27 lines · Hugo Mills

Quoted 7 lines · Martin Steigerwald

Quoted 2 lines

Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White:

Quoted 1 line · Martin Steigerwald

Now, since you're seeing lockups when the space on your disks is
all allocated I'd say that's a bug. However, you're the *only* person who's reported this as a regular occurrence. Does this happen with all filesystems you have, or just this one?

I do see something similar, but there are so many problems going on I have no idea which ones to report, and which ones are my own doing. :-P

I see lots of CPU being burned when all the disk space is allocated to chunks, but there is still lots of space free (multiple GB) inside the chunks.

iotop shows a crapton of disk writes (1-5MB/sec) from one kworker. There are maybe a few kB/sec of writes through the filesystem at the time.

The filesystem where I see this most is on a laptop, so the disk writes also hit the CPU again for encryption. There's so much CPU usage it's worth mentioning twice. :-(

'watch cat /proc/12345/stack' on the active processes shows the kernel fairly often in that new chunk deallocator function whose name escapes me at the moment.

Quoted 7 lines

Deleting a bunch of data then running balance helps return to sane CPU usage...for a while (maybe a week?).

It's not technically "locked up" per se, but when a 5KB download takes a minute or more, most users won't wait around to see the difference.

Kernel versions I'm using are 3.17.7 and 3.18.1.

OK, so I'd like to change my statement above.

When I first read Martin's problem, I thought that he was referring
to a complete, hit-the-power-button kind of lock-up. Given that (erroneous) assumption, I stand by my (now pointless) statement. :)

I realised during a brief conversation on IRC that Martin was
actually referring to long but temporary periods where the machine is unusable by any process requiring disk activity. There's clearly a number of people seeing that.

It doesn't stop it being a major problem, but it does change the
interpretation considerably.

My simple test case didn´t trigger it, and I so not have another twice 160 GiB available on this SSDs available to try with a copy of my home filesystem. Then I could safely test without bringing the desktop session to an halt. Maybe someone has an idea on how to "enhance" my test case in order to reliably trigger the issue.

It may be challenging tough. My /home is quite a filesystem. It has a maildir with at least one million of files (yeah, I am performance testing KMail and Akonadi as well to the limit!), and it has git repos and this one VM image, and the desktop search and the Akonadi database. In other words: It has been hit nicely with various mostly random I think workloads over the last about six months. I bet its not that easy to simulate that. Maybe some runs of compilebench to age the filesystem before the fio test?

That said, BTRFS performs a lot better. The complete lockups without any CPU usage of 3.15 and 3.16 have gone for sure. Thats wonderful. But there is this kworker issue now. I noticed it that gravely just while trying to complete this tax returns stuff with the Windows XP VM. Otherwise it may have happened, I have seen some backtraces in kern.log, but it didn´t last for minutes. So this indeed is of less severity than the full lockups with 3.15 and 3.16.

Zygo, was is the characteristics of your filesystem. Do you use compress=lzo and skinny metadata as well? How are the chunks allocated? What kind of data you have on it?

Quoted 22 lines

My simple test case didn´t trigger it, and I so not have another twice 160 GiB available on this SSDs available to try with a copy of my home filesystem. Then I could safely test without bringing the desktop session to an halt. Maybe someone has an idea on how to "enhance" my test case in order to reliably trigger the issue.

It may be challenging tough. My /home is quite a filesystem. It has a maildir with at least one million of files (yeah, I am performance testing KMail and Akonadi as well to the limit!), and it has git repos and this one VM image, and the desktop search and the Akonadi database. In other words: It has been hit nicely with various mostly random I think workloads over the last about six months. I bet its not that easy to simulate that. Maybe some runs of compilebench to age the filesystem before the fio test?

That said, BTRFS performs a lot better. The complete lockups without any CPU usage of 3.15 and 3.16 have gone for sure. Thats wonderful. But there is this kworker issue now. I noticed it that gravely just while trying to complete this tax returns stuff with the Windows XP VM. Otherwise it may have happened, I have seen some backtraces in kern.log, but it didn´t last for minutes. So this indeed is of less severity than the full lockups with 3.15 and 3.16.

Quoted 3 lines

Zygo, was is the characteristics of your filesystem. Do you use compress=lzo and skinny metadata as well? How are the chunks allocated? What kind of data you have on it?

compress-force (default zlib), no skinny-metadata. Chunks are d=single, m=dup. Data is a mix of various desktop applications, most active file sizes from a few hundred K to a few MB, maybe 300k-400k files. No database or VM workloads. Filesystem is 100GB and is usually between 98 and 99% full (about 1-2GB free).

I have another filesystem which has similar problems when it's 99.99% full (it's 13TB, so 0.01% is 1.3GB). That filesystem is RAID1 with skinny-metadata and no-holes.

On various filesystems I have the above CPU-burning problem, a bunch of irreproducible random crashes, and a hang with a kernel stack that goes through SyS_unlinkat and btrfs_evict_inode.

Am Sonntag, 28. Dezember 2014, 21:07:05 schrieb Zygo Blaxell:

Quoted 33 lines · Martin Steigerwald

Quoted 22 lines

My simple test case didn´t trigger it, and I so not have another twice 160 GiB available on this SSDs available to try with a copy of my home filesystem. Then I could safely test without bringing the desktop session to an halt. Maybe someone has an idea on how to "enhance" my test case in order to reliably trigger the issue.

It may be challenging tough. My /home is quite a filesystem. It has a maildir with at least one million of files (yeah, I am performance testing KMail and Akonadi as well to the limit!), and it has git repos and this one VM image, and the desktop search and the Akonadi database. In other words: It has been hit nicely with various mostly random I think workloads over the last about six months. I bet its not that easy to simulate that. Maybe some runs of compilebench to age the filesystem before the fio test?

That said, BTRFS performs a lot better. The complete lockups without any CPU usage of 3.15 and 3.16 have gone for sure. Thats wonderful. But there is this kworker issue now. I noticed it that gravely just while trying to complete this tax returns stuff with the Windows XP VM. Otherwise it may have happened, I have seen some backtraces in kern.log, but it didn´t last for minutes. So this indeed is of less severity than the full lockups with 3.15 and 3.16.

Quoted 3 lines

Zygo, was is the characteristics of your filesystem. Do you use compress=lzo and skinny metadata as well? How are the chunks allocated? What kind of data you have on it?

compress-force (default zlib), no skinny-metadata. Chunks are d=single, m=dup. Data is a mix of various desktop applications, most active file sizes from a few hundred K to a few MB, maybe 300k-400k files. No database or VM workloads. Filesystem is 100GB and is usually between 98 and 99% full (about 1-2GB free).

Quoted 7 lines

I have another filesystem which has similar problems when it's 99.99% full (it's 13TB, so 0.01% is 1.3GB). That filesystem is RAID1 with skinny-metadata and no-holes.

On various filesystems I have the above CPU-burning problem, a bunch of irreproducible random crashes, and a hang with a kernel stack that goes through SyS_unlinkat and btrfs_evict_inode.

Zygo, thanks. That desktop filesystem sounds a bit similar to my usecase, with the interesting difference that you have no databases or VMs on it.

That said, I use the Windows XP rarely, but using it was what made the issue so visible for me. Is your desktop filesystem on SSD?

Do you have the chance to extend one of the affected filesystems to check my theory that this does not happen as long as BTRFS can still allocate new data chunks? If its right, your FS should be fluent again as long as you see more than 1 GiB free

Label: none uuid: 53bdf47c-4298-45bc-a30f-8a310c274069
Total devices 2 FS bytes used 512.00KiB
devid 1 size 10.00GiB used 6.53GiB path /dev/mapper/sata-btrfsraid1
devid 2 size 10.00GiB used 6.53GiB path /dev/mapper/msata-btrfsraid1

between "size" and "used" in btrfs fi sh. I suggest going with at least 2-3 GiB, as BTRFS may allocate just one chunk so quickly that you do not have the chance to recognize the difference.

Well, and if thats works for you, we are back to my recommendation:

More so than with other filesystems give BTRFS plenty of free space to operate with. At best as much, that you always have a mininum of 2-3 GiB unused device space for chunk reservation left. One could even do some Nagios/Icinga monitoring plugin for that :)

Signature · 2 lines

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Quoted 2 lines

Am Sonntag, 28. Dezember 2014, 21:07:05 schrieb Zygo Blaxell:

Quoted 1 line · Martin Steigerwald

Quoted 19 lines

Quoted 11 lines

Quoted 4 lines

Zygo, was is the characteristics of your filesystem. Do you use compress=lzo and skinny metadata as well? How are the chunks allocated?
What kind of data you have on it?

compress-force (default zlib), no skinny-metadata. Chunks are d=single, m=dup. Data is a mix of various desktop applications, most active file sizes from a few hundred K to a few MB, maybe 300k-400k files. No database or VM workloads. Filesystem is 100GB and is usually between 98 and 99% full (about 1-2GB free).

Quoted 7 lines

I have another filesystem which has similar problems when it's 99.99% full (it's 13TB, so 0.01% is 1.3GB). That filesystem is RAID1 with skinny-metadata and no-holes.

On various filesystems I have the above CPU-burning problem, a bunch of irreproducible random crashes, and a hang with a kernel stack that goes through SyS_unlinkat and btrfs_evict_inode.

Quoted 5 lines

Zygo, thanks. That desktop filesystem sounds a bit similar to my usecase, with the interesting difference that you have no databases or VMs on it.

That said, I use the Windows XP rarely, but using it was what made the issue so visible for me. Is your desktop filesystem on SSD?

No, but I recently stumbled across the same symptoms on an 8GB SD card on kernel 3.12.24 (raspberry pi). When the filesystem hit over ~97% full, all accesses were blocked for several minutes. I was able to work around it by adjusting the threshold on a garbage collector daemon (i.e. deleting a lot of expendable files) to keep usage below 90%. I didn't try to balance the filesystem, and didn't seem to need to.

ext3 has a related problem when it's nearly full: it will try to search gigabytes of block allocation bitmaps searching for a free block, which can result in a single 'mkdir' call spending 45 minutes reading a large slow 99.5% full filesystem.

I'd expect a btrfs filesystem that was nearly full to have a small tree of cached free space extents and be able to search it quickly even if the result is negative (i.e. there's no free space). It seems to be doing something else... :-P

Quoted 14 lines

Do you have the chance to extend one of the affected filesystems to check my theory that this does not happen as long as BTRFS can still allocate new
data chunks? If its right, your FS should be fluent again as long as you see more than 1 GiB free

Label: none uuid: 53bdf47c-4298-45bc-a30f-8a310c274069

Total devices 2 FS bytes used 512.00KiB
devid 1 size 10.00GiB used 6.53GiB path
/dev/mapper/sata-btrfsraid1
devid 2 size 10.00GiB used 6.53GiB path
/dev/mapper/msata-btrfsraid1

Quoted 4 lines

between "size" and "used" in btrfs fi sh. I suggest going with at least 2-3
GiB, as BTRFS may allocate just one chunk so quickly that you do not have the chance to recognize the difference.

So far I've found that problems start when space drops below 1GB free (although it can go as low as 400MB) and problems stop when space gets above 1GB free, even without resizing or balancing the filesystem. I've adjusted free space monitoring thresholds accordingly for now, and it seems to be keeping things working so far.

Quoted 6 lines

Well, and if thats works for you, we are back to my recommendation:

More so than with other filesystems give BTRFS plenty of free space to operate with. At best as much, that you always have a mininum of 2-3 GiB unused device space for chunk reservation left. One could even do some Nagios/Icinga monitoring plugin for that :)

Am Dienstag, 6. Januar 2015, 15:03:23 schrieb Zygo Blaxell:

Quoted 4 lines

ext3 has a related problem when it's nearly full: it will try to search gigabytes of block allocation bitmaps searching for a free block, which can result in a single 'mkdir' call spending 45 minutes reading a large slow 99.5% full filesystem.

Ok, thats for bitmap access. Ext4 uses extens.

Quoted 5 lines

So far I've found that problems start when space drops below 1GB free (although it can go as low as 400MB) and problems stop when space gets above 1GB free, even without resizing or balancing the filesystem. I've adjusted free space monitoring thresholds accordingly for now, and it seems to be keeping things working so far.

Just to see whether we are on the same terms: You talk about space that BTRFS has not yet reserved for chunks, i.e. the difference between size and used in btrfs fi sh, right?

No BTRFS developers commented yet on this, neither in this thread nor in the bug report at kernel.org I made.

Martin Steigerwald posted on Wed, 07 Jan 2015 20:08:50 +0100 as excerpted:

Quoted 2 lines

No BTRFS developers commented yet on this, neither in this thread nor in the bug report at kernel.org I made.

Just a quick general note on this point...

There has in the past (and I believe referenced on the wiki) been dev comment to the effect that on the list they tend to find particular reports/threads and work on them until they find and either fix the issue or (when not urgent) decide it must wait for something else, first. During the time they're busy pursuing such a report, they don't read others on the list very closely, and such list-only bug reports may thus get dropped on the floor and never worked on.

The recommendation, then, is to report it to the list, and if not picked up right away and you plan on being around in a few weeks/months when they potentially get to it, file a bug on it, so it doesn't get dropped on the floor.

Duncan, I *did* file a bug.

Quoted 3 lines

But the recommendation is to file the bugzilla report precisely so it does /not/ get lost, and you've done that, so... you've done your part there and now comes the enforced patience bit of waiting [...]