Skip to contentSkip to conversation

ENOSPC on mostly empty file system

DISCUSSION Arnd Bergmann 5 participants 11 messages Sep 9, 2014 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi Chris,

As I mentioned at the kernel summit, I have a file system that I use mostly for storing my one kernel git tree and occasionally some build trees (those are normally on a tmpfs), and I have again run into the problem where the file system is only partially full (I think 18% in this case) but I am unable to create new files.

As you recommended, I have created an image using btrfs-image and uploaded it to google drive for inspection.

I also played around with it some more. After removing a few small files, I could create new files with up to 20-60MB again before hitting ENOSPC. I then did a 'make clean' in all the object directories I had and after that could fill the file system up to 100% by creating a 62GB file using the same command (dd if=/dev/zero of=x).

While probably unrelated, I also saw a few older 'hung task' messages with btrfs call chains in dmesg and I'm adding them here in this mail as well. The kernel messages are ten days old, but I did not reboot in the meantime and I never saw them again before or after that. The kernel is the stock Ubuntu 3.16.0 image from Ubuntu, I upgrade to that after running an old 3.11 kernel that I never rebooted.

The 82 MB image file is at
https://drive.google.com/file/d/0B_XQwQ5KlfJAWDJfS1E0TG1CLTA/edit?usp=sharing

Arnd

[407459.475639] INFO: task BrowserBlocking:3282 blocked for more than 120 seconds.
[407459.475649]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407459.475652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407459.475656] BrowserBlocking D ffff88042fc94800     0  3282   2580 0x00000000
[407459.475664]  ffff88040d9f7db0 0000000000000086 ffff88041215b2a0 0000000000014800
[407459.475671]  ffff88040d9f7fd8 0000000000014800 ffff88041215b2a0 ffff8800bbb12af8
[407459.475677]  ffff8800bbb12afc ffff88041215b2a0 00000000ffffffff ffff8800bbb12b00
[407459.475683] Call Trace:
[407459.475699]  [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407459.475707]  [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407459.475714]  [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407459.475771]  [<ffffffffc0546535>] btrfs_log_inode_parent+0xd5/0x550 [btrfs]
[407459.475804]  [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407459.475834]  [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407459.475844]  [<ffffffff81210461>] do_fsync+0x51/0x80
[407459.475849]  [<ffffffff812106f0>] SyS_fsync+0x10/0x20
[407459.475856]  [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407459.475861] INFO: task BrowserBlocking:3283 blocked for more than 120 seconds.
[407459.475864]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407459.475866] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407459.475869] BrowserBlocking D ffff88042fd54800     0  3283   2580 0x00000000
[407459.475875]  ffff88040bb13db0 0000000000000086 ffff88041215ef60 0000000000014800
[407459.475881]  ffff88040bb13fd8 0000000000014800 ffff88041215ef60 ffff880429b43590
[407459.475886]  ffff880429b43594 ffff88041215ef60 00000000ffffffff ffff880429b43598
[407459.475892] Call Trace:
[407459.475899]  [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407459.475906]  [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407459.475912]  [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407459.475943]  [<ffffffffc0546895>] btrfs_log_inode_parent+0x435/0x550 [btrfs]
[407459.475976]  [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407459.476005]  [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407459.476012]  [<ffffffff81210461>] do_fsync+0x51/0x80
[407459.476018]  [<ffffffff81210713>] SyS_fdatasync+0x13/0x20
[407459.476023]  [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407459.476099] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407459.476102]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407459.476104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407459.476107] kworker/u65:6   D ffff88042fd14800     0 2090685      2 0x00000000
[407459.476131] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407459.476134]  ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407459.476140]  ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407459.476145]  ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407459.476151] Call Trace:
[407459.476158]  [<ffffffff8177b089>] schedule+0x29/0x70
[407459.476164]  [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407459.476171]  [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407459.476177]  [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407459.476184]  [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407459.476191]  [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407459.476198]  [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407459.476221]  [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407459.476243]  [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407459.476266]  [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407459.476273]  [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407459.476278]  [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407459.476284]  [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407459.476290]  [<ffffffff81094a1b>] kthread+0xdb/0x100
[407459.476297]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407459.476302]  [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407459.476308]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407579.528238] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407579.528247]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407579.528251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407579.528255] kworker/u65:6   D ffff88042fd14800     0 2090685      2 0x00000000
[407579.528308] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407579.528312]  ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407579.528318]  ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407579.528324]  ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407579.528330] Call Trace:
[407579.528345]  [<ffffffff8177b089>] schedule+0x29/0x70
[407579.528352]  [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407579.528360]  [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407579.528366]  [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407579.528373]  [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407579.528380]  [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407579.528388]  [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407579.528411]  [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407579.528434]  [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407579.528458]  [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407579.528464]  [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407579.528470]  [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407579.528475]  [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407579.528481]  [<ffffffff81094a1b>] kthread+0xdb/0x100
[407579.528488]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407579.528494]  [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407579.528501]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407699.580802] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407699.580811]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407699.580815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407699.580818] kworker/u65:6   D ffff88042fd14800     0 2090685      2 0x00000000
[407699.580872] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407699.580877]  ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407699.580883]  ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407699.580889]  ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407699.580895] Call Trace:
[407699.580910]  [<ffffffff8177b089>] schedule+0x29/0x70
[407699.580917]  [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407699.580925]  [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407699.580930]  [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407699.580938]  [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407699.580945]  [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407699.580952]  [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407699.580976]  [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407699.580999]  [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407699.581022]  [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407699.581029]  [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407699.581034]  [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407699.581040]  [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407699.581046]  [<ffffffff81094a1b>] kthread+0xdb/0x100
[407699.581053]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407699.581059]  [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407699.581066]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407819.633225] INFO: task BrowserBlocking:3282 blocked for more than 120 seconds.
[407819.633234]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633238] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633242] BrowserBlocking D ffff88042fc94800     0  3282   2580 0x00000000
[407819.633250]  ffff88040d9f7db0 0000000000000086 ffff88041215b2a0 0000000000014800
[407819.633256]  ffff88040d9f7fd8 0000000000014800 ffff88041215b2a0 ffff8800bbb12af8
[407819.633262]  ffff8800bbb12afc ffff88041215b2a0 00000000ffffffff ffff8800bbb12b00
[407819.633269] Call Trace:
[407819.633285]  [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407819.633292]  [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407819.633299]  [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407819.633357]  [<ffffffffc0546535>] btrfs_log_inode_parent+0xd5/0x550 [btrfs]
[407819.633390]  [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407819.633419]  [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407819.633429]  [<ffffffff81210461>] do_fsync+0x51/0x80
[407819.633435]  [<ffffffff812106f0>] SyS_fsync+0x10/0x20
[407819.633440]  [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407819.633446] INFO: task BrowserBlocking:3284 blocked for more than 120 seconds.
[407819.633449]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633451] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633454] BrowserBlocking D ffff88042fc14800     0  3284   2580 0x00000000
[407819.633460]  ffff880416d4fdb0 0000000000000086 ffff88041215e540 0000000000014800
[407819.633466]  ffff880416d4ffd8 0000000000014800 ffff88041215e540 ffff880429b43590
[407819.633471]  ffff880429b43594 ffff88041215e540 00000000ffffffff ffff880429b43598
[407819.633477] Call Trace:
[407819.633485]  [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407819.633491]  [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407819.633498]  [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407819.633530]  [<ffffffffc0546895>] btrfs_log_inode_parent+0x435/0x550 [btrfs]
[407819.633561]  [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407819.633590]  [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407819.633597]  [<ffffffff81210461>] do_fsync+0x51/0x80
[407819.633603]  [<ffffffff81210713>] SyS_fdatasync+0x13/0x20
[407819.633608]  [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407819.633687] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407819.633690]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633693] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633695] kworker/u65:6   D ffff88042fd14800     0 2090685      2 0x00000000
[407819.633719] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407819.633722]  ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407819.633728]  ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407819.633734]  ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407819.633739] Call Trace:
[407819.633747]  [<ffffffff8177b089>] schedule+0x29/0x70
[407819.633753]  [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407819.633760]  [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407819.633765]  [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407819.633772]  [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407819.633779]  [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407819.633786]  [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407819.633809]  [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407819.633832]  [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407819.633854]  [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407819.633861]  [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407819.633866]  [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407819.633872]  [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407819.633878]  [<ffffffff81094a1b>] kthread+0xdb/0x100
[407819.633885]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407819.633890]  [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407819.633897]  [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407819.633907] INFO: task find:2091978 blocked for more than 120 seconds.
[407819.633910]       Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633915] find            D ffff88042fc94800     0 2091978 2091977 0x00000000
[407819.633921]  ffff8805c8c83ab0 0000000000000086 ffff880271929e60 0000000000014800
[407819.633926]  ffff8805c8c83fd8 0000000000014800 ffff880271929e60 ffff8805c8c83bd8
[407819.633932]  ffff8805c8c83bd0 7fffffffffffffff ffff880271929e60 0000000000000001
[407819.633937] Call Trace:
[407819.633944]  [<ffffffff8177b089>] schedule+0x29/0x70
[407819.633950]  [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407819.633955]  [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407819.633976]  [<ffffffffc04eb215>] ? block_rsv_release_bytes+0x115/0x180 [btrfs]
[407819.633981]  [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407819.633988]  [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407819.633994]  [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407819.633999]  [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407819.634023]  [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407819.634046]  [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407819.634068]  [<ffffffffc04eff83>] reserve_metadata_bytes+0x203/0x490 [btrfs]
[407819.634075]  [<ffffffff8177dcd2>] ? mutex_lock+0x12/0x30
[407819.634099]  [<ffffffffc04f0786>] btrfs_block_rsv_add+0x26/0x50 [btrfs]
[407819.634126]  [<ffffffffc0507e03>] start_transaction+0x453/0x590 [btrfs]
[407819.634154]  [<ffffffffc0507f5b>] btrfs_start_transaction+0x1b/0x20 [btrfs]
[407819.634182]  [<ffffffffc050d4a8>] btrfs_dirty_inode+0xb8/0xf0 [btrfs]
[407819.634210]  [<ffffffffc050d55f>] btrfs_update_time+0x7f/0xc0 [btrfs]
[407819.634217]  [<ffffffff811fb285>] update_time+0x25/0xd0
[407819.634223]  [<ffffffff811fb409>] touch_atime+0xd9/0x140
[407819.634228]  [<ffffffff811f3b60>] iterate_dir+0xe0/0x130
[407819.634234]  [<ffffffff811f3fb9>] SyS_getdents+0x89/0x100
[407819.634238]  [<ffffffff811f3c80>] ? fillonedir+0xd0/0xd0
[407819.634244]  [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[408133.512919] audit: type=1400 audit(1409380699.459:52): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/cups/backend/cups-pdf" pid=2092471 comm="apparmor_parser"
[408133.512930] audit: type=1400 audit(1409380699.459:53): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/cupsd" pid=2092471 comm="apparmor_parser"
[408133.513460] audit: type=1400 audit(1409380699.459:54): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/cupsd" pid=2092471 comm="apparmor_parser"
[408299.843229] INFO: task BrowserBlocking:3282 blocked for more than 120 seconds.
[408299.843238]       Not tainted 3.16.0-10-generic #15-Ubuntu
[408299.843241] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[408299.843245] BrowserBlocking D ffff88042fc14800     0  3282   2580 0x00000000
[408299.843253]  ffff88040d9f7db0 0000000000000086 ffff88041215b2a0 0000000000014800
[408299.843260]  ffff88040d9f7fd8 0000000000014800 ffff88041215b2a0 ffff880429b43590
[408299.843266]  ffff880429b43594 ffff88041215b2a0 00000000ffffffff ffff880429b43598
[408299.843272] Call Trace:
[408299.843288]  [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[408299.843296]  [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[408299.843303]  [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[408299.843360]  [<ffffffffc0546895>] btrfs_log_inode_parent+0x435/0x550 [btrfs]
[408299.843393]  [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[408299.843423]  [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[408299.843432]  [<ffffffff81210461>] do_fsync+0x51/0x80
[408299.843438]  [<ffffffff812106f0>] SyS_fsync+0x10/0x20
[408299.843444]  [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
#1 lore raw
Quoted 5 lines · Arnd Bergmann

As I mentioned at the kernel summit, I have a file system that I use mostly for storing my one kernel git tree and occasionally some build trees (those are normally on a tmpfs), and I have again run into the problem where the file system is only partially full (I think 18% in this case) but I am unable to create new files.

You didn't give us any "btrfs fs show" or "btrfs fs df" outpu to look at, so chances are that a lot of empty data chunks have made themselves comfortable and prevent new metadata chunks from being allocated.

The usual fix is to run balance with -dusage=x, where 0 <= x <= ~20.

Quoted 2 lines

The 82 MB image file is at
https://drive.google.com/file/d/0B_XQwQ5KlfJAWDJfS1E0TG1CLTA/edit?usp=sharing

This is >780 MB..

-h

#2 lore raw
Quoted 1 line · Holger Hoffstätte

You didn't give us any "btrfs fs show" or "btrfs fs df" outpu to look at,

Of course "fi", not "fs"..

-h

#3 lore raw
In reply to Arnd Bergmann
Quoted 6 lines · On Tuesday 09 September 2014 16:29:05 Arnd Bergmann wrote:

I also played around with it some more. After removing a few small files, I could create new files with up to 20-60MB again before hitting ENOSPC. I then did a 'make clean' in all the object directories I had and after that could fill the file system up to 100% by creating a 62GB file using the same command (dd if=/dev/zero of=x).

Ok, one more data point: I was accidentally building randconfig kernels in the same partition all the time, meaning I had lots of traffic with small files using the same names all over all the time. After I got the file system back to a usable state today, building some more randconfig kernels broke it in the same way.

Arnd

#4 lore raw

Hi Arnd,

Quoted 1 line

Ok, one more data point:

Why don't you provide the data point you were specifically asked for, "btrfs fi df" ;)

Regards, Clemens

#5 lore raw
Quoted 6 lines · Clemens Eisserer

Hi Arnd,

Quoted 1 line

Ok, one more data point:

Why don't you provide the data point you were specifically asked for, "btrfs fi df" ;)

btrfs fi show is important as well -- it's hard to work out the
state of the FS from just one of them.

Hugo.

Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- You can get more with a kind word and a two-by-four than you ---   
                       can with just a kind word.                        
#6 lore raw
In reply to Clemens Eisserer
Quoted 6 lines · On Tuesday 09 September 2014 21:49:12 Clemens Eisserer wrote:

Hi Arnd,

Quoted 1 line

Ok, one more data point:

Why don't you provide the data point you were specifically asked for, "btrfs fi df" ;)

I've cleaned it up again already. At the moment, it's working fine, with this data:

Data: total=65.11GB, used=7.41GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=799.54MB
Metadata: total=8.00MB, used=0.00
: total=184.00MB, used=0.00

I also realized that the file I uploaded was the uncompressed 800MB version, not the compressed 82MB file, which is now at
https://drive.google.com/file/d/0B_XQwQ5KlfJAYzY5c1lJRFVQNEE/edit?usp=sharing
Sorry about that.

Ok, now I'm in the bad state again (after running a 'make allmodconfig' kernel build:

Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 67.14GB path /dev/sdc6

Data: total=65.11GB, used=7.99GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=821.48MB
Metadata: total=8.00MB, used=0.00
: total=200.00MB, used=0.00

Not much extra space used up. Also, a bit from my last shell session after getting into this state:

arnd@wuerfel:~/arm-soc$ make -skj40
../arch/arm/mach-mmp/devices.c:81:21: warning: 'u2o_get' defined but not used [-Wunused-function]
 static unsigned int u2o_get(void __iomem *base, unsigned int offset)
                     ^
../arch/arm/mach-mmp/devices.c:86:13: warning: 'u2o_set' defined but not used [-Wunused-function]
 static void u2o_set(void __iomem *base, unsigned int offset,
             ^
../arch/arm/mach-mmp/devices.c:97:13: warning: 'u2o_clear' defined but not used [-Wunused-function]
 static void u2o_clear(void __iomem *base, unsigned int offset,
             ^
../arch/arm/mach-mmp/devices.c:108:13: warning: 'u2o_write' defined but not used [-Wunused-function]
 static void u2o_write(void __iomem *base, unsigned int offset,
             ^
mv: cannot move 'security/selinux/ss/.tmp_conditional.o' to 'security/selinux/ss/conditional.o': No space left on device
make[4]: *** [security/selinux/ss/conditional.o] Error 1
mv: cannot move 'kernel/power/.snapshot.o.tmp' to 'kernel/power/.snapshot.o.cmd': No space left on device
make[4]: *** [kernel/power/snapshot.o] Error 1
mv: cannot move 'drivers/ata/.pata_hpt366.o.tmp' to 'drivers/ata/.pata_hpt366.o.cmd': No space left on device
make[4]: *** [drivers/ata/pata_hpt366.o] Error 1
mv: cannot move 'kernel/irq/.proc.o.tmp' to 'kernel/irq/.proc.o.cmd': No space left on device
make[4]: *** [kernel/irq/proc.o] Error 1
...
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
416642+0 records in
416641+0 records out
213320192 bytes (213 MB) copied, 2.97814 s, 71.6 MB/s
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ rm x
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
366053+0 records in
366052+0 records out
187418624 bytes (187 MB) copied, 3.04426 s, 61.6 MB/s
arnd@wuerfel:~/arm-soc$ rm x
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
363482+0 records in
363481+0 records out
186102272 bytes (186 MB) copied, 2.80644 s, 66.3 MB/s
arnd@wuerfel:~/arm-soc$ rm x
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
366677+0 records in
366676+0 records out
187738112 bytes (188 MB) copied, 2.86939 s, 65.4 MB/s
arnd@wuerfel:~/arm-soc$ sync
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device

I believe that there were no other processes writing to this partition at the time.

Arnd

#7 lore raw
Quoted 6 lines · Arnd Bergmann

Ok, now I'm in the bad state again (after running a 'make allmodconfig' kernel build:

Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 67.14GB path /dev/sdc6

All the space on the FS has been allocated to some purpose or other.

Quoted 1 line

Data: total=65.11GB, used=7.99GB

Here, you have 65 GiB allocated to data, but only 8 GiB of that
used. The FS won't automatically free up any of that (yet -- it's one of the project ideas).

Quoted 3 lines

System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=821.48MB

Here, you're running close to full with metadata -- the FS needs
some space to write new copies of metadata block in order to modify anything. It can't get enough space to do that, because there's nowhere for any more metadata allocation to come from (because it's all allocated --see my first comment).

So... you need to free up some data chunks. You can do this with:

# btrfs balance start -dusage=5 /mountpoint

Take a look at the output of btrfs fi df and btrfs fi show
afterwards, and see how much the Data allocation has reduced by, and how much unallocated space you have left afterwards. You may want to increase the number in the above balance command to some higher value, to free up even more chunks (it limits the balance to chunks less than n% full -- so the command above will only touch chunks with 5% actual data or less). This is in the FAQ.

Hugo.

Quoted 2 lines

Metadata: total=8.00MB, used=0.00
: total=200.00MB, used=0.00

Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Comic Sans goes into a bar,  and the barman says, "We don't ---   
                         serve your type here."                          
#8 lore raw
Quoted 8 lines · On Tuesday 09 September 2014 22:57:25 Hugo Mills wrote:
Quoted 6 lines · Arnd Bergmann

Ok, now I'm in the bad state again (after running a 'make allmodconfig' kernel build:

Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 67.14GB path /dev/sdc6

Quoted 7 lines

All the space on the FS has been allocated to some purpose or other.

Quoted 1 line

Data: total=65.11GB, used=7.99GB

Here, you have 65 GiB allocated to data, but only 8 GiB of that
used. The FS won't automatically free up any of that (yet -- it's one of the project ideas).

Ok, I see.

Quoted 11 lines

So... you need to free up some data chunks. You can do this with:

# btrfs balance start -dusage=5 /mountpoint

Take a look at the output of btrfs fi df and btrfs fi show
afterwards, and see how much the Data allocation has reduced by, and how much unallocated space you have left afterwards. You may want to increase the number in the above balance command to some higher value, to free up even more chunks (it limits the balance to chunks less than n% full -- so the command above will only touch chunks with 5% actual data or less). This is in the FAQ.

This is the output afterwards:

$ time sudo btrfs fi balance start  -dusage=5 /git/
Done, had to relocate 44 out of 70 chunks
real    0m22.026s
user    0m0.014s
sys     0m8.302s
$ sudo btrfs fi show /dev/sdc6
Label: none  uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
        Total devices 1 FS bytes used 8.79GB
        devid    1 size 67.14GB used 23.14GB path /dev/sdc6
$ sudo btrfs fi df /git
Data: total=21.01GB, used=7.99GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.05GB, used=821.32MB
Metadata: total=8.00MB, used=0.00
: total=200.00MB, used=0.00

I'm pretty sure I read the FAQ one of the last times this happened, but I thought it would only apply to mostly full file systems. I can usually recover by moving all the data to tmpfs and doing a new mkfs before moving the data back, and that normally lasts for a few months before I run out of space again.

I'll try to see how long it takes until the problem comes back now.

Arnd

#9 lore raw
In reply to Hugo Mills
Quoted 8 lines · On Tue, 9 Sep 2014 22:57:25 +0100 Hugo Mills <[email protected]> wrote:
Quoted 6 lines · Arnd Bergmann

Ok, now I'm in the bad state again (after running a 'make allmodconfig' kernel build:

Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 67.14GB path /dev/sdc6

Quoted 7 lines

All the space on the FS has been allocated to some purpose or other.

Quoted 1 line

Data: total=65.11GB, used=7.99GB

Here, you have 65 GiB allocated to data, but only 8 GiB of that
used. The FS won't automatically free up any of that (yet -- it's one of the project ideas).

If you find yourself constantly running into this situation, rather than incorporating periodical btrfs 'balances' into your scripts, you could consider going with the "mixed" allocation mode of the FS, using the -M option of mkfs.

Seeing as this is not the largest-sized partition in the world in the first place, the mixed mode would allow for a more efficient and flexible space allocation on it; at the cost of some performance penalty (you could compare your compilation times, but I don't think it's too significant).

Signature · 2 lines
With respect,
Roman
#10 lore raw
In reply to Arnd Bergmann
Quoted 3 lines · Arnd Bergmann

Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 67.14GB path /dev/sdc6

                          ^^^^^^^^^^^^^^^^^^^^ *ding ding ding*

The term "used" is confusing here and really should be called "allocated", since it's more than the number of actually used bytes. Note that df shows that you "have" ~66GB data and ~1GB metadata (giving approx. ~67GB), despite the fact that of those 65GB data allocated only ~8GB are actually used.

However all these reserved data chunks are preventing the metadata space from expanding, which is pretty full:
Metadata, DUP: total=1.00GB, used=799.54MB

Here "total" also should be "allocated/reserved" and is 1GB, but btrfs feels it's necessary to allocate a new metadata chunk (which would also increase "total") and fails.

Running balance with -dusage=x will garbage-collect the reserved but mostly-empty data chunks with "up to x" percent actual usage.

-h