No space left, with 80 GB space free
I've been trying to run PostgreSQL with a ~20-30 GB database on a btrfs filesystem (rest of the filesystem filled with KVM images, compressed archives etc.). Snapshots are cool, and let you rollback to a different database version easily ;)
Unfortunately, running PostgreSQL on btrfs wasn't very reliable as I was getting rather frequent "No space left" for some disk intensive operations (CREATE INDEX, REINDEX, VACUUM ANALYZE).
As the error happens, I still have around 80 GB free space on the filesystem:
Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 255G 80G 77% /mnt/btrfs
What's even more interesting, after the error happened, I'm able to write a ~85 GB file with dd before it exits with "out of space" (with and without the oflag=direct):
# dd if=/dev/urandom of=bigfile bs=1M dd: writing `bigfile': No space left on device 81056+0 records in 81055+0 records out 84992327680 bytes (85 GB) copied, 9429.28 s, 9.0 MB/s
The kernel used was 2.6.39-rc5.
If it matters, I had the filesystem mounted with compress-force flag:
/dev/sdb4 on /mnt/btrfs type btrfs (rw,noatime,compress-force)
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 3 lines · Tomasz Chmielewski
The kernel used was 2.6.39-rc5.
If it matters, I had the filesystem mounted with compress-force flag:
Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 257G 78G 77% /mnt/btrfs
coredb=# DELETE FROM core_trevent where "date" < E'2011-05-07 00:00';
ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device CONTEXT: writing block 1014999 of relation base/16384/16521
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 2 lines · On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski <[email protected]> = wrote:
Nobody has a clue what makes btrfs run out of space when used with Po=
stgreSQL, even when there is plenty of free space left?
Quoted 5 lines
# df -h Filesystem =A0 =A0 =A0 =A0 =A0 =A0Size =A0Used Avail Use% Mounted on /dev/sdb4 =A0 =A0 =A0 =A0 =A0 =A0 336G =A0257G =A0 78G =A077% /mnt/bt=
rfs
Quoted 3 lines
coredb=3D# DELETE FROM
core_treventwhere "date" < E'2011-05-07 00:00=
';
Quoted 1 line
ERROR: =A0could not write block 1014999 of relation base/16384/16521:=
No space left on device
Quoted 2 lines
CONTEXT: =A0writing block 1014999 of relation base/16384/16521
Hi Tomasz,
What does btrfs fi df /mountpoint show?
Please see this link:
https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_=
out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21 As well as this one:
https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_=
to_check_the_amount_of_free_space.3F
--=20
MG
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Miguel Garrido
Quoted 9 lines · Tomasz Chmielewski
Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 257G 78G 77% /mnt/btrfs
Quoted 8 lines
Quoted 4 lines
coredb=# DELETE FROM
core_treventwhere "date"< E'2011-05-07 00:00';ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device CONTEXT: writing block 1014999 of relation base/16384/16521Hi Tomasz,
What does btrfs fi df /mountpoint show?
Used space is almost identical as with normal df.
Quoted 2 lines
Please see this link:
https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21
Doesn't apply in my case: kernel is 2.6.39, device is relatively large.
It is *probably* related to compression used (compress-force - disabled the option, and didn't see out of space since).
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 5 lines · Miguel Garrido
Quoted 4 lines · Tomasz Chmielewski
Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
Quoted 8 lines
Quoted 8 lines
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 257G 78G 77% /mnt/btrfscoredb=# DELETE FROM
core_treventwhere "date"< E'2011-05-07 00:00';ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device CONTEXT: writing block 1014999 of relation base/16384/16521
Hmm, this "No space left on device" only shows up if I write to the btrfs filesystem relatively fast.
For example, this way, I can only write around 1 GB (~60 MB/s write):
# cp -v acer.img.xz /mnt/btrfs/ `ac.img.xz' -> `/mnt/btrfs/ac.img.xz' cp: writing `/mnt/btrfs/ac.img.xz': No space left on device
This way, when the writes are CPU-bound, I'm able to fully fill the btrfs filesystem (~8 MB/s):
# dd if=/dev/urandom of=/mnt/btrfs/bigfile
Delayed allocation?
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 5 lines · Miguel Garrido
Quoted 4 lines · Tomasz Chmielewski
Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
Quoted 8 lines
Quoted 8 lines
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb4 336G 257G 78G 77% /mnt/btrfscoredb=# DELETE FROM
core_treventwhere "date"< E'2011-05-07 00:00';ERROR: could not write block 1014999 of relation base/16384/16521: No space left on device CONTEXT: writing block 1014999 of relation base/16384/16521
Just a follow up - I'm able to reproduce the issue with 3.0.1 kernel, with plain rsync. I'm trying to write a ~ 4 GB file to btrfs.
"Fast" writes will make btrfs output "No space left on device".
On the other hand, if I write slowly, btrfs will be happy and write the whole file.
cp ("full speed copy") fails after a few seconds, so let's try to write with ~5 MB/s - still fails:
# rsync -a -v --progress --bwlimit=5000 . /mnt/btrfs/temp sending incremental file list ./ randomfile 645136384 13% 4.87MB/s 0:13:32 rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: write failed on "/mnt/btrfs/temp/randomfile": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7] rsync: connection unexpectedly closed (31 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]
Let's write with ~1 MB/s then - succeeds:
# rsync -a -v --progress --bwlimit=1000 . /mnt/btrfs/temp sending incremental file list ./ randomfile 4693411469 100% 988.30kB/s 1:17:17 (xfer#1, to-check=6/8) (...) sent 4758749321 bytes received 112 bytes 1012176.84 bytes/sec total size is 4758167972 speedup is 1.00
I don't think it's expected?
I see these in dmesg while copying the files:
[ 1703.978600] btrfs: free space inode generation (0) did not match free space cache generation (22496) [ 1703.978605] btrfs: failed to load free space cache for block group 92371156992 [ 1704.026476] btrfs: free space inode generation (0) did not match free space cache generation (22502) [ 1704.026481] btrfs: failed to load free space cache for block group 114919735296 [ 1704.035239] btrfs: free space inode generation (0) did not match free space cache generation (22503) [ 1704.035243] btrfs: failed to load free space cache for block group 118140960768 [ 1704.035252] btrfs: free space inode generation (0) did not match free space cache generation (22481) [ 1704.035255] btrfs: failed to load free space cache for block group 119214702592 [ 1704.294757] btrfs: free space inode generation (0) did not match free space cache generation (22503) [ 1704.294761] btrfs: failed to load free space cache for block group 243231883264
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org