Where is the disk space?
root@polgara:/mnt/btrfs_root# du -sh * 28G @ 28G @_hourly.20151113_08:04:01 4.0K @_last 4.0K @_last_rw 28G @_rw.20151113_00:02:01 root@polgara:/mnt/btrfs_root# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb5 56G 40G 5.4G 89% /mnt/btrfs_root
root@polgara:/mnt/btrfs_root# btrfs fi df . Data, single: total=39.85GiB, used=38.52GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=6.00GiB, used=579.17MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=208.00MiB, used=0.00B
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root' uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
Total devices 1 FS bytes used 39.09GiB
devid 1 size 55.88GiB used 51.88GiB path /dev/sdb5btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root#
root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=80 Done, had to relocate 1 out of 55 chunks
Sadly, it's only running 3.17.8 because of complicated reasons, but still,
1) I have 28GB used (modulo a few files between the btrfs send snapshots and current status)
2) fi show shows I'm using 39GB, not sure where the extra 11GB came from
3) fi df agrees with fi show
4) regular df agrees on used too, but shows 5GB free instead of 15GB despite the filesystem being balanced.
I did have a bunch of snapshots that I did delete a while ago now, but it looks like their blocks aren't being reclaimed.
Any ideas?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Marc MERLIN posted on Fri, 13 Nov 2015 09:41:01 -0800 as excerpted:
Quoted 1 line
Any ideas?
Without addressing the main question, a couple targets of opportunity:
1) A quick balance with -mprofiles=single should kill those unused single metadata and system mkfs.btrfs legacies so you don't have to see them in fi df.
2) While fi df reports the data total vs. used spread being close enough that it's unsurprising a balance -dusage=80 didn't give you anything back, the spread on metadata is rather higher, dup, 6.0 gig total, less than 1 gig used, so a balance -musage=50 or 80 should return a few gigs, 4-5 I'd guess, leaving you 1-2 gig metadata total.
Maybe that's part of the missing snapshot overhead?
(The -musage run should kill the unused single profile metadata/system as well, but for a different reason, so I made it a different point and used -mprofiles=single there to avoid conflating the two.)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 10 lines · Duncan
Marc MERLIN posted on Fri, 13 Nov 2015 09:41:01 -0800 as excerpted:
Quoted 1 line
Any ideas?
Without addressing the main question, a couple targets of opportunity:
1) A quick balance with -mprofiles=single should kill those unused single metadata and system mkfs.btrfs legacies so you don't have to see them in fi df.
Quoted 5 lines
2) While fi df reports the data total vs. used spread being close enough that it's unsurprising a balance -dusage=80 didn't give you anything back, the spread on metadata is rather higher, dup, 6.0 gig total, less than 1 gig used, so a balance -musage=50 or 80 should return a few gigs, 4-5 I'd guess, leaving you 1-2 gig metadata total.
Well, -musage helped, I didn't know about that one, thanks:
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root' uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
Total devices 1 FS bytes used 39.09GiB
devid 1 size 55.88GiB used 51.88GiB path /dev/sdb5btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root# btrfs balance start -musage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x6, state 0x0, force is off
METADATA (flags 0x2): balancing, usage=80
SYSTEM (flags 0x2): balancing, usage=80
Done, had to relocate 14 out of 55 chunks
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root' uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
Total devices 1 FS bytes used 39.75GiB
devid 1 size 55.88GiB used 41.91GiB path /dev/sdb5btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb5 56G 41G 15G 74% /mnt/btrfs_root
I'm still seeing 39GB used for 28GB of actual data, but I definitely fixed one bit already thanks to you.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Marc MERLIN posted on Fri, 13 Nov 2015 12:01:58 -0800 as excerpted:
Quoted 2 lines
I'm still seeing 39GB used for 28GB of actual data, but I definitely fixed one bit already thanks to you.
For the data side, I think I understand what's going on with the space, but am not in sufficient mastery of the concept to feel confident that I can explain it well. Never-the-less, here's a go at it. One of the devs did a post complete with nice ascii diagrams if you're interested in trying to look it up.
What happens is that with larger files (particularly VM images and the like if they're not set nocow or if they're nocow but cow1-ed due to snapshotting, which AFAIK fits your use-case directly) originally setup in a few reasonably large extents, as rewrites occur, they cow and thus unmap random smaller extents from within the larger extents. But, btrfs doesn't do extent splitting, so those larger extents remain pinned as long as at least some of the data within them remains referenced. The result can eventually be rather cavernous mostly empty original extents still pinned in place by the few 4K blocks that haven't ever been rewritten.
If you believe you know what files are likely to be the culprit, and if you're doing VMs that's exactly where I'd look first, try temporarily moving them (and any old snapshots of them) out of the filesystem, doing a quick fi df to see if it has indeed emptied out some data chunks, and if so, a balance -dusage=80 or whatever to try to reclaim them, then move the file(s) back.
Of course if you still have other snapshots of the file pinning down those extents it won't free them, so you have to either delete the snapshots or at least delete the culprit file(s) from within them (obviously only with writable snapshots) in ordered for this to work.
In theory, defrag can do much the same thing, assuming of course there aren't snapshots containing the same file and still locking down its old extents, but that doesn't get the file off the filesystem for the cleanup, so it's likely to be somewhat less effective, perhaps considerably less effective if the remaining free space inside existing data chunks is itself highly fragmented.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Hi,
Quoted 40 lines · Marc MERLIN
root@polgara:/mnt/btrfs_root# du -sh * 28G @ 28G @_hourly.20151113_08:04:01 4.0K @_last 4.0K @_last_rw 28G @_rw.20151113_00:02:01 root@polgara:/mnt/btrfs_root# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb5 56G 40G 5.4G 89% /mnt/btrfs_rootroot@polgara:/mnt/btrfs_root# btrfs fi df . Data, single: total=39.85GiB, used=38.52GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=6.00GiB, used=579.17MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=208.00MiB, used=0.00Broot@polgara:/mnt/btrfs_root# btrfs fi show . Label: 'btrfs_root' uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf Total devices 1 FS bytes used 39.09GiB devid 1 size 55.88GiB used 51.88GiB path /dev/sdb5btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root#root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=80 Done, had to relocate 1 out of 55 chunksSadly, it's only running 3.17.8 because of complicated reasons, but still,
1) I have 28GB used (modulo a few files between the btrfs send snapshots and current status)
2) fi show shows I'm using 39GB, not sure where the extra 11GB came from
3) fi df agrees with fi show
Quoted 8 lines
4) regular df agrees on used too, but shows 5GB free instead of 15GB despite the filesystem being balanced.
I did have a bunch of snapshots that I did delete a while ago now, but it looks like their blocks aren't being reclaimed.
Any ideas?
Since you said you have some snapshots in between...I can think of one case to prove where the space goes,
Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).
This comes from our COW and extent references implementation, so you get the benefit of COW, meanwhile have to live with the un-reclaimed space.
It's sort of something I was trying to fix, but I found that my approach led to other problems so I decided to give it up.
Thanks,
-liubo
Quoted 11 lines
Thanks,
MarcSignature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Liu Bo
Since you said you have some snapshots in between...I can think of one case to prove where the space goes,
Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).
This comes from our COW and extent references implementation, so you get the benefit of COW, meanwhile have to live with the un-reclaimed space.
It's sort of something I was trying to fix, but I found that my approach led to other problems so I decided to give it up.
That's an interesting scenario. Thanks for explaining this.
MArc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Liu Bo wrote on 2015/11/14 22:35 -0800:
Quoted 60 lines
Hi,
Quoted 40 lines · Marc MERLIN
root@polgara:/mnt/btrfs_root# du -sh * 28G @ 28G @_hourly.20151113_08:04:01 4.0K @_last 4.0K @_last_rw 28G @_rw.20151113_00:02:01 root@polgara:/mnt/btrfs_root# df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb5 56G 40G 5.4G 89% /mnt/btrfs_rootroot@polgara:/mnt/btrfs_root# btrfs fi df . Data, single: total=39.85GiB, used=38.52GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00B Metadata, DUP: total=6.00GiB, used=579.17MiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=208.00MiB, used=0.00Broot@polgara:/mnt/btrfs_root# btrfs fi show . Label: 'btrfs_root' uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf Total devices 1 FS bytes used 39.09GiB devid 1 size 55.88GiB used 51.88GiB path /dev/sdb5btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root#root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=80 Done, had to relocate 1 out of 55 chunksSadly, it's only running 3.17.8 because of complicated reasons, but still,
1) I have 28GB used (modulo a few files between the btrfs send snapshots and current status)
2) fi show shows I'm using 39GB, not sure where the extra 11GB came from
3) fi df agrees with fi show
Quoted 8 lines
4) regular df agrees on used too, but shows 5GB free instead of 15GB despite the filesystem being balanced.
I did have a bunch of snapshots that I did delete a while ago now, but it looks like their blocks aren't being reclaimed.
Any ideas?
Since you said you have some snapshots in between...I can think of one case to prove where the space goes,
Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).
This comes from our COW and extent references implementation, so you get the benefit of COW, meanwhile have to live with the un-reclaimed space.
Quoted 6 lines
It's sort of something I was trying to fix, but I found that my approach led to other problems so I decided to give it up.
Thanks,
-liubo
The case is quite right, but the example maybe a little incorrect BTW.
For 10M file in one subvolume and rewrite [3M, 5M](including the last byte) in the snapshot will only increase used space to 12M + 4K, as above numbers except the last byte are already sectorsize aligned.
The root cause is, btrfs' lazy extent freeing behavior. Use a new 12M case to describe:
======
Step 0:
Subv A:
[0,12M): Shared between A and B (Extent 1)
on disk:
Extent 1: record the above [0,12M) data
Subv B:
[0,4M): Shared between A and B (Extent 1) [4M,8M): Exclusive in B (Extent 2) [8M,12M): Shared between A and B (Extent 1)
on disk:
Extent 1: is reused in [0,4M) and [8M,12M) ranges Extent 2: record data of [4M,8M)
------
Step 1:
Write [4M,8M) of subv A:
[0,4M): Shared between A and B (Extent 1) [4M,8M): Exclusive in A (Extent 3) [8M,12M): Shared between A and B (Extent 1)
Extent 3: record the above [4M,8M) data And Extent 1 is not changed.
Subv B:
[0,4M): Shared between A and B (Extent 1) [4M,8M): Exclusive in B (Extent 2) [8M,12M): Shared between A and B (Extent 1)
======
After step 2, the used space will be 12M (Extent 1) + 4M (Extent 2) + 4M (Extent 3) = 20M.
Even only 8M of extent 1 is really referred by.
So the middle 4M of extent 1 is totally wasted and Btrfs won't free it, until *ALL* of the extent is not referred by any one.
One solution would be defrag, but IIRC defrag under multi-subvolume case is not supported yet...
BTW, personally speaking, to find out how much space a subvolume takes, btrfs qgroup would be quite handy (after 4.2 kernel). Shows how much space a subvolume takes exclusively.
Thanks,
Qu
Quoted 17 lines
Quoted 12 lines
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html