Skip to contentSkip to conversation

Where is the disk space?

DISCUSSION Marc MERLIN 4 participants 7 messages Nov 13, 2015 – Nov 16, 2015 Btrfs file system

lore
Share & tools
lore
#0 lore raw
root@polgara:/mnt/btrfs_root# du -sh *
28G     @
28G     @_hourly.20151113_08:04:01
4.0K    @_last
4.0K    @_last_rw
28G     @_rw.20151113_00:02:01
root@polgara:/mnt/btrfs_root# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb5        56G   40G  5.4G  89% /mnt/btrfs_root
root@polgara:/mnt/btrfs_root# btrfs fi df .
Data, single: total=39.85GiB, used=38.52GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=6.00GiB, used=579.17MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=208.00MiB, used=0.00B
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.09GiB
        devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5

btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root#

root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=80
Done, had to relocate 1 out of 55 chunks

Sadly, it's only running 3.17.8 because of complicated reasons, but still,

1) I have 28GB used (modulo a few files between the btrfs send snapshots and current status)

2) fi show shows I'm using 39GB, not sure where the extra 11GB came from

3) fi df agrees with fi show

4) regular df agrees on used too, but shows 5GB free instead of 15GB despite the filesystem being balanced.

I did have a bunch of snapshots that I did delete a while ago now, but it looks like their blocks aren't being reclaimed.

Any ideas?

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#1 lore raw

Marc MERLIN posted on Fri, 13 Nov 2015 09:41:01 -0800 as excerpted:

Quoted 1 line

Any ideas?

Without addressing the main question, a couple targets of opportunity:

1) A quick balance with -mprofiles=single should kill those unused single metadata and system mkfs.btrfs legacies so you don't have to see them in fi df.

2) While fi df reports the data total vs. used spread being close enough that it's unsurprising a balance -dusage=80 didn't give you anything back, the spread on metadata is rather higher, dup, 6.0 gig total, less than 1 gig used, so a balance -musage=50 or 80 should return a few gigs, 4-5 I'd guess, leaving you 1-2 gig metadata total.

Maybe that's part of the missing snapshot overhead?

(The -musage run should kill the unused single profile metadata/system as well, but for a different reason, so I made it a different point and used -mprofiles=single there to avoid conflating the two.)

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#2 lore raw
Quoted 10 lines · Duncan

Marc MERLIN posted on Fri, 13 Nov 2015 09:41:01 -0800 as excerpted:

Quoted 1 line

Any ideas?

Without addressing the main question, a couple targets of opportunity:

1) A quick balance with -mprofiles=single should kill those unused single metadata and system mkfs.btrfs legacies so you don't have to see them in fi df.

Quoted 5 lines

2) While fi df reports the data total vs. used spread being close enough that it's unsurprising a balance -dusage=80 didn't give you anything back, the spread on metadata is rather higher, dup, 6.0 gig total, less than 1 gig used, so a balance -musage=50 or 80 should return a few gigs, 4-5 I'd guess, leaving you 1-2 gig metadata total.

Well, -musage helped, I didn't know about that one, thanks:

root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.09GiB
        devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5

btrfs-progs v4.0-dirty

root@polgara:/mnt/btrfs_root# btrfs balance start -musage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=80
  SYSTEM (flags 0x2): balancing, usage=80
Done, had to relocate 14 out of 55 chunks
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.75GiB
        devid    1 size 55.88GiB used 41.91GiB path /dev/sdb5

btrfs-progs v4.0-dirty

root@polgara:/mnt/btrfs_root# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb5        56G   41G   15G  74% /mnt/btrfs_root

I'm still seeing 39GB used for 28GB of actual data, but I definitely fixed one bit already thanks to you.

Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#3 lore raw

Marc MERLIN posted on Fri, 13 Nov 2015 12:01:58 -0800 as excerpted:

Quoted 2 lines

I'm still seeing 39GB used for 28GB of actual data, but I definitely fixed one bit already thanks to you.

For the data side, I think I understand what's going on with the space, but am not in sufficient mastery of the concept to feel confident that I can explain it well. Never-the-less, here's a go at it. One of the devs did a post complete with nice ascii diagrams if you're interested in trying to look it up.

What happens is that with larger files (particularly VM images and the like if they're not set nocow or if they're nocow but cow1-ed due to snapshotting, which AFAIK fits your use-case directly) originally setup in a few reasonably large extents, as rewrites occur, they cow and thus unmap random smaller extents from within the larger extents. But, btrfs doesn't do extent splitting, so those larger extents remain pinned as long as at least some of the data within them remains referenced. The result can eventually be rather cavernous mostly empty original extents still pinned in place by the few 4K blocks that haven't ever been rewritten.

If you believe you know what files are likely to be the culprit, and if you're doing VMs that's exactly where I'd look first, try temporarily moving them (and any old snapshots of them) out of the filesystem, doing a quick fi df to see if it has indeed emptied out some data chunks, and if so, a balance -dusage=80 or whatever to try to reclaim them, then move the file(s) back.

Of course if you still have other snapshots of the file pinning down those extents it won't free them, so you have to either delete the snapshots or at least delete the culprit file(s) from within them (obviously only with writable snapshots) in ordered for this to work.

In theory, defrag can do much the same thing, assuming of course there aren't snapshots containing the same file and still locking down its old extents, but that doesn't get the file off the filesystem for the cleanup, so it's likely to be somewhat less effective, perhaps considerably less effective if the remaining free space inside existing data chunks is itself highly fragmented.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#4 lore raw
In reply to Marc MERLIN

Hi,

Quoted 40 lines · Marc MERLIN
root@polgara:/mnt/btrfs_root# du -sh *
28G     @
28G     @_hourly.20151113_08:04:01
4.0K    @_last
4.0K    @_last_rw
28G     @_rw.20151113_00:02:01
root@polgara:/mnt/btrfs_root# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb5        56G   40G  5.4G  89% /mnt/btrfs_root
root@polgara:/mnt/btrfs_root# btrfs fi df .
Data, single: total=39.85GiB, used=38.52GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=6.00GiB, used=579.17MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=208.00MiB, used=0.00B
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
        Total devices 1 FS bytes used 39.09GiB
        devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5

btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root#

root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=80
Done, had to relocate 1 out of 55 chunks

Sadly, it's only running 3.17.8 because of complicated reasons, but still,

1) I have 28GB used (modulo a few files between the btrfs send snapshots and current status)

2) fi show shows I'm using 39GB, not sure where the extra 11GB came from

3) fi df agrees with fi show

Quoted 8 lines

4) regular df agrees on used too, but shows 5GB free instead of 15GB despite the filesystem being balanced.

I did have a bunch of snapshots that I did delete a while ago now, but it looks like their blocks aren't being reclaimed.

Any ideas?

Since you said you have some snapshots in between...I can think of one case to prove where the space goes,

Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).

This comes from our COW and extent references implementation, so you get the benefit of COW, meanwhile have to live with the un-reclaimed space.

It's sort of something I was trying to fix, but I found that my approach led to other problems so I decided to give it up.

Thanks,

-liubo

Quoted 11 lines

Thanks,
Marc

Signature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw
Quoted 10 lines · Liu Bo

Since you said you have some snapshots in between...I can think of one case to prove where the space goes,

Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).

This comes from our COW and extent references implementation, so you get the benefit of COW, meanwhile have to live with the un-reclaimed space.

It's sort of something I was trying to fix, but I found that my approach led to other problems so I decided to give it up.

That's an interesting scenario. Thanks for explaining this.

MArc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#6 lore raw
In reply to Liu Bo

Liu Bo wrote on 2015/11/14 22:35 -0800:

Quoted 60 lines

Hi,

Quoted 40 lines · Marc MERLIN
root@polgara:/mnt/btrfs_root# du -sh *
28G     @
28G     @_hourly.20151113_08:04:01
4.0K    @_last
4.0K    @_last_rw
28G     @_rw.20151113_00:02:01
root@polgara:/mnt/btrfs_root# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb5        56G   40G  5.4G  89% /mnt/btrfs_root
root@polgara:/mnt/btrfs_root# btrfs fi df .
Data, single: total=39.85GiB, used=38.52GiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=6.00GiB, used=579.17MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=208.00MiB, used=0.00B
root@polgara:/mnt/btrfs_root# btrfs fi show .
Label: 'btrfs_root'  uuid: a2a1ed7b-6bfe-4e83-bc10-727126ed17bf
         Total devices 1 FS bytes used 39.09GiB
         devid    1 size 55.88GiB used 51.88GiB path /dev/sdb5

btrfs-progs v4.0-dirty
root@polgara:/mnt/btrfs_root#

root@polgara:/mnt/btrfs_root# btrfs balance start -dusage=80 -v /mnt/btrfs_root
Dumping filters: flags 0x1, state 0x0, force is off
   DATA (flags 0x2): balancing, usage=80
Done, had to relocate 1 out of 55 chunks

Sadly, it's only running 3.17.8 because of complicated reasons, but still,

1) I have 28GB used (modulo a few files between the btrfs send snapshots and current status)

2) fi show shows I'm using 39GB, not sure where the extra 11GB came from

3) fi df agrees with fi show

Quoted 8 lines

4) regular df agrees on used too, but shows 5GB free instead of 15GB despite the filesystem being balanced.

I did have a bunch of snapshots that I did delete a while ago now, but it looks like their blocks aren't being reclaimed.

Any ideas?

Since you said you have some snapshots in between...I can think of one case to prove where the space goes,

Say, you have a file with size=10M on a freshly created partition(the total used data space is 10M), and you have a snapshot which owns this file, then you modify the original file by overwrite the range [3M, 5M], and right now you can find that the total used data space increases to 15M or maybe more (because of unaliged write and extent pads to 4K length).

This comes from our COW and extent references implementation, so you get the benefit of COW, meanwhile have to live with the un-reclaimed space.

Quoted 6 lines

It's sort of something I was trying to fix, but I found that my approach led to other problems so I decided to give it up.

Thanks,

-liubo

The case is quite right, but the example maybe a little incorrect BTW.

For 10M file in one subvolume and rewrite [3M, 5M](including the last byte) in the snapshot will only increase used space to 12M + 4K, as above numbers except the last byte are already sectorsize aligned.

The root cause is, btrfs' lazy extent freeing behavior. Use a new 12M case to describe:

======
Step 0:
Subv A:
[0,12M): Shared between A and B (Extent 1)

on disk:
Extent 1: record the above [0,12M) data

Subv B:
[0,4M): Shared between A and B (Extent 1) [4M,8M): Exclusive in B (Extent 2) [8M,12M): Shared between A and B (Extent 1)

on disk:
Extent 1: is reused in [0,4M) and [8M,12M) ranges Extent 2: record data of [4M,8M)

------
Step 1:
Write [4M,8M) of subv A:
[0,4M): Shared between A and B (Extent 1) [4M,8M): Exclusive in A (Extent 3) [8M,12M): Shared between A and B (Extent 1)

Extent 3: record the above [4M,8M) data And Extent 1 is not changed.

Subv B:
[0,4M): Shared between A and B (Extent 1) [4M,8M): Exclusive in B (Extent 2) [8M,12M): Shared between A and B (Extent 1)
======

After step 2, the used space will be 12M (Extent 1) + 4M (Extent 2) + 4M (Extent 3) = 20M.

Even only 8M of extent 1 is really referred by.
So the middle 4M of extent 1 is totally wasted and Btrfs won't free it, until *ALL* of the extent is not referred by any one.

One solution would be defrag, but IIRC defrag under multi-subvolume case is not supported yet...

BTW, personally speaking, to find out how much space a subvolume takes, btrfs qgroup would be quite handy (after 4.2 kernel). Shows how much space a subvolume takes exclusively.

Thanks,
Qu

Quoted 17 lines
Quoted 12 lines

Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html