Skip to contentSkip to conversation

fstrim silently does nothing on dev add/dev rem'd filesystem

DISCUSSION Chris West 4 participants 5 messages Sep 27, 2015 – Sep 28, 2015 Btrfs file system

lore
Share & tools
lore

Chris West

author
#0 lore raw

I have a filesystem for which fstrim won't do anything.
The filesystem has a history of abuse; dev add, dev remove, dding, ...

There's nothing wrong with the kernel or the disc; other btrfs volumes on the same disc trim fine, and the volume used to trim fine.

By "won't trim", I mean that it always, instantly returns 0 bytes trimmed:

% time sudo fstrim -v .
.: 0 B (0 bytes) trimmed
sudo fstrim -v . 0.00s user 0.00s system 0% cpu 0.004 total

It used to, as expected, take a while, then report a trimmed size. Currently, it's a single-device setup, but with weird devids because of its history:

% sudo btrfs f show
Label: none uuid: 672510b8-afb9-425f-a58d-bd325f7c987a
Total devices 1 FS bytes used 226.86GiB
devid 2 size 475.94GiB used 237.03GiB path /dev/sda2
[..]

% mount | fgrep sda2
/dev/sda2 on / type btrfs (rw,relatime,compress=lzo,ssd,space_cache,user_subvol_rm_allowed,subvolid=257,subvol=/@) /dev/sda2 on /home type btrfs (rw,relatime,compress=lzo,ssd,space_cache,user_subvol_rm_allowed,subvolid=258,subvol=/@home)

% uname -a
Linux a 4.2.0-10-generic #12-Ubuntu SMP Tue Sep 15 19:43:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
(Ubuntu Wily pre-release.)

While fstrimming probably has minimal value for such an idle/empty volume, I'd still be interested in working out what's wrong (if anything is wrong).

Any suggestions?

Chris (Faux on IRC).

#1 lore raw

Hi Chris,

Quoted 8 lines

I have a filesystem for which fstrim won't do anything.
The filesystem has a history of abuse; dev add, dev remove, dding, ...

There's nothing wrong with the kernel or the disc; other btrfs volumes on the same disc trim fine, and the volume used to trim fine.

By "won't trim", I mean that it always, instantly returns 0 bytes trimmed:

you probably suffer from the same problem I had a few years ago. It is a bug in how btrfs implements fstrim. Adding and removing devices is the way I got my btrfs filesystem into the same situation then.

See for the background:
http://comments.gmane.org/gmane.comp.file-systems.btrfs/15597

I wrote a patch, see here:
http://www.mail-archive.com/[email protected]/msg40618.html

I understand that in kernel 4.3 there will be an extension so that fstrim trims free space outside of allocated chunks. This is orthogonal; my patch allows to trim free space inside allocated chunks (which, somewhat accidentally, works for most btrfs filesystems since a long time) even with filesystems like yours.

Kind regards,

Lutz Euler

#2 lore raw
In reply to Chris West

Chris West posted on Sun, 27 Sep 2015 18:52:52 +0100 as excerpted:

Quoted 6 lines

I have a filesystem for which fstrim won't do anything.
The filesystem has a history of abuse; dev add, dev remove, dding, ...

There's nothing wrong with the kernel or the disc; other btrfs volumes on the same disc trim fine, and the volume used to trim fine.

Quoted 8 lines

By "won't trim", I mean that it always, instantly returns 0 bytes trimmed:

% time sudo fstrim -v .
.: 0 B (0 bytes) trimmed sudo fstrim -v . 0.00s user 0.00s system 0% cpu 0.004 total

It used to, as expected, take a while, then report a trimmed size.

[replying to list and OP both]

As lutz suggests, fstrim on btrfs is known to be buggy at present. Solutions are in general known and patches available, but they aren't in 4.2, and while 4.3 will be better, AFAIK the patch there is only a partial solution.

As I overprovisioned my SSDs such that they're only something like 55% partitioned at all, leaving the other ~45% entirely unpartitioned empty space, there's plenty of room for the FTL to do its thing there and while I know fstrim is scheduled to run weekly, I've not personally worried too much about it beyond seeing the on-list discussion, but I understand why others with much tighter provisioning/partitioning would be more worried about it.

But I think part of reasoning behind the relatively low priority this issue has received is that it's a low visibility issue not really affecting most people running btrfs, either because they're not running on ssd or because they simply don't have a particularly high awareness of what trim does and thus about how it's failing to work here and what that means to them. If we get a rash of people posting on-list that it's affecting them, that relative priority is likely to go up, and with it the patch testing and integration schedule for the affected patches.

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman
#3 lore raw
Quoted 8 lines · Duncan

But I think part of reasoning behind the relatively low priority this issue has received is that it's a low visibility issue not really affecting most people running btrfs, either because they're not running on ssd or because they simply don't have a particularly high awareness of what trim does and thus about how it's failing to work here and what that means to them. If we get a rash of people posting on-list that it's affecting them, that relative priority is likely to go up, and with it the patch testing and integration schedule for the affected patches.

I've never actually seen fstrim do anything on btrfs (0 bytes trimmed). I stopped using it a few months ago when the news came out about all the issues with its implementation, and I believe my drive is still blacklisted anyway.

It really should be fixed, but right now that goes all around - if btrfs fixed it tomorrow I'd still be stuck until somebody figures out how to reliably do it on a Samsung 850.

--
Rich

#4 lore raw

Rich Freeman posted on Sun, 27 Sep 2015 23:08:49 -0400 as excerpted:

Quoted 11 lines · Duncan
Quoted 9 lines

But I think part of reasoning behind the relatively low priority this issue has received is that it's a low visibility issue not really affecting most people running btrfs, either because they're not running on ssd or because they simply don't have a particularly high awareness of what trim does and thus about how it's failing to work here and what that means to them. If we get a rash of people posting on-list that it's affecting them, that relative priority is likely to go up, and with it the patch testing and integration schedule for the affected patches.

Quoted 8 lines

I've never actually seen fstrim do anything on btrfs (0 bytes trimmed). I stopped using it a few months ago when the news came out about all the issues with its implementation, and I believe my drive is still blacklisted anyway.

It really should be fixed, but right now that goes all around - if btrfs fixed it tomorrow I'd still be stuck until somebody figures out how to reliably do it on a Samsung 850.

FWIW, just checked here, and fstrim -v says it trims rather more than I expected, actually -- tho I should mention that I *am* on 4.3-rc2+, so I should be running those patches mentioned as going into 4.3 that at least partially address the problem.

On my home filesystem (actually /h here):

Btrfs raid1 on two 20-gig partitions:

# btrfs fi sh /h
Label: 'hm0238gcnx+35l0'  uuid: c9f93efc-5ae1-4d89-b501-2d7944856e98
        Total devices 2 FS bytes used 13.24GiB
        devid    1 size 20.00GiB used 14.78GiB path /dev/sda6
        devid    2 size 20.00GiB used 14.78GiB path /dev/sdb6
# btrfs fi usage /h

Overall:

    Device size:                  40.00GiB
    Device allocated:             29.56GiB
    Device unallocated:           10.44GiB

Device missing: 0.00B
Used: 26.48GiB
Free (estimated): 6.44GiB (min: 6.44GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 160.00MiB (used: 0.00B)

Data,RAID1: Size:14.00GiB, Used:12.78GiB
/dev/sda6 14.00GiB
/dev/sdb6 14.00GiB

Metadata,RAID1: Size:768.00MiB, Used:476.02MiB
/dev/sda6 768.00MiB
/dev/sdb6 768.00MiB

System,RAID1: Size:32.00MiB, Used:16.00KiB
/dev/sda6 32.00MiB
/dev/sdb6 32.00MiB

Unallocated:
/dev/sda6 5.22GiB
/dev/sdb6 5.22GiB

# fstrim -v /h
/h: 13.3 GiB (14255448064 bytes) trimmed

OK, so I'm used to thinking in per-device sizes, IOW ratio two, here. And I guess that 13.3 GiB trimmed as reported by fstrim isn't divided by two, which is why it looks so unexpectedly large to me on a filesystem of 20 GiB capacity, 2*20 GiB partitions in raid1. That'd be 6.5+ GiB per partition trimmed, on a 20 GiB partition that's 5.22 GiB unallocated. That's not to bad, actually. =:^)

At first I was thinking wait a minute, 20 GiB per partition, 12.78 GiB of actual data on each, and 13.3 GiB trimmed. WTF? That doesn't add up! Of course not, because all those figures but trimmed are cut in half, per partition numbers, while trimmed is total for the filesystem. So either trimmed needs cut in half too, or the other numbers need doubled to reflect filesystem totals. /Then/ it makes sense! =:^)

FWIW, the SSDs are both Corsair Neutrons, 256 GB (238.5 GiB) capacity, partitioned up with the largest partitions 24 GiB each, but only just under 51% (121 GiB of the 238.5) actually partitioned, 117.5 GiB of unpartitioned free space. (Upthread I said about 55% partitioned, seems I was a bit conservative.)

As I said upthread, that leaves the FTL plenty of room to do wear- leveling, even without active fstrim, so working fstrim is simply "nice to have" for me, not anything like necessary.

So obviously my Corsair Neutrons aren't blacklisted. =:^) I've actually been rather pleasantly surprised indeed with 'em, as I didn't know half what I do now about SSDs when I purchased them, but it seems I made a pretty good choice. The controllers don't do the invisible dedup/ compression that sandforce controllers do so I can rely on what I write, including dup-mode copies, actually being written; they're aimed at the server market where consistent speed is more important than burst speed and the behavior demonstrates exactly that, etc.

The one bad point, and it's probably luck of the draw, is that of the three identical units I purchased together, from Fry's Electronics, one was apparently returned by someone else and I too really should have returned it, as it gave me problems almost from the beginning, but I didn't.

OTOH, I ran for several months with the bad one and one of the good ones paired in the raid1, using btrfs' data integrity features and scrub to repair the data as sectors went bad and the scrub forced the FTL to mark a bad sector and replace it with one of the spares, and thus have far more practical experience now with btrfs raid1 in failing-data conditions, btrfs scrub restoring from the good copy, and the behavior of ssds as more and more sectors go bad, along with how smartctl -A reports the damage, than I ever would have gotten had I simply returned the defective device.

In fact, I was able to watch as over time, smartctl -A dropped that stat line cooked value from 255 "clean", to 100 immediately as the first spare sector was swapped in, down thru several hundred sectors per percentage point, until I finally decided I had enough, at 85 percent (with a threshold of 36% for that value, so I was still nowhere near threshold fail for that value, tho the more immediate per-session read-retry-rate would go thru the roof and drop that cooked value to 1, with a threshold of 6 so definitely failing right then, any time I did a big update or balance and then a scrub).

I just replaced it a few days ago, less than a week, I think, with the still unused third one of the three, originally bought for a netbook I never installed it in and eventually lost, with the third one so far seemed every bit as healthy as the good one of the pair, no bad blocks reported by scrub or spare sectors reported swapped in by smartctl -A, on either one, so far. Which of course was why I was able to let the bad one go so long, since the good one of the pair was absolutely reliable. =:^)

Signature · 3 lines
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman