Options for SSD - autodefrag etc?

Duncan, thank you for this outstanding explanation. It was very informative and helpful. I only have one follow-up question.

I followed your advice on NOCOW for virtualbox images and torrents like so:
chattr -v /home/juha/VirtualBox\ VMs/
chattr -RC /home/juha/Downloads/torrent/#unfinished

As you can see, i used the recursive flag. However, I do not know whether this will automatically apply to files that will be created in the future in subfolders that do not yet exist.

Also, how can I confirm whether a file/folder has a NOCOW attribute set on it?

The C attribute is also inherited by newly created directories. But keep in mind that, at the time applied, it only has effects on existing files if they are empty (read: never written to yet). Newly created files will inherit the attribute from its directory and then behave as expected.

You can use lsattr to confirm the C attribute was set. But again keep in
mind: it does not reflect the file is actually nocow because of the above caveat. So in your use-case you may want to be sure by doing this (quit all VirtualBox instances beforehand):

# mkdir "VirtualBox VMs.new"
# chattr +C "VirtualBox VMs.new"
# rsync -aSv "VirtualBox VMs"/. "VirtualBox VMs.new"/.
# mv "VirtualBox VMs" "VirtualBox VMs.bak"
# mv "VirtualBox VMs.new" "VirtualBox VMs"

Then ensure everything is working, you can use lsattr to see the C attribute has been inherited. You should immediatly notice the effects of this by seeing better performing IO in VirtualBox (at least this was what I noticed). If everything was copied correctly, you can delete the backups. You could compare md5sums to be sure, of course before running a VM. ;-)

Replies to list only preferred.

Kai Krakow posted on Fri, 24 Jan 2014 21:27:19 +0100 as excerpted:

Excellent reply (including what I snipped). I don't actually work with VMs or other huge internal-write files much here, and don't otherwise work with extended attributes much, so would have had to lookup lsattr, and wasn't actually sure on the nested subdirs inheritance point myself tho I thought it /should/ work that way.

And your chattr/rsync routine ensures all data will be newly copied in AFTER the chattr on the dir, thus nicely addressing the very critical point about NEW DATA ONLY coverage I was most worried about communicating correctly. =:^)

Which is why I so enjoy mailing lists and newsgroups. Time and again I've seen one person's answer simply not getting the whole job done no matter how mightily they struggle to do so, but because it's a public list/group, someone else steps in with a followup that addresses the gaps left by the first answer. It nicely takes the pressure off any one person to have the "perfect" reply "every" time, as well as benefiting untold numbers of lurkers who now understand something they didn't know before, but may have never thought to ask themselves. =:^)

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Every write on a SSD block reduces its data retension capability.

No concrete figures but it is assumed to be
- 10 years for new devices
- 1 year at rated usage. (There are much lower figures around)

Hence, I would not trade retension time and wear for autodefrag with no/minor benefits on SSD. (which means at least +2x write amplification on fragments)

On hard disks, we've experienced temporary freezes (about 10secs to 3mins) during background autodefrag.

Regards,
Imran

Am Samstag, 25. Januar 2014, 15:33:08 schrieb Imran Geriskovan:

Where do you have these figures from?

For the Intel SSD 320 in this ThinkPad T520 I read about a minimal usable live of 5 years with 20 GB host writes each day in the tech specs. Thats 7300 GB a year or 7,3 TB. I assume metric system here.

According to smartctl it has written

241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always
- 360158

360158 * 32 MiB (hmmm, now according to smartctl output this is MiB) which gives almost 11 TiB (10,99).

The SSD is over 2,5 years old. Thats less than 5 TiB a year.

So that would lay within the range you say. Although the Intel SSD 320 isn´t basically a new device in my eyes.

Thats with KDE session with Akonadi and desktop search, sometimes even two KDE sessions and a load of applications running at times.

Anyway that SSDs still thinks it is well *new*:

233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always
- 0

Thats the same media wearout indicator (which takes into account the amount of Erase cycles according to Intel docs) it had at the first day I used it.

So I am basically not concerned.

While autodefrag mal cause additional writes… that would not even be the main reason for me not to use it at the moment. I am just not convinced that it gives any noticable benefit. And given that… of course it doesn´t make sense to me to have it cause additional writes to the SSD.

But I am not using it due to avoiding those additional writes in the first place.

My most important recommendation regarding SSDs still is: Keep some space free. Yeah, SSD manufacturers are doing this. But in another Intel SSD PDF I saw some graphs that convinced me in an instant that leaving free about 20% is a good idea. But heck, due to the current fill status of this SSD I do not even adhere to my own recommendation at the moment.

Then a occasional fstrim, maybe mount with noatime (cause who cares about it at all?)…

Ciao,

Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Martin Steigerwald posted on Sat, 25 Jan 2014 15:01:13 +0100 as excerpted:

The two of you are talking about two entirely different things.

1) SSD's limited write-cycle thing, which you're talking about, is widely known, and must be considered, but with modern wear-leveling it's not a /horrible/ concern under /reasonable/ (that is, not constant write/erase as if to benchmark or prove the point) usage. While it's a real issue, I think it has been blown out of proportion, potentially by old-style spinning-rust manufacturers in ordered to maintain a market when it looked like SSD prices were going to drop to and below spinning rust prices within a few years (which they didn't do).

I don't remember the exact numbers I saw given at one point, but they were in the context of worry over using SSD for swap. Suffice it to say that the level of constant writing to blow the write-cycle rating within a feasible swap-usage lifetime of 5 years was well beyond anything most people even with low memory would be doing. Once I saw those numbers, I more or less quit /worrying/ about that, and started /considering/ it, but in a far more "yes, this is practical to use without excessive worry" context.

2) What (I think) Imran was talking about was something very different altho somewhat related, which has seemed to get far less attention, the actual memory cell on-the-shelf-archival data retention lifetime.

For comparison purposes and to make crystal clear that we're not talking about rewriting, it's well known that commercially pressed CDs have a useful lifetime of perhaps a few decades (15-25 years is what I've seen quoted) if treated /well/ (practically "well", still actually using them, not atmosphere-controlled file away for a decade and bring out to read once test then file away again data-archiving well), while CD-ROMs burnt at full rated 24x speed may retain their data for only perhaps 2-5 years, but reducing the write-speed to say 4x can often double or triple that, thus yielding a very reasonable decade or so of retention, midline, approaching commercial press lifetimes of a quarter century or so on the long end.

With current-use common SSD MLC-flash-memory technology, the cell-data- retention lifetimes numbers I've seen are as Imran said, perhaps 10 years powered-off when new, a year at rated write-cycles, down as far as days or even hours past rating shortly before cell write failure.

*HOWEVER*, that's *UNPOWERED* data retention time. Flash technology, like DRAM but on an order of hours/days/years instead of milliseconds, requires refreshing cell charge occasionally to maintain state. Plug in that USB thumbdrive that you've written to a couple of times then forgotten until you find it again several years later, and it'll probably still work. If the same thumbdrive was used as swap (impractical perhaps, but this is just a thought experiment example) on a low-memory machine for a year, such that it reached lifetime write rating, then unplugged and lost for a few years, then found and plugged in to see what's on it, very likely it'd be unreadable.

OTOH, plug that same thumbdrive into an internal USB connector on a regularly used machine, use it as swap for a year, then reconfigure not to use it as swap any longer but keep it in the machine and keep using the machine regularly, so the thumbdrive continues to receive power but isn't actually used to store anything for a few years, and when that machine dies and you're salvaging it before throwing out the dead hulk, and you find that forgotten thumb drive still plugged into its internal slot, the data from its last use may very well still be readable, because the thumb drive was regularly powered and the cells recharged the whole time it wasn't otherwise used.

Now apply that same idea to a standard SSD instead of a thumb drive.

But with SSDs still relatively expensive compared to spinning rust, such sit around unpowered for years, or even weeks, usage, just isn't that common. And if the flash (in SSD or thumbdrive form) is regularly powered, the cells recharge and data should be retained.

So again, as long as SSDs remain more expensive and lower capacity than spinning rust (and as long as capacity doesn't reach petabytes for under $100 at near current data usage, such that the difference in cost is so trivial it ceases to be a factor), they're relatively unlikely to be used for archival storage where unpowered data retention under say a year is that much of a factor. Sure, if unpowered retention life drops to weeks, someone might go on vacation and not power their work laptop for long enough to be a problem, but as long as unpowered retention remains a year or so at minimum, the issue isn't likely to hit the common person enough to hit the radar.

Still, as can be seen by Imram's post, it's a real concern for some, perhaps because the technology is new enough and unproven enough that they're worried the numbers aren't actually that good, and that they'll find themselves on the wrong end of an outlier, dead in the water after taking a week off.

But to quote you admittedly now out of context (since I happened to glance down and see your sentence, just waiting to be quoted in my new context! =:^) :

Particularly since I still have bootable spinning rust backups at this point in any case. I might lose a few months of work as I'm not exactly keeping those backups current, but the risk is low enough and the work I'd lose uncritical enough, that's a risk I'm willing to take...

Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[I don't see this on the list (which I read/reply-to using nntp via gmane.org's list2news service) yet, so I'll reply to both the list and you directly via mail...]

The default is now space_cache -- formerly a btrfs needed mounted with it once, after which the option was "sticky" and applied by default so it didn't need to be used again, and I believe the wiki mount-option documentation at least still says that, but for at least several kernels, the option seems to be on by default -- I never mounted with space_cache specifically given here, yet all my btrfs have it listed in /proc/self/mounts.

So space-cache is now the default unless specifically turned off. And while I don't have a specific reason to use it on ssd, I don't have a good reason not to either, so not knowing anything specific I figured I'd be best sticking with the defaults.

So on my btrfs on SSDs, space_cache is default-on simply because it's the default and I know no good reason to mess with the defaults in that case. Actually I hadn't even thought of it in the context of something else to record and thus to contribute to write-cycles, if there's no real benefit to it on SSDs otherwise, but I guess there is, or it'd default to off when ssds are detected, just like the ssd option is automatically turned on in that case. (In general, btrfs should in most cases be able to detect ssd if it's on the "bare metal" physical device or a partition on it. If the btrfs is on top of lvm or mdraid, however, or on some other mid-layer virtual device, it's less likely to properly detect ssd, and you'd likely need to turn that option on manually.)

Duncan - No HTML messages please, as they are filtered as spam.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman