SSD Optimizations
I'm looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Specifically, on ext2 (journal means more writes, so I don't use ext3 on SSDs, since fsck typically only takes a few seconds when access time is < 100us), I usually apply the -b 4096 -E stripe-width = (erase_block/4096)
parameters to mkfs in order to reduce the multiple erase cycles on the same underlying block.
Are there similar optimizations available in BTRFS?
Gordan
Hi there,
Quoted 2 lines · Gordan Bobic
[...]
Are there similar optimizations available in BTRFS?
There is an SSD mount option available[1].
Cheers,
Marcus
[1] http://btrfs.wiki.kernel.org/index.php/Getting_started#Mount_Options
Erm... You know, sorry for the noise.
Cheers,
Marcus
Quoted 6 lines · Gordan Bobic
I'm looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Quoted 8 lines
Specifically, on ext2 (journal means more writes, so I don't use ext3 on SSDs, since fsck typically only takes a few seconds when access time is < 100us), I usually apply the
-b 4096 -E stripe-width = (erase_block/4096)
parameters to mkfs in order to reduce the multiple erase cycles on the same underlying block.Are there similar optimizations available in BTRFS?
I think you'll get more out of btrfs, but another thing you can look into is ext4 without the journal. Support was added for that recently (thanks to google).
Quoted 9 lines · Marcus Fritzsch
Hi there,
Quoted 2 lines · Gordan Bobic
[...]
Are there similar optimizations available in BTRFS?There is an SSD mount option available[1].
[1] http://btrfs.wiki.kernel.org/index.php/Getting_started#Mount_Options
But what _exactly_ does it do?
Is there a way to leverage any knowledge of erase block size at file system creation time?
Are there any special parameters that might affect locations of superblocks and metadata?
Is there a way to ensure they don't span erase block boundaries?
What about ATA TRIM command support? Is this available? Is it included in the version in Fedora 13?
Gordan
Quoted 14 lines · Mike Fedyk
Quoted 13 lines · Gordan Bobic
I'm looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Specifically, on ext2 (journal means more writes, so I don't use ext3 on SSDs, since fsck typically only takes a few seconds when access time is < 100us), I usually apply the
-b 4096 -E stripe-width = (erase_block/4096)
parameters to mkfs in order to reduce the multiple erase cycles on the same underlying block.
Quoted 5 lines
Quoted 1 line
Are there similar optimizations available in BTRFS?
I think you'll get more out of btrfs, but another thing you can look into is ext4 without the journal. Support was added for that recently (thanks to google).
How is this different to using mkfs.ext2 from e4fsprogs?
And while I appreciate hopeful remarks along the lines of "I think you'll get more out of btrfs", I am really after specifics of what the ssd mount option does, and what features comparable to the optimizations that can be done with ext2/3/4 (e.g. the mentioned stripe-width option) are available to get the best possible alignment of data and metadata to increase both performance and life expectancy of a SSD.
Also, for drives that don't support TRIM, is there a way to make the FS apply aggressive re-use of erased space (in order to help the drive's internal wear-leveling)?
I have looked through the documentation and the wiki, but it provides very little of actual substance.
Gordan
Hello Gordan,
Gordan Bobic wrote (ao):
Quoted 4 lines
Quoted 3 lines · Mike Fedyk
Quoted 2 lines · Gordan Bobic
Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
With SSDs you don't have to worry about wear.
Quoted 7 lines
And while I appreciate hopeful remarks along the lines of "I think you'll get more out of btrfs", I am really after specifics of what the ssd mount option does, and what features comparable to the optimizations that can be done with ext2/3/4 (e.g. the mentioned stripe-width option) are available to get the best possible alignment of data and metadata to increase both performance and life expectancy of a SSD.
Alignment is about the partition, not the fs, and thus taken care of with fdisk and the like.
If you don't create a partition, the fs is aligned with the SSD.
Quoted 3 lines
Also, for drives that don't support TRIM, is there a way to make the FS apply aggressive re-use of erased space (in order to help the drive's internal wear-leveling)?
TRIM has nothing to do with wear-leveling (although it helps reducing wear).
TRIM lets the OS tell the disk which blocks are not in use anymore, and thus don't have to be copied during a rewrite of the blocks. Wear-leveling is the SSD making sure all blocks are more or less equally written to avoid continuous load on the same blocks.
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 6 lines · Gordan Bobic
Quoted 5 lines · Marcus Fritzsch
Hi there,
Quoted 1 line · Gordan Bobic
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
[...]
Are there similar optimizations available in BTRFS?There is an SSD mount option available[1].
[1] http://btrfs.wiki.kernel.org/index.php/Getting_started#Mount_Options
But what _exactly_ does it do?
Chris explains the change to favour spatial locality in allocator behaviour in with '-o ssd'. '-o ssd_spread' does the opposite, where RMW cycles are higher penalty. Elsewhere IIRC, Chris also said BTRFS attempts to submit 128KB BIOs where possible (or wishful thinking?):
http://markmail.org/message/4sq4uco2lghgxzzz
Signature · 1 line
Daniel J Blueman
Quoted 5 lines · On Thursday 11 March 2010 08:38:53 Sander wrote:
Hello Gordan,
=20
Gordan Bobic wrote (ao):Quoted 2 lines
Quoted 1 line · Mike Fedyk
On Wed, Mar 10, 2010 at 11:49 AM, Gordan Bobic <[email protected]>=
Quoted 1 line · wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Are there options available comparable to ext2/ext3 to help reduc=
e
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
wear and improve performance?
=20
With SSDs you don't have to worry about wear.
Sorry, but you do have to worry about wear. I was able to destroy a rel= atively=20
new SD card (2007 or early 2008) just by writing on the first 10MiB ove= r and=20
over again for two or three days. The end of the card still works witho= ut=20
problems but about 10 sectors on the beginning give write errors.
And with journaled file systems that write over and over again on the s= ame spot=20
you do have to worry about wear leveling. It depends on the underlying = block=20
allocation algorithm, but I'm sure that most of the cheap SSDs do wear=20 leveling only inside big blocks, not on whole hard drive, making it muc= h=20
easier to hit the 10 000-100 000 erase cycles boundary.
Still, I think that if you can prolong the life of hardware without not= icable=20
performance degradation, you should do it. Just because it may help the=
drive=20
with some defects last those 3-5years between upgreades without any pro= blems.
Quoted 7 lines
=20
Quoted 6 lines
And while I appreciate hopeful remarks along the lines of "I think you'll get more out of btrfs", I am really after specifics of what the ssd mount option does, and what features comparable to the optimizations that can be done with ext2/3/4 (e.g. the mentioned stripe-width option) are available to get the best possible alignment of data and metadata to increase both performance and lif=
e
Quoted 6 lines
Quoted 1 line
expectancy of a SSD.
=20
Alignment is about the partition, not the fs, and thus taken care of with fdisk and the like.
=20
If you don't create a partition, the fs is aligned with the SSD.
But it does not align internal FS structures to the SSD erase block siz= e and=20
that's what Gordon asked for.
And sorry Gordon, I don't know. But there's a 'ssd_spread' option that = tries=20
to allocate blocks as far as possible (within reason) from themselfs. = That=20
should, in most cases, make the fs structures reside on an erase block = by =20
themself.
I'm afraid that you'll need to dive into the code to know about block=20 alignment or one of the developers will need to provide us with info.
Quoted 2 lines
=20
Quoted 1 line
Also, for drives that don't support TRIM, is there a way to make th=
e
Quoted 6 lines
Quoted 2 lines
FS apply aggressive re-use of erased space (in order to help the drive's internal wear-leveling)?
=20
TRIM has nothing to do with wear-leveling (although it helps reducing wear).
TRIM lets the OS tell the disk which blocks are not in use anymore, a=
nd
Quoted 2 lines
thus don't have to be copied during a rewrite of the blocks. Wear-leveling is the SSD making sure all blocks are more or less equa=
lly
Quoted 1 line
written to avoid continuous load on the same blocks.
Isn't this all about wear leveling? TRIM has no meaning for magnetic me= dia.=20
It's used to tell the drive which parts of medium contain only junk dat= a and=20
can be used in block rotation, making the wear-leveling easier and more= =20
effective.
--=20
Hubert Kario
QBS - Quality Business Software
ul. Ksawer=F3w 30/85
02-656 Warszawa
POLAND
tel. +48 (22) 646-61-51, 646-74-24
fax +48 (22) 646-61-50
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · On Thu, 11 Mar 2010 11:59:57 +0100 Hubert Kario <[email protected]> wrote:
Quoted 8 lines · On Thursday 11 March 2010 08:38:53 Sander wrote:
Hello Gordan,
Gordan Bobic wrote (ao):
Quoted 4 lines
Quoted 3 lines · Mike Fedyk
Quoted 2 lines · Gordan Bobic
Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Quoted 6 lines
Quoted 1 line
With SSDs you don't have to worry about wear.
Sorry, but you do have to worry about wear. I was able to destroy a relatively new SD card (2007 or early 2008) just by writing on the first 10MiB over and over again for two or three days. The end of the card still works without problems but about 10 sectors on the beginning give write errors.
Sorry, the topic was SSD, not SD. SSDs have controllers that contain heavy closed magic to circumvent all kinds of troubles you get when using classical flash and SD cards.
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
Of course you may well discuss about an option for passive flash devices like ide-CF/SD or the like. There is no controller involved so your fs implementation may well work out.
Signature · 2 lines
Regards, Stephan
Quoted 4 lines · Sander
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
With SSDs you don't have to worry about wear.
And if you believe that you clearly swallowed the marketing spiel hook line and sinker without enough real world exprience to show you otherwise. But I'm not going to go off on a tangent now enumerating various victories of marketing over mathematics and empirical evidence relating to currently popular technologies.
In short - I have several dead SSDs of various denominations that demonstrate otherwise - and all within their warranty period, and not having been used in pathologically write-heavy environments. You do have to worry about wear. Operations that increase wear also reduce speed (erasing a block is slow, and if the disk is fully tainted you cannot write without erasing), so you doubly have to worry about it.
Also remember that hardware sectors are 512 bytes, and FS blocks tend to be 4096 bytes. It is thus entirely plausible that if you aren't careful you'll end up with blocks straddling two erase block boundaries. If that happens, you'll make wear twice as bad because you are facing a situation where you may need to erase and write two blocks rather than one. Half the performance, twice the wear.
Quoted 8 lines
Quoted 7 lines
And while I appreciate hopeful remarks along the lines of "I think you'll get more out of btrfs", I am really after specifics of what the ssd mount option does, and what features comparable to the optimizations that can be done with ext2/3/4 (e.g. the mentioned stripe-width option) are available to get the best possible alignment of data and metadata to increase both performance and life expectancy of a SSD.
Quoted 4 lines
Alignment is about the partition, not the fs, and thus taken care of with fdisk and the like.
If you don't create a partition, the fs is aligned with the SSD.
I'm talking about internal FS data structures, not the partition alignment.
Quoted 6 lines
Quoted 3 lines
Also, for drives that don't support TRIM, is there a way to make the FS apply aggressive re-use of erased space (in order to help the drive's internal wear-leveling)?
TRIM has nothing to do with wear-leveling (although it helps reducing wear).
That's self contradictory. If it helps reduce wear it has something to do with wear leveling.
Quoted 4 lines
TRIM lets the OS tell the disk which blocks are not in use anymore, and thus don't have to be copied during a rewrite of the blocks. Wear-leveling is the SSD making sure all blocks are more or less equally written to avoid continuous load on the same blocks.
And thus it is impossible to do wear leveling when all blocks have been written to once without TRIM. So I'd say that in the long term, without TRIM there is no wear leveling. That makes them pretty related.
So considering that there are various nuggets of opinion floating around (correct or otherwise) saying that ext4 has support for TRIM, I'd like to know whether there similar support in BTRFS at the moment?
Gordan
Quoted 6 lines · Daniel J Blueman
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
[...]
Are there similar optimizations available in BTRFS?There is an SSD mount option available[1].
[1]
http://btrfs.wiki.kernel.org/index.php/Getting_started#Mount_Options
Quoted 9 lines
Quoted 2 lines
But what _exactly_ does it do?
Chris explains the change to favour spatial locality in allocator behaviour in with '-o ssd'. '-o
ssd_spread' does the opposite, where RMW cycles are higher penalty. Elsewhere IIRC, Chris also said BTRFS attempts to submit 128KB BIOs where possible (or wishful thinking?):
Thanks, that's useful info.
What about FS block and metadata alignment, though? Is there a way to leverage the knowledge of erase block size in order to reduce wear and increase performance?
Gordan
Quoted 6 lines · Hubert Kario
Still, I think that if you can prolong the life of hardware without noticable
performance degradation, you should do it. Just because it may help the drive
with some defects last those 3-5years between upgreades without any problems.
I couldn't agree more. Not only that but working with wear leveling in mind (especially on devices without TRIM support) can increase performance, too, by avoiding having to wait for the slow erase operation on writes.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Also, for drives that don't support TRIM, is there a way to make the FS apply aggressive re-use of erased space (in order to help the drive's internal wear-leveling)?
TRIM has nothing to do with wear-leveling (although it helps reducing wear).
TRIM lets the OS tell the disk which blocks are not in use anymore, and thus don't have to be copied during a rewrite of the blocks. Wear-leveling is the SSD making sure all blocks are more or less
equally
Quoted 4 lines
Quoted 1 line
written to avoid continuous load on the same blocks.
Isn't this all about wear leveling? TRIM has no meaning for magnetic media.
I fully agree that it's important for wear leveling on flash media, but from the security point of view, I think TRIM would be a useful feature on all storage media. If the erased blocks were trimmed it would provide a potentially useful feature of securely erasing the sectors that are no longer used. It would be useful and much more transparent than the secure erase features that only operate on the entire disk. Just MHO.
Gordan
Quoted 3 lines · Stephan von Krawczynski
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line · Gordan Bobic
Are there options available comparable to ext2/ext3 to help
reduce
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
wear and improve performance?
With SSDs you don't have to worry about wear.
Sorry, but you do have to worry about wear. I was able to destroy a relatively
new SD card (2007 or early 2008) just by writing on the first 10MiB
over
Quoted 6 lines
Quoted 4 lines
and
over again for two or three days. The end of the card still works without
problems but about 10 sectors on the beginning give write errors.Sorry, the topic was SSD, not SD.
SD == SSD with an SD interface.
Quoted 3 lines
SSDs have controllers that contain heavy
closed magic to circumvent all kinds of troubles you get when using classical flash and SD cards.
There is absolutely no basis for thinking that SD cards don't contain wear leveling logic. SD standard, and thus SD cards support a lot of fancy copy protection capabilities, which means there is a lot of firmware involvement on SD cards. It is unlikely that any reputable SD card manufacturer wouldn't also build wear leveling logic into it.
Quoted 2 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in
the
Quoted 1 line
end you cannot really tell if the option does something constructive and
not
Quoted 1 line
destructive in combination with a SSD controller.
You can make an educated guess. For starters given that visible sector sizes are not equal to FS block sizes, it means that FS block sizes can straddle erase block boundaries without the flash controller, no matter how fancy, being able to determine this. Thus, at the very least, aligning FS structures so that they do not straddle erase block boundaries is useful in ALL cases. Thinking otherwise is just sticking your head in the sand because you cannot be bothered to think.
Quoted 3 lines
Of course you may well discuss about an option for passive flash devices like ide-CF/SD or the like. There is no controller involved so your fs implementation may well work out.
I suggest you educate yourself on the nature of IDE and CF (which is just IDE with a different connector). There most certainly are controllers involved. The days when disks (mechanical or solid state) didn't integrate controllers ended with MFM/RLL and ESDI disks some 20+ years ago.
Gordan
Quoted 14 lines · On Thu, 11 Mar 2010 12:17:30 +0000 Gordan Bobic <[email protected]> wrote:
Quoted 3 lines · Stephan von Krawczynski
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line · Gordan Bobic
Are there options available comparable to ext2/ext3 to help
reduce
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
wear and improve performance?
With SSDs you don't have to worry about wear.
Sorry, but you do have to worry about wear. I was able to destroy a relatively
new SD card (2007 or early 2008) just by writing on the first 10MiBover
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
and
over again for two or three days. The end of the card still works without
problems but about 10 sectors on the beginning give write errors.Sorry, the topic was SSD, not SD.
SD == SSD with an SD interface.
That really is quite a statement. You really talk of a few-bucks SD card (like the one in my android handy) as an SSD comparable with Intel XE only with different interface? Come on, stay serious. The product is not only made of SLCs and some raw logic.
Quoted 9 lines
Quoted 3 lines
SSDs have controllers that contain heavy
closed magic to circumvent all kinds of troubles you get when using classical flash and SD cards.There is absolutely no basis for thinking that SD cards don't contain wear leveling logic. SD standard, and thus SD cards support a lot of fancy copy protection capabilities, which means there is a lot of firmware involvement on SD cards. It is unlikely that any reputable SD card manufacturer wouldn't also build wear leveling logic into it.
I really don't guess about what is built into an SD or even CF card. But we hopefully agree that there is a significant difference compared to a product that calls itself a _disk_.
Quoted 7 lines
Quoted 2 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in
the
Quoted 1 line
end you cannot really tell if the option does something constructive and
not
Quoted 1 line
destructive in combination with a SSD controller.
Quoted 7 lines
You can make an educated guess. For starters given that visible sector sizes are not equal to FS block sizes, it means that FS block sizes can straddle erase block boundaries without the flash controller, no matter how fancy, being able to determine this. Thus, at the very least, aligning FS structures so that they do not straddle erase block boundaries is useful in ALL cases. Thinking otherwise is just sticking your head in the sand because you cannot be bothered to think.
And your guess is that intel engineers had no glue when designing the XE including its controller? You think they did not know what you and me know and therefore pray every day that some smart fs designer falls from heaven and saves their product from dying in between? Really?
Quoted 8 lines
Quoted 3 lines
Of course you may well discuss about an option for passive flash devices like ide-CF/SD or the like. There is no controller involved so your fs implementation may well work out.
I suggest you educate yourself on the nature of IDE and CF (which is just IDE with a different connector). There most certainly are controllers involved. The days when disks (mechanical or solid state) didn't integrate controllers ended with MFM/RLL and ESDI disks some 20+ years ago.
I suggest you don't talk to someone administering some hundred boxes based on CF and SSD mediums for _years_ about pro and con of the respective implementation and its long term usage.
Sorry, the world is not built out of paper, sometimes you meet the hard facts. And one of it is that the ssd option in fs is very likely already overrun by the ssd controller designers and mostly _superfluous_. The market has already decided to make SSDs compatible to standard fs layouts.
Signature · 2 lines
Regards, Stephan
Quoted 18 lines · Stephan von Krawczynski
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line · Gordan Bobic
Are there options available comparable to ext2/ext3 to help
reduce
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
wear and improve performance?
With SSDs you don't have to worry about wear.
Sorry, but you do have to worry about wear. I was able to destroy a relatively
new SD card (2007 or early 2008) just by writing on the first 10MiBover
Quoted 5 lines
Quoted 4 lines
and
over again for two or three days. The end of the card still works without
problems but about 10 sectors on the beginning give write errors.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Sorry, the topic was SSD, not SD.
SD == SSD with an SD interface.
That really is quite a statement. You really talk of a few-bucks SD card (like the one in my android handy) as an SSD comparable with Intel XE
only with
Quoted 1 line
different interface? Come on, stay serious. The product is not only made
of
Quoted 1 line
SLCs and some raw logic.
I am saying that there is no reason for the firmware in an SD card to not be as advanced. If the manufacturer has some advanced logic in their SATA SSD, I cannot see any valid engineering reason to not apply the same logic in a SD product.
Quoted 13 lines
Quoted 4 lines
Quoted 3 lines
SSDs have controllers that contain heavy
closed magic to circumvent all kinds of troubles you get when using classical flash and SD cards.Quoted 8 lines
There is absolutely no basis for thinking that SD cards don't contain wear
leveling logic. SD standard, and thus SD cards support a lot of fancy copy
protection capabilities, which means there is a lot of firmware involvement
on SD cards. It is unlikely that any reputable SD card manufacturer wouldn't also build wear leveling logic into it.
Quoted 1 line
I really don't guess about what is built into an SD or even CF card. But
we
Quoted 2 lines
hopefully agree that there is a significant difference compared to a product that calls itself a _disk_.
Wo don't agree on that. Not at all. I don't see any reason why a CF card and an IDE SSD made by the same manufacturer should have any difference between them other than capacity and the physical package.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Honestly I would just drop the idea of an SSD option simply because
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
vendors implement all kinds of neat strategies in their devices. So
in
Quoted 9 lines
Quoted 9 lines
the
Quoted 2 lines
end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
You can make an educated guess. For starters given that visible sector sizes are not equal to FS block sizes, it means that FS block sizes can straddle erase block boundaries without the flash controller, no matter how
fancy, being able to determine this. Thus, at the very least, aligning
FS
Quoted 1 line
Quoted 1 line
structures so that they do not straddle erase block boundaries is
useful
Quoted 6 lines
Quoted 3 lines
in
ALL cases. Thinking otherwise is just sticking your head in the sand because you cannot be bothered to think.And your guess is that intel engineers had no glue when designing the XE including its controller? You think they did not know what you and me
know
Quoted 2 lines
and
therefore pray every day that some smart fs designer falls from heaven
and
Quoted 1 line
saves their product from dying in between? Really?
I am saying that there are problems that CANNOT be solved on the disk firmware level. Some problems HAVE to be addressed higher up the stack.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Of course you may well discuss about an option for passive flash devices
like ide-CF/SD or the like. There is no controller involved so your
fs
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
implementation may well work out.
I suggest you educate yourself on the nature of IDE and CF (which is
just
Quoted 6 lines
Quoted 4 lines
IDE with a different connector). There most certainly are controllers involved. The days when disks (mechanical or solid state) didn't integrate
controllers ended with MFM/RLL and ESDI disks some 20+ years ago.I suggest you don't talk to someone administering some hundred boxes
based
Quoted 6 lines
on
CF and SSD mediums for _years_ about pro and con of the respective implementation and its long term usage.
Sorry, the world is not built out of paper, sometimes you meet the hard facts.
And one of it is that the ssd option in fs is very likely already
overrun
Quoted 3 lines
by
the ssd controller designers and mostly _superfluous_. The market has already decided to make SSDs compatible to standard fs layouts.
Seems to me that you haven't done any analysis of comparative long term failure rates between SSDs used with default layouts (Default? Really? You mean you don't apply any special partitioning on your hundreds of servers?) and those with carefully aligned FS-es. Just because defaults may be good enough for your use case, doesn't mean that somebody with a use case that's harder on the flash will observe the same reliability, or deem the the unoptimized performance figures good enough.
Gordan
Quoted 8 lines · On Thursday 11 March 2010 14:20:23 Gordan Bobic wrote:
Quoted 5 lines · On Thu, 11 Mar 2010 13:59:09 +0100, Stephan von Krawczynski =20…
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
On Wed, Mar 10, 2010 at 11:49 AM, Gordan Bobic
<[email protected]>Quoted 1 line · wrote:
Are there options available comparable to ext2/ext3 to he=
lp
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
reduce
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
wear and improve performance?
With SSDs you don't have to worry about wear.
Sorry, but you do have to worry about wear. I was able to destr=
oy a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
relatively
new SD card (2007 or early 2008) just by writing on the first 1=
0MiB
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
over
Quoted 2 lines
Quoted 2 lines
and
over again for two or three days. The end of the card still wor=
ks
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
without
problems but about 10 sectors on the beginning give write error=
s.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Sorry, the topic was SSD, not SD.
SD =3D=3D SSD with an SD interface.
That really is quite a statement. You really talk of a few-bucks SD=
card
Quoted 1 line
Quoted 1 line
(like the one in my android handy) as an SSD comparable with Intel =
XE
Quoted 4 lines
=20
only with
=20Quoted 1 line
different interface? Come on, stay serious. The product is not only=
made
Quoted 6 lines
=20
of
=20Quoted 1 line
SLCs and some raw logic.
=20
I am saying that there is no reason for the firmware in an SD card to=
not
Quoted 1 line
be as advanced. If the manufacturer has some advanced logic in their =
SATA
Quoted 1 line
SSD, I cannot see any valid engineering reason to not apply the same =
logic
Quoted 1 line
in a SD product.
The _SD_standard_ states that the media has to implement wear-leveling. So any card with an SD logo implements it.
As I stated previously, the algorithms used in SD cards may not be as a= dvanced=20
as those in top-of-the-line Intel SSDs, but I bet they don't differ by = much to=20
the ones used in cheapest SSD drives.
Besides, why shouldn't we help the drive firmware by=20
- writing the data only in erase-block sizes
- trying to write blocks that are smaller than the erase-block in a way= that=20
won't cross the erase-block boundary
- using TRIM on deallocated parts of the drive
This will not only increase the life of the SSD but also increase its=20 performance.
Quoted 2 lines
=20
Quoted 1 line
Quoted 1 line
Quoted 1 line
Honestly I would just drop the idea of an SSD option simply beca=
use
Quoted 4 lines
=20
the
=20Quoted 1 line
Quoted 1 line
Quoted 1 line
vendors implement all kinds of neat strategies in their devices.=
So
Quoted 6 lines
=20
in
=20Quoted 3 lines
Quoted 3 lines
the
Quoted 1 line
end you cannot really tell if the option does something construc=
tive
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
and not destructive in combination with a SSD controller.
You can make an educated guess. For starters given that visible se=
ctor
Quoted 1 line
Quoted 1 line
Quoted 1 line
sizes are not equal to FS block sizes, it means that FS block size=
s can
Quoted 1 line
Quoted 1 line
Quoted 1 line
straddle erase block boundaries without the flash controller, no m=
atter
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
how
fancy, being able to determine this. Thus, at the very least, alig=
ning
Quoted 9 lines
=20
FS
=20Quoted 1 line
Quoted 1 line
structures so that they do not straddle erase block boundaries is
=20
useful
=20Quoted 2 lines
Quoted 2 lines
in
ALL cases. Thinking otherwise is just sticking your head in the sa=
nd
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
because you cannot be bothered to think.
And your guess is that intel engineers had no glue when designing t=
he XE
Quoted 1 line
Quoted 1 line
including its controller? You think they did not know what you and =
me
Quoted 5 lines
=20
know
=20Quoted 2 lines
and
therefore pray every day that some smart fs designer falls from hea=
ven
Quoted 7 lines
=20
and
=20Quoted 1 line
saves their product from dying in between? Really?
=20
I am saying that there are problems that CANNOT be solved on the disk firmware level. Some problems HAVE to be addressed higher up the stac=
k.
Exactly, you can't assume that the SSDs firmware understands any and al= l file=20
system layouts, especially if they are on fragmented LVM or other logic= al=20
volume manager partitions.
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=C3=B3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
System Zarz=C4=85dzania Jako=C5=9Bci=C4=85
zgodny z norm=C4=85 ISO 9001:2000
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Gordan Bobic
I'm looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Quoted 8 lines
Specifically, on ext2 (journal means more writes, so I don't use ext3 on SSDs, since fsck typically only takes a few seconds when access time is < 100us), I usually apply the
-b 4096 -E stripe-width = (erase_block/4096)
parameters to mkfs in order to reduce the multiple erase cycles on the same underlying block.Are there similar optimizations available in BTRFS?
All devices (raid, ssd, single spindle) tend to benefit from big chunks of writes going down close together on disk. This is true for different reasons on each one, but it is still the easiest way to optimize writes. COW filesystems like btrfs are very well suited to send down lots of big writes because we're always reallocating things.
For traditional storage, we also need to keep blocks from one file (or files in a directory) close together to reduce seeks during reads. SSDs have no such restrictions, and so the mount -o ssd related options in btrfs focus on tossing out tradeoffs that slow down writes in hopes of reading faster later.
Someone already mentioned the mount -o ssd and ssd_spread options. Mount -o ssd is targeted at faster SSD that is good at wear leveling and generally just benefits from having a bunch of data sent down close together. In mount -o ssd, you might find a write pattern like this:
block N, N+2, N+3, N+4, N+6, N+7, N+16, N+17, N+18, N+19, N+20 ...
It's a largely contiguous chunk of writes, but there may be gaps. Good ssds don't really care about the gaps, and they benefit more from the fact that we're preferring to reuse blocks that had once been written than to go off and find completely contiguous areas of the disk to write (which are more likely to have never been written at all).
mount -o ssd_spread is much more strict. You'll get N,N+2,N+3,N+4,N+5 etc because crummy ssds really do care about the gaps.
Now, btrfs could go off and probe for the erasure size and work very hard to align things to it. As someone said, alignment of the partition table is very important here as well. But for modern ssd this generally matters much less than just doing big ios and letting the little log structured squirrel inside the device figure things out.
For trim, we do have mount -o discard. It does introduce a run time performance hit (this varies wildly from device to device) and we're tuning things as discard capable devices become more common. If anyone is looking for a project it would be nice to have an ioctl that triggers free space discards in bulk.
-chris
Stephan von Krawczynski wrote (ao):
Quoted 4 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
My understanding of the ssd mount option is also that the fs doens't try to do all kinds of smart (and potential expensive) things which make sense for rotating media to reduce seeks and the like.
Sander
Signature · 2 lines
Humilis IT Services and Solutions http://www.humilis.net
Quoted 7 lines · On Thu, 11 Mar 2010 15:01:55 +0100 Hubert Kario <[email protected]> wrote:
[...]
The _SD_standard_ states that the media has to implement wear-leveling. So any card with an SD logo implements it.As I stated previously, the algorithms used in SD cards may not be as advanced as those in top-of-the-line Intel SSDs, but I bet they don't differ by much to the ones used in cheapest SSD drives.
Well, we are all pretty sure about that. And that is exactly the reason why these are not surviving the market pressure. Why should one care about bad products that are possibly already extincted because of their bad performance when the fs is production ready some day?
Quoted 4 lines
Besides, why shouldn't we help the drive firmware by
- writing the data only in erase-block sizes
- trying to write blocks that are smaller than the erase-block in a way that won't cross the erase-block boundary
Because if the designing engineer of a good SSD controller wasn't able to cope with that he will have no chance to design a second one.
Quoted 1 line
- using TRIM on deallocated parts of the drive
Another story. That is a designed part of a software interface between fs and drive bios on which both agreed in its usage pattern. Whereas the above points are pure guess based on dumb and old hardware and its behaviour.
Quoted 2 lines
This will not only increase the life of the SSD but also increase its performance.
TRIM: maybe yes. Rest: pure handwaving.
Quoted 7 lines
[...]
Quoted 6 lines
Quoted 5 lines
And your guess is that intel engineers had no glue when designing the XE including its controller? You think they did not know what you and me know and
therefore pray every day that some smart fs designer falls from heaven and saves their product from dying in between? Really?
Quoted 6 lines
Quoted 2 lines
I am saying that there are problems that CANNOT be solved on the disk firmware level. Some problems HAVE to be addressed higher up the stack.
Exactly, you can't assume that the SSDs firmware understands any and all file system layouts, especially if they are on fragmented LVM or other logical volume manager partitions.
Hopefully the firmware understands exactly no fs layout at all. That would be braindead. Instead it should understand how to arrange incoming and outgoing data in a way that its own technical requirements are met as perfect as possible. This is no spinning disk, it is completely irrelevant what the data layout looks like as long as the controller finds its way through and copes best with read/write/erase cycles. It may well use additional RAM for caching and data reordering.
Do you really believe ascending block numbers are placed in ascending addresses inside the disk (as an example)? Why should they? What does that mean for fs block ordering? If you don't know anyway what a controller does to your data ordering, how do you want to help it with its job? Please accept that we are _not_ talking about trivial flash mem here or pseudo-SSDs consisting of sd cards. The market has already evolved better products. The dinosaurs are extincted even if some are still looking alive.
Quoted 2 lines
Signature · 1 line
Hubert Kario
Signature · 2 lines
Regards, Stephan
Quoted 10 lines · Gordan Bobic
Quoted 5 lines
TRIM lets the OS tell the disk which blocks are not in use anymore, and thus don't have to be copied during a rewrite of the blocks. Wear-leveling is the SSD making sure all blocks are more or less equally written to avoid continuous load on the same blocks.
And thus it is impossible to do wear leveling when all blocks have been written to once without TRIM. So I'd say that in the long term, without TRIM there is no wear leveling. That makes them pretty related.
I'm no expert of SSDs, however
1- I think the SSD would rewrite once-written blocks to other locations, so to reuse the same physical blocks for wear levelling. The written-once blocks are very good candidates because their write-count is "1"
2- I think SSDs show you a smaller usable size than what they physically have. In this way they always have some more blocks for moving data to, so to free blocks which have a low write-count.
3- If you think #2 is not enough you can leave part of the SSD disk unused, by leaving unused space after the last partition.
Actually, after considering #1 and #2, I don't think TRIM is really needed for SSDs, are you sure it is really needed? I think it's more a kind of optimization, but it needs to be very fast for it to be useful as an optimization, faster than an internal block rewrite by the SSD wear levelling, and so fast as SATA/SAS command that the computer is not significantly slowed down by using it. Instead IIRC I read something about it being slow and maybe was even requiring FUA or barrier or flush? I don't remember exactly.
There is one place where TRIM would be very useful though, and it's not for SSDs, but it's in virtualization: if the Virtual Machine frees space, the VM file system should use TRIM to signal to the host that some space is unused. The host should have a way to tell its filesystem that the VM-disk-file has a new hole in that position, so that disk space can be freed on the host for use for another VM. This would allow much greater overcommit of disk spaces to virtual machines.
There's probably no need for "TRIM" support itself on the host filesystem, but another mechanism is needed that allows to sparsify an existing file creating a hole in it (which I think is not possible with the filesystems syscalls we have now, correct me if I'm wrong). There *is* need for TRIM support in the guest filesystem though.
Quoted 6 lines · Stephan von Krawczynski
Quoted 4 lines
Besides, why shouldn't we help the drive firmware by
- writing the data only in erase-block sizes
- trying to write blocks that are smaller than the erase-block in a way that won't cross the erase-block boundaryBecause if the designing engineer of a good SSD controller wasn't able
to
Quoted 1 line
cope with that he will have no chance to design a second one.
You seem to be confusing quality of implementation with theoretical possibility.
Quoted 9 lines
Quoted 2 lines
This will not only increase the life of the SSD but also increase its performance.
TRIM: maybe yes. Rest: pure handwaving.
Quoted 4 lines
[...]
Quoted 3 lines
Quoted 3 lines
And your guess is that intel engineers had no glue when designing the XE
including its controller? You think they did not know what you and
me
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
know and
therefore pray every day that some smart fs designer falls from heaven
and saves their product from dying in between? Really?I am saying that there are problems that CANNOT be solved on the disk firmware level. Some problems HAVE to be addressed higher up the
stack.
Quoted 2 lines
Quoted 2 lines
Exactly, you can't assume that the SSDs firmware understands any and
all
Quoted 6 lines
Quoted 4 lines
file
system layouts, especially if they are on fragmented LVM or other logical
volume manager partitions.Hopefully the firmware understands exactly no fs layout at all. That
would
Quoted 7 lines
be
braindead. Instead it should understand how to arrange incoming and outgoing
data in a way that its own technical requirements are met as perfect as possible. This is no spinning disk, it is completely irrelevant what the data
layout looks like as long as the controller finds its way through and
copes
Quoted 4 lines
best with read/write/erase cycles. It may well use additional RAM for caching and data reordering.
Do you really believe ascending block numbers are placed in ascending addresses inside the disk (as an example)? Why should they? What does
that
Quoted 5 lines
mean for fs block ordering? If you don't know anyway what a controller does to
your data ordering, how do you want to help it with its job? Please accept that we are _not_ talking about trivial flash mem here or pseudo-SSDs consisting of sd cards. The market has already evolved
better
Quoted 1 line
products. The dinosaurs are extincted even if some are still looking
alive.
I am assuming that you are being deliberately facetious here (the alternative is less kind). The simple fact is that you cannot come up with some magical data (re)ordering method that nullifies problems of common use-cases that are quite nasty for flash based media.
For example - you have a disk that has had all it's addressable blocks tainted. A new write comes in - what do you do with it? Worse, a write comes in spanning two erase blocks as a consequence of the data re-alignment in the firmware. You have no choice but to wipe them both and re-write the data. You'd be better off not doing the magic and assuming that the FS is sensibly aligned.
Having a large chunk of spare non-addressable space for this doesn't necessarily help you, either, unless it is about the same size as the addressable space (worse case scenario, if you accept that the vast majority of FS-es use 4KB block sizes, you can cut a corner there by a factor of 8). All of that adds to cost - flash is still expensive.
The bottom line is that you _cannot_ solve wear-leveling completely just in firmware. There is no doubt you can get some of the way there, but it is mathematically impossible to solve completely without intervention from further up the stack. Since some black-box firmware optimizations may quite concievably make the wear problem worse, it makes perfect sense to just hopefully assume that the FS is trying to help - it's unlikely to make things worse and may well make things a lot better.
Gordan
Quoted 5 lines · Asdo
1- I think the SSD would rewrite once-written blocks to other locations,
so to reuse the same physical blocks for wear levelling. The written-once blocks are very good candidates because their write-count is "1"
There are likely to be millions of blocks with the same write count. How do you pick the optimal ones?
Quoted 4 lines
2- I think SSDs show you a smaller usable size than what they physically
have. In this way they always have some more blocks for moving data to, so to free blocks which have a low write-count.
I'm pretty sure that is the case, too. However, to be able to deal with the worst case scenario you would have to effectively double the amoutn of flash (only expose half of it as addressable). With some corner cutting and assumptions about file systems' block sizes (usually 4KB these days), you can cut a corner, but that's dodgy until we get bigger hardware sectors as standard.
Quoted 2 lines
3- If you think #2 is not enough you can leave part of the SSD disk unused, by leaving unused space after the last partition.
That is true, but I'd rather apply some higher level logic to this rather than expect the firmware do make a guess about things it has absolutely no way of knowing for sure.
Quoted 2 lines
Actually, after considering #1 and #2, I don't think TRIM is really needed for SSDs, are you sure it is really needed?
I don't think there's any doubt that trim helps flash longevity.
Quoted 8 lines
I think it's more a
kind of optimization, but it needs to be very fast for it to be useful as an optimization, faster than an internal block rewrite by the SSD wear levelling, and so fast as SATA/SAS command that the computer is notsignificantly slowed down by using it. Instead IIRC I read something about it being slow and maybe was even requiring FUA or barrier or flush? I don't remember exactly.
There is no obligation on part of the disk to do anything in response to the trim command, IIRC. It is advisory. It doesn't have to clear the blocks online. In fact, trim is sector based, and it is unlikely an SSD would act until it can sensibly free an entire erase block.
Quoted 7 lines
There is one place where TRIM would be very useful though, and it's not for SSDs, but it's in virtualization: if the Virtual Machine frees space, the VM file system should use TRIM to signal to the host that some space is unused. The host should have a way to tell its filesystem that the VM-disk-file has a new hole in that position, so that disk space can be freed on the host for use for another VM. This would allow much greater overcommit of disk spaces to virtual machines.
Indeed, I brought this very point up on the KVM mailing list a while back.
Gordan
Quoted 14 lines · Chris Mason
Quoted 13 lines · Gordan Bobic
I'm looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Specifically, on ext2 (journal means more writes, so I don't use ext3 on SSDs, since fsck typically only takes a few seconds when access time is < 100us), I usually apply the
-b 4096 -E stripe-width = (erase_block/4096)
parameters to mkfs in order to reduce the multiple erase cycles on the same underlying block.
Quoted 7 lines
Quoted 1 line
Are there similar optimizations available in BTRFS?
All devices (raid, ssd, single spindle) tend to benefit from big chunks of writes going down close together on disk. This is true for different reasons on each one, but it is still the easiest way to optimize writes. COW filesystems like btrfs are very well suited to send down lots of big writes because we're always reallocating things.
Doesn't this mean _more_ writes? If that's the case, then that would make btrfs a _bad_ choice for flash based media with limite write cycles.
Quoted 19 lines
For traditional storage, we also need to keep blocks from one file (or files in a directory) close together to reduce seeks during reads. SSDs have no such restrictions, and so the mount -o ssd related options in btrfs focus on tossing out tradeoffs that slow down writes in hopes of reading faster later.
Someone already mentioned the mount -o ssd and
ssd_spreadoptions. Mount -o ssd is targeted at faster SSD that is good at wear leveling and generally just benefits from having a bunch of data sent down close together. In mount -o ssd, you might find a write pattern like this:block N, N+2, N+3, N+4, N+6, N+7, N+16, N+17, N+18, N+19, N+20 ...
It's a largely contiguous chunk of writes, but there may be gaps. Good ssds don't really care about the gaps, and they benefit more from the fact that we're preferring to reuse blocks that had once been written than to go off and find completely contiguous areas of the disk to write (which are more likely to have never been written at all).
Quoted 8 lines
mount -o
ssd_spreadis much more strict. You'll get N,N+2,N+3,N+4,N+5 etc because crummy ssds really do care about the gaps.Now, btrfs could go off and probe for the erasure size and work very hard to align things to it. As someone said, alignment of the partition table is very important here as well. But for modern ssd this generally matters much less than just doing big ios and letting the little log structured squirrel inside the device figure things out.
Thanks, that's quite helpful. Can you provide any insight into alignment of FS structures in such a way that they do not straddle erase block boundaries?
Quoted 5 lines
For trim, we do have mount -o discard. It does introduce a run time performance hit (this varies wildly from device to device) and we're tuning things as discard capable devices become more common. If anyone is looking for a project it would be nice to have an ioctl that triggers free space discards in bulk.
Are you saying that -o discard implements trim support?
Gordan
Quoted 70 lines · Gordan Bobic
Quoted 6 lines · Stephan von Krawczynski
Quoted 4 lines
Besides, why shouldn't we help the drive firmware by
- writing the data only in erase-block sizes
- trying to write blocks that are smaller than the erase-block in a way that won't cross the erase-block boundaryBecause if the designing engineer of a good SSD controller wasn't able
to
Quoted 1 line
cope with that he will have no chance to design a second one.
You seem to be confusing quality of implementation with theoretical possibility.
Quoted 9 lines
Quoted 2 lines
This will not only increase the life of the SSD but also increase its performance.
TRIM: maybe yes. Rest: pure handwaving.
Quoted 4 lines
[...]
Quoted 3 lines
Quoted 3 lines
And your guess is that intel engineers had no glue when designing the XE
including its controller? You think they did not know what you andme
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
know and
therefore pray every day that some smart fs designer falls from heaven
and saves their product from dying in between? Really?I am saying that there are problems that CANNOT be solved on the disk firmware level. Some problems HAVE to be addressed higher up the
stack.
Quoted 2 lines
Quoted 2 lines
Exactly, you can't assume that the SSDs firmware understands any and
all
Quoted 6 lines
Quoted 4 lines
file
system layouts, especially if they are on fragmented LVM or other logical
volume manager partitions.Hopefully the firmware understands exactly no fs layout at all. That
would
Quoted 7 lines
be
braindead. Instead it should understand how to arrange incoming and outgoing
data in a way that its own technical requirements are met as perfect as possible. This is no spinning disk, it is completely irrelevant what the data
layout looks like as long as the controller finds its way through andcopes
Quoted 4 lines
best with read/write/erase cycles. It may well use additional RAM for caching and data reordering.
Do you really believe ascending block numbers are placed in ascending addresses inside the disk (as an example)? Why should they? What doesthat
Quoted 5 lines
mean for fs block ordering? If you don't know anyway what a controller does to
your data ordering, how do you want to help it with its job? Please accept that we are _not_ talking about trivial flash mem here or pseudo-SSDs consisting of sd cards. The market has already evolvedbetter
Quoted 1 line
products. The dinosaurs are extincted even if some are still looking
alive.
I am assuming that you are being deliberately facetious here (the alternative is less kind). The simple fact is that you cannot come up with some magical data (re)ordering method that nullifies problems of common use-cases that are quite nasty for flash based media.
Quoted 6 lines
For example - you have a disk that has had all it's addressable blocks tainted. A new write comes in - what do you do with it? Worse, a write comes in spanning two erase blocks as a consequence of the data re-alignment in the firmware. You have no choice but to wipe them both and re-write the data. You'd be better off not doing the magic and assuming that the FS is sensibly aligned.
Ok, how exactly would the FS help here? We have a device with a 256kb erasure size, and userland does a 4k write followed by an fsync.
If the FS were to be smart and know about the 256kb requirement, it would do a read/modify/write cycle somewhere and then write the 4KB.
The underlying implementation is the same in the device. It picks a destination, reads it then writes it back. You could argue (and many people do) that this operation is risky and has a good chance of destroying old data. Perhaps we're best off if the FS does the rmw cycle instead into an entirely safe location.
It's a great place for research and people are definitely looking at it.
But with all of that said, it has nothing to do with alignment or trim. Modern ssds are a raid device with a large stripe size, and someone somewhere is going to do a read/modify/write to service any small write. You can force this up to the FS or the application, it'll happen somewhere.
The filesystem metadata writes are a very small percentage of the problem overall. Sure we can do better and try to force larger metadata blocks. This was the whole point behind btrfs' support for large tree blocks, which I'll be enabling again shortly.
-chris
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Gordan" == Gordan Bobic <[email protected]> writes:
Gordan> I fully agree that it's important for wear leveling on flash Gordan> media, but from the security point of view, I think TRIM would Gordan> be a useful feature on all storage media. If the erased blocks Gordan> were trimmed it would provide a potentially useful feature of Gordan> securely erasing the sectors that are no longer used. It would Gordan> be useful and much more transparent than the secure erase Gordan> features that only operate on the entire disk. Just MHO.
Except there are no guarantees that TRIM does anything, even if the drive claims to support it.
There are a couple of IDENTIFY DEVICE knobs that indicate whether the drive deterministically returns data after a TRIM. And whether the resulting data is zeroes. We query these values and report them to the filesystem.
However, testing revealed several devices that reported the right thing but which did in fact return the old data afterwards.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 18 lines · Gordan Bobic
Quoted 16 lines · Chris Mason
Quoted 14 lines · Gordan Bobic
I'm looking to try BTRFS on a SSD, and I would like to know what SSD optimizations it applies. Is there a comprehensive list of what ssd mount option does? How are the blocks and metadata arranged? Are there options available comparable to ext2/ext3 to help reduce wear and improve performance?
Specifically, on ext2 (journal means more writes, so I don't use ext3 on SSDs, since fsck typically only takes a few seconds when access time is < 100us), I usually apply the
-b 4096 -E stripe-width = (erase_block/4096)
parameters to mkfs in order to reduce the multiple erase cycles on the same underlying block.Are there similar optimizations available in BTRFS?
Quoted 8 lines
Quoted 5 lines
All devices (raid, ssd, single spindle) tend to benefit from big chunks of writes going down close together on disk. This is true for different reasons on each one, but it is still the easiest way to optimize writes. COW filesystems like btrfs are very well suited to send down lots of big writes because we're always reallocating things.
Doesn't this mean _more_ writes? If that's the case, then that would make btrfs a _bad_ choice for flash based media with limite write cycles.
It just means that when we do write, we don't overwrite the existing data in the file. We allocate a new block instead and write there (freeing the old one)
.
This gives us a lot of control over grouping writes together, instead of being restricted to the layout from when the file was first created.
It also fragments the files much more, but this isn't an issue on ssd.
Quoted 29 lines
Quoted 20 lines
For traditional storage, we also need to keep blocks from one file (or files in a directory) close together to reduce seeks during reads. SSDs have no such restrictions, and so the mount -o ssd related options in btrfs focus on tossing out tradeoffs that slow down writes in hopes of reading faster later.
Someone already mentioned the mount -o ssd and
ssd_spreadoptions. Mount -o ssd is targeted at faster SSD that is good at wear leveling and generally just benefits from having a bunch of data sent down close together. In mount -o ssd, you might find a write pattern like this:block N, N+2, N+3, N+4, N+6, N+7, N+16, N+17, N+18, N+19, N+20 ...
It's a largely contiguous chunk of writes, but there may be gaps. Good ssds don't really care about the gaps, and they benefit more from the fact that we're preferring to reuse blocks that had once been written than to go off and find completely contiguous areas of the disk to write (which are more likely to have never been written at all).
Quoted 8 lines
mount -o
ssd_spreadis much more strict. You'll get N,N+2,N+3,N+4,N+5 etc because crummy ssds really do care about the gaps.Now, btrfs could go off and probe for the erasure size and work very hard to align things to it. As someone said, alignment of the partition table is very important here as well. But for modern ssd this generally matters much less than just doing big ios and letting the little log structured squirrel inside the device figure things out.
Quoted 3 lines
Thanks, that's quite helpful. Can you provide any insight into alignment of FS structures in such a way that they do not straddle erase block boundaries?
We align on 4k (but partition alignment can defeat this). We don't attempt to understand or guess at erasure blocks. Unless the filesystem completely takes over the FTL duties, I don't think it makes sense to do more than send large writes whenever we can.
The raid 5/6 patches will add more knobs for strict alignment, but I'd be very surprised if they made a big difference on modern ssd.
Quoted 8 lines
Quoted 6 lines
For trim, we do have mount -o discard. It does introduce a run time performance hit (this varies wildly from device to device) and we're tuning things as discard capable devices become more common. If anyone is looking for a project it would be nice to have an ioctl that triggers free space discards in bulk.
Are you saying that -o discard implements trim support?
Yes, it sends trim/discards down to devices that support it.
-chris
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Gordan" == Gordan Bobic <[email protected]> writes:
Gordan> SD == SSD with an SD interface.
No, not really.
It is true that conceivably you could fit a sophisticated controller in an SD card form factor. But fact is that takes up space which could otherwise be used for flash. There may also be power consumption/heat dissipation concerns.
Most SD card controllers have very, very simple wear leveling that in most cases rely on the filesystem being FAT. These cards are aimed at cameras, MP3 players, etc. after all. And consequently it's trivial to wear out an SD card by writing a block over and over.
The same is kind of true for Compact Flash. There are two types of cards, I prefer to think of them as camera grade and industrial. Camera grade CF is really no different from SD cards or any other consumer flash form factor.
Industrial CF cards have controllers with sophisticated wear leveling. Usually this is not quite as clever as a "big" SSD, but it is close enough that you can treat the device as a real disk drive. I.e. it has multiple channels working in parallel unlike the consumer devices.
As a result of the smarter controller logic and the bigger bank of spare flash, industrial cards are much smaller in capacity. Typically in the 1-4 GB range. But they are in many cases indistinguishable from a real SSD in terms of performance and reliability.
Gordan> You can make an educated guess. For starters given that visible Gordan> sector sizes are not equal to FS block sizes, it means that FS Gordan> block sizes can straddle erase block boundaries without the Gordan> flash controller, no matter how fancy, being able to determine Gordan> this. Thus, at the very least, aligning FS structures so that Gordan> they do not straddle erase block boundaries is useful in ALL Gordan> cases. Thinking otherwise is just sticking your head in the sand Gordan> because you cannot be bothered to think.
There are no means of telling what the erase block size is. None. We have no idea. The vendors won't talk. It's part of their IP.
Also, there is no point in being hung up on the whole erase block thing. Only crappy SSDs use block mapping where that matters. These devices will die a horrible death soon enough. Good SSDs use a technique akin to logging filesystems in which the erase block size and all other other physical characteristics don't matter (from a host perspective).
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 11 lines · On Thu, 11 Mar 2010 15:39:05 +0100 Sander <[email protected]> wrote:
Stephan von Krawczynski wrote (ao):
Quoted 4 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
My understanding of the ssd mount option is also that the fs doens't try to do all kinds of smart (and potential expensive) things which make sense for rotating media to reduce seeks and the like.
Sander
Such an optimization sounds valid on first sight. But re-think closely: how does the fs really know about seeks needed during some operation? If your disk is a single plate one your seeks are completely different from multi plate. So even a simple case is more or less unpredictable. If you consider a RAID or SAN as device base it should be clear that trying to optimize for certain device types is just a fake. What does that tell you? The optimization was a pure loss of work hours in the first place. In fact if you look at this list a lot of talks going on are highly academic and have no real usage scenario.
Sometimes trying to be super-smart is indeed not useful (for a fs) ...
Signature · 2 lines
Regards, Stephan
Quoted 9 lines · Stephan von Krawczynski
Quoted 6 lines · On Thu, 11 Mar 2010 15:39:05 +0100 Sander <[email protected]> wrote:
Stephan von Krawczynski wrote (ao):
Quoted 4 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
Quoted 8 lines
Quoted 5 lines
My understanding of the ssd mount option is also that the fs doens't try to do all kinds of smart (and potential expensive) things which make sense for rotating media to reduce seeks and the like.
Sander
Such an optimization sounds valid on first sight. But re-think closely: how does the fs really know about seeks needed during some operation?
Well the FS makes a few assumptions (in the nonssd case). First it assumes the storage is not a memory device. If things would fit in memory we wouldn't need filesytems in the first place.
Then it assumes that adjacent blocks are cheap to read and blocks that are far away are expensive to read. Given expensive raid controllers, cache, and everything else, you're correct that sometimes this assumption is wrong. But, on average seeking hurts. Really a lot.
We try to organize files such that files that are likely to be read together are found together on disk. Btrfs is fairly good at this during file creation and not as good as ext*/xfs as files over overwritten and modified again and again (due to cow).
If you turn mount -o ssd on for your drive and do a test, you might not notice much difference right away. ssds tend to be pretty good right out of the box. Over time it tends to help, but it is a very hard thing to benchmark in general.
-chris
Quoted 7 lines · On Thursday 11 March 2010 17:19:32 Chris Mason wrote:
Quoted 6 lines · Gordan Bobic
Quoted 3 lines · On Thu, 11 Mar 2010 16:35:33 +0100, Stephan von Krawczynski =20…
Quoted 3 lines
Besides, why shouldn't we help the drive firmware by
- writing the data only in erase-block sizes
- trying to write blocks that are smaller than the erase-block i=
n a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
way that won't cross the erase-block boundary
=20
Because if the designing engineer of a good SSD controller wasn't=
able
Quoted 9 lines
Quoted 9 lines
=20
to
=20Quoted 1 line
cope with that he will have no chance to design a second one.
=20
You seem to be confusing quality of implementation with theoretical possibility.
=20Quoted 1 line
Quoted 1 line
This will not only increase the life of the SSD but also increas=
e its
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
performance.
=20
TRIM: maybe yes. Rest: pure handwaving.
=20Quoted 3 lines
[...]
=20Quoted 1 line
Quoted 1 line
And your guess is that intel engineers had no glue when desi=
gning
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the XE
including its controller? You think they did not know what y=
ou and
Quoted 5 lines
Quoted 5 lines
=20
me
=20Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
know and
therefore pray every day that some smart fs designer falls f=
rom
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
heaven
and saves their product from dying in between? Really?=20
I am saying that there are problems that CANNOT be solved on t=
he
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
disk firmware level. Some problems HAVE to be addressed higher=
up
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
the
=20
stack.
=20Quoted 1 line
Quoted 1 line
Exactly, you can't assume that the SSDs firmware understands any=
and
Quoted 5 lines
Quoted 5 lines
=20
all
=20Quoted 2 lines
Quoted 2 lines
file
system layouts, especially if they are on fragmented LVM or othe=
r
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
logical
volume manager partitions.=20
Hopefully the firmware understands exactly no fs layout at all. T=
hat
Quoted 5 lines
Quoted 5 lines
=20
would
=20Quoted 2 lines
be
braindead. Instead it should understand how to arrange incoming a=
nd
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
outgoing
data in a way that its own technical requirements are met as perf=
ect as
Quoted 1 line
Quoted 1 line
Quoted 1 line
possible. This is no spinning disk, it is completely irrelevant w=
hat
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the data
layout looks like as long as the controller finds its way through=
and
Quoted 4 lines
Quoted 4 lines
=20
copes
=20Quoted 1 line
best with read/write/erase cycles. It may well use additional RAM=
for
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
caching and data reordering.
Do you really believe ascending block numbers are placed in ascen=
ding
Quoted 1 line
Quoted 1 line
Quoted 1 line
addresses inside the disk (as an example)? Why should they? What =
does
Quoted 4 lines
Quoted 4 lines
=20
that
=20Quoted 1 line
mean for fs block ordering? If you don't know anyway what a contr=
oller
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
does to
your data ordering, how do you want to help it with its job? Please accept that we are _not_ talking about trivial flash mem h=
ere or
Quoted 1 line
Quoted 1 line
Quoted 1 line
pseudo-SSDs consisting of sd cards. The market has already evolve=
d
Quoted 4 lines
Quoted 4 lines
=20
better
=20Quoted 1 line
products. The dinosaurs are extincted even if some are still look=
ing
Quoted 2 lines
Quoted 2 lines
=20
alive.
You seem to be forgetting that CEOs like to save 10 cents per drive to = show=20
"millions of dollars saved" by their work, I highly doubt that we won't=
see=20
SSDs with half assed wear leveling implementations 10 years from now.
And no, I don't think that the linear storage that we see at the ATA le= vel is=20
any linear on the drive itself. But erase blocks are still erase blocks= =2E I=20
highly doubt that the abstraction layer works over sector sizes (512B) = and not=20
over whole erase block sizes -- just because it would make it much more= =20
complicated, thus slower.
This way, even if the writes to the flash cells are made in fashion sim= ilar to=20
a LogFS, one will still get r/m/w cycle if the write is 512B in size on=
a=20
block that has also other data.
Quoted 3 lines
Quoted 3 lines
=20
I am assuming that you are being deliberately facetious here (the alternative is less kind). The simple fact is that you cannot come =
up
Quoted 1 line
Quoted 1 line
with some magical data (re)ordering method that nullifies problems =
of
Quoted 3 lines
Quoted 3 lines
common use-cases that are quite nasty for flash based media. =20
For example - you have a disk that has had all it's addressable blo=
cks
Quoted 1 line
Quoted 1 line
tainted. A new write comes in - what do you do with it? Worse, a wr=
ite
Quoted 2 lines
Quoted 2 lines
comes in spanning two erase blocks as a consequence of the data re-alignment in the firmware. You have no choice but to wipe them b=
oth
Quoted 4 lines
Quoted 2 lines
and re-write the data. You'd be better off not doing the magic and assuming that the FS is sensibly aligned.
=20
Ok, how exactly would the FS help here? We have a device with a 256k=
b
Quoted 1 line
erasure size, and userland does a 4k write followed by an fsync.
I assume here that the FS knows about erasure size and does implement T= RIM.
Quoted 2 lines
If the FS were to be smart and know about the 256kb requirement, it would do a read/modify/write cycle somewhere and then write the 4KB.
If all the free blocks have been TRIMmed, FS should pick a completely f= ree=20
erasure size block and write those 4KiB of data.
Correct implementation of wear leveling in the drive should notice that=
the=20
write is entirely inside a free block and make just a write cycle addin= g zeros=20
to the end of supplied data.
Quoted 5 lines
The underlying implementation is the same in the device. It picks a destination, reads it then writes it back. You could argue (and many people do) that this operation is risky and has a good chance of destroying old data. Perhaps we're best off if the FS does the rmw cycle instead into an entirely safe location.
And IMO that's the idea behind TRIM -- not to force the device do do rm= w=20
cycles, only write cycle or erase cycle, provided there's free space an= d the=20
free space doesn't have considerably more write cycles than the alread= y=20
allocated data.
Quoted 2 lines
=20
It's a great place for research and people are definitely looking at =
it.
Quoted 2 lines
=20
But with all of that said, it has nothing to do with alignment or tri=
m.
Quoted 2 lines
Modern ssds are a raid device with a large stripe size, and someone somewhere is going to do a read/modify/write to service any small wri=
te.
Quoted 2 lines
You can force this up to the FS or the application, it'll happen somewhere.
Yes, and if the parition is full rmw will happen in the drive. But if t= he=20
partition is far from full, free space is TRIMmed then than the r/m/w c= ycle=20
will happen inside btrfs and the SSD won't have to do its magic -- maki= ng the=20
process faster.
The effect will be a FS that behaves consistently over a broad range of=
SSDs,=20
provided there's free space left.
Quoted 2 lines
The filesystem metadata writes are a very small percentage of the problem overall. Sure we can do better and try to force larger metad=
ata
Quoted 1 line
blocks. This was the whole point behind btrfs' support for large tre=
e
Quoted 1 line
blocks, which I'll be enabling again shortly.
--=20
Hubert Kario
QBS - Quality Business Software
ul. Ksawer=F3w 30/85
02-656 Warszawa
POLAND
tel. +48 (22) 646-61-51, 646-74-24
fax +48 (22) 646-61-50
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 42 lines · Hubert Kario
Quoted 9 lines
Quoted 6 lines
For example - you have a disk that has had all it's addressable blocks tainted. A new write comes in - what do you do with it? Worse, a write comes in spanning two erase blocks as a consequence of the data re-alignment in the firmware. You have no choice but to wipe them both and re-write the data. You'd be better off not doing the magic and assuming that the FS is sensibly aligned.
Ok, how exactly would the FS help here? We have a device with a 256kb erasure size, and userland does a 4k write followed by an fsync.
I assume here that the FS knows about erasure size and does implement TRIM.
Quoted 2 lines
If the FS were to be smart and know about the 256kb requirement, it would do a read/modify/write cycle somewhere and then write the 4KB.
If all the free blocks have been TRIMmed, FS should pick a completely free erasure size block and write those 4KiB of data.
Correct implementation of wear leveling in the drive should notice that the write is entirely inside a free block and make just a write cycle adding zeros to the end of supplied data.
Quoted 5 lines
The underlying implementation is the same in the device. It picks a destination, reads it then writes it back. You could argue (and many people do) that this operation is risky and has a good chance of destroying old data. Perhaps we're best off if the FS does the rmw cycle instead into an entirely safe location.
And IMO that's the idea behind TRIM -- not to force the device do do rmw cycles, only write cycle or erase cycle, provided there's free space and the free space doesn't have considerably more write cycles than the already allocated data.
Quoted 8 lines
It's a great place for research and people are definitely looking at it.
But with all of that said, it has nothing to do with alignment or trim. Modern ssds are a raid device with a large stripe size, and someone somewhere is going to do a read/modify/write to service any small write. You can force this up to the FS or the application, it'll happen somewhere.
Quoted 4 lines
Yes, and if the parition is full rmw will happen in the drive. But if the partition is far from full, free space is TRIMmed then than the r/m/w cycle will happen inside btrfs and the SSD won't have to do its magic -- making the process faster.
The filesystem cannot do read/modify/write faster or better than the drive. The drive is pushing data around internally and the FS has to pull it in and out of the sata bus. The drive is much faster.
The FS can be safer than the drive because it is able to do more consistency checks on the data as it reads. But this also has a cost because the crcs for the blocks might not be adjacent to the block.
If the FS is the FTL, we don't need trim because the FS already knows which blocks are in use. So, there isn't as much complexity in finding the free erasure block. The FS FTL does win there.
-chris
Quoted 10 lines · On Fri, 12 Mar 2010 02:07:40 +0100 Hubert Kario <[email protected]> wrote:
Quoted 3 lines
[...]
If the FS were to be smart and know about the 256kb requirement, it would do a read/modify/write cycle somewhere and then write the 4KB.If all the free blocks have been TRIMmed, FS should pick a completely free erasure size block and write those 4KiB of data.
Correct implementation of wear leveling in the drive should notice that the write is entirely inside a free block and make just a write cycle adding zeros to the end of supplied data.
Your assumption here is that your _addressed_ block layout is completely identical to the SSDs "disk" layout. Else you cannot know where a "free erasure block" is located and how to address it from FS.
I really wonder what this assumption is based on. You still think a SSD is a true disk with linear addressing. I doubt that very much. Even on true spinning disks your assumption is wrong for relocated sectors. Which basically means that every disk controller firmware fiddles around with the physical layout since decades. Please accept that you cannot do a disks' job in FS. The more advanced technology gets the more disks become black boxes with a defined software interface. Use this interface and drop the idea of having inside knowledge of such a device. That's other peoples' work. If you want to design smart SSD controllers hire at a company that builds those.
Signature · 2 lines
Regards, Stephan
Quoted 5 lines · On Friday 12 March 2010 10:15:28 Stephan von Krawczynski wrote:
Quoted 2 lines · On Fri, 12 Mar 2010 02:07:40 +0100 =20 Hubert Kario <[email protected]> wrote:
Quoted 2 lines
[...]
If the FS were to be smart and know about the 256kb requirement, =
it
Quoted 1 line
Quoted 1 line
Quoted 1 line
would do a read/modify/write cycle somewhere and then write the 4=
KB.
Quoted 2 lines
Quoted 2 lines
=20
If all the free blocks have been TRIMmed, FS should pick a complete=
ly
Quoted 3 lines
Quoted 3 lines
free erasure size block and write those 4KiB of data.
=20
Correct implementation of wear leveling in the drive should notice =
that
Quoted 1 line
Quoted 1 line
the write is entirely inside a free block and make just a write cyc=
le
Quoted 3 lines
Quoted 1 line
adding zeros to the end of supplied data.
=20
Your assumption here is that your _addressed_ block layout is complet=
ely
Quoted 4 lines
identical to the SSDs "disk" layout.
Else you cannot know where a "free
erasure block" is located and how to address it from FS.
I really wonder what this assumption is based on. You still think a S=
SD is
Quoted 1 line
a true disk with linear addressing. I doubt that very much.
I made no such assumptions.
Im sure that the linearity on the ATA LBA level isn't so linear on the = device=20
level, especially after wear-leveling takes its toll, but I assume that=
the=20
smallest block of data that the translation layer can address is erase-= block=20
sized and that all the erase-block are equal in size. Otherwise the alg= orithm=20
would be needlessly complicated which would make it both slower and mor= e error=20
prone.
Quoted 2 lines
Even on true
spinning disks your assumption is wrong for relocated sectors.
Which we don't have to worry about because if the drive has less than 5=
of=20
'em, the impact of hitting them is marginal and if there are more, the = user=20
has much more pressing problem than the performance of the drive or FS.
Quoted 2 lines
Which
basically means that every disk controller firmware fiddles around wi=
th
Quoted 2 lines
the physical layout since decades. Please accept that you cannot do a disks' job in FS. The more advanced technology gets the more disks be=
come
Quoted 1 line
black boxes with a defined software interface. Use this interface and=
drop
Quoted 3 lines
the idea of having inside knowledge of such a device. That's other peoples' work. If you want to design smart SSD controllers hire at a company that builds those.
And I don't think that doing disks' job in the FS is good idea, but I t= hink=20
that we should be able to minimise the impact of the translation layer.
The way to do this, is to threat the device as a block device with sect= ors the=20
size of erase-blocks. That's nothing too fancy, don't you think?
--=20
Hubert Kario
QBS - Quality Business Software
ul. Ksawer=F3w 30/85
02-656 Warszawa
POLAND
tel. +48 (22) 646-61-51, 646-74-24
fax +48 (22) 646-61-50
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · On Thu, 11 Mar 2010 13:00:17 -0500 Chris Mason <[email protected]> wrote:
Quoted 13 lines · Stephan von Krawczynski
Quoted 10 lines · On Thu, 11 Mar 2010 15:39:05 +0100 Sander <[email protected]> wrote:
Stephan von Krawczynski wrote (ao):
Quoted 4 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
My understanding of the ssd mount option is also that the fs doens't try to do all kinds of smart (and potential expensive) things which make sense for rotating media to reduce seeks and the like.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Sander
Such an optimization sounds valid on first sight. But re-think closely: how does the fs really know about seeks needed during some operation?
Well the FS makes a few assumptions (in the nonssd case). First it assumes the storage is not a memory device. If things would fit in memory we wouldn't need filesytems in the first place.
Ok, here is the bad news. This assumption everything from right to completely wrong, and you cannot really tell the mainstream answer.
Two examples from opposite parts of the technology world:
- History: way back in the 80's there was a 3rd party hardware for C=1541 (floppy drive for C=64) that read in the complete floppy and served all incoming requests from the ram buffer. So your assumption can already be wrong for a trivial floppy drive from ancient times.
- Nowadays: being a linux installation today chances are that the matrix has you. Quite a lot of installations are virtualized. So your storage is a virtual one either, which means it is likely being a fs buffer from the host system, i.e. RAM.
And sorry to say: "if things would fit in memory" you probably still need a fs simply because there is no actual way to organize data (be it executable or not) in RAM without a fs layer. You can't save data without an abstract file data type. To have one accessible you need a fs.
Btw the other way round is as interesting: there is currently no fs for linux that knows how to execute in place. Meaning if you really had only RAM and you have a fs to organize your data it would be just logical to have ways to _not_ load data (in other parts of the RAM), but to use it in its original storage (RAM-)space.
Quoted 4 lines
Then it assumes that adjacent blocks are cheap to read and blocks that are far away are expensive to read. Given expensive raid controllers, cache, and everything else, you're correct that sometimes this assumption is wrong.
As already mentioned this assumption may be completely wrong even without a raid controller, being within a virtual environment. Even far away blocks can be one byte away in the next fs buffer of the underlying host fs (assuming your device is in fact a file on the host;-).
Quoted 1 line
But, on average seeking hurts. Really a lot.
Yes, seeking hurts. But there is no way to know if there is seeking at all. On the other hand, if your storage is a netblock device seeking on the server is probably your smallest problem, compared to the network latency in between.
Quoted 4 lines
We try to organize files such that files that are likely to be read together are found together on disk. Btrfs is fairly good at this during file creation and not as good as ext*/xfs as files over overwritten and modified again and again (due to cow).
You are basically saying that btrfs perfectly organizes write-once devices ;-)
Quoted 4 lines
If you turn mount -o ssd on for your drive and do a test, you might not notice much difference right away. ssds tend to be pretty good right out of the box. Over time it tends to help, but it is a very hard thing to benchmark in general.
Honestly, this sounds like "I give up" to me ;-)
You just said that generally it is "very hard to benchmark". Which means "nobody can see or feel it in real world" in non-tech language.
Please understand that I am the last one critizing your and others' brillant work and the time you spend for btrfs. Only I do believe that if you spent one hour on some fs like glusterfs for every 10 hours you spend on btrfs you would be both king and queen for the linux HA community :-)
(but probably unemployed, so I can't really beat you for it)
Quoted 1 line
-chris
Signature · 2 lines
Regards, Stephan
Quoted 6 lines · On Fri, 12 Mar 2010 17:00:08 +0100 Hubert Kario <[email protected]> wrote:
Quoted 2 lines
Even on true
spinning disks your assumption is wrong for relocated sectors.Which we don't have to worry about because if the drive has less than 5 of 'em, the impact of hitting them is marginal and if there are more, the user has much more pressing problem than the performance of the drive or FS.
Are you really sure that a drive firmware tells you about the true number of relocated sectors? I mean if it makes the product look better in comparison to another product, are you really sure that the firmware will not tell you what you expect to see only to make you content and happy with your drive?
Quoted 9 lines
Quoted 8 lines
Which
basically means that every disk controller firmware fiddles around with the physical layout since decades. Please accept that you cannot do a disks' job in FS. The more advanced technology gets the more disks become black boxes with a defined software interface. Use this interface and drop the idea of having inside knowledge of such a device. That's other peoples' work. If you want to design smart SSD controllers hire at a company that builds those.
Quoted 5 lines
And I don't think that doing disks' job in the FS is good idea, but I think that we should be able to minimise the impact of the translation layer.
The way to do this, is to threat the device as a block device with sectors the size of erase-blocks. That's nothing too fancy, don't you think?
I don't believe anyone is able to tell the size of erase-blocks of some device
- current and future - for sure. I do believe that making this guess only reduces the future design options for new devices - if its creators care at all about your guess.
Why not let the fs designer take his creative options in fs layer and let the device designer use his brain on the device level and all meet at the predefined software interface in between - and nowhere _else_.
Quoted 2 lines
Signature · 1 line
Hubert Kario
Signature · 2 lines
Regards, Stephan
Quoted 6 lines · On Saturday 13 March 2010 18:02:10 Stephan von Krawczynski wrote:
Quoted 4 lines · On Fri, 12 Mar 2010 17:00:08 +0100 Hubert Kario <[email protected]> wrote:
Quoted 2 lines
Even on true
spinning disks your assumption is wrong for relocated sectors.=20
Which we don't have to worry about because if the drive has less th=
an 5
Quoted 1 line
Quoted 1 line
of 'em, the impact of hitting them is marginal and if there are mor=
e,
Quoted 4 lines
Quoted 2 lines
the user has much more pressing problem than the performance of the drive or FS.
=20
Are you really sure that a drive firmware tells you about the true nu=
mber
Quoted 2 lines
of relocated sectors? I mean if it makes the product look better in comparison to another product, are you really sure that the firmware =
will
Quoted 1 line
not tell you what you expect to see only to make you content and happ=
y
Quoted 1 line
with your drive?
because Joe Sixpack reads SMART values, and even if he does, he will be=
much=20
more angry when a drive that has no or few relocations fails, that when=
a=20
drive that reports that's failing fails.
If the drive arrives with badsectors, it goes where it came from the sa= me day=20
if it meets an IT guy worth its salt, any IT guy knows that some HDDs d= evelop=20
badsectors no matter the make and model, but if they do, you replace th= em.
And as the Google disk survey showed, the SMART has very high percentag= e of=20
Type I errors, but very few Type II errors.
But we're off-topic here
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Which
basically means that every disk controller firmware fiddles aroun=
d with
Quoted 1 line
Quoted 1 line
Quoted 1 line
the physical layout since decades. Please accept that you cannot =
do a
Quoted 1 line
Quoted 1 line
Quoted 1 line
disks' job in FS. The more advanced technology gets the more disk=
s
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
become black boxes with a defined software interface. Use this interface and drop the idea of having inside knowledge of such a device. That's other peoples' work. If you want to design smart S=
SD
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
controllers hire at a company that builds those.
=20
And I don't think that doing disks' job in the FS is good idea, but=
I
Quoted 1 line
Quoted 1 line
think that we should be able to minimise the impact of the translat=
ion
Quoted 4 lines
Quoted 4 lines
layer.
=20
The way to do this, is to threat the device as a block device with sectors the size of erase-blocks. That's nothing too fancy, don't y=
ou
Quoted 3 lines
Quoted 1 line
think?
=20
I don't believe anyone is able to tell the size of erase-blocks of so=
me
Quoted 1 line
device - current and future - for sure.
Well, if the engeneer that designed it doesn't know this, I don't know = how he=20
got his degree.
Just because it isn't publicised now, doesn't mean it won't be in near = future.
Besides that, to detect how big the erase-blocks are in size is easy, i= f they=20
have any impact on the performance, if they don't have any impact (what= ever=20
the reason) tunning for their size is pointless anyway.=20
Quoted 3 lines
I do believe that making this
guess only reduces the future design options for new devices - if its creators care at all about your guess.
Did I, or any one else, say that we want to hardwire a specific erase-b= lock=20
size to the design of the FS?! That would be utter stupidity!
Quoted 1 line
Why not let the fs designer take his creative options in fs layer and=
let
Quoted 1 line
the device designer use his brain on the device level and all meet at=
the
Quoted 1 line
predefined software interface in between - and nowhere _else_.
We (well, at least Gordon and I) just want a "stripe_width" option adde= d to=20
the mkfs.btrfs, just like it is there for ext2/3/4, reiserfs, xfs and j= fs to=20
name a few. It would need very few additional tweaks to make it SSD fri= endly,=20
hardly any considering how -o ssd or -o ssd_spread already work.
You're forgetting there's an elephant in the room that won't to talk to= =20
devices that don't have sectors 512B in size. If not for it, there woul= dn't=20
even _be_ SSDs with 512B sectors.
It's not the way Flash memory works.
The 512B abstraction is there to be compatible, to work with one curren= t OS,=20
it's not there because it describes better the way Flash memory works o= r is=20
the best way to address the data on the device itself.
There are already consumer HDDs with 4kiB sector size, so the situation=
is =20
getting better. We can only hope that in few years time the SSDs will h= ave=20
sectors the size of erase-blocks. But in the mean time, stripe_width wo= uld be=20
enough.
Besides, the stripe_width option will be not only useful for the SSDs b= ut also=20
in environments where btrfs is on a device that is a RAID5/6 array=20 (reconfiguring a server with many virtual machines is far from easy and= =20
sometimes just can't be done because of heterogeneous virtualised OSs t= hat=20
need the data protection provided by lower layers).
--=20
Hubert Kario
QBS - Quality Business Software
ul. Ksawer=F3w 30/85
02-656 Warszawa
POLAND
tel. +48 (22) 646-61-51, 646-74-24
fax +48 (22) 646-61-50
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · On Saturday 13 March 2010 17:43:59 Stephan von Krawczynski wrote:
Quoted 1 line · On Thu, 11 Mar 2010 13:00:17 -0500 Chris Mason <[email protected]> wrote:
On Thu, Mar 11, 2010 at 06:35:06PM +0100, Stephan von Krawczynski w=
rote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines · On Thu, 11 Mar 2010 15:39:05 +0100 Sander <[email protected]> wrote:
Stephan von Krawczynski wrote (ao):
Quoted 1 line
Honestly I would just drop the idea of an SSD option simply b=
ecause
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option d=
oes
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
something constructive and not destructive in combination wit=
h a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
SSD controller.
=20
My understanding of the ssd mount option is also that the fs do=
ens't
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
try to do all kinds of smart (and potential expensive) things w=
hich
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
make sense for rotating media to reduce seeks and the like. =20
Sander=20
Such an optimization sounds valid on first sight. But re-think cl=
osely:
Quoted 1 line
Quoted 1 line
Quoted 1 line
how does the fs really know about seeks needed during some operat=
ion?
Quoted 9 lines
Quoted 4 lines
=20
Well the FS makes a few assumptions (in the nonssd case). First it assumes the storage is not a memory device. If things would fit in memory we wouldn't need filesytems in the first place.=20
Ok, here is the bad news. This assumption everything from right to completely wrong, and you cannot really tell the mainstream answer. Two examples from opposite parts of the technology world:
- History: way back in the 80's there was a 3rd party hardware for C=3D=
1541
Quoted 1 line
(floppy drive for C=3D64) that read in the complete floppy and served=
all
Quoted 1 line
incoming requests from the ram buffer. So your assumption can already=
be
Quoted 1 line
wrong for a trivial floppy drive from ancient times.
such assumption doesn't make it work slower on such device
Quoted 1 line
- Nowadays: being a linux installation today chances are that the mat=
rix
Quoted 1 line
has you. Quite a lot of installations are virtualized. So your storag=
e is
Quoted 1 line
a virtual one either, which means it is likely being a fs buffer from=
the
Quoted 1 line
host system, i.e. RAM.
Buffers use read_ahead and are smaller than the underlaying device, sti= ll, such=20
assumption doesn't make the FS perform worse in this situation.=20
Quoted 1 line
And sorry to say: "if things would fit in memory" you probably still =
need a
Quoted 2 lines
fs simply because there is no actual way to organize data (be it executable or not) in RAM without a fs layer. You can't save data wit=
hout
Quoted 1 line
an abstract file data type. To have one accessible you need a fs.
yes, that's why there is tmpfs, btrfs isn't meant to be all and end all=
as far=20
as FSs go
Quoted 1 line
Btw the other way round is as interesting: there is currently no fs f=
or
Quoted 1 line
linux that knows how to execute in place. Meaning if you really had o=
nly
Quoted 1 line
RAM and you have a fs to organize your data it would be just logical =
to
Quoted 1 line
have ways to _not_ load data (in other parts of the RAM), but to use =
it in
Quoted 1 line
its original storage (RAM-)space.
at least ext2 does support XIP on platform that support it...
Quoted 2 lines
=20
Quoted 1 line
Then it assumes that adjacent blocks are cheap to read and blocks t=
hat
Quoted 1 line
Quoted 1 line
are far away are expensive to read. Given expensive raid controlle=
rs,
Quoted 4 lines
Quoted 2 lines
cache, and everything else, you're correct that sometimes this assumption is wrong.
=20
As already mentioned this assumption may be completely wrong even wit=
hout a
Quoted 1 line
raid controller, being within a virtual environment. Even far away bl=
ocks
Quoted 2 lines
can be one byte away in the next fs buffer of the underlying host fs (assuming your device is in fact a file on the host;-).
and again, such assumption doesn't reduce the performance
Quoted 4 lines
=20
Quoted 1 line
But, on average seeking hurts. Really a lot.
=20
Yes, seeking hurts. But there is no way to know if there is seeking a=
t all.
Quoted 1 line
On the other hand, if your storage is a netblock device seeking on th=
e
Quoted 1 line
server is probably your smallest problem, compared to the network lat=
ency
Quoted 1 line
in between.
and because of that, there's read ahead and support for big packets on = the TCP=20
level, so the assumption does make the FS perform better with it than w= ithout=20
it.
It's one of the assumptions that you _have_ to make, just like the assu= mption=20
that the computer counts in binary, or there's more disk space than RAM= =2E But=20
those assumptions _don't_ make the performance (much) worse when they d= on't=20
hold true for known devices that can impersonate rotating magnetic medi= a.
Quoted 6 lines
Quoted 4 lines
We try to organize files such that files that are likely to be read together are found together on disk. Btrfs is fairly good at this during file creation and not as good as ext*/xfs as files over overwritten and modified again and again (due to cow).
=20
You are basically saying that btrfs perfectly organizes write-once de=
vices
Quoted 3 lines
;-)
=20Quoted 1 line
If you turn mount -o ssd on for your drive and do a test, you might=
not
Quoted 1 line
Quoted 1 line
notice much difference right away. ssds tend to be pretty good rig=
ht
Quoted 1 line
Quoted 1 line
out of the box. Over time it tends to help, but it is a very hard =
thing
Quoted 4 lines
Quoted 1 line
to benchmark in general.
=20
Honestly, this sounds like "I give up" to me ;-)
You just said that generally it is "very hard to benchmark". Which me=
ans
Quoted 1 line
"nobody can see or feel it in real world" in non-tech language.
No, it's not this. When a SSD is fresh, the undeling write leveling has=
many=20
blocks to choose from, so it's blaizing fast. The same holds true when = the=20
test uses small amount of data (relative to SSD size).
"very hard to benchmark" means just that -- the benchmark is much more=20 complicated, must take into account much more variables and takes much = more=20
time compared to rotating magnetic media benchmark.
To test SSD performance you need to benchmark both the speed of flash m= emory=20
_and_ the speed and performance of the write leveling algorithm (becaus= e it=20
shows its ugly head only after specific workloads or when all blocks ar= e=20
allocated), and that's non trivial to say the least. Add FS on top of i= t and=20
you have a nice dissertation right there.
--=20
Hubert Kario
QBS - Quality Business Software
ul. Ksawer=F3w 30/85
02-656 Warszawa
POLAND
tel. +48 (22) 646-61-51, 646-74-24
fax +48 (22) 646-61-50
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 26 lines · Stephan von Krawczynski
Quoted 14 lines · On Thu, 11 Mar 2010 13:00:17 -0500 Chris Mason <[email protected]> wrote:
Quoted 13 lines · Stephan von Krawczynski
Quoted 10 lines · On Thu, 11 Mar 2010 15:39:05 +0100 Sander <[email protected]> wrote:
Stephan von Krawczynski wrote (ao):
Quoted 4 lines
Honestly I would just drop the idea of an SSD option simply because the vendors implement all kinds of neat strategies in their devices. So in the end you cannot really tell if the option does something constructive and not destructive in combination with a SSD controller.
My understanding of the ssd mount option is also that the fs doens't try to do all kinds of smart (and potential expensive) things which make sense for rotating media to reduce seeks and the like.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Sander
Such an optimization sounds valid on first sight. But re-think closely: how does the fs really know about seeks needed during some operation?
Well the FS makes a few assumptions (in the nonssd case). First it assumes the storage is not a memory device. If things would fit in memory we wouldn't need filesytems in the first place.
Quoted 7 lines
Ok, here is the bad news. This assumption everything from right to completely wrong, and you cannot really tell the mainstream answer.
Two examples from opposite parts of the technology world:
- History: way back in the 80's there was a 3rd party hardware for C=1541 (floppy drive for C=64) that read in the complete floppy and served all incoming requests from the ram buffer. So your assumption can already be wrong for a trivial floppy drive from ancient times.
Agreed, I'll try my best not to tune btrfs for trivial floppies from ancient times ;)
Quoted 9 lines
Quoted 4 lines
Then it assumes that adjacent blocks are cheap to read and blocks that are far away are expensive to read. Given expensive raid controllers, cache, and everything else, you're correct that sometimes this assumption is wrong.
As already mentioned this assumption may be completely wrong even without a raid controller, being within a virtual environment. Even far away blocks can be one byte away in the next fs buffer of the underlying host fs (assuming your device is in fact a file on the host;-).
Ok, there are roughly three environments at play here.
1) Seeking hurts, and you have no idea if adjacent block numbers are close together on the device.
2) Seeking doesn't hurt and you have no idea if adjacent block numbers are close together on the device. (SSD).
3) Seeking hurts and you can assume adjacent block numbers are close together on the device (disks).
Type one is impossible to tune, and so it isn't interesting in this discussion. There are an infinite number of ways to actually store data you care about, and just because one of those ways can't be tuned doesn't mean we should stop trying to tune for the ones that most people actually use.
Quoted 7 lines
Quoted 2 lines
But, on average seeking hurts. Really a lot.
Yes, seeking hurts. But there is no way to know if there is seeking at all. On the other hand, if your storage is a netblock device seeking on the server is probably your smallest problem, compared to the network latency in between.
Very true, and if I were using such a setup in performance critical applications, I would:
1) Tune the network so that seeks mattered again
2) Tune the seeks.
Quoted 6 lines
Quoted 4 lines
We try to organize files such that files that are likely to be read together are found together on disk. Btrfs is fairly good at this during file creation and not as good as ext*/xfs as files over overwritten and modified again and again (due to cow).
You are basically saying that btrfs perfectly organizes write-once devices ;-)
Storage is all about trade offs, and optimizing read access for write once vs write many is a very different thing. It's surprising how many of your files are written once and never read, let alone written and then never changed.
Quoted 9 lines
Quoted 5 lines
If you turn mount -o ssd on for your drive and do a test, you might not notice much difference right away. ssds tend to be pretty good right out of the box. Over time it tends to help, but it is a very hard thing to benchmark in general.
Honestly, this sounds like "I give up" to me ;-)
You just said that generally it is "very hard to benchmark". Which means "nobody can see or feel it in real world" in non-tech language.
No, it just means it is hard to benchmark. SSDs, even really good
ssds, are not deterministic. Sometimes they are faster than others and the history of how you've abused it in the past factors into how well it performs in the future.
A simple graph that talks about the performance of one drive in one workload needs a lot of explanation.
Quoted 6 lines
Please understand that I am the last one critizing your and others' brillant work and the time you spend for btrfs. Only I do believe that if you spent one hour on some fs like glusterfs for every 10 hours you spend on btrfs you would be both king and queen for the linux HA community :-)
(but probably unemployed, so I can't really beat you for it)
Grin, the list of things I wish I had time to work on is quite long.
-chris
Quoted 5 lines · Stephan von Krawczynski
- Nowadays: being a linux installation today chances are that the matrix has you. Quite a lot of installations are virtualized. So your storage is a virtual one either, which means it is likely being a fs buffer from the host system, i.e. RAM.
That would be a strictly amateur-hour implementation. It is very important for data integrity that at least all writes are synchronous, and ideally all IO should be uncached in the host. In that case the performance of the guest's virtual IO device will be broadly similar to a real hardware device.
J