Skip to contentSkip to conversation

UBIFS vs Logfs (was [RFC PATCH] UBIFS - new flash file system)

DISCUSSION Tomasz Chmielewski 7 participants 18 messages Apr 1, 2008 – Apr 10, 2008

lore
Share & tools
lore
#0 lore raw
Quoted 7 lines · Artem Bityutskiy

I've renamed the thread because I do not like this flamish discussion to me mixed with the technical one.

Quoted 2 lines · Jörn Engel

Shiny numbers! Performance has improved significantly in the last six month. Still worth a closer look.

We'll re-run them. Does logfs support write-back? Does it support compression?

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

1. It works on normal block devices and it supports transparent compression

Today, a 64 GB SSD/flash-based media costs ~about the same as a 1 TB hard disk. This makes flash very expensive to use; compression can compensate that cost a bit (will depend on the usage, of course).

I believe there is no other Linux filesystem which can do transparent compression on block devices.

2. It does wear-levelling also on normal block devices

Although it doesn't sound normal to do wear-levelling twice (most flash-based block devices do wear-levelling on their own), I had a flash corruption after just ~one month of using RAID bitmap on a IDE-flash disk formatted with ext3. Apparently, device-level wear-levelling wasn't spreading updates of RAID bitmap file well enough.

(...)

Quoted 5 lines

This basically means it is unfinished. Handling dynamic bad blocks is a *must* if you are going to work on NAND, especially on MLC NAND which are not as reliable as SLC.
I think you should bluntly say about this when you submit patches to prevent people from starting using it in production.

I too wouldn't use LogFS today in a production environment - it is still not feature complete and not widely tested. I wouldn't use btrfs or ext4 today for the very same reason.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#1 lore raw
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file systems for block devices. UBIFS may work on top of a block device as well (just needs few hacks to make it possible) - it is not a problem at all, it is just _senseless_.

JFFS2/UBIFS/LogFS is a separate _class_ of file-systems. The are designed for _flash_, which has completely different work model then block device. They are _native_ flash file systems.
Here are more details: http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd

The traditional FSes _cannot_ work on top of flash. The solution for this is using FTL, which emulates a block device on top of flash. It _hides_ the real device, and fakes a block device for you. And you can use traditional FSes on top of that fake block device.

The whole _point_ of this separate class of FSes is because we believe we may do much _better_ job if we use flash _natively_, instead of using FTL. FTL is the place where you loose performance, reliability, and so on.

And you are saying about using a native flash FS on top of a block device like an SD card. This is just not sane: SD card first emulates a block device for you, looses performance at this point, then you again emulate a flash on top of this, and suffer from this again.

Signature · 2 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
#2 lore raw
Quoted 24 lines · Artem Bityutskiy
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file systems for block devices. UBIFS may work on top of a block device as well (just needs few hacks to make it possible) - it is not a problem at all, it is just _senseless_.

JFFS2/UBIFS/LogFS is a separate _class_ of file-systems. The are designed for _flash_, which has completely different work model then block device. They are _native_ flash file systems.
Here are more details:
http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd

The traditional FSes _cannot_ work on top of flash. The solution for this is using FTL, which emulates a block device on top of flash. It _hides_ the real device, and fakes a block device for you. And you can use traditional FSes on top of that fake block device.

The whole _point_ of this separate class of FSes is because we believe we may do much _better_ job if we use flash _natively_, instead of using FTL. FTL is the place where you loose performance, reliability, and so on.

Quoted 5 lines

And you are saying about using a native flash FS on top of a block device like an SD card. This is just not sane: SD card first emulates a block device
for you, looses performance at this point, then you again emulate a flash on top of this, and suffer from this again.

About a year ago I would have used roughly the same argument. But in the meanwhile I have accepted the fact that any piece of flash I'll use in a notebook in the next five years will likely use SATA or USB as a transport medium.

Jörn

Signature · 2 lines
...one more straw can't possibly matter...
-- Kirby Bakken
#3 lore raw
Quoted 4 lines · Jörn Engel

About a year ago I would have used roughly the same argument. But in the meanwhile I have accepted the fact that any piece of flash I'll use in a notebook in the next five years will likely use SATA or USB as a transport medium.

Well, Thomasz was talking not about the transport I believe. If you have a raw flash access via SATA or USB, then this is handled on the driver level and does not any require FS support. JFFS2 or UBIFS may also work with such a device, because it is just raw flash. But I know that you know that most of the FTL-enabled stuff does not give use raw flash access.

Signature · 2 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
#4 lore raw
Quoted 7 lines · Artem Bityutskiy
Quoted 4 lines · Jörn Engel

About a year ago I would have used roughly the same argument. But in the meanwhile I have accepted the fact that any piece of flash I'll use in a notebook in the next five years will likely use SATA or USB as a transport medium.

Well, Thomasz was talking not about the transport I believe.

When the day comes and I can actually use raw flash with a SATA or USB transport medium, I will be more careful and not imply that they all expose a block device interface. Today they do.

[ With the exception of Alauda, which has a userbase of about 1 - me. ]

Jörn

Signature · 2 lines
It does not matter how slowly you go, so long as you do not stop.
-- Confucius
#5 lore raw
In reply to Artem Bityutskiy
Quoted 6 lines · Artem Bityutskiy
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file systems for block devices.

Such as?
Flash (also on block devices) is slow and expensive (when compared to modern hard disks) and therefore compression is *very* useful here.
Not only it can potentially save you money; reads and writes will be smaller/faster (unless you're editing music and videos, where one won't use flash anyway), flash will wear out slower etc.

Is there a filesystem for Linux which can provide transparent compression and works for block devices (other than user-space NTFS or some outdated ext2 patches)? I don't think so.

Quoted 3 lines

UBIFS may work on top of a block device as
well (just needs few hacks to make it possible) - it is not a problem at all, it is just _senseless_.

Do you mean using hacks like block2mtd? It's hacky, and pretty hard to boot a system this way (need to build own initramfs, with a static block2mtd or loads of libraries - not something an average user would like to do; no distro supports it; updating a kernel would be a pain etc.).

Quoted 4 lines

JFFS2/UBIFS/LogFS is a separate _class_ of file-systems. The are designed for _flash_, which has completely different work model then block device. They are _native_ flash file systems.
Here are more details: http://www.linux-mtd.infradead.org/faq/general.html#L_mtd_vs_hdd

It's a good comparison, but it doesn't show disadvantages of using traditional filesystems on flash-based block devices.

I just mentioned the reasons why a filesystem like LogFS would be useful on block devices: there are valid reasons to do it.

(...)

Quoted 3 lines

The whole _point_ of this separate class of FSes is because we believe we may do much _better_ job if we use flash _natively_, instead of using FTL. FTL is the place where you loose performance, reliability, and so on.

True.
Unfortunately, there is no way to access flash directly on flash-based block devices (USB-sticks, IDE-flash disks, SSD disks etc.).

Therefore, could an answer be: use a traditional filesystem?

Unfortunately, traditional filesystems were rather designed for rotating media / cheap disks (no transparent compression; tend to accumulate writes in one area of the disk - more on that - below).

Quoted 4 lines

And you are saying about using a native flash FS on top of a block device like an SD card. This is just not sane: SD card first emulates a block device for you, looses performance at this point, then you again emulate a flash on top of this, and suffer from this again.

Performance is only one factor in the equation. Other factors are: cost and reliability.

I speak from experience: flash-based block devices tend to have poor wear-levelling (at least Transcend IDE-flash disks). To reproduce:
- format a 2 GB Transcend IDE-flash disk with ext3
- write a small file (50-100 kB)
- update that file ~several hundred thousand times - as you finish, IDE-flash disk will have 200-300 badblocks

If wear-levelling on the underlying IDE-flash device was any decent, writes would be spread on the whole ~2GB surface, totalling in many more successful writes.

Again: this is my experience, although it may contradict the theory underlined in the link you gave earlier. You have much more experience with flash file systems: correct me where I'm wrong.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#6 lore raw
Quoted 1 line · Tomasz Chmielewski

Such as?

ext3 for example.

Quoted 1 line

Flash (also on block devices) is slow and expensive (when compared to modern hard disks) and therefore compression is *very* useful here.

Well, if you are ready to trade performance to compression, then well, go ahead :-) May be I used too strong wording, but I wanted to say then use raw flash then. But I'd also consider implementing compression support for a block based FS. Reiser4 claimed to have it for example.

Quoted 1 line

Do you mean using hacks like block2mtd? It's hacky, and pretty hard to boot a system this way (need to build own initramfs, with a static block2mtd or loads of libraries - not something an average user would like to do; no distro supports it; updating a kernel would be a pain etc.).

Well, ok, it still sounds strange for me, but you may use JFFS2 and UBIFS with block2mtd as well if you really want to.

Quoted 2 lines

True.
Unfortunately, there is no way to access flash directly on flash-based block devices (USB-sticks, IDE-flash disks, SSD disks etc.).

Yeah, that's a pity :-(

Quoted 1 line

Unfortunately, traditional filesystems were rather designed for rotating media / cheap disks (no transparent compression; tend to accumulate writes in one area of the disk - more on that - below).

Sure.

Quoted 8 lines

Performance is only one factor in the equation. Other factors are: cost and reliability.

I speak from experience: flash-based block devices tend to have poor wear-levelling (at least Transcend IDE-flash disks). To reproduce:
- format a 2 GB Transcend IDE-flash disk with ext3
- write a small file (50-100 kB)
- update that file ~several hundred thousand times - as you finish,
IDE-flash disk will have 200-300 badblocks

Yeah, that's bad. But if you have a bad FTL, surely there is not guarantee a flash FS will help? Isn't it better to use better hardware?

We did some experiments with MMC cards and we were unable to wear them out with re-writing the same sectors again and again. This suggests there _is_ better FTL hardware then that USB stick you was using.

Anyway, your original mail said Logfs can work with block devices. My answer - UBIFS too, but this is very strange to do this IMO. But OK, it might is not senseless, sorry for the wording. :-)

Signature · 2 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
#9 lore raw
In reply to Artem Bityutskiy
Quoted 6 lines · Artem Bityutskiy
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file systems for block devices.

I think he refers to flash disks appearing as block devices, like usb sticks or similar.

-Andi

#10 lore raw
Quoted 10 lines · Andi Kleen
Quoted 5 lines · Artem Bityutskiy
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file systems for block devices.

I think he refers to flash disks appearing as block devices, like usb sticks or similar.

Right, I also meant that in my opinion it makes more sense to use traditional file-systems like ext3 on USB-key/MMC and the like stuff (which I confusingly referred as "block devices"), or may be something more "heavy-weight" like XFS or JFS (never tried them, though).

Signature · 2 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
#11 lore raw
Quoted 9 lines · Artem Bityutskiy
Quoted 8 lines · Andi Kleen
Quoted 5 lines · Artem Bityutskiy
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file systems for block devices.

Quoted 8 lines
Quoted 2 lines

I think he refers to flash disks appearing as block devices, like usb sticks or similar.

Right, I also meant that in my opinion it makes more sense to use traditional file-systems like ext3 on USB-key/MMC and the like stuff (which I confusingly referred as "block devices"), or may be something more "heavy-weight" like XFS or JFS (never tried them, though).

Well, even auto-levelling storage should benefit from a filesystem which minimizes the total number of flash sectors churned, which means doing as few writes as possible and to large, contiguous sections.

-hpa

#12 lore raw
Quoted 22 lines · H. Peter Anvin
Quoted 13 lines · Artem Bityutskiy
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Artem Bityutskiy
Quoted 2 lines · Tomasz Chmielewski

For me, the motivators to wait for LogFS are mainly the facts that it can work on traditional block devices, and not only on pure flash:

Sorry Thomasz, for me this makes zero sense. There are _much_ better file
systems for block devices.

I think he refers to flash disks appearing as block devices, like usb sticks or similar.

Quoted 7 lines

Right, I also meant that in my opinion it makes more sense to use traditional
file-systems like ext3 on USB-key/MMC and the like stuff (which I confusingly
referred as "block devices"), or may be something more "heavy-weight" like XFS or JFS (never tried them, though).

Quoted 3 lines

Well, even auto-levelling storage should benefit from a filesystem which minimizes the total number of flash sectors churned, which means doing as few writes as possible and to large, contiguous sections.

Exactly. At exosec, we ship one appliance which writes statistics to a partition on a compactflash every 5 minutes. We preferred to go with JFFS2 exactly because of this reason. We never had any problem proceeding this way. I'm not sure if it would have been the same with ext2 though.

Willy

#13 lore raw
Quoted 7 lines · Willy Tarreau
Quoted 1 line

Well, even auto-levelling storage should benefit from a filesystem which minimizes the total number of flash sectors churned, which means doing as few writes as possible and to large, contiguous sections.

Exactly. At exosec, we ship one appliance which writes statistics to a partition on a compactflash every 5 minutes. We preferred to go with JFFS2 exactly because of this reason. We never had any problem proceeding this way. I'm not sure if it would have been the same with ext2 though.

Yes, as I agreed in a previous mail this may make sense in some cases.

But in general it is not a good approach. Basically, it is wastage of resources. Indeed, first the firmware on MMC/SD/etc makes efforts to make flash look like a block device. It gives you in-place updates, but by cost of performance and reliability. Then you just drop this nice property, and use JFFS2, which assumes it has only out-of-place updates. But if this solves the task you have
- fine!

Signature · 2 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
#14 lore raw
Quoted 12 lines · Artem Bityutskiy
Quoted 10 lines · Willy Tarreau
Quoted 3 lines

Well, even auto-levelling storage should benefit from a filesystem which minimizes the total number of flash sectors churned, which means doing as few writes as possible and to large, contiguous sections.

Exactly. At exosec, we ship one appliance which writes statistics to a partition on a compactflash every 5 minutes. We preferred to go with JFFS2 exactly because of this reason. We never had any problem proceeding this way.
I'm not sure if it would have been the same with ext2 though.

Quoted 6 lines

Yes, as I agreed in a previous mail this may make sense in some cases.

But in general it is not a good approach. Basically, it is wastage of resources.
Indeed, first the firmware on MMC/SD/etc makes efforts to make flash look like a block device. It gives you in-place updates,

One of the problem is that unless you crash-test your flash cards, you will never know if their wear-leveling algorithm is fine or not. And I suspect that nowadays, due to very large consumer demand, flash cards price drop at the cost of reliability. I think that most of those not flagged "industrial-grade" do absolutely zero wear-leveling, because they are sold to people using them in digital cameras, and they will never kill their device with such a usage.

Quoted 4 lines

but by cost of performance
and reliability. Then you just drop this nice property, and use JFFS2, which assumes it has only out-of-place updates. But if this solves the task you have

I'm certainly not the only one with this requirement. A lot of embedded motherboards come with IDE compactflash connectors. This is very convenient, but if you need to keep informations between reboots, you have to write to the device anyway. If you need to do that very often, either you pray for the device to be very reliable, or you take all the chances on your side by adding your own wear-leveling "just in case".

Cheers,
Willy

#15 lore raw
Quoted 7 lines · Willy Tarreau

One of the problem is that unless you crash-test your flash cards, you will never know if their wear-leveling algorithm is fine or not. And I suspect that nowadays, due to very large consumer demand, flash cards price drop at the cost of reliability. I think that most of those not flagged "industrial-grade" do absolutely zero wear-leveling, because they are sold to people using them in digital cameras, and they will never kill their device with such a usage.

Sure, I know about this problem. My point was that in this case it is wiser to use bare flash and put JFFS2 on it, instead of using this black box MMC/etc and then put JFFS2 on it.

Quoted 6 lines

I'm certainly not the only one with this requirement. A lot of embedded motherboards come with IDE compactflash connectors. This is very convenient, but if you need to keep informations between reboots, you have to write to the device anyway. If you need to do that very often, either you pray for the device to be very reliable, or you take all the chances on your side by adding your own wear-leveling "just in case".

OK. Fair enough. Although stuff exists, but this does not necessarily mean this a good design :-)

Signature · 2 lines
Best Regards,
Artem Bityutskiy (Артём Битюцкий)
#7 lore raw
In reply to Artem Bityutskiy
Quoted 1 line · Artem Bityutskiy

Tomasz Chmielewski wrote:

(...)

Quoted 11 lines
Quoted 8 lines

Performance is only one factor in the equation. Other factors are: cost and reliability.

I speak from experience: flash-based block devices tend to have poor wear-levelling (at least Transcend IDE-flash disks). To reproduce:
- format a 2 GB Transcend IDE-flash disk with ext3
- write a small file (50-100 kB)
- update that file ~several hundred thousand times - as you finish,
IDE-flash disk will have 200-300 badblocks

Yeah, that's bad. But if you have a bad FTL, surely there is not guarantee a flash FS will help? Isn't it better to use better hardware?

Quoted 7 lines

We did some experiments with MMC cards and we were unable to wear them out with re-writing the same sectors again and again. This suggests there _is_ better FTL hardware then that USB stick you was using.

Anyway, your original mail said Logfs can work with block devices. My answer - UBIFS too, but this is very strange to do this IMO. But OK, it might is not senseless, sorry for the wording. :-)

I was thinking why my IDE-flash disk died so soon[1] and how efficient can an internal wear-levelling be in devices which hide its "flashiness" (USB-sticks, IDE-flash disks etc.).

Internal wear-levelling mechanism doesn't have a clue about free space on the filesystem - it that case, how can it do any efficient wear-levelling?

[1] Well, it didn't die, really. Once I removed the file which was showing I/O errors and did "dd if=/dev/zero of=bigfile", there are no badblocks anymore - probably remapped.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#8 lore raw
Quoted 4 lines · Tomasz Chmielewski

[1] Well, it didn't die, really. Once I removed the file which was showing I/O errors and did "dd if=/dev/zero of=bigfile", there are no badblocks anymore - probably remapped.

Could have been soft errors. Heaven knows how those devices report uncorrectable errors, but claiming the block to be bad is a good guess. Once blocks with uncorrectable errors are rewritten (the physical blocks, not what the device reports), they are perfectly usable again.

Jörn

Signature · 3 lines
They laughed at Galileo.  They laughed at Copernicus.  They laughed at
Columbus. But remember, they also laughed at Bozo the Clown.
-- unknown
#17 lore raw
In reply to Pavel Machek
Quoted 4 lines · Pavel Machek

I'd like compressed filesystem for maps and lingvistic data... but will the flash flesystems have 'reasonable' performance when used on harddrive?

If someone implemented readpages(), that might be possible - depending on your definition of 'reasonable' and your workload.

Writes will hit the disk roughly in order. When readahead requests a bunch of pages, there is some chance of them being adjacent and having the block layer combine most of the bios into a few large ones.

The main drawbacks are:
- No reservations. If data is written in random order or several writers chew away in parallel, write order will be fairly pessimal.
- No readahead yet. Wouldn't be hard to do.
- Garbage collection completely ignores fragmentation. If segments are needed and one contains a nice long extend from a single file, that data will be written elsewhere, often split between two segments. Rinse, repeat and fragmentation will increase over time.
- File creation/deletion currently will cause disk heads to jump in triangles. This hurts write performance on most flash media as well, so it will get changed reasonably soon.

All of those are solvable. Some will definitely be solved because the help performance on flash media as well. Other may or may not.

Jörn

Signature · 3 lines
When people work hard for you for a pat on the back, you've got
to give them that pat.
-- Robert Heinlein
#16 lore raw
In reply to Tomasz Chmielewski

Hi!

Quoted 15 lines

For me, the motivators to wait for LogFS are mainly the facts that it
can work on traditional block devices, and not only on pure flash:

1. It works on normal block devices and it supports transparent compression

Today, a 64 GB SSD/flash-based media costs ~about the same as a 1 TB
hard disk. This makes flash very expensive to use; compression can
compensate that cost a bit (will depend on the usage, of course).

Quoted 3 lines

I believe there is no other Linux filesystem which can do transparent
compression on block devices.

I'd like compressed filesystem for maps and lingvistic data... but will the flash flesystems have 'reasonable' performance when used on harddrive?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html