Some very basic questions

reading the list for a while it looks like all kinds of implementational topics are covered but no basic user requests or talks are going on. Since I have found no other list on vger covering these issues I choose this one, forgive my ignorance if it is the wrong place.
Like many people on the planet we try to handle quite some amounts of data (TBs) and try to solve this with several linux-based fileservers. Years of (mostly bad) experience led us to the following minimum requirements for a new fs on our servers:

Quoted 8 lines · Stephan von Krawczynski

reading the list for a while it looks like all kinds of implementational topics are covered but no basic user requests or talks are going on. Since I have found no other list on vger covering these issues I choose this one, forgive my ignorance if it is the wrong place.
Like many people on the planet we try to handle quite some amounts of data (TBs) and try to solve this with several linux-based fileservers. Years of (mostly bad) experience led us to the following minimum requirements for a new fs on our servers:

If that are the minimum requirements, what are the maximum ones?

Also you realize that some of the requirements (like parallel read/write aka a full cluster file system) are extremly hard?

Perhaps it would make more sense if you extracted the top 10 items and ranked them by importance and posted again.

thanks for your feedback. Understand "minimum requirement" as "minimum requirement to drop the current installation and migrate the data to a new fs platform".

Of course you are right, dealing with multiple/parallel mounts can be quite a nasty job if the fs was not originally planned with this feature in mind. On the other hand I cannot really imagine how to deal with TBs of data in the future without such a feature.
If you look at the big picture the things I mentioned allow you to have redundant front-ends for the fileservice doing the same or completely different applications. You can use one mount (host) for tape backup purposes only without heavy loss in standard file service. You can even mount for filesystem check purposes, a box that does nothing else but check the structure and keep you informed what is really going on with your data - and your data is still in production in the meantime.
Whatever happens you have a real chance of keeping your file service up, even if parts of your fs go nuts because some underlying hd got partially damaged. Keeping it up and running is the most important part, performance is only second on the list.
If you take a close look there are not really 10 different items on my list, depending on the level of abstraction you prefer, nevertheless:

1) parallel mounts

2) mounting must not delay the system startup significantly
3) errors in parts of the fs are no reason for a fs to go offline as a whole
4) power loss at any time must not corrupt the fs
5) fsck on a mounted fs, interactively, not part of the mount (all fsck features)

Hearing what user's think they want is always good, but...

Quoted 4 lines · Stephan von Krawczynski

thanks for your feedback. Understand "minimum requirement" as "minimum requirement to drop the current installation and migrate the data to a new fs platform".

I would sure like to know what existing platform and filesystem you have that you think has all 10 of your features.

Quoted 2 lines

[...]
1) parallel mounts

What I see from that explanation is you have a "system design" idea using parallel machines to fix problems you have had in the past. To implement your design, you need a filesystem to fit it.

I think
it is better to just design a filesystem without the problems and configure the hardware to handle the necessary load.

Quoted 5 lines

2) mounting must not delay the system startup significantly
3) errors in parts of the fs are no reason for a fs to go offline as a whole
4) power loss at any time must not corrupt the fs
5) fsck on a mounted fs, interactively, not part of the mount (all fsck features)

I think all of these are part of the "reliability" goal for btrfs and when you say "fsck" it is probably misleading if I understand your real requirement to be the same as my customers:

- *NO* fsck
- filesystem design "prevents problems we have had before"
- filesystem autodetects, isolates, and (possibly) repairs errors
- online "scan, check, repair filesystem" tool initiated by admin
- Reliability so high that they never run that check-and-fix tool

Note that I personally have never seen a first release meet the "no problems, no need to fix" criteria that would obviate any need for a check/fix tool.

jim

Quoted 5 lines

- filesystem autodetects, isolates, and (possibly) repairs errors
- online "scan, check, repair filesystem" tool initiated by admin
- Reliability so high that they never run that check-and-fix tool

That is _wrong_ (to a certain extent). You _want to run_ diagnostic tools to make sure that there is no problem. And you don't want some software (not even HAL) to repair errors without prior admin knowledge/permission

Quoted 6 lines · Stephan von Krawczynski

Quoted 5 lines

- filesystem autodetects, isolates, and (possibly) repairs errors
- online "scan, check, repair filesystem" tool initiated by admin
- Reliability so high that they never run that check-and-fix tool

Quoted 3 lines

That is _wrong_ (to a certain extent). You _want to run_ diagnostic tools to make sure that there is no problem. And you don't want some software (not even HAL) to repair errors without prior admin knowledge/permission

I think there's a place for a scrubber to continuously verify filesystem data and metadata, at low io priority, and correct any correctable errors. The admin can read the error correction report at their leisure, and then take any action that's outside the filesystem's capabilities (like ordering and installing new disks).

Quoted 13 lines · Avi Kivity

Quoted 11 lines · Stephan von Krawczynski

Quoted 5 lines

- filesystem autodetects, isolates, and (possibly) repairs errors
- online "scan, check, repair filesystem" tool initiated by admin
- Reliability so high that they never run that check-and-fix tool

That is _wrong_ (to a certain extent). You _want to run_ diagnostic tools to
make sure that there is no problem. And you don't want some software (not even
HAL) to repair errors without prior admin knowledge/permission

Quoted 6 lines

I think there's a place for a scrubber to continuously verify filesystem data and metadata, at low io priority, and correct any correctable errors. The admin can read the error correction report at their leisure, and then take any action that's outside the filesystem's capabilities (like ordering and installing new disks).

Scrubbing is key for many scenarios since errors can "grow" even in places where previous IO has been completed without flagging an error.

Scrubbing is key for many scenarios since errors can "grow" even in places where previous IO has been completed without flagging an error.

Some neat tricks are:

(1) use block level scrubbing to detect any media errors. If you
can map that sector level error into a file system object (meta data, file data or unallocated space), tools can recover (fsck, get another copy of the file or just ignore it!). There is a special command called "READ_VERIFY" that can be used to validate the sectors without actually moving data from the target to the host, so you can scrub without consuming page cache, etc.

(2) sign and validate the object at the file level, say by
validating a digital signature. This can catch high level errors (say the app messed up).

Note that this scrubbing needs to be carefully tuned to not interfere with the foreground workload, using something like IO nice or the other IO controllers being kicked about might help :-)

- parallel mounts (very important!)

- journaling

- map out dead blocks

btrfs has many of the same goals... but they are goals not code so when you might see them is indeterminate.

I believe these should not be in btrfs:

Quoted 1 line · Stephan von Krawczynski

- parallel mounts (very important!)

as Andi said, you want a cluster or distributed fs. There are layered designs (CRFS or network filesystems) that can do the job and trying to do it in btrfs causes too many problems.

Quoted 1 line

- journaling

I assume you *do not* mean metadata journaling, you mean sending all file updates to a single output stream (as in one disk, tape, or network link). I've done that, but would not recommend it in btrfs because it limits the total fs bandwidth to what the single stream can support. This is normally done today by applications like databases, not in the filesystem.

Quoted 1 line

- map out dead blocks

Useless... a waste of time, code, and metadata structures. With current device technology, any device reporting bad blocks the device can not map out is about to die and needs replaced!

jim

Sure, but what you say only reflects the ideal world. On a file service, you never have that. In fact you do not even have good control about what is going on. Lets say you have a setup that creates, reads and deletes files 24h a day from numerous clients. At two o'clock in the morning some hd decides to partially die. Files get created on it, fill data up to errors, get deleted and another bunch of data arrives and yet again fs tries to allocate the same dead areas. You loose a lot more data only because the fs did not map out the already known dead blocks. Of course you would replace the dead drive later on, but in the meantime you have a lot of fun.
In other words: give me a tool to freeze the world right at the time the errors show up, or map out dead blocks (only because it is a lot easier).

Quoted 12 lines · Stephan von Krawczynski

Sure, but what you say only reflects the ideal world. On a file service, you never have that. In fact you do not even have good control about what is going on. Lets say you have a setup that creates, reads and deletes files 24h a day from numerous clients. At two o'clock in the morning some hd decides to partially die. Files get created on it, fill data up to errors, get deleted and another bunch of data arrives and yet again fs tries to allocate the same dead areas. You loose a lot more data only because the fs did not map out the already known dead blocks. Of course you would replace the dead drive later on, but in the meantime you have a lot of fun.
In other words: give me a tool to freeze the world right at the time the errors show up, or map out dead blocks (only because it is a lot easier).

When modern disks can't solve the problems with their internal driver remapping anymore you better replace it ASAP as it is a very strong disk failure indication. Last years FAST has some very interesting statitics showing this in the field.

[...]
If you have remapped a big chunk of the sectors (say more than 10%), you should grab the data off the disk asap and replace it. Worry less about errors during read, writes indicate more serious errors.

[...]
ric

Quoted 4 lines · On Tue, 21 Oct 2008 13:31:37 -0400 Ric Wheeler <[email protected]> wrote:

[...]
If you have remapped a big chunk of the sectors (say more than 10%), you should grab the data off the disk asap and replace it. Worry less about errors during read, writes indicate more serious errors.

Ok, now for the bad news: money is invented.
If you replace a disk before real failure you won't get replacement from the manufacturer. That may sound irrelevant to someone handling 5 disks, but is significant if handling 500 or more. The replacement rate is indeed much higher than people think from their home pcs.

Quoted 4 lines · Stephan von Krawczynski

On Tue, 21 Oct 2008 13:31:37 -0400
Ric Wheeler <[email protected]> wrote:

Quoted 9 lines

Quoted 3 lines

[...]
If you have remapped a big chunk of the sectors (say more than 10%), you should grab the data off the disk asap and replace it. Worry less about errors during read, writes indicate more serious errors.

Ok, now for the bad news: money is invented.
If you replace a disk before real failure you won't get replacement from the manufacturer. That may sound irrelevant to someone handling 5 disks, but is significant if handling 500 or more. The replacement rate is indeed much higher than people think from their home pcs.

Hardware vendors already do replace disks based on policies defined by their own array hardware. These are already predictive.

-chris

One key is not to replace the drives too early - you often can recover significant amounts of data from a drive that is on its last legs. This can be useful even in RAID rebuilds since with today's enormous drive capacities, you might hit a latent error during the rebuild on one of the presumed healthy drives.

Of course, if you don't have a spare drive in your configuration, this is not practical...

Quoted 6 lines · Ric Wheeler

One key is not to replace the drives too early - you often can recover significant amounts of data from a drive that is on its last legs. This can be useful even in RAID rebuilds since with today's enormous drive capacities, you might hit a latent error during the rebuild on one of the presumed healthy drives.

Quoted 2 lines

Of course, if you don't have a spare drive in your configuration, this is not practical...

Why would you have a spare drive? That's a wasted spindle.

You want to have spare capacity, enough for one or two (or fifteen) drives' worth of data. When a drive goes bad, you rebuild into the spare capacity you have.

Quoted 3 lines

You want to have spare capacity, enough for one or two (or fifteen) drives' worth of data. When a drive goes bad, you rebuild into the spare capacity you have.

You want spare capacity that does not degrade your raid levels if you move the data onto it. In some configs, this will be a hot spare, in others it'll just be free space.

Quoted 3 lines · Ric Wheeler

One key is not to replace the drives too early - you often can recover significant amounts of data from a drive that is on its last legs. This can be useful even in RAID rebuilds since with today's enormous drive capacities, you might hit a latent error during the rebuild on one of the presumed healthy drives.

Of course, if you don't have a spare drive in your configuration, this is not practical...

Why would you have a spare drive? That's a wasted spindle.

You want to have spare capacity, enough for one or two (or fifteen) drives' worth of data. When a drive goes bad, you rebuild into the spare capacity you have.

When you replace the drive, the filesystem moves data into the new drive to take advantage of the new spindle.

Quoted 1 line

You want to have spare capacity, enough for one or two (or fifteen) drives' worth of data. When a drive goes bad, you rebuild into the spare capacity you have.

That is a different model (and one that makes sense, we used that in Centera for object level protection schemes). It is a nice model as well, but not how most storage works today.

Quoted 3 lines

When you replace the drive, the filesystem moves data into the new drive to take advantage of the new spindle.

When you buy a storage solution (hardware or software), the key here is "utilized capacity." If you have an enclosure that can host say 12-15 drives in a 2U enclosure, people normally leave one drive as spare. RAID6 is another way to do this. You can do a 4+2 and 4+2 with 66% utilized capacity in RAID 6 or possibly a RAID5 scheme using like 5+1 and 4+1 with one global spare (75% utilized capacity).

That gives you the chance to do rebuild your RAID group without having to physically visit the data center. You can also do fancy stuff with the spare (like migrate as many blocks as possible before the RAID rebuild to that spare) which reduces your exposure to the 2nd drive failure and speeds up your rebuild time.

In the end, whether you use a block based RAID solution or an object based solution, you just need to figure out how to balance your utilized capacity against performance and data integrity needs.

Quoted 3 lines · Ric Wheeler

Quoted 1 line

You want to have spare capacity, enough for one or two (or fifteen) drives' worth of data. When a drive goes bad, you rebuild into the spare capacity you have.

That is a different model (and one that makes sense, we used that in Centera for object level protection schemes). It is a nice model as well, but not how most storage works today.

Well, btrfs is not about duplicating how most storage works today. Spare capacity has significant advantages over spare disks, such as being able to mix disk sizes, RAID levels, and better performance.

Quoted 5 lines

Quoted 4 lines

When you replace the drive, the filesystem moves data into the new drive to take advantage of the new spindle.

Quoted 5 lines

When you buy a storage solution (hardware or software), the key here is "utilized capacity." If you have an enclosure that can host say 12-15 drives in a 2U enclosure, people normally leave one drive as spare. RAID6 is another way to do this. You can do a 4+2 and 4+2 with 66% utilized capacity in RAID 6 or possibly a RAID5 scheme using like 5+1 and 4+1 with one global spare (75% utilized capacity).

That gives you the chance to do rebuild your RAID group without having to physically visit the data center. You can also do fancy stuff with the spare (like migrate as many blocks as possible before the RAID rebuild to that spare) which reduces your exposure to the 2nd drive failure and speeds up your rebuild time.

In the end, whether you use a block based RAID solution or an object based solution, you just need to figure out how to balance your utilized capacity against performance and data integrity needs.

In both models (spare disk and spare capacity) the storage utilization is the same, or nearly so. But with spare capacity you get better performance since you have more spindles seeking for your data, and since less of the disk surface is occupied by data, making your seeks shorter.

Quoted 2 lines

Well, btrfs is not about duplicating how most storage works today. Spare capacity has significant advantages over spare disks, such as being able to mix disk sizes, RAID levels, and better performance.

Sure, there are advantages that go in favour of one or the other approaches. But btrfs is also about being able to use common hardware configurations without having to reinvent where we can avoid it (if we have a working RAID or enough drives to do RAID5 with spares or RAID6, we want to be able to delegate that off to something else if we can).

The major difficulty with the spare capacity model is that your recovery is not as simple and well understood as RAID rebuilds.

If you assume that whole drives fail under btrfs mirroring, you are not really doing anything more than simple RAID, or do I misunderstand your suggestion?

I don't see the point about head seeking. In RAID, you also have the same layout so you minimize head movement (just move more heads per IO in parallel).

Quoted 4 lines · Ric Wheeler

Quoted 2 lines

Well, btrfs is not about duplicating how most storage works today. Spare capacity has significant advantages over spare disks, such as being able to mix disk sizes, RAID levels, and better performance.

Sure, there are advantages that go in favour of one or the other approaches. But btrfs is also about being able to use common hardware configurations without having to reinvent where we can avoid it (if we have a working RAID or enough drives to do RAID5 with spares or RAID6, we want to be able to delegate that off to something else if we can).

Well, if you have an existing RAID (or have lots of $$$ to buy a new one), you needn't tell Btrfs about it. Just be sure not to enable Btrfs data redundancy, or you'll have redundant redundancy, which is expensive.

What Btrfs enables with its multiple device capabilities is to assemble a JBOD into a filesystem-level data redundancy system, which is cheaper, more flexible (per-file data redundancy levels), and faster (no need for RMW, since you're always COWing).

Quoted 2 lines

The major difficulty with the spare capacity model is that your recovery is not as simple and well understood as RAID rebuilds.

That's Chris's problem. :-)

Quoted 2 lines

If you assume that whole drives fail under btrfs mirroring, you are not really doing anything more than simple RAID, or do I misunderstand your suggestion?

I do assume that whole drives fail, but RAIDing and rebuilding is file level. So one extent on a failed disk might be part of a mirrored file, while another extent can be part of a 14-member RAID6 extent.

A rebuild would iterate over all disk extents (making use of the backref tree), determine which file contains that extent, and rebuild that extent using spare storage on other disks.

Quoted 1 line

I don't see the point about head seeking. In RAID, you also have the same layout so you minimize head movement (just move more heads per IO in parallel).

Suppose you have 5 disks with 1 spare. Suppose you are reading from a full fs. On a disk-level RAID, all disks are full. So you have 5 spindles seeking over 100% of the disk surface. With spare capacity, you have 6 disks which are 5/6 full (retaining the same utilization as old-school RAID). So you have 6 spindles, each with a seek range that is 5/6 of a whole disk, so more seek heads _and_ faster individual seeks.

Quoted 18 lines · Avi Kivity

Quoted 11 lines · Ric Wheeler

Quoted 4 lines

Well, btrfs is not about duplicating how most storage works today. Spare capacity has significant advantages over spare disks, such as being able to mix disk sizes, RAID levels, and better performance.

Sure, there are advantages that go in favour of one or the other approaches. But btrfs is also about being able to use common hardware configurations without having to reinvent where we can avoid it (if we have a working RAID or enough drives to do RAID5 with spares or RAID6, we want to be able to delegate that off to something else if we can).

Well, if you have an existing RAID (or have lots of $$$ to buy a new one), you needn't tell Btrfs about it. Just be sure not to enable Btrfs data redundancy, or you'll have redundant redundancy, which is expensive.

Quoted 4 lines

What Btrfs enables with its multiple device capabilities is to assemble a JBOD into a filesystem-level data redundancy system, which is cheaper, more flexible (per-file data redundancy levels), and faster (no need for RMW, since you're always COWing).

I think that the btrfs plan is still to push more complicated RAID schemes off to MD (RAID6, etc) so this is an issue even with a JBOD.

One problem with the spare
capacity model is the general trend where drives from the same batch that get hammered on in the same way tend to die at the same time. Some shops will sleep better knowing there's a hot spare and that's fine by me.

Quoted 6 lines · Chris Mason

One problem with the spare
capacity model is the general trend where drives from the same batch that get hammered on in the same way tend to die at the same time. Some shops will sleep better knowing there's a hot spare and that's fine by me.

How does hot sparing help? All your disks die except the spare.

Of course, I've no objection to disk sparing as an additional option; I just feel that capacity sparing is superior.

For any given set of disks, you "just" need to do the math to compute the utilized capacity, the expected rate of drive failure, the rebuild time and then see whether you can recover from your first failure before a 2nd disk dies.

In practice, this is not an academic question since drives do occasionally fail in batches (and drives from the same batch get stuffed into the same system).

I suspect that what will be used in mission critical deployments will be more conservative than what is used in less critical path systems

I think that the btrfs plan is still to push more complicated RAID schemes off to MD (RAID6, etc) so this is an issue even with a JBOD. It will be interesting to map out the possible ways to use built in mirroring, etc vs the external RAID and actually measure the utilized capacity and performance (online & during rebuilds).

Quoted 8 lines · Stephan von Krawczynski

Quoted 4 lines · On Tue, 21 Oct 2008 13:31:37 -0400 Ric Wheeler <[email protected]> wrote:

[...]
If you have remapped a big chunk of the sectors (say more than 10%), you should grab the data off the disk asap and replace it. Worry less about errors during read, writes indicate more serious errors.

Quoted 5 lines

Ok, now for the bad news: money is invented.
If you replace a disk before real failure you won't get replacement from the manufacturer. That may sound irrelevant to someone handling 5 disks, but is significant if handling 500 or more. The replacement rate is indeed much higher than people think from their home pcs.

Hardware vendors already do replace disks based on policies defined by their own array hardware. These are already predictive.

-chris

Though it would be nice to have a tool that would provide enough information to make a warranty claim -- does btrfs keep enough information for such a tool to be written?

Quoted 6 lines

[...]
Lets agree that the market for drives, arrays and related stuff is big and contains just about any example one needs for arguing :-) Nevertheless we probably agree that if john doe meets big-player and tries to warranty-replace a non-dead drive he will have troubles.

If John Doe is using redundant storage in the first place, he just needs an emergency disk that can be swapped-in for a failing disk, and then stress-test the failing disk to death, get it replaced by manufacturer, and the replacement becomes the next standby/emergency disk.

Quoted 11 lines · Stephan von Krawczynski

Sure, but what you say only reflects the ideal world. On a file service, you never have that. In fact you do not even have good control about what is going on. Lets say you have a setup that creates, reads and deletes files 24h a day from numerous clients. At two o'clock in the morning some hd decides to partially die. Files get created on it, fill data up to errors, get deleted and another bunch of data arrives and yet again fs tries to allocate the same dead areas. You loose a lot more data only because the fs did not map out the already known dead blocks. Of course you would replace the dead drive later on, but in the meantime you have a lot of fun.
In other words: give me a tool to freeze the world right at the time the errors show up, or map out dead blocks (only because it is a lot easier).

When modern disks can't solve the problems with their internal driver remapping anymore you better replace it ASAP as it is a very strong disk failure indication. Last years FAST has some very interesting statitics showing this in the field.

Hello all,

reading the list for a while it looks like all kinds of implementational topics are covered but no basic user requests or talks are going on. Since I have found no other list on vger covering these issues I choose this one, forgive my ignorance if it is the wrong place.
Like many people on the planet we try to handle quite some amounts of data (TBs) and try to solve this with several linux-based fileservers. Years of (mostly bad) experience led us to the following minimum requirements for a new fs on our servers:

1. filesystem-check
1.1 it should not
- delay boot process (we have to wait for hours currently)
- prevent mount in case of errors
- be a part of the mount process at all
- always check the whole fs

1.2 it should be able
- to always be started interactively by user
- to check parts/subtrees of the fs
- to run purely informational (reporting, non-modifying)
- to run on a mounted fs

2. general requirements
- fs errors without file/dir names are useless
- errors in parts of the fs are no reason for a fs to go offline as a whole

- mounting must not delay the system startup significantly

- resizing during runtime (up and down)

- parallel mounts (very important!)
(two or more hosts mount the same fs concurrently for reading and writing)

- journaling

- versioning (file and dir)

From a data structure point of view, version control is fairly easy. From a user interface and policy point of view, it gets difficult very

- undelete (file and dir)

- snapshots

- run into hd errors more than once for the same file (as an option)

- map out dead blocks
(and of course display of the currently mapped out list)

- no size limitations (more or less)
- performant handling of large numbers of files inside single dirs (to check that use > 100.000 files in a dir, understand that it is no good idea to spread inode-blocks over the whole hd because of seek times)

- power loss at any time must not corrupt the fs (atomic fs modification) (new-data loss is acceptable)

Remember, this is not meant to be a request for features, it is a list that built up over 10 years of handling data and the failures we experienced. To our knowledge no fs meets this list, but hey, is that a reason for not talking about it? Our goal is pretty simple: maximize fs uptime.
How does btrfs match?

Started interactively? I'm not entirely sure what that means, but in general when you ask the user a question about if/how to fix a corruption, they will have no idea what the correct answer is.

While that's true today, I'm not sure it has to be true always. I always thought traditional fsck user interfaces were a UI desaster and could be done much better with some simple tweaks. [...]

-Andi

Thanks for this input and for taking the time to post it.

Quoted 6 lines

1. filesystem-check
1.1 it should not
- delay boot process (we have to wait for hours currently)
- prevent mount in case of errors
- be a part of the mount process at all
- always check the whole fs

For this, you have to define filesystem-check very carefully. In reality, corruptions can prevent mounting. We can try very very hard to limit the class of corruptions that prevent mounting, and use duplication and replication to create configurations that address the remaining cases.

In general, we'll be able to make things much better than they are today.

Quoted 5 lines

1.2 it should be able
- to always be started interactively by user
- to check parts/subtrees of the fs
- to run purely informational (reporting, non-modifying)
- to run on a mounted fs

Started interactively? I'm not entirely sure what that means, but in general when you ask the user a question about if/how to fix a corruption, they will have no idea what the correct answer is.

Quoted 3 lines

2. general requirements
- fs errors without file/dir names are useless
- errors in parts of the fs are no reason for a fs to go offline as a whole

These two are in progress. Btrfs won't always be able to give a file and directory name, but it will be able to give something that can be turned into a file or directory name. You don't want important diagnostic messages delayed by name lookup.

Quoted 2 lines

- mounting must not delay the system startup significantly

Mounts are fast

Quoted 1 line

- resizing during runtime (up and down)

Resize is done

Quoted 3 lines

- parallel mounts (very important!)
(two or more hosts mount the same fs concurrently for reading and writing)

As Jim and Andi have said, parallel mounts are not in the feature list for Btrfs. Network filesystems will provide these features.

Quoted 1 line

- journaling

Btrfs doesn't journal. The tree logging code is close, it provides optimized fsync and O_SYNC operations. The same basic structures could be used for remote replication.

Quoted 1 line

- versioning (file and dir)

Quoted 2 lines

From a data structure point of view, version control is fairly easy. From a user interface and policy point of view, it gets difficult very

quickly. Aside from snapshotting, version control is outside the scope of btrfs.

There are lots of good version control systems available, I'd suggest you use them instead.

Quoted 1 line

- undelete (file and dir)

Undelete is easy

but I think best done at a layer above the FS.

Quoted 1 line

- snapshots

Done

Quoted 1 line

- run into hd errors more than once for the same file (as an option)

Sorry, I'm not sure what you mean here.

Quoted 2 lines

- map out dead blocks
(and of course display of the currently mapped out list)

I agree with Jim on this one. Drives remap dead sectors, and when they stop remapping them, the drive should be replaced.

Quoted 6 lines

- no size limitations (more or less)
- performant handling of large numbers of files inside single dirs (to check that use > 100.000 files in a dir, understand that it is no good idea to spread inode-blocks over the whole hd because of seek times)

Everyone has different ideas on "large" numbers of files inside a single dir. The directory indexing done by btrfs can easily handle 100,000

Quoted 2 lines

- power loss at any time must not corrupt the fs (atomic fs modification) (new-data loss is acceptable)

Done. Btrfs already uses barriers as required for sata drives. [...]
-chris

Yes, we hear and say that all the time, name one linux fs doing it, please.

Quoted 2 lines · Stephan von Krawczynski

Yes, we hear and say that all the time, name one linux fs doing it, please.

ext[234] support it to some extent. It has some limitations (especially when the files are large and you shouldn't do too much followon IO to prevent the data from being overwriten) and the user frontends are not very nice, but it it's there

-Andi

Quoted 8 lines

Quoted 3 lines

2. general requirements
- fs errors without file/dir names are useless
- errors in parts of the fs are no reason for a fs to go offline as a whole

These two are in progress. Btrfs won't always be able to give a file and directory name, but it will be able to give something that can be turned into a file or directory name. You don't want important diagnostic messages delayed by name lookup.

That's a point I really never understood. Why is it non-trivial for a fs to know what file or dir (name) it is currently working on?

Quoted 13 lines

Quoted 2 lines

- mounting must not delay the system startup significantly

Mounts are fast

Quoted 1 line

- resizing during runtime (up and down)

Resize is done

Quoted 3 lines

- parallel mounts (very important!)
(two or more hosts mount the same fs concurrently for reading and writing)

Quoted 2 lines

As Jim and Andi have said, parallel mounts are not in the feature list for Btrfs. Network filesystems will provide these features.

Can you explain what "network filesystems" stands for in this statement, please name two or three examples.

Quoted 13 lines

Quoted 1 line

- journaling

Btrfs doesn't journal. The tree logging code is close, it provides optimized fsync and O_SYNC operations. The same basic structures could be used for remote replication.

Quoted 4 lines

- versioning (file and dir)

From a data structure point of view, version control is fairly easy. From a user interface and policy point of view, it gets difficult very

quickly. Aside from snapshotting, version control is outside the scope of btrfs.

Quoted 2 lines

There are lots of good version control systems available, I'd suggest you use them instead.

To me versioning sounds like a not-so-easy-to-implement feature. Nevertheless I trust your experience. If a basic implementation is possible and not too complex, why deny a feature?

Quoted 3 lines

Quoted 1 line

- undelete (file and dir)

Undelete is easy

Yes, we hear and say that all the time, name one linux fs doing it, please.

Quoted 1 line

but I think best done at a layer above the FS.

Before we got into the linux community we used n.vell netware. Undelete has been there since about the first day. More then ten years later (nowadays) it is still missing in linux. I really do suggest to provide _some_ solution and _then_ lets talk about the _better_ solution.

Quoted 6 lines

Quoted 1 line

- snapshots

Done

Quoted 1 line

- run into hd errors more than once for the same file (as an option)

Quoted 1 line

Sorry, I'm not sure what you mean here.

If your hd is going dead you often find out that touching broken files takes ages. If the fs finds out a file is corrupt because the device has errors it could just flag the file as broken and not re-read the same error a thousand times more. Obviously you want that as an option, because there can be good reasons for re-reading dead files...

Quoted 6 lines

Quoted 3 lines

- map out dead blocks
(and of course display of the currently mapped out list)

I agree with Jim on this one. Drives remap dead sectors, and when they stop remapping them, the drive should be replaced.

If your life depends on it, would you use one rope or two to secure yourself?

Quoted 7 lines

Quoted 6 lines

- no size limitations (more or less)
- performant handling of large numbers of files inside single dirs (to check that use > 100.000 files in a dir, understand that it is no good idea to spread inode-blocks over the whole hd because of seek times)

Quoted 2 lines

Everyone has different ideas on "large" numbers of files inside a single dir. The directory indexing done by btrfs can easily handle 100,000

The story is not really about if it can but how fast it can. You know that most time is spent in seeks these days. If you have 100000 blocks to read right across the whole disk for scanning through a dir (fstat every file) you will see quite a difference to a situation where the relevant data can be read within few (or zero) seeks. Its a question of fs layout on the disk.

Quoted 11 lines · Stephan von Krawczynski

Quoted 8 lines

Quoted 3 lines

2. general requirements
- fs errors without file/dir names are useless
- errors in parts of the fs are no reason for a fs to go offline as a whole

These two are in progress. Btrfs won't always be able to give a file and directory name, but it will be able to give something that can be turned into a file or directory name. You don't want important diagnostic messages delayed by name lookup.

That's a point I really never understood. Why is it non-trivial for a fs to know what file or dir (name) it is currently working on?

The name lives in block A, but you might find a corruption while processing block B. Block A might not be in ram anymore, or it might be in ram but locked by another process.

On top of all of that, when we print errors it's because things haven't gone well. They are deep inside of various parts of the filesystem, and we might not be able to take the required locks or read from the disk in order to find the name of the thing we're operating on.

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

- parallel mounts (very important!)
(two or more hosts mount the same fs concurrently for reading and writing)

Quoted 6 lines

Quoted 2 lines

As Jim and Andi have said, parallel mounts are not in the feature list for Btrfs. Network filesystems will provide these features.

Can you explain what "network filesystems" stands for in this statement, please name two or three examples.

NFS (done) CRFS (under development), maybe ceph as well which is also under development.

Quoted 7 lines

Quoted 7 lines

Quoted 4 lines

- versioning (file and dir)

From a data structure point of view, version control is fairly easy. From a user interface and policy point of view, it gets difficult very

quickly. Aside from snapshotting, version control is outside the scope of btrfs.

Quoted 7 lines

Quoted 2 lines

There are lots of good version control systems available, I'd suggest you use them instead.

To me versioning sounds like a not-so-easy-to-implement feature. Nevertheless I trust your experience. If a basic implementation is possible and not too complex, why deny a feature?

In general I think snapshotting solves enough of the problem for most of the people most of the time. I'd love for Btrfs to be the perfect FS, but I'm afraid everyone has a different definition of perfect.

Storing multiple versions of something is pretty easy. Making a usable interface around those versions is the hard part, especially because you need groups of files to be versioned together in atomic groups (something that looks a lot like a snapshot).

Versioning is solved in userspace. We would never be able to implement everything that git or mercurial can do inside the filesystem.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

- undelete (file and dir)

Undelete is easy

Yes, we hear and say that all the time, name one linux fs doing it, please.

The fact that nobody is doing it is not a good argument for why it should be done ;)

Undelete is a policy decision about what to do with
files as they are removed. I'd much rather see it implemented above the filesystems instead of individually in each filesystem.

This doesn't mean I'll never code it, it just means it won't get implemented directly inside of Btrfs. In comparison with all of the other features pending, undelete is pretty far down on the list.

Quoted 5 lines

If your hd is going dead you often find out that touching broken files takes ages. If the fs finds out a file is corrupt because the device has errors it could just flag the file as broken and not re-read the same error a thousand times more. Obviously you want that as an option, because there can be good reasons for re-reading dead files...

I really agree that we want to avoid beating on a dead drive.

Btrfs will record some error information about the drive so it can decide what to do with failures. But, remembering that sector #12345768 is bad doesn't help much. When the drive returned the IO error it remapped the sector and the next write will probably succeed.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

- map out dead blocks
(and of course display of the currently mapped out list)

I agree with Jim on this one. Drives remap dead sectors, and when they stop remapping them, the drive should be replaced.

If your life depends on it, would you use one rope or two to secure yourself?

Btrfs will keep the dead drive around as a fallback for sectors that fail on the other mirrors when data is being rebuilt. Beyond that, we'll expect you to toss the bad drive once the rebuild has finished.

There's an interesting paper about how netapp puts the drive into rehab and is able to avoid service calls by rewriting the bad sectors and checking them over. That's a little ways off for Btrfs.

[...]
-chris

Quoted 14 lines · On Tue, 21 Oct 2008 13:49:43 -0400 Chris Mason <[email protected]> wrote:

Quoted 11 lines · Stephan von Krawczynski

Quoted 8 lines

Quoted 3 lines

2. general requirements
- fs errors without file/dir names are useless
- errors in parts of the fs are no reason for a fs to go offline as a whole

These two are in progress. Btrfs won't always be able to give a file and directory name, but it will be able to give something that can be turned into a file or directory name. You don't want important diagnostic messages delayed by name lookup.

That's a point I really never understood. Why is it non-trivial for a fs to know what file or dir (name) it is currently working on?

Quoted 8 lines

The name lives in block A, but you might find a corruption while processing block B. Block A might not be in ram anymore, or it might be in ram but locked by another process.

On top of all of that, when we print errors it's because things haven't gone well. They are deep inside of various parts of the filesystem, and we might not be able to take the required locks or read from the disk in order to find the name of the thing we're operating on.

Ok, this is interesting. In another thread I was told parallel mounts are really complex and you cannot do good things in such an environment that you can do with single mount. Well, then, why don't we do it? All boxes I know have tons of RAM, but fs finds no place in RAM to put large parts (if not all) of the structural fs data including filenames? Besides the simple fact that RAM is always faster than any known disk be it rotating or not, and that RAM is just there, whats the word for not doing it?

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

- parallel mounts (very important!)
(two or more hosts mount the same fs concurrently for reading and writing)

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

As Jim and Andi have said, parallel mounts are not in the feature list for Btrfs. Network filesystems will provide these features.

Can you explain what "network filesystems" stands for in this statement, please name two or three examples.

NFS (done) CRFS (under development), maybe ceph as well which is also under development.

NFS is a good example for a fs that never got redesigned for modern world. I hope it will, but currently it's like Model T on a highway.
You have a NFS server with clients. Your NFS server dies, your backup server cannot take over the clients without them resetting their NFS-link (which means reboot to many applications) - no way.
Besides that you still need another fs below NFS to bring your data onto some medium, which means you still have the problem how to create redundancy in your server architecture.

Quoted 8 lines

NFS is a good example for a fs that never got redesigned for modern world. I hope it will, but currently it's like Model T on a highway.
You have a NFS server with clients. Your NFS server dies, your backup server cannot take over the clients without them resetting their NFS-link (which means reboot to many applications) - no way.
Besides that you still need another fs below NFS to bring your data onto some medium, which means you still have the problem how to create redundancy in your server architecture.

You are somewhat misinformed on this. Perhaps the Linux nfs server can't cope, but I doubt it. NFS was designed to be stateless. I've got a fair amount of experience
with a dual head netapp architecture. When 1 head dies, the other transparently fails
over. During the brief downtime, the clients will go into I/O wait if at all instead of being
disconnected. You might be able to do something similar using nfsd and keepalived if
both servers were connected to the same storage. Setting that up would be trivial. You
just need the clients mounting the vip and a reliable mechanism to provide the data from
that vip. You could use heartbeat, but it is overly complex. Also look at clustered nfs or
pnfs, both of which are nfs redesigns like you speak of.

Signature · 1 line

Jeff Schroeder

Quoted 14 lines · On Tue, 21 Oct 2008 13:49:43 -0400 Chris Mason <[email protected]> wrote:

Quoted 11 lines · Stephan von Krawczynski

Quoted 8 lines

Quoted 3 lines

2. general requirements
- fs errors without file/dir names are useless
- errors in parts of the fs are no reason for a fs to go offline as a whole

These two are in progress. Btrfs won't always be able to give a file and directory name, but it will be able to give something that can be turned into a file or directory name. You don't want important diagnostic messages delayed by name lookup.

That's a point I really never understood. Why is it non-trivial for a fs to know what file or dir (name) it is currently working on?

Quoted 8 lines

The name lives in block A, but you might find a corruption while processing block B. Block A might not be in ram anymore, or it might be in ram but locked by another process.

On top of all of that, when we print errors it's because things haven't gone well. They are deep inside of various parts of the filesystem, and we might not be able to take the required locks or read from the disk in order to find the name of the thing we're operating on.

Ok, this is interesting. In another thread I was told parallel mounts are really complex and you cannot do good things in such an environment that you can do with single mount. Well, then, why don't we do it? All boxes I know have tons of RAM, but fs finds no place in RAM to put large parts (if not all) of the structural fs data including filenames?

Besides the simple fact that
RAM is always faster than any known disk be it rotating or not, and that RAM is just there, whats the word for not doing it?

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

- parallel mounts (very important!)
(two or more hosts mount the same fs concurrently for reading and writing)

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

As Jim and Andi have said, parallel mounts are not in the feature list for Btrfs. Network filesystems will provide these features.

Can you explain what "network filesystems" stands for in this statement, please name two or three examples.

NFS (done) CRFS (under development), maybe ceph as well which is also under development.

NFS is a good example for a fs that never got redesigned for modern world. I hope it will, but currently it's like Model T on a highway.
You have a NFS server with clients. Your NFS server dies, your backup server cannot take over the clients without them resetting their NFS-link (which means reboot to many applications) - no way.
Besides that you still need another fs below NFS to bring your data onto some medium, which means you still have the problem how to create redundancy in your server architecture.

Quoted 19 lines

Quoted 7 lines

Quoted 7 lines

Quoted 4 lines

- versioning (file and dir)

From a data structure point of view, version control is fairly easy. From a user interface and policy point of view, it gets difficult very

quickly. Aside from snapshotting, version control is outside the scope of btrfs.

Quoted 7 lines

Quoted 2 lines

There are lots of good version control systems available, I'd suggest you use them instead.

To me versioning sounds like a not-so-easy-to-implement feature. Nevertheless I trust your experience. If a basic implementation is possible and not too complex, why deny a feature?

In general I think snapshotting solves enough of the problem for most of the people most of the time. I'd love for Btrfs to be the perfect FS, but I'm afraid everyone has a different definition of perfect.

Quoted 7 lines

Storing multiple versions of something is pretty easy. Making a usable interface around those versions is the hard part, especially because you need groups of files to be versioned together in atomic groups (something that looks a lot like a snapshot).

Versioning is solved in userspace. We would never be able to implement everything that git or mercurial can do inside the filesystem.

Well, quite often the question is not about whole trees of data to be versioned. Even single (few) files or dirs can be of interest. And you want people to set up a complete user space monster to version three openoffice documents (only a rather flawed example of course)?
Lots of people need a basic solution, not the groundbreaking answer to all questions.

Quoted 8 lines

Quoted 5 lines

If your hd is going dead you often find out that touching broken files takes ages. If the fs finds out a file is corrupt because the device has errors it could just flag the file as broken and not re-read the same error a thousand times more. Obviously you want that as an option, because there can be good reasons for re-reading dead files...

I really agree that we want to avoid beating on a dead drive.

Quoted 4 lines

Btrfs will record some error information about the drive so it can decide what to do with failures. But, remembering that sector #12345768 is bad doesn't help much. When the drive returned the IO error it remapped the sector and the next write will probably succeed.

Problem with probability is that software is pretty bad in judging. That's why my proposal was, lets do it and make it configurable for an admin that has a better idea of the current probability.

Quoted 13 lines

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

- map out dead blocks
(and of course display of the currently mapped out list)

I agree with Jim on this one. Drives remap dead sectors, and when they stop remapping them, the drive should be replaced.

If your life depends on it, would you use one rope or two to secure yourself?

Btrfs will keep the dead drive around as a fallback for sectors that fail on the other mirrors when data is being rebuilt. Beyond that, we'll expect you to toss the bad drive once the rebuild has finished.

Quoted 3 lines

There's an interesting paper about how netapp puts the drive into rehab and is able to avoid service calls by rewriting the bad sectors and checking them over. That's a little ways off for Btrfs.

It will become more interesting what remapping means in a world full of flash-disks. Does it mean a disk must be replaced when some or even lots of sectors are dead?

Let me reword my answer ;). The next write will always succeed unles=

the drive is out of remapping sectors. If the drive is out, it is on=

good for reads and holding down paper on your desk.

On 10/22/2008 3:50 PM, Chris Mason wrote:

=20

Quoted 1 line

Let me reword my answer ;). The next write will always succeed unle=

Quoted 1 line

the drive is out of remapping sectors. If the drive is out, it is o=

Quoted 2 lines

good for reads and holding down paper on your desk.
=20

I have a fairly new SATA disk with about 3000 hours of 24/7 duty (very light load), 0 remapped sectors and 8 consecutive sectors with read/write errors. Still, it did not perform remapping facing heavy writes on the bad sectors. Now what? For whatever reason, remapping not always works (or mine was produced with a total of zero remapping sectors=E2=80=A6).

- Matthias
=20

Quoted 4 lines · Matthias W=C3=A4chter wrote:

On 10/22/2008 3:50 PM, Chris Mason wrote:

=20

Quoted 1 line

Let me reword my answer ;). The next write will always succeed unl=

Quoted 1 line

Quoted 1 line

the drive is out of remapping sectors. If the drive is out, it is =

Quoted 10 lines

Quoted 2 lines

good for reads and holding down paper on your desk.
=20

I have a fairly new SATA disk with about 3000 hours of 24/7 duty (very light load), 0 remapped sectors and 8 consecutive sectors with read/write errors. Still, it did not perform remapping facing heavy writes on the bad sectors. Now what? For whatever reason, remapping not always works (or mine was produced with a total of zero remapping sectors=E2=80=A6).

Quoted 2 lines

- Matthias
=20

=20
It sounds like this drive is actually fine, you might have seen some=20 transient issues.
=20
Are you positive that the writes went directly to the sectors in=20 question - that should either clear the error or cause it to remap th=

sectors internally. (Reads will continue to fail).
=20
Mark Lord has added some options to hdparm that you might be able to =

to expressly clear the sectors in question in a more direct way.

As someone else replied, NFS is stateless

You have a NFS server with clients. Your NFS server dies, your backup server cannot take over the clients without them resetting their NFS-link (which means reboot to many applications) - no way.

Quoted 2 lines

- power loss at any time must not corrupt the fs (atomic fs modification) (new-data loss is acceptable)

Done. Btrfs already uses barriers as required for sata drives.

Quoted 3 lines · Chris Mason

Quoted 3 lines

- power loss at any time must not corrupt the fs (atomic fs modification) (new-data loss is acceptable)

Quoted 2 lines

Done. Btrfs already uses barriers as required for sata drives.

Aren't there situations in which write barriers don't do what they're supposed to do?

Cheers,
Eric

Quoted 11 lines · Eric Anopolsky

Quoted 5 lines · Chris Mason

Quoted 3 lines

- power loss at any time must not corrupt the fs (atomic fs modification) (new-data loss is acceptable)

Done. Btrfs already uses barriers as required for sata drives.

Aren't there situations in which write barriers don't do what they're supposed to do?

Quoted 4 lines

Cheers,
Eric

If the drive effectively "lies" to you about flushing the write cache, you might have an issue. I have not seen that first hand with recent disk drives (and I have seen a lot :-))

Quoted 15 lines · Ric Wheeler

Quoted 13 lines · Eric Anopolsky

Quoted 7 lines · Chris Mason

Quoted 4 lines

- power loss at any time must not corrupt the fs (atomic fs modification) (new-data loss is acceptable)

Done. Btrfs already uses barriers as required for sata drives.

Aren't there situations in which write barriers don't do what they're supposed to do?

Quoted 7 lines

Quoted 5 lines

Cheers,
Eric

If the drive effectively "lies" to you about flushing the write cache, you might have an issue. I have not seen that first hand with recent disk drives (and I have seen a lot :-))

That does not match the understanding I get from reading the notes/caveats section of Documentation/block/barrier.txt:

"Note that block drivers must not requeue preceding requests while completing latter requests in an ordered sequence. Currently, no error checking is done against this."

and perhaps more importantly:

"[a technical scenario involving disk writes]
The problem here is that the barrier request is *supposed* to indicate that filesystem update requests [2] and [3] made it safely to the physical medium and, if the machine crashes after the barrier is written, filesystem recovery code can depend on that. Sadly, that isn't true in this case anymore. IOW, the success of a I/O barrier should also be dependent on success of some of the preceding requests, where only upper layer (filesystem) knows what 'some' is.

This can be solved by implementing a way to tell the block layer which requests affect the success of the following barrier request and making lower lever drivers to resume operation on error only after block layer tells it to do so.

As the probability of this happening is very low and the drive should be faulty, implementing the fix is probably an overkill. But, still, it's there."

Cheers,
Eric

The cache flush command for ATA devices will block and wait until all of the device's write cache has been written back.

What I assume Tejun was referring to here is that some IO might have been written out to the device and an error happened when the device tried to write the cache back (say due to normal drive microcode cache destaging). The problem with this is that there is no outstanding IO context between the host and the storage to report the error to (i.e., the drive has already ack'ed the write).

If this is what is being described, there is a non-zero chance that this might happen, but it is extremely infrequent. The checksumming that we have in btrfs will catch these bad writes when you replay the journal after a crash (or even when you read data blocks) so I would contend that this is about as good as we can do.

Quoted 10 lines · Ric Wheeler

The cache flush command for ATA devices will block and wait until all of the device's write cache has been written back.

What I assume Tejun was referring to here is that some IO might have been written out to the device and an error happened when the device tried to write the cache back (say due to normal drive microcode cache destaging). The problem with this is that there is no outstanding IO context between the host and the storage to report the error to (i.e., the drive has already ack'ed the write).

Quoted 6 lines

If this is what is being described, there is a non-zero chance that this might happen, but it is extremely infrequent. The checksumming that we have in btrfs will catch these bad writes when you replay the journal after a crash (or even when you read data blocks) so I would contend that this is about as good as we can do.

Please consider the following scenario.

1. FS issues lots of writes which are queued in the block elevator.
2. FS issues barrier.
3. Elevator pushes out all the writes.
4. One of the writes fails for some reason. Media failure or what not. Failure is propagated to upper layer.
5. Whether there was preceding failure or not, block queue processing continues and writes out all the pending requests.
6. Elevator issues FLUSH and it gets executed by the device.
7. Elevator issues barrier write and it gets executed by the device.
8. *POWER LOSS*

The thing is that currently there is no defined way for FS to take action after #4 once happens unless it waits for all outstanding writes to complete before issuing the barrier. One way to solve this would be to make the failure status sticky such that any barrier following any number of uncleared errors will fail too, so that the filesystem can think about what it should do with the write failure.

Thanks.

Quoted 10 lines · Ric Wheeler

The cache flush command for ATA devices will block and wait until all of the device's write cache has been written back.

What I assume Tejun was referring to here is that some IO might have been written out to the device and an error happened when the device tried to write the cache back (say due to normal drive microcode cache destaging). The problem with this is that there is no outstanding IO context between the host and the storage to report the error to (i.e., the drive has already ack'ed the write).

Quoted 6 lines

If this is what is being described, there is a non-zero chance that this might happen, but it is extremely infrequent. The checksumming that we have in btrfs will catch these bad writes when you replay the journal after a crash (or even when you read data blocks) so I would contend that this is about as good as we can do.

Please consider the following scenario.

1. FS issues lots of writes which are queued in the block elevator.
2. FS issues barrier.
3. Elevator pushes out all the writes.
4. One of the writes fails for some reason. Media failure or what not. Failure is propagated to upper layer.
5. Whether there was preceding failure or not, block queue processing continues and writes out all the pending requests.
6. Elevator issues FLUSH and it gets executed by the device.
7. Elevator issues barrier write and it gets executed by the device.
8. *POWER LOSS*

The thing is that currently there is no defined way for FS to take action after #4 once happens unless it waits for all outstanding writes to complete before issuing the barrier. One way to solve this would be to make the failure status sticky such that any barrier following any number of uncleared errors will fail too, so that the filesystem can think about what it should do with the write failure.

Thanks.

I think that we do handle a failure in the case that you outline above since the FS will be able to notice the error before it sends a commit down (and that commit is wrapped in the barrier flush calls). This is the easy case since we still have the context for the IO.

It is more challenging (and kind of related) if the IO done in (4) has been ack'ed by drive, the drive later destages (not as part of the flush) its write cache and then an error happens. In this case, there is nothing waiting on the initiator side to receive the IO error. We have effectively lost the context for that IO.

The only way to detect this is on replay (if the journal has checksums enabled or the error will be flagged as a media error).

Quoted 4 lines · Ric Wheeler

I think that we do handle a failure in the case that you outline above since the FS will be able to notice the error before it sends a commit down (and that commit is wrapped in the barrier flush calls). This is the easy case since we still have the context for the IO.

I'm no FS guy but for that to be true FS should be waiting for all the outstanding IOs to finish before issuing a barrier and actually doesn't need barriers at all - it can do the same with flush_cache.

Quoted 12 lines · Tejun Heo

Quoted 5 lines · Ric Wheeler

I think that we do handle a failure in the case that you outline above since the FS will be able to notice the error before it sends a commit down (and that commit is wrapped in the barrier flush calls). This is the easy case since we still have the context for the IO.

I'm no FS guy but for that to be true FS should be waiting for all the outstanding IOs to finish before issuing a barrier and actually doesn't need barriers at all - it can do the same with flush_cache.

We wait and then barrier. If the barrier returned status that a previously ack'd IO had actually failed, we could do something to make sure the FS was consistent.

-chris

Quoted 15 lines · Chris Mason

Quoted 12 lines · Tejun Heo

Quoted 5 lines · Ric Wheeler

I think that we do handle a failure in the case that you outline above since the FS will be able to notice the error before it sends a commit down (and that commit is wrapped in the barrier flush calls). This is the easy case since we still have the context for the IO.

I'm no FS guy but for that to be true FS should be waiting for all the outstanding IOs to finish before issuing a barrier and actually doesn't need barriers at all - it can do the same with flush_cache.

Quoted 4 lines

We wait and then barrier. If the barrier returned status that a previously ack'd IO had actually failed, we could do something to make sure the FS was consistent.

As I mentioned in a reply to Tejun, I am not sure that we can count on the barrier op giving us status for IO's that failed to destage cleanly.

Waiting and then doing the FLUSH seems to give us the best coverage for normal failures (and your own testing shows that it is hugely effective in reducing some types of corruption at least :-)).

If you look at the types of common drive failures, I would break them into two big groups.

The first group would be transient errors - i.e., this IO fails (usually a read), but a subsequent IO will succeed with or without a sector remapping happening. Causes might be:

(1) just a bad read due to dirt on the surface of the drive - the
read will always fail, a write might clean the surface and restore it to useful life.
(2) vibrations (dropping your laptop, rolling a big machine down the
data center, passing trains :-))
(3) adjacent sector writes - hot spotting on drives can degrade the
data on adjacent tracks. This causes IO errors on reads for data that was successfully written before, but the track itself is still perfectly fine.

Quoted 31 lines · Ric Wheeler

Quoted 18 lines · Chris Mason

Quoted 9 lines · Tejun Heo

Quoted 6 lines · Ric Wheeler

I think that we do handle a failure in the case that you outline above since the FS will be able to notice the error before it sends a commit down (and that commit is wrapped in the barrier flush calls). This is the easy case since we still have the context for the IO.

Quoted 6 lines

I'm no FS guy but for that to be true FS should be waiting for all the outstanding IOs to finish before issuing a barrier and actually doesn't need barriers at all - it can do the same with flush_cache.

Quoted 5 lines

We wait and then barrier. If the barrier returned status that a previously ack'd IO had actually failed, we could do something to make sure the FS was consistent.

As I mentioned in a reply to Tejun, I am not sure that we can count on the barrier op giving us status for IO's that failed to destage cleanly.

Waiting and then doing the FLUSH seems to give us the best coverage for normal failures (and your own testing shows that it is hugely effective in reducing some types of corruption at least :-)).

If you look at the types of common drive failures, I would break them into two big groups.

Quoted 10 lines

The first group would be transient errors - i.e., this IO fails (usually a read), but a subsequent IO will succeed with or without a sector remapping happening. Causes might be:

(1) just a bad read due to dirt on the surface of the drive - the
read will always fail, a write might clean the surface and restore it to useful life.
(2) vibrations (dropping your laptop, rolling a big machine down the
data center, passing trains :-))
(3) adjacent sector writes - hot spotting on drives can degrade the
data on adjacent tracks. This causes IO errors on reads for data that was successfully written before, but the track itself is still perfectly fine.

4) Transient conditions such as heat or other problems made the drive give errors.

Combine your matrix with the single drive install vs the mirrored configuration and we get a lot of variables. What I'd love to have is a rehab tool for drives that works it over and decides if it should stay or go.

It is somewhat difficult to run the rehab on a mounted single disk install, but we can start with the multi-device config and work out way out from there.

For barrier flush, io errors reported back by the barrier flush would allow us to know when corrective action was required.

-chris

Quoted 5 lines · Ric Wheeler

I think that we do handle a failure in the case that you outline above since the FS will be able to notice the error before it sends a commit down (and that commit is wrapped in the barrier flush calls). This is the easy case since we still have the context for the IO.

I'm no FS guy but for that to be true FS should be waiting for all the outstanding IOs to finish before issuing a barrier and actually doesn't need barriers at all - it can do the same with flush_cache.

Quoted 7 lines

It is more challenging (and kind of related) if the IO done in (4) has been ack'ed by drive, the drive later destages (not as part of the flush) its write cache and then an error happens. In this case, there is nothing waiting on the initiator side to receive the IO error. We have effectively lost the context for that IO.

IIUC, that should be detectable from FLUSH whether the destaging occurred as part of flush or not, no?

Quoted 4 lines

The only way to detect this is on replay (if the journal has checksums enabled or the error will be flagged as a media error).

If it's not reported on FLUSH, it basically amounts to silent data corruption and only checksums can help.

Thanks.

Waiting for the target to ack an IO is not sufficient, since the target ack does not (with write cache enabled) mean that it is on persistent storage.

The key is to make your transaction commit insure that the commit block itself is not written out of sequence without flushing the dependent IO from the transaction.

If we disable the write cache, then file systems effectively do exactly the right thing today as you describe :-)

Quoted 3 lines

IIUC, that should be detectable from FLUSH whether the destaging occurred as part of flush or not, no?

I am not sure what happens to a write that fails to get destaged from cache. It probably depends on the target firmware, but I imagine that the target cannot hold onto it forever (or all subsequent flushes would always fail).

Quoted 4 lines · Ric Wheeler

Waiting for the target to ack an IO is not sufficient, since the target ack does not (with write cache enabled) mean that it is on persistent storage.

FS waiting for completion of all the dependent writes isn't too good latency and throughput-wise tho. It would be best if FS can indicate dependencies between write commands and barrier so that barrier doesn't have to empty the whole queue. Hmm... Can someone tell me how much such scheme would help?

Quoted 7 lines

The key is to make your transaction commit insure that the commit block itself is not written out of sequence without flushing the dependent IO from the transaction.

If we disable the write cache, then file systems effectively do exactly the right thing today as you describe :-)

For most SATA drives, disabling write back cache seems to take high toll on write throughput. :-(

Quoted 10 lines

Quoted 5 lines

IIUC, that should be detectable from FLUSH whether the destaging occurred as part of flush or not, no?

I am not sure what happens to a write that fails to get destaged from cache. It probably depends on the target firmware, but I imagine that the target cannot hold onto it forever (or all subsequent flushes would always fail).

As long as the error status is sticky, it doesn't have to hold on to the data, it's not gonna be able to write it anyway. The drive has to hold onto the failure information only. Yeah, but fully agreed on that it's most likely dependent on the specific firmware. There isn't any requirement on how to handle write back failure in the ATA spec. It wouldn't be too surprising if there are some drives which happily report the old data after silent write failure followed by flush and power loss at the right timing.

Thanks.

Quoted 7 lines

FS waiting for completion of all the dependent writes isn't too good latency and throughput-wise tho. It would be best if FS can indicate dependencies between write commands and barrier so that barrier doesn't have to empty the whole queue. Hmm... Can someone tell me how much such scheme would help?

I think that this is where SCSI ordered tags come in (or similar schemes). The idea would be to have tag all IO. You bump the tag, for example after you send down the journal data blocks to a new tag which is used for the commit block data sequence.

The ordering would require that lower ranked tags must all be destaged to persistent storage before a subsequent tag is written out.

The T13 had a microsoft proposal that is in this area:

http://www.t13.org/Documents/UploadedDocuments/docs2007/e07174r0-Write_Barrier_Command_Proposal.doc

For most SATA drives, disabling write back cache seems to take high toll on write throughput. :-(

As long as the error status is sticky, it doesn't have to hold on to the data, it's not gonna be able to write it anyway. The drive has to hold onto the failure information only. Yeah, but fully agreed on that it's most likely dependent on the specific firmware. There isn't any requirement on how to handle write back failure in the ATA spec. It wouldn't be too surprising if there are some drives which happily report the old data after silent write failure followed by flush and power loss at the right timing.

Quoted 2 lines · Tejun Heo

For most SATA drives, disabling write back cache seems to take high toll on write throughput. :-(

I measured this yesterday. This is true for pure write workloads; for mixed read/write workloads the throughput decrease is negligible.

Quoted 8 lines

As long as the error status is sticky, it doesn't have to hold on to the data, it's not gonna be able to write it anyway. The drive has to hold onto the failure information only. Yeah, but fully agreed on that it's most likely dependent on the specific firmware. There isn't any requirement on how to handle write back failure in the ATA spec. It wouldn't be too surprising if there are some drives which happily report the old data after silent write failure followed by flush and power loss at the right timing.

I got flamed for this on another list, but let's disable the write cache and live with the performance drop.

Quoted 4 lines

Quoted 2 lines

For most SATA drives, disabling write back cache seems to take high toll on write throughput. :-(

I measured this yesterday. This is true for pure write workloads; for mixed read/write workloads the throughput decrease is negligible.

Different tests on different hardware
give different results at different times!

Quoted 2 lines

I got flamed for this on another list, but let's disable the write cache and live with the performance drop.

We don't get to decide this, customers do.

As they say in the raid forum... fast, cheap, good - pick any 2

P.S. no flames because we chose no-battery == disable-write-cache

Quoted 4 lines · Tejun Heo

For most SATA drives, disabling write back cache seems to take high toll on write throughput. :-(

I measured this yesterday. This is true for pure write workloads; for mixed read/write workloads the throughput decrease is negligible.

Quoted 8 lines

As long as the error status is sticky, it doesn't have to hold on to the data, it's not gonna be able to write it anyway. The drive has to hold onto the failure information only. Yeah, but fully agreed on that it's most likely dependent on the specific firmware. There isn't any requirement on how to handle write back failure in the ATA spec. It wouldn't be too surprising if there are some drives which happily report the old data after silent write failure followed by flush and power loss at the right timing.

I got flamed for this on another list, but let's disable the write cache and live with the performance drop.

Quoted 3 lines · Ric Wheeler

Waiting for the target to ack an IO is not sufficient, since the target ack does not (with write cache enabled) mean that it is on persistent storage.

FS waiting for completion of all the dependent writes isn't too good latency and throughput-wise tho. It would be best if FS can indicate dependencies between write commands and barrier so that barrier doesn't have to empty the whole queue. Hmm... Can someone tell me how much such scheme would help?

Quoted 6 lines · Tejun Heo

Quoted 4 lines · Ric Wheeler

Waiting for the target to ack an IO is not sufficient, since the target ack does not (with write cache enabled) mean that it is on persistent storage.

Quoted 6 lines

FS waiting for completion of all the dependent writes isn't too good latency and throughput-wise tho. It would be best if FS can indicate dependencies between write commands and barrier so that barrier doesn't have to empty the whole queue. Hmm... Can someone tell me how much such scheme would help?

The extent of my coding for ZFS on FUSE was in this area. Solaris has a generic ioctl to flush the write cache on a block device but Linux does not. I wrote a few routines to detect the type of block device and flush the cache by talking to the hardware via an ioctl.

Tests with bonnie++ on my laptop showed that throughput and metadata operations per second were not noticeably affected by completely flushing the write cache when necessary versus never flushing the write cache or using any kind of IO barrier.

Caveats:
*Not every HDD is a laptop HDD.
*ZFS on FUSE got average to poor results for metadata operations per second since it hadn't been optimized for that yet.

Maybe fancier schemes aren't necessary?

Cheers,
Eric