Tux3 Report: How fast can we fsync?

Where does tux3 live? What I found looked abandoned.

Quoted 2 lines · Mike Galbraith

Where does tux3 live? What I found looked abandoned.

Current work is here:

https://github.com/OGAWAHirofumi/linux-tux3

Note, the new fsync code isn't pushed to that tree yet, however Hirofumi's optimized syncfs is already in there, which isn't a lot slower.

Quoted 1 line · Daniel Phillips

On Tuesday, April 28, 2015 7:21:11 PM PDT, Mike Galbraith wrote: ...

Where can I find the fsync code?
IOW how to reproduce your results? :)

Quoted 1 line · Mike Galbraith

Where does tux3 live? What I found looked abandoned.

Current work is here:

https://github.com/OGAWAHirofumi/linux-tux3

Note, the new fsync code isn't pushed to that tree yet, however Hirofumi's optimized syncfs is already in there, which isn't a lot slower.

Quoted 5 lines · Daniel Phillips

Quoted 1 line · Mike Galbraith

Where does tux3 live? What I found looked abandoned.

Current work is here:

Quoted 4 lines

https://github.com/OGAWAHirofumi/linux-tux3

Note, the new fsync code isn't pushed to that tree yet, however Hirofumi's optimized syncfs is already in there, which isn't a lot slower.

Ah, I did find the right spot, it's just been idle a while. Where does one find mkfs.tux3?

Quoted 6 lines · Mike Galbraith

Quoted 5 lines · Daniel Phillips

Quoted 1 line · Mike Galbraith

Where does tux3 live? What I found looked abandoned.

Current work is here:

Quoted 7 lines

Quoted 4 lines

https://github.com/OGAWAHirofumi/linux-tux3

Note, the new fsync code isn't pushed to that tree yet, however Hirofumi's optimized syncfs is already in there, which isn't a lot slower.

Ah, I did find the right spot, it's just been idle a while. Where does one find mkfs.tux3?

Hi Mike,

See my reply to Richard. You are right, we have been developing on Hirofumi's
branch and master is getting old. Short version:

checkout hirofumi-user
cd fs/tux3/user
make
./tux3 mkfs <volume>

Here's something that _might_ interest xfs folks.

cd git (source repository of git itself)
make clean
echo 3 > /proc/sys/vm/drop_caches
time make -j8 test

ext4    2m20.721s
xfs     6m41.887s <-- ick
btrfs   1m32.038s
tux3    1m30.262s

Testing by Aunt Tilly: mkfs, no fancy switches, mount the thing, test.

Are defaults for mkfs.xfs such that nobody sane uses them, or does xfs really hate whatever git selftests are doing this much?

-Mike

Here's something that _might_ interest xfs folks.

cd git (source repository of git itself)
make clean
echo 3 > /proc/sys/vm/drop_caches
time make -j8 test

ext4    2m20.721s
xfs     6m41.887s <-- ick
btrfs   1m32.038s
tux3    1m30.262s

Testing by Aunt Tilly: mkfs, no fancy switches, mount the thing, test.

Are defaults for mkfs.xfs such that nobody sane uses them, or does xfs really hate whatever git selftests are doing this much?

Btrfs appears to optimize tiny files by storing them in its big btree, the equivalent of our itree, and Tux3 doesn't do that yet, so we are a bit hobbled for a make load.

Here's something that _might_ interest xfs folks.

cd git (source repository of git itself)
make clean
echo 3 > /proc/sys/vm/drop_caches
time make -j8 test

ext4    2m20.721s
xfs     6m41.887s <-- ick
btrfs   1m32.038s
tux3    1m30.262s

Testing by Aunt Tilly: mkfs, no fancy switches, mount the thing, test.

Are defaults for mkfs.xfs such that nobody sane uses them, or does xfs really hate whatever git selftests are doing this much?

IOWS, XFS just hates your disk. Spend $50 and buy a cheap SSD and the problem goes away. :)

----

And now in more detail.

Quoted 7 lines

Here's something that _might_ interest xfs folks.

cd git (source repository of git itself)
make clean
echo 3 > /proc/sys/vm/drop_caches
time make -j8 test

Quoted 6 lines

ext4    2m20.721s
xfs     6m41.887s <-- ick
btrfs   1m32.038s
tux3    1m30.262s

Testing by Aunt Tilly: mkfs, no fancy switches, mount the thing, test.

TL;DR: Results are *very different* on a 256GB Samsung 840 EVO SSD with slightly slower CPUs (E5-4620 @ 2.20GHz)i, all filesystems using defaults:

	real		user		sys

xfs 3m16.138s 7m8.341s 14m32.462s
ext4 3m18.045s 7m7.840s 14m32.994s
btrfs 3m45.149s 7m10.184s 16m30.498s

What you are seeing is physical seek distances impacting read performance. XFS does not optimise for minimal physical seek distance, and hence is slower than filesytsems that do optimise for minimal seek distance. This shows up especially well on slow single spindles.

XFS is *adequate* for the use on slow single drives, but it is really designed for best performance on storage hardware that is not seek distance sensitive.

IOWS, XFS just hates your disk. Spend $50 and buy a cheap SSD and the problem goes away. :)

----

And now in more detail.

It's easy to be fast on empty filesystems. XFS does not aim to be fast in such situations - it aims to have consistent performance across the life of the filesystem.

Thing is, once you've abused those filesytsems for a couple of months, the files in ext4, btrfs and tux3 are not going to be laid out perfectly on the outer edge of the disk. They'll be spread all over the place and so all the filesystems will be seeing large seeks on read. The thing is, XFS will have roughly the same performance as when the filesystem is empty because the spreading of the allocation allows it to maintain better locality and separation and hence doesn't fragment free space nearly as badly as the oher filesystems. Free space fragmentation is what leads to performance degradation in filesystems, and all the other filesystem will have degraded to be *much worse* than XFS.

Put simply: empty filesystem benchmarking does not show the real performance of the filesystem under sustained production workloads. Hence benchmarks like this - while interesting from a theoretical point of view and are widely used for bragging about whose got the fastest - are mostly irrelevant to determining how the filesystem will perform in production environments.

We can also look at this algorithm in a different way: take a large filesystem (say a few hundred TB) across a few tens of disks in a linear concat. ext4, btrfs and tux3 will only hit the first disk in the concat, and so go no faster because they are still bound by physical seek times. XFS, however, will spread the load across many (if not all) of the disks, and so effectively reduce the average seek time by the number of disks doing concurrent IO. Then you'll see that application level IO concurrency becomes the performance limitation, not the physical seek time of the hardware.

Quoted 2 lines

IOWS, XFS just hates your disk. Spend $50 and buy a cheap SSD and the problem goes away. :)

I am quite surprised that a traditional filesystem that was created in the age of rotating media does not like this kind of media and even seems to excel on BTRFS on the new non rotating media available.

Quoted 11 lines

Thing is, once you've abused those filesytsems for a couple of months, the files in ext4, btrfs and tux3 are not going to be laid out perfectly on the outer edge of the disk. They'll be spread all over the place and so all the filesystems will be seeing large seeks on read. The thing is, XFS will have roughly the same performance as when the filesystem is empty because the spreading of the allocation allows it to maintain better locality and separation and hence doesn't fragment free space nearly as badly as the oher filesystems. Free space fragmentation is what leads to performance degradation in filesystems, and all the other filesystem will have degraded to be *much worse* than XFS.

That are the allocation groups. I always wondered how it can be beneficial to spread the allocations onto 4 areas of one partition on expensive seek media. Now that makes better sense for me. I always had the gut impression that XFS may not be the fastest in all cases, but it is one of the filesystem with the most consistent performance over time, but never was able to fully explain why that is.

This is one of the reasons why I find head-to-head "competitions" between file systems to be not very helpful for anything other than benchmarketing. It's almost certain that the benchmark won't be "fair" in some way, and it doesn't really matter whether the person doing the benchmark was doing it with malice aforethought, or was just incompetent and didn't understand the issues --- or did understand the issues and didn't really care, because what they _really_ wanted to do was to market their file system.

And even if the benchmark is fair, it might not match up with the end user's hardware, or their use case. There will always be some use case where file system A is better than file system B, for pretty much any file system. Don't get me wrong --- I will do comparisons between file systems, but only so I can figure out ways of making _my_ file system better. And more often than not, it's comparisons of the same file system before and after adding some new feature which is the most interesting.

Quoted 6 lines

That are the allocation groups. I always wondered how it can be beneficial to spread the allocations onto 4 areas of one partition on expensive seek media. Now that makes better sense for me. I always had the gut impression that XFS may not be the fastest in all cases, but it is one of the filesystem with the most consistent performance over time, but never was able to fully explain why that is.

Yep, pretty much all of the traditional update-in-place file systems since the BSD FFS have done this, and for the same reason. For COW file systems which are are constantly moving data and metadata blocks around, they will need different strategies for trying to avoid the free space fragmentation problem as the file system ages.

One of the problems is that it's *hard* to get good benchmarking numbers that take into account file system aging and measure how well the free space has been fragmented over time. Most of the benchmark results that I've seen do a really lousy job at this, and the vast majority don't even try.

This is one of the reasons why I find head-to-head "competitions" between file systems to be not very helpful for anything other than benchmarketing. It's almost certain that the benchmark won't be "fair" in some way, and it doesn't really matter whether the person doing the benchmark was doing it with malice aforethought, or was just incompetent and didn't understand the issues --- or did understand the issues and didn't really care, because what they _really_ wanted to do was to market their file system.

It's easy to be fast on empty filesystems. XFS does not aim to be fast in such situations - it aims to have consistent performance across the life of the filesystem.

In this case, ext4, btrfs and tux3 have optimal allocation filling from the outside of the disk, while XFS is spreading the files across (at least) 4 separate regions of the whole disk. Hence XFS is seeing seek times on read are much larger than the other filesystems when the filesystem is empty as it is doing full disk seeks rather than being confined to the outer edges of spindle.

Thing is, once you've abused those filesytsems for a couple of months, the files in ext4, btrfs and tux3 are not going to be laid out perfectly on the outer edge of the disk. They'll be spread all over the place and so all the filesystems will be seeing large seeks on read. The thing is, XFS will have roughly the same performance as when the filesystem is empty because the spreading of the allocation allows it to maintain better locality and separation and hence doesn't fragment free space nearly as badly as the oher filesystems. Free space fragmentation is what leads to performance degradation in filesystems, and all the other filesystem will have degraded to be *much worse* than XFS.

Put simply: empty filesystem benchmarking does not show the real performance of the filesystem under sustained production workloads. Hence benchmarks like this - while interesting from a theoretical point of view and are widely used for bragging about whose got the fastest - are mostly irrelevant to determining how the filesystem will perform in production environments.

We can also look at this algorithm in a different way: take a large filesystem (say a few hundred TB) across a few tens of disks in a linear concat. ext4, btrfs and tux3 will only hit the first disk in the concat, and so go no faster because they are still bound by physical seek times. XFS, however, will spread the load across many (if not all) of the disks, and so effectively reduce the average seek time by the number of disks doing concurrent IO. Then you'll see that application level IO concurrency becomes the performance limitation, not the physical seek time of the hardware.

IOWs, what you don't see here is that the XFS algorithms that make your test slow will keep *lots* of disks busy. i.e. testing empty filesystem performance a single, slow disk demonstrates that an algorithm designed for scalability isn't designed to acheive physical seek distance minimisation. Hence your storage makes XFS look particularly poor in comparison to filesystems that are being designed and optimised for the limitations of single slow spindles...

To further demonstrate that it is physical seek distance that is the issue here, lets take the seek time out of the equation (e.g. use a SSD). Doing that will result in basically no difference in performance between all 4 filesystems as performance will now be determined by application level concurrency and that is the same for all tests.

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

Quoted 8 lines · Mike Galbraith

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

This has nothing to do with big storage. The proposition was that seek time is the reason for Tux3's fsync performance. That claim was easily falsified by removing the seek time.

Dave's big storage words are there to draw attention away from the fact that XFS ran the Git tests four times slower than Tux3 and three times slower than Ext4. Whatever the big storage excuse is for that, the fact is, XFS obviously sucks at little storage.

Quoted 14 lines · Daniel Phillips

Quoted 8 lines · Mike Galbraith

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

This has nothing to do with big storage. The proposition was that seek time is the reason for Tux3's fsync performance. That claim was easily falsified by removing the seek time.

Quoted 4 lines

Dave's big storage words are there to draw attention away from the fact that XFS ran the Git tests four times slower than Tux3 and three times slower than Ext4. Whatever the big storage excuse is for that, the fact is, XFS obviously sucks at little storage.

If you allocate spanning the disk from start of life, you're going to eat seeks that others don't until later. That seemed rather obvious and straight forward.

He flat stated that xfs has passable performance on
single bit of rust, and openly explained why. I see no misdirection, only some evidence of bad blood between you two.

No, I won't be switching to xfs any time soon, but then it would take a hell of a lot of evidence to get me to move away from ext4. I trust ext[n] deeply because it has proven many times over the years that it can take one hell of a lot (of self inflicted wounds;).

Quoted 20 lines · Mike Galbraith

Quoted 18 lines · Daniel Phillips

Quoted 8 lines · Mike Galbraith

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

This has nothing to do with big storage. The proposition was that seek time is the reason for Tux3's fsync performance. That claim was easily falsified by removing the seek time.

Dave's big storage words are there to draw attention away from the fact that XFS ran the Git tests four times slower than Tux3 and three times slower than Ext4. Whatever the big storage excuse is for that, the fact is, XFS obviously sucks at little storage.

Quoted 3 lines

If you allocate spanning the disk from start of life, you're going to eat seeks that others don't until later. That seemed rather obvious and straight forward.

It is a logical falacy. It mixes a grain of truth (spreading all over the disk causes extra seeks) with an obvious falsehood (it is not necessarily the only possible way to avoid long term fragmentation).

Quoted 3 lines

He flat stated that xfs has passable performance on
single bit of rust, and openly explained why. I see no misdirection, only some evidence of bad blood between you two.

Raising the spectre of theoretical fragmentation issues when we have not even begun that work is a straw man and intellectually dishonest. You have to wonder why he does it. It is destructive to our community image and harmful to progress.

Quoted 23 lines · Daniel Phillips

Quoted 20 lines · Mike Galbraith

Quoted 18 lines · Daniel Phillips

Quoted 8 lines · Mike Galbraith

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

This has nothing to do with big storage. The proposition was that seek time is the reason for Tux3's fsync performance. That claim was easily falsified by removing the seek time.

Dave's big storage words are there to draw attention away from the fact that XFS ran the Git tests four times slower than Tux3 and three times slower than Ext4. Whatever the big storage excuse is for that, the fact is, XFS obviously sucks at little storage.

Quoted 7 lines

Quoted 3 lines

If you allocate spanning the disk from start of life, you're going to eat seeks that others don't until later. That seemed rather obvious and straight forward.

It is a logical falacy. It mixes a grain of truth (spreading all over the disk causes extra seeks) with an obvious falsehood (it is not necessarily the only possible way to avoid long term fragmentation).

You're reading into it what isn't there. Spreading over the disk isn't (just) about avoiding fragmentation - it's about delivering consistent and predictable latency. It is undeniable that if you start by only allocating from the fastest portion of the platter, you are going to see performance slow down over time. If you start by spreading allocations across the entire platter, you make the worst-case and average-case latency equal, which is exactly what a lot of folks are looking for.

Quoted 8 lines

Quoted 3 lines

He flat stated that xfs has passable performance on
single bit of rust, and openly explained why. I see no misdirection, only some evidence of bad blood between you two.

Raising the spectre of theoretical fragmentation issues when we have not even begun that work is a straw man and intellectually dishonest. You have to wonder why he does it. It is destructive to our community image and harmful to progress.

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

Quoted 6 lines · Howard Chu

You're reading into it what isn't there. Spreading over the disk isn't (just) about avoiding fragmentation - it's about delivering consistent and predictable latency. It is undeniable that if you start by only allocating from the fastest portion of the platter, you are going to see performance slow down over time. If you start by spreading allocations across the entire platter, you make the worst-case and average-case latency equal, which is exactly what a lot of folks are looking for.

Another fallacy: intentionally running slower than necessary is not necessarily the only way to deliver consistent and predictable latency.

Not only that, but
intentionally running slower than necessary does not necessarily guarantee performing better than some alternate strategy later.

Anyway, let's not be silly. Everybody in the room who wants Git to run 4 times slower with no guarantee of any benefit in the future, please raise your hand.

Quoted 9 lines

Quoted 8 lines

Quoted 3 lines

He flat stated that xfs has passable performance on
single bit of rust, and openly explained why. I see no misdirection, only some evidence of bad blood between you two.

Raising the spectre of theoretical fragmentation issues when we have not even begun that work is a straw man and intellectually dishonest. You have to wonder why he does it. It is destructive to our community image and harmful to progress.

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Now here is where you lose the bet: we already know that linear allocation with wrap ends horribly right? However, as above, we start linear, without compromise, but because of the gaps we leave, we are able to switch to a slower strategy, but not nearly as slow as the ugly tangle we get with simple wrap. So impact over the lifetime of the filesystem is positive, not negative, and what seemed to be self evident to you turns out to be wrong.

In short, we would rather deliver as much performance as possible, all the time. I really don't need to think about it very hard to know that is what I want, and what most users want.

I will make you a bet in return: when we get to doing that part properly, the quality of the work will be just as high as everything else we have completed so far. Why would we suddenly get lazy?

Quoted 26 lines

Quoted 19 lines · Daniel Phillips

Quoted 17 lines · Mike Galbraith

Quoted 16 lines · Daniel Phillips

Quoted 7 lines · Mike Galbraith

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

This has nothing to do with big storage. The proposition was that seek time is the reason for Tux3's fsync performance. That claim was easily falsified by removing the seek time.

Dave's big storage words are there to draw attention away from the fact that XFS ran the Git tests four times slower than Tux3 and three times slower than Ext4. Whatever the big storage excuse is for that, the fact is, XFS obviously sucks at little storage.

Quoted 6 lines

Quoted 3 lines

If you allocate spanning the disk from start of life, you're going to eat seeks that others don't until later. That seemed rather obvious and straight forward.

It is a logical falacy. It mixes a grain of truth (spreading all over the disk causes extra seeks) with an obvious falsehood (it is not necessarily the only possible way to avoid long term fragmentation).

Quoted 6 lines

You're reading into it what isn't there. Spreading over the disk isn't (just) about avoiding fragmentation - it's about delivering consistent and predictable latency. It is undeniable that if you start by only allocating from the fastest portion of the platter, you are going to see performance slow down over time. If you start by spreading allocations across the entire platter, you make the worst-case and average-case latency equal, which is exactly what a lot of folks are looking for.

Another fallacy: intentionally running slower than necessary is not necessarily the only way to deliver consistent and predictable latency. Not only that, but intentionally running slower than necessary does not necessarily guarantee performing better than some alternate strategy later.

Anyway, let's not be silly. Everybody in the room who wants Git to run 4 times slower with no guarantee of any benefit in the future, please raise your hand.

Quoted 11 lines

Quoted 7 lines

Quoted 3 lines

He flat stated that xfs has passable performance on
single bit of rust, and openly explained why. I see no misdirection, only some evidence of bad blood between you two.

Raising the spectre of theoretical fragmentation issues when we have not even begun that work is a straw man and intellectually dishonest. You have to wonder why he does it. It is destructive to our community image and harmful to progress.

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Now here is where you lose the bet: we already know that linear allocation with wrap ends horribly right? However, as above, we start linear, without compromise, but because of the gaps we leave, we are able to switch to a slower strategy, but not nearly as slow as the ugly tangle we get with simple wrap. So impact over the lifetime of the filesystem is positive, not negative, and what seemed to be self evident to you turns out to be wrong.

In short, we would rather deliver as much performance as possible, all the time. I really don't need to think about it very hard to know that is what I want, and what most users want.

I will make you a bet in return: when we get to doing that part properly, the quality of the work will be just as high as everything else we have completed so far. Why would we suddenly get lazy?

Regards,

Daniel
--

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

Quoted 3 lines · Pavel Machek

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

It's still true. The difference between O.D. and I.D. (outer diameter vs inner diameter) LBA's is typically a factor of 2. This is why "short-stroking" works as a technique,

and another way that people
doing competitive benchmarking can screw up and produce misleading numbers.

(If you use partitions instead of the whole disk, you have to use the same partition in order to make sure you aren't comparing apples with oranges.)

Anyway, everybody but you loves competitive benchmarks, that is why I

post them. They are not only useful for tracking down performance bugs, but as you point out, they help us advertise the reasons why Tux3 is interesting and ought to be merged.

I think Ted and I are on the same page here. "Competitive benchmarks" only matter to the people who are trying to sell something. You're trying to sell Tux3, but....

The "except page forking design" statement is your biggest hurdle for getting tux3 merged, not performance.

Without page forking, tux3
cannot be merged at all. But it's not filesystem developers you need to convince about the merits of the page forking design and implementation - it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

IOWs, you need to focus on the important things needed to acheive your stated goal of getting tux3 merged. New filesystems should be faster than those based on 20-25 year old designs, so you don't need to waste time trying to convince people that tux3, when complete, will be fast.

Quoted 3 lines · Dave Chinner

I think Ted and I are on the same page here. "Competitive benchmarks" only matter to the people who are trying to sell something. You're trying to sell Tux3, but....

By "same page", do you mean "transparently obvious about obstructing other projects"?

Quoted 2 lines

The "except page forking design" statement is your biggest hurdle for getting tux3 merged, not performance.

No, the "except page forking design" is because the design is already good and effective. The small adjustments needed in core are well worth merging because the benefits are proved by benchmarks. So benchmarks are key and will not stop just because you don't like the attention they bring to XFS issues.

Quoted 5 lines

Without page forking, tux3
cannot be merged at all. But it's not filesystem developers you need to convince about the merits of the page forking design and implementation - it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

Please do not say "we" when you know that I am just as much a "we" as you are. Merging Tux3 is not your decision. The people whose decision it actually is are perfectly capable of recognizing your agenda for what it is.

http://www.phoronix.com/scan.php?page=news_item&px=MTA0NzM
"XFS Developer Takes Shots At Btrfs, EXT4"

The real question is, has the Linux development process become so political and toxic that worthwhile projects fail to benefit from supposed grassroots community support. You are the poster child for that.

Quoted 5 lines

IOWs, you need to focus on the important things needed to acheive your stated goal of getting tux3 merged. New filesystems should be faster than those based on 20-25 year old designs, so you don't need to waste time trying to convince people that tux3, when complete, will be fast.

You know that Tux3 is already fast. Not just that of course. It has a higher standard of data integrity than your metadata-only journalling filesystem and a small enough code base that it can be reasonably expected to reach the quality expected of an enterprise class filesystem, quite possibly before XFS gets there.

Quoted 8 lines · Daniel Phillips

Quoted 2 lines

...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

Please do not say "we" when you know that I am just as much a "we" as you are. Merging Tux3 is not your decision. The people whose decision it actually is are perfectly capable of recognizing your agenda for what it is.

Quoted 2 lines

http://www.phoronix.com/scan.php?page=news_item&px=MTA0NzM
"XFS Developer Takes Shots At Btrfs, EXT4"

umm, Phoronix has no input on what gets merged into the kernel. they also hae a reputation for trying to turn anything into click-bait by making it sound like a fight when it isn't.

Quoted 4 lines

The real question is, has the Linux development process become so political and toxic that worthwhile projects fail to benefit from supposed grassroots community support. You are the poster child for that.

The linux development process is making code available, responding to concerns from the experts in the community, and letting the code talk for itself.

There have been many people pushing code for inclusion that has not gotten into the kernel, or has not been used by any distros after it's made it into the kernel, in spite of benchmarks being posted that seem to show how wonderful the new code is. ReiserFS was one of the first, and part of what tarnished it's reputation with many people was how much they were pushing the benchmarks that were shown to be faulty (the one I remember most vividly was that the entire benchmark completed in <30 seconds, and they had the FS tuned to not start flushing data to disk for 30 seconds, so the entire 'benchmark' ran out of ram without ever touching the disk)

So when Ted and Dave point out problems with the benchmark (the difference in behavior between a single spinning disk, different partitions on the same disk, SSDs, and ramdisks), you would be better off acknowledging them and if you can't adjust and re-run the benchmarks, don't start attacking them as a result.

As Dave says above, it's not the other filesystem people you have to convince, it's the core VFS and Memory Mangement folks you have to convince. You may need a little benchmarking to show that there is a real advantage to be gained, but the real discussion is going to be on the impact that page forking is going to have on everything else (both in complexity and in performance impact to other things)

Quoted 6 lines

You know that Tux3 is already fast. Not just that of course. It has a higher standard of data integrity than your metadata-only journalling filesystem and a small enough code base that it can be reasonably expected to reach the quality expected of an enterprise class filesystem, quite possibly before XFS gets there.

We wouldn't expect anyone developing a new filesystem to believe any differently.

If they didn't
believe this, why would they be working on the filesystem instead of just using an existing filesystem.

The ugly reality is that everyone's early versions of their new filesystem looks really good. The problem is when they extend it to cover the corner cases and when it gets stressed by real-world (as opposed to benchmark) workloads. This isn't saying that you are wrong in your belief, just that you may not be right, and nobody will know until you are to a usable state and other people can start beating on it.

Quoted 9 lines · David Lang

Quoted 8 lines · Daniel Phillips

Quoted 2 lines

...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

Please do not say "we" when you know that I am just as much a "we" as you are. Merging Tux3 is not your decision. The people whose decision it actually is are perfectly capable of recognizing your agenda for what it is.

Quoted 5 lines

Quoted 2 lines

http://www.phoronix.com/scan.php?page=news_item&px=MTA0NzM
"XFS Developer Takes Shots At Btrfs, EXT4"

umm, Phoronix has no input on what gets merged into the kernel. they also hae a reputation for trying to turn anything into click-bait by making it sound like a fight when it isn't.

Perhaps you misunderstood. Linus decides what gets merged. Andrew decides. Greg decides. Dave Chinner does not decide, he just does his level best to create the impression that our project is unfit to merge. Any chance there might be an agenda?

Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it.

Quoted 5 lines

Quoted 4 lines

The real question is, has the Linux development process become so political and toxic that worthwhile projects fail to benefit from supposed grassroots community support. You are the poster child for that.

Quoted 2 lines

The linux development process is making code available, responding to concerns from the experts in the community, and letting the code talk for itself.

Nice idea, but it isn't working. Did you let the code talk to you? Right, you let the code talk to Dave Chinner, then you listen to what Dave Chinner has to say about it. Any chance that there might be some creative licence acting somewhere in that chain?

Quoted 7 lines

There have been many people pushing code for inclusion that has not gotten into the kernel, or has not been used by any distros after it's made it into the kernel, in spite of benchmarks being posted that seem to show how wonderful the new code is. ReiserFS was one of the first, and part of what tarnished it's reputation with many people was how much they were pushing the benchmarks that were shown to be faulty (the one I remember most vividly was that the entire benchmark completed in <30 seconds, and they had the FS tuned to not start flushing data to disk for 30 seconds, so the entire 'benchmark' ran out of ram without ever touching the disk)

You know what to do about checking for faulty benchmarks.

Quoted 4 lines

So when Ted and Dave point out problems with the benchmark (the difference in behavior between a single spinning disk, different partitions on the same disk, SSDs, and ramdisks), you would be better off acknowledging them and if you can't adjust and re-run the benchmarks, don't start attacking them as a result.

Ted and Dave failed to point out any actual problem with any benchmark. They invented issues with benchmarks and promoted those as FUD.

Quoted 5 lines

As Dave says above, it's not the other filesystem people you have to convince, it's the core VFS and Memory Mangement folks you have to convince. You may need a little benchmarking to show that there is a real advantage to be gained, but the real discussion is going to be on the impact that page forking is going to have on everything else (both in complexity and in performance impact to other things)

Yet he clearly wrote "we" as if he believes he is part of it.

Now that ENOSPC is done to a standard way beyond what Btrfs had when it was merged, the next item on the agenda is writeback. That involves us and VFS people as you say, and not Dave Chinner, who only intends to obstruct the process as much as he possibly can. He should get back to work on his own project. Nobody will miss his posts if he doesn't make them. They contribute nothing of value, create a lot of bad blood, and just serve to further besmirch the famously tarnished reputation of LKML.

Quoted 8 lines

Quoted 6 lines

You know that Tux3 is already fast. Not just that of course. It has a higher standard of data integrity than your metadata-only journalling filesystem and a small enough code base that it can be reasonably expected to reach the quality expected of an enterprise class filesystem, quite possibly before XFS gets there.

We wouldn't expect anyone developing a new filesystem to believe any differently.

It is not a matter of belief, it is a matter of testable fact. For example, you can count the lines. You can run the same benchmarks.

Proving the data consistency claims would be a little harder, you need tools for that, and some of those aren't built yet. Or, if you have technical ability, you can read the code and the copious design material that has been posted and convince yourself that, yes, there is something cool here, why didn't anybody do it that way before? But of course that starts to sound like work. Debating nontechnical issues and playing politics seems so much more like fun.

Quoted 2 lines

If they didn't
believe this, why would they be working on the filesystem instead of just using an existing filesystem.

Right, and it is my job to convince you that what I believe for perfectly valid, demonstrable technical reasons, is really true. I do not see why you feel it is your job to convince me that the obviously broken Linux community process is not in fact broken, and that a certain person who obviously has an agenda, is not actually obstructing.

Quoted 5 lines

The ugly reality is that everyone's early versions of their new filesystem looks really good. The problem is when they extend it to cover the corner cases and when it gets stressed by real-world (as opposed to benchmark) workloads. This isn't saying that you are wrong in your belief, just that you may not be right, and nobody will know until you are to a usable state and other people can start beating on it.

With ENOSPC we are at that state. Tux3 would get more testing and advance faster if it was merged. Things like ifdefs, grandiose new schemes for writeback infrastructure, dumb little hooks in the mkwrite path, those are all just manufactured red herrings. Somebody wanted those to be issues, so now they are issues. Fake ones.

Nobody is trying to trick you. Just stating a fact. You ought to be able to figure out by now that Tux3 is worth merging.

You might possibly have an argument that merging a filesystem that crashes as soon as it fills the disk is just sheer stupidity than can only lead to embarrassment in the long run, but then you would need to explain why Btrfs was merged. As I recall, it went something like, Chris had it on a laptop, so it must be a filesystem, and wow look at that feature list. Then it got merged in a completely unusable state and got worked on. If it had not been merged, Btrfs would most likely be dead right now. After all, who cares about an out of tree filesystem?

By the way, I gave my Tux3 presentation at SCALE 7x in Los Angeles in 2009, with Tux3 running as my root filesystem. By the standard applied to Btrfs, Tux3 should have been merged then, right? After all, our nospace handling worked just as well as theirs at that time.

Regards,

Daniel

Quoted 3 lines · Daniel Phillips

Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it.

Phoronix turns any correction or criticism into an attack.

You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies.

They assume that you are working in good faith (but are
inexperienced compared to them), and you need to assume that they are working in good faith. If they ever do resort to underhanded means to sabotage you, Linus and the other kernel developers will take action. But pointing out limits in your current implementation, problems in your benchmarks based on how they are run, and concepts that are going to be difficult to merge is not underhanded, it's exactly the type of assistance that you should be greatful for in friendly competition.

You were the one who started crowing about how badly XFS performed.

explination about the reasons for the differences, and showing benchmarks on other hardware that showed that XFS works very well there. That's not an attack on EXT4 (or Tux3), it's an explination.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

The real question is, has the Linux development process become so political and toxic that worthwhile projects fail to benefit from supposed grassroots community support. You are the poster child for that.

Quoted 7 lines

Quoted 2 lines

The linux development process is making code available, responding to concerns from the experts in the community, and letting the code talk for itself.

Nice idea, but it isn't working. Did you let the code talk to you? Right, you let the code talk to Dave Chinner, then you listen to what Dave Chinner has to say about it. Any chance that there might be some creative licence acting somewhere in that chain?

I have my own concerns about how things are going to work (I've voiced some of them), but no, I haven't tried running Tux3 because you say it's not ready yet.

Quoted 9 lines

Quoted 7 lines

There have been many people pushing code for inclusion that has not gotten into the kernel, or has not been used by any distros after it's made it into the kernel, in spite of benchmarks being posted that seem to show how wonderful the new code is. ReiserFS was one of the first, and part of what tarnished it's reputation with many people was how much they were pushing the benchmarks that were shown to be faulty (the one I remember most vividly was that the entire benchmark completed in <30 seconds, and they had the FS tuned to not start flushing data to disk for 30 seconds, so the entire 'benchmark' ran out of ram without ever touching the disk)

You know what to do about checking for faulty benchmarks.

That requires that the code be readily available, which last I heard, Tux3 wasn't. Has this been fixed?

Quoted 8 lines

Quoted 4 lines

So when Ted and Dave point out problems with the benchmark (the difference in behavior between a single spinning disk, different partitions on the same disk, SSDs, and ramdisks), you would be better off acknowledging them and if you can't adjust and re-run the benchmarks, don't start attacking them as a result.

Ted and Dave failed to point out any actual problem with any benchmark. They invented issues with benchmarks and promoted those as FUD.

They pointed out problems with using ramdisk to simulate a SSD and huge differences between spinning rust and an SSD (or disk array). Those aren't FUD.

Quoted 7 lines

Quoted 5 lines

As Dave says above, it's not the other filesystem people you have to convince, it's the core VFS and Memory Mangement folks you have to convince. You may need a little benchmarking to show that there is a real advantage to be gained, but the real discussion is going to be on the impact that page forking is going to have on everything else (both in complexity and in performance impact to other things)

Yet he clearly wrote "we" as if he believes he is part of it.

He is part of the group of people who use and work with this stuff, so he is part of it.

Quoted 8 lines

Now that ENOSPC is done to a standard way beyond what Btrfs had when it was merged, the next item on the agenda is writeback. That involves us and VFS people as you say, and not Dave Chinner, who only intends to obstruct the process as much as he possibly can. He should get back to work on his own project. Nobody will miss his posts if he doesn't make them. They contribute nothing of value, create a lot of bad blood, and just serve to further besmirch the famously tarnished reputation of LKML.

BTRFS is a perfect example of how not to introduce a new filesystem. Lots of hype, the presumption that is is going to replace all the existing filesystems because it's so much better (especially according to benchmarks). But then progress stalled before it was really ready, and it's still something most people avoid.

Quoted 12 lines

Quoted 8 lines

Quoted 6 lines

You know that Tux3 is already fast. Not just that of course. It has a higher standard of data integrity than your metadata-only journalling filesystem and a small enough code base that it can be reasonably expected to reach the quality expected of an enterprise class filesystem, quite possibly before XFS gets there.

We wouldn't expect anyone developing a new filesystem to believe any differently.

It is not a matter of belief, it is a matter of testable fact. For example, you can count the lines. You can run the same benchmarks.

Quoted 7 lines

Proving the data consistency claims would be a little harder, you need tools for that, and some of those aren't built yet. Or, if you have technical ability, you can read the code and the copious design material that has been posted and convince yourself that, yes, there is something cool here, why didn't anybody do it that way before? But of course that starts to sound like work. Debating nontechnical issues and playing politics seems so much more like fun.

why are you picking a fight? there was no attack in my statement?

Quoted 9 lines

Quoted 3 lines

If they didn't
believe this, why would they be working on the filesystem instead of just using an existing filesystem.

Right, and it is my job to convince you that what I believe for perfectly valid, demonstrable technical reasons, is really true. I do not see why you feel it is your job to convince me that the obviously broken Linux community process is not in fact broken, and that a certain person who obviously has an agenda, is not actually obstructing.

You will need to have a fully working, usable system before you can convince people that you are right. A partial system may look good, but how much is fixing the corner cases that you haven't gotten to yet going to hurt it? That there are going to be such cases is pretty much a given, and that changing things to add code to work around the pathalogical conditions is going to hurt the common case is pretty close to a given (it's one of those things that isn't mathamatically guaranteed, but happens on 99.99999+% of projects)

Quoted 11 lines

Quoted 5 lines

The ugly reality is that everyone's early versions of their new filesystem looks really good. The problem is when they extend it to cover the corner cases and when it gets stressed by real-world (as opposed to benchmark) workloads. This isn't saying that you are wrong in your belief, just that you may not be right, and nobody will know until you are to a usable state and other people can start beating on it.

With ENOSPC we are at that state. Tux3 would get more testing and advance faster if it was merged. Things like ifdefs, grandiose new schemes for writeback infrastructure, dumb little hooks in the mkwrite path, those are all just manufactured red herrings. Somebody wanted those to be issues, so now they are issues. Fake ones.

Ok, so you are happy with your allocation strategy? you didn't seem to be a few e-mail ago.

but if you think it's ready for users, then start working to submit it in the next merge window. Dave said that except for one part, there was no reason not to merge it. That's pretty good. So you need to be discussing that one part with the the folks that Dave pointed you at.

Quoted 11 lines

Nobody is trying to trick you. Just stating a fact. You ought to be able to figure out by now that Tux3 is worth merging.

You might possibly have an argument that merging a filesystem that crashes as soon as it fills the disk is just sheer stupidity than can only lead to embarrassment in the long run, but then you would need to explain why Btrfs was merged. As I recall, it went something like, Chris had it on a laptop, so it must be a filesystem, and wow look at that feature list. Then it got merged in a completely unusable state and got worked on. If it had not been merged, Btrfs would most likely be dead right now. After all, who cares about an out of tree filesystem?

As I said above, Btrfs is a perfect example of how not to do things.

The other think you need to realize is that getting something in the kernel isn't a one-time effort, the code needs to be maintained over time (especially for a filesystem), and it's very possible for a developer/team/company to be so toxic and hostile to others that the Linux folks don't want to deal with the hassle of dealing with them. You are starting out on a path to put yourself into that category. Calm down and stop taking offense at everything. Your succeeding doesn't require that other people loose, so stop talking as if it's a zero sum game and you have to beat down the enemy to get your code accepted.

David Lang

Quoted 9 lines

By the way, I gave my Tux3 presentation at SCALE 7x in Los Angeles in 2009, with Tux3 running as my root filesystem. By the standard applied to Btrfs, Tux3 should have been merged then, right? After all, our nospace handling worked just as well as theirs at that time.

Regards,

Daniel

Quoted 6 lines · David Lang

Quoted 3 lines · Daniel Phillips

Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it.

Phoronix turns any correction or criticism into an attack.

Phoronix gets attacked in an unseemly way by a number of people in the developer community who should behave better. You are doing it yourself, seemingly oblivious to the valuable role that the publication plays in our community. Google for filesystem benchmarks. Where do you find them? Right. Not to mention the Xorg coverage, community issues, etc etc. The last thing we need is a monoculture in Linux news, and we are dangerously close to that now.

So, how is "EXT4 is not as stable or as well tested as most people think" not a cheap shot? By my first hand experience, that claim is absurd. Add to that the first hand experience of roughly two billion other people. Seems to be a bit self serving too, or was that just an accident.

Quoted 2 lines

You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies.

You are wrong about Dave These are not the words of any friend:

"I don't think I'm alone in my suspicion that there was something stinky about your numbers." -- Dave Chinner

Basically allegations of cheating. And wrong. Maybe Dave just lives in his own dreamworld where everybody is out to get him, so he has to attack people he views as competitors first.

I happen to think that it's correct. It's not that Ext4 isn't tested, but that people's expectations of how much it's been tested, and at what scale don't match the reality.

Quoted 4 lines · Daniel Phillips

Quoted 2 lines · David Lang

You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies.

Quoted 4 lines

You are wrong about Dave These are not the words of any friend:

"I don't think I'm alone in my suspicion that there was something stinky about your numbers." -- Dave Chinner

you are looking for offense. That just means that something is wrong with them, not that they were deliberatly falsified.

Quoted 3 lines

Basically allegations of cheating. And wrong. Maybe Dave just lives in his own dreamworld where everybody is out to get him, so he has to attack people he views as competitors first.

you are the one doing the attacking.

Please stop. Take a break if needed, and then get back to
producing software rather than complaining about how everyone is out to get you.

Quoted 11 lines · Daniel Phillips

Quoted 7 lines · David Lang

Quoted 2 lines

You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies.

You are wrong about Dave These are not the words of any friend:
"I don't think I'm alone in my suspicion that there was something stinky about your numbers." -- Dave Chinner

Quoted 2 lines

you are looking for offense. That just means that something is wrong with them, not that they were deliberatly falsified.

I am not mistaken. Dave made sure to eliminate any doubt about what he meant. He said "Oh, so nicely contrived. But terribly obvious now that I've found it" among other things.

Daniel, what are you trying to achieve here?

I thought you wanted to create interest for your filesystem and acceptance for merging it.

What I see you are actually creating tough is something different.

Is what you see after you send your mails really what you want to see? If not… why not? And if you seek change, where can you create change?

I really like to see Tux3 inside the kernel for easier testing, yet I also see that the way you, in your oppinion, "defend" it, does not seem to move that goal any closer, quite the opposite. It triggers polarity and resistance.

I believe it to be more productive to work together with the people who will decide about what goes into the kernel and the people whose oppinions are respected by them, instead of against them.

"Assume good faith" can help here. No amount of accusing people of bad intention will change them. The only thing you have the power to change is your approach. You absolutely and ultimately do not have the power to change other people. You can´t force Tux3 in by sheer willpower or attacking people.

On any account for anyone discussing here: I believe that any personal attacks, counter-attacks or "you are wrong" kind of speech will not help to move this discussion out of the circling it seems to be in at the moment.

Quoted 9 lines · Martin Steigerwald

"Assume good faith" can help here. No amount of accusing people of bad intention will change them. The only thing you have the power to change is your approach. You absolutely and ultimately do not have the power to change other people. You can´t force Tux3 in by sheer willpower or attacking people.

On any account for anyone discussing here: I believe that any personal attacks, counter-attacks or "you are wrong" kind of speech will not help to move this discussion out of the circling it seems to be in at the moment.

Thanks for the sane commentary. I have the power to change my behavior. But if nobody else changes their behavior, the process remains just as unpleasant for us as it ever was (not just me!). Obviously, this is not the first time I have been through this, and it has never been pleasant. After a while, contributors just get tired of the grind and move on to something more fun. I know I did, and I am far from the only one.

Quoted 2 lines · Daniel Phillips

On 05/13/2015 12:09 PM, Martin Steigerwald wrote:
...

Daniel, please listen to Martin. He speaks a fundamental truth here.

As you know, I am also interested in Tux3, and would love to see it as a filesystem option for NAS servers using Samba. But please think about the way you're interacting with people on the list, and whether that makes this outcome more or less likely.

Quoted 10 lines

Daniel, what are you trying to achieve here?

I thought you wanted to create interest for your filesystem and acceptance for merging it.

What I see you are actually creating tough is something different.

Is what you see after you send your mails really what you want to see? If
not… why not? And if you seek change, where can you create change?

That is the question indeed, whether to try and change the system while merging, or just keep smiling and get the job done. The problem is, I am just too stupid to realize that I can't change the system, which is famously unpleasant for submitters.

Quoted 8 lines

I really like to see Tux3 inside the kernel for easier testing, yet I also see that the way you, in your oppinion, "defend" it, does not seem to move that goal any closer, quite the opposite. It triggers polarity and resistance.

I believe it to be more productive to work together with the people who will decide about what goes into the kernel and the people whose oppinions are respected by them, instead of against them.

Obviously true.

Quoted 10 lines

"Assume good faith" can help here. No amount of accusing people of bad intention will change them. The only thing you have the power to change is your approach. You absolutely and ultimately do not have the power to change other people. You can´t force Tux3 in by sheer willpower or attacking people.

On any account for anyone discussing here: I believe that any personal attacks, counter-attacks or "you are wrong" kind of speech will not help to move this discussion out of the circling it seems to be in at the moment.

Thanks for the sane commentary. I have the power to change my behavior. But if nobody else changes their behavior, the process remains just as unpleasant for us as it ever was (not just me!). Obviously, this is not the first time I have been through this, and it has never been pleasant. After a while, contributors just get tired of the grind and move on to something more fun. I know I did, and I am far from the only one.

Am Mittwoch, 13. Mai 2015, 12:37:41 schrieb Daniel Phillips:

Quoted 1 line

On 05/13/2015 12:09 PM, Martin Steigerwald wrote: ...

Daniel, if you want to change the process of patch review and inclusion into the kernel, model an example of how you would like it to be. This has way better chances to inspire others to change their behaviors themselves than accusing them of bad faith.

Its yours to choose.

What outcome do you want to create?

Quoted 10 lines

Am Mittwoch, 13. Mai 2015, 12:37:41 schrieb Daniel Phillips:

Quoted 1 line

On 05/13/2015 12:09 PM, Martin Steigerwald wrote: ...

Daniel, if you want to change the process of patch review and inclusion into
the kernel, model an example of how you would like it to be. This has way
better chances to inspire others to change their behaviors themselves than accusing them of bad faith.

Quoted 3 lines

Its yours to choose.

What outcome do you want to create?

The outcome I would like is:

* Everybody has a good think about what has gone wrong in the past, not only with troublesome submitters, but with mutual respect and collegial conduct.

* Tux3 is merged on its merits so we get more developers and testers and move it along faster.

* I left LKML better than I found it.

* Group hugs

Well, group hugs are optional, that one would be situational.

Quoted 3 lines · Daniel Phillips

Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it.

Phoronix turns any correction or criticism into an attack.

You need to get out of the mindset that Ted and Dave are Enemies that you need to overcome, they are friendly competitors, not Enemies.

They assume that you are working in good faith (but are
inexperienced compared to them), and you need to assume that they are working in good faith. If they ever do resort to underhanded means to sabotage you, Linus and the other kernel developers will take action. But pointing out limits in your current implementation, problems in your benchmarks based on how they are run, and concepts that are going to be difficult to merge is not underhanded, it's exactly the type of assistance that you should be greatful for in friendly competition.

You were the one who started crowing about how badly XFS performed.

Dave gave a long and detailed explination about the reasons for the differences, and showing

showed that XFS works very well there. That's not an attack on EXT4 (or Tux3), it's an explination.

I have my own concerns about how things are going to work (I've voiced some of them), but no, I haven't tried running Tux3 because you say it's not ready yet.

Quoted 1 line

You know what to do about checking for faulty benchmarks.

That requires that the code be readily available, which last I heard, Tux3 wasn't. Has this been fixed?

They pointed out problems with using ramdisk to simulate a SSD and huge differences between spinning rust and an SSD (or disk array). Those aren't FUD.

Quoted 7 lines

Quoted 5 lines

As Dave says above, it's not the other filesystem people you have to convince, it's the core VFS and Memory Mangement folks you have to convince. You may need a little benchmarking to show that there is a real advantage to be gained, but the real discussion is going to be on the impact that page forking is going to have on everything else (both in complexity and in performance impact to other things)

Yet he clearly wrote "we" as if he believes he is part of it.

He is part of the group of people who use and work with this stuff, so he is part of it.

BTRFS is a perfect example of how not to introduce a new filesystem. Lots of hype, the presumption that is is going to replace all the existing filesystems because it's so much better (especially according to benchmarks). But then progress stalled before it was really ready, and it's still something most people avoid.

Quoted 7 lines

Proving the data consistency claims would be a little harder, you need tools for that, and some of those aren't built yet. Or, if you have technical ability, you can read the code and the copious design material that has been posted and convince yourself that, yes, there is something cool here, why didn't anybody do it that way before? But of course that starts to sound like work. Debating nontechnical issues and playing politics seems so much more like fun.

why are you picking a fight? there was no attack in my statement?

Quoted 4 lines

Quoted 3 lines

If they didn't
believe this, why would they be working on the filesystem instead of just using an existing filesystem.

Quoted 5 lines

Right, and it is my job to convince you that what I believe for perfectly valid, demonstrable technical reasons, is really true. I do not see why you feel it is your job to convince me that the obviously broken Linux community process is not in fact broken, and that a certain person who obviously has an agenda, is not actually obstructing.

You will need to have a fully working, usable system before you can convince people that you are right.

gotten to yet going to hurt it?

that changing things to add code to work around the pathalogical conditions is going to hurt the common case is pretty close to a given (it's one of those things that isn't mathamatically guaranteed, but happens on 99.99999+% of projects)

Quoted 6 lines

Quoted 5 lines

The ugly reality is that everyone's early versions of their new filesystem looks really good. The problem is when they extend it to cover the corner cases and when it gets stressed by real-world (as opposed to benchmark) workloads. This isn't saying that you are wrong in your belief, just that you may not be right, and nobody will know until you are to a usable state and other people can start beating on it.

Quoted 5 lines

With ENOSPC we are at that state. Tux3 would get more testing and advance faster if it was merged. Things like ifdefs, grandiose new schemes for writeback infrastructure, dumb little hooks in the mkwrite path, those are all just manufactured red herrings. Somebody wanted those to be issues, so now they are issues. Fake ones.

Ok, so you are happy with your allocation strategy? you didn't seem to be a few e-mail ago.

but if you think it's ready for users, then start working to submit it in the next merge window.

Dave said that except for one part, there was no reason not to merge it. That's pretty good. So you need to be discussing that one part with the the folks that Dave pointed you at.

As I said above, Btrfs is a perfect example of how not to do things.

The other think you need to realize is that getting something in the kernel isn't a one-time effort, the code needs to be maintained over time (especially for a filesystem), and it's very possible for a developer/team/company to be so toxic and hostile to others that the Linux folks don't want to deal with the hassle of dealing with them. You are starting out on a path to put yourself into that category. Calm down and stop taking offense at everything. Your succeeding doesn't require that other people loose, so stop talking as if it's a zero sum game and you have to beat down the enemy to get your code accepted.

Quoted 8 lines · David Lang

Quoted 7 lines · Daniel Phillips

Quoted 2 lines

...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

Please do not say "we" when you know that I am just as much a "we" as you are. Merging Tux3 is not your decision. The people whose decision it actually is are perfectly capable of recognizing your agenda for what it is.

Quoted 4 lines

Quoted 2 lines

http://www.phoronix.com/scan.php?page=news_item&px=MTA0NzM
"XFS Developer Takes Shots At Btrfs, EXT4"

umm, Phoronix has no input on what gets merged into the kernel. they also hae a reputation for trying to turn anything into click-bait by making it sound like a fight when it isn't.

Perhaps you misunderstood. Linus decides what gets merged. Andrew decides. Greg decides. Dave Chinner does not decide, he just does his level best to create the impression that our project is unfit to merge. Any chance there might be an agenda?

Phoronix published a headline that identifies Dave Chinner as someone who takes shots at other projects. Seems pretty much on the money to me, and it ought to be obvious why he does it.

Quoted 6 lines

Quoted 4 lines

The real question is, has the Linux development process become so political and toxic that worthwhile projects fail to benefit from supposed grassroots community support. You are the poster child for that.

The linux development process is making code available, responding to concerns from the experts in the community, and letting the code talk for itself.

Nice idea, but it isn't working. Did you let the code talk to you? Right, you let the code talk to Dave Chinner, then you listen to what Dave Chinner has to say about it. Any chance that there might be some creative licence acting somewhere in that chain?

Quoted 7 lines

There have been many people pushing code for inclusion that has not gotten into the kernel, or has not been used by any distros after it's made it into the kernel, in spite of benchmarks being posted that seem to show how wonderful the new code is. ReiserFS was one of the first, and part of what tarnished it's reputation with many people was how much they were pushing the benchmarks that were shown to be faulty (the one I remember most vividly was that the entire benchmark completed in<30 seconds, and they had the FS tuned to not start flushing data to disk for 30 seconds, so the entire 'benchmark' ran out of ram without ever touching the disk)

You know what to do about checking for faulty benchmarks.

Quoted 4 lines

So when Ted and Dave point out problems with the benchmark (the difference in behavior between a single spinning disk, different partitions on the same disk, SSDs, and ramdisks), you would be better off acknowledging them and if you can't adjust and re-run the benchmarks, don't start attacking them as a result.

Ted and Dave failed to point out any actual problem with any benchmark. They invented issues with benchmarks and promoted those as FUD.

Quoted 5 lines

As Dave says above, it's not the other filesystem people you have to convince, it's the core VFS and Memory Mangement folks you have to convince. You may need a little benchmarking to show that there is a real advantage to be gained, but the real discussion is going to be on the impact that page forking is going to have on everything else (both in complexity and in performance impact to other things)

Yet he clearly wrote "we" as if he believes he is part of it.

Now that ENOSPC is done to a standard way beyond what Btrfs had when it was merged, the next item on the agenda is writeback. That involves us and VFS people as you say, and not Dave Chinner, who only intends to obstruct the process as much as he possibly can. He should get back to work on his own project. Nobody will miss his posts if he doesn't make them. They contribute nothing of value, create a lot of bad blood, and just serve to further besmirch the famously tarnished reputation of LKML.

Quoted 7 lines

Quoted 6 lines

You know that Tux3 is already fast. Not just that of course. It has a higher standard of data integrity than your metadata-only journalling filesystem and a small enough code base that it can be reasonably expected to reach the quality expected of an enterprise class filesystem, quite possibly before XFS gets there.

We wouldn't expect anyone developing a new filesystem to believe any differently.

It is not a matter of belief, it is a matter of testable fact. For example, you can count the lines. You can run the same benchmarks.

Proving the data consistency claims would be a little harder, you need tools for that, and some of those aren't built yet. Or, if you have technical ability, you can read the code and the copious design material that has been posted and convince yourself that, yes, there is something cool here, why didn't anybody do it that way before? But of course that starts to sound like work. Debating nontechnical issues and playing politics seems so much more like fun.

Quoted 2 lines

If they didn't
believe this, why would they be working on the filesystem instead of just using an existing filesystem.

Right, and it is my job to convince you that what I believe for perfectly valid, demonstrable technical reasons, is really true. I do not see why you feel it is your job to convince me that the obviously broken Linux community process is not in fact broken, and that a certain person who obviously has an agenda, is not actually obstructing.

Quoted 5 lines

The ugly reality is that everyone's early versions of their new filesystem looks really good. The problem is when they extend it to cover the corner cases and when it gets stressed by real-world (as opposed to benchmark) workloads. This isn't saying that you are wrong in your belief, just that you may not be right, and nobody will know until you are to a usable state and other people can start beating on it.

With ENOSPC we are at that state. Tux3 would get more testing and advance faster if it was merged. Things like ifdefs, grandiose new schemes for writeback infrastructure, dumb little hooks in the mkwrite path, those are all just manufactured red herrings. Somebody wanted those to be issues, so now they are issues. Fake ones.

Nobody is trying to trick you. Just stating a fact. You ought to be able to figure out by now that Tux3 is worth merging.

You might possibly have an argument that merging a filesystem that crashes as soon as it fills the disk is just sheer stupidity than can only lead to embarrassment in the long run, but then you would need to explain why Btrfs was merged. As I recall, it went something like, Chris had it on a laptop, so it must be a filesystem, and wow look at that feature list. Then it got merged in a completely unusable state and got worked on. If it had not been merged, Btrfs would most likely be dead right now. After all, who cares about an out of tree filesystem?

By the way, I gave my Tux3 presentation at SCALE 7x in Los Angeles in 2009, with Tux3 running as my root filesystem. By the standard applied to Btrfs, Tux3 should have been merged then, right? After all, our nospace handling worked just as well as theirs at that time.

Regards,

Daniel

Quoted 12 lines · David Lang

Quoted 10 lines · Daniel Phillips

Quoted 2 lines

...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

Please do not say "we" when you know that I am just as much a "we" as you are. Merging Tux3 is not your decision. The people whose decision it actually is are perfectly capable of recognizing your agenda for what it is.

http://www.phoronix.com/scan.php?page=news_item&px=MTA0NzM
"XFS Developer Takes Shots At Btrfs, EXT4"

Quoted 2 lines

umm, Phoronix has no input on what gets merged into the kernel. they also hae a reputation for trying to turn anything into click-bait by making it sound like a fight when it isn't.

Perhaps you misunderstood. Linus decides what gets merged. Andrew decides. Greg decides. Dave Chinner does not decide, he just does his level best to create the impression that our project is unfit to merge. Any chance there might be an agenda?

Quoted 5 lines

As Dave says above, it's not the other filesystem people you have to convince, it's the core VFS and Memory Mangement folks you have to convince. You may need a little benchmarking to show that there is a real advantage to be gained, but the real discussion is going to be on the impact that page forking is going to have on everything else (both in complexity and in performance impact to other things)

Yet he clearly wrote "we" as if he believes he is part of it.

Now that ENOSPC is done to a standard way beyond what Btrfs had when it was merged, the next item on the agenda is writeback. That involves us and VFS people as you say, and not Dave Chinner, who only intends to obstruct the process as much as he possibly can. He

Quoted 10 lines · Daniel Phillips

Quoted 9 lines · David Lang

Quoted 8 lines · Daniel Phillips

Quoted 2 lines

...it's the mm and core kernel developers that need to review and accept that code *before* we can consider merging tux3.

Please do not say "we" when you know that I am just as much a "we" as you are. Merging Tux3 is not your decision. The people whose decision it actually is are perfectly capable of recognizing your agenda for what it is.

Quoted 4 lines

Quoted 4 lines

Quoted 2 lines

http://www.phoronix.com/scan.php?page=news_item&px=MTA0NzM
"XFS Developer Takes Shots At Btrfs, EXT4"

umm, Phoronix has no input on what gets merged into the kernel. they

also hae a reputation for

Quoted 1 line

Quoted 1 line

trying to turn anything into click-bait by making it sound like a

fight when it isn't.

Quoted 5 lines

Perhaps you misunderstood. Linus decides what gets merged. Andrew decides. Greg decides. Dave Chinner does not decide, he just does his level best to create the impression that our project is unfit to merge. Any chance there might be an agenda?

Dunno. _Your_ agenda seems to be "attack other maintainers so much that you can later claim they are biased".

Not going to work, sorry.

Quoted 1 line

Quoted 1 line

As Dave says above, it's not the other filesystem people you have to

convince, it's the core VFS and

Quoted 1 line

Quoted 1 line

Memory Mangement folks you have to convince. You may need a little

benchmarking to show that there

Quoted 1 line

Quoted 1 line

is a real advantage to be gained, but the real discussion is going to

be on the impact that page

Quoted 1 line

Quoted 1 line

forking is going to have on everything else (both in complexity and in

performance impact to other

Quoted 8 lines

Quoted 1 line

things)

Yet he clearly wrote "we" as if he believes he is part of it.

Now that ENOSPC is done to a standard way beyond what Btrfs had when it was merged, the next item on the agenda is writeback. That involves us and VFS people as you say, and not Dave Chinner, who only intends to obstruct the process as much as he possibly can. He

Why would he do that? Aha, maybe because you keep attacking him all the time. Or maybe because your code is not up to the kernel standards. You want to claim it is the former, but it really looks like the latter.

Just stop doing that. You are not creating nice atmosphere and you are not getting tux3 being merged in any way.

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures)
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

_______________________________________________
Tux3 mailing list
[email protected]
http://phunq.net/mailman/listinfo/tux3

Quoted 8 lines · Theodore Ts'o

Quoted 3 lines · Pavel Machek

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

It's still true. The difference between O.D. and I.D. (outer diameter vs inner diameter) LBA's is typically a factor of 2. This is why "short-stroking" works as a technique,

That is true, and the effect is not dominant compared to introducing a lot of extra seeks.

Quoted 3 lines

and another way that people
doing competitive benchmarking can screw up and produce misleading numbers.

If you think we screwed up or produced misleading numbers, could you please be up front about it instead of making insinuations and continuing your tirade against benchmarking and those who do it.

The ram disk removes seek overhead and greatly reduces media transfer overhead. This does not change things much: it confirms that Tux3 is significantly faster than the others at synchronous loads. This is apparently true independently of media type, though to be sure SSD remains to be tested.

The really interesting result is how much difference there is between filesystems, even on a ram disk. Is it just CPU or is it synchronization strategy and lock contention? Does our asynchronous front/back design actually help a lot, instead of being a disadvantage as you predicted?

It is too bad that fs_mark caps number of tasks at 64, because I am sure that some embarrassing behavior would emerge at high task counts, as with my tests on spinning disk.

Quoted 8 lines · Daniel Phillips

Quoted 3 lines · Theodore Ts'o

and another way that people
doing competitive benchmarking can screw up and produce misleading numbers.

If you think we screwed up or produced misleading numbers, could you please be up front about it instead of making insinuations and continuing your tirade against benchmarking and those who do it.

Are not you little harsh with Ted? He was polite.

Quoted 11 lines

The ram disk removes seek overhead and greatly reduces media transfer overhead. This does not change things much: it confirms that Tux3 is significantly faster than the others at synchronous loads. This is apparently true independently of media type, though to be sure SSD remains to be tested.

The really interesting result is how much difference there is between filesystems, even on a ram disk. Is it just CPU or is it synchronization strategy and lock contention? Does our asynchronous front/back design actually help a lot, instead of being a disadvantage as you predicted?

Quoted 3 lines

It is too bad that fs_mark caps number of tasks at 64, because I am sure that some embarrassing behavior would emerge at high task counts, as with my tests on spinning disk.

I'd call system with 65 tasks doing heavy fsync load at the some time "embarrassingly misconfigured" :-). It is nice if your filesystem can stay fast in that case, but...

Quoted 3 lines · Pavel Machek

I'd call system with 65 tasks doing heavy fsync load at the some time "embarrassingly misconfigured" :-). It is nice if your filesystem can stay fast in that case, but...

Well, Tux3 wins the fsync race now whether it is 1 task, 64 tasks or 10,000 tasks. At the high end, maybe it is just a curiosity, or maybe it tells us something about how Tux3 is will scale on the big machines that XFS currently lays claim to. And Java programmers are busy doing all kinds of wild and crazy things with lots of tasks. Java almost makes them do it. If they need their data durable then they can easily create loads like my test case.

Suppose you have a web server meant to serve 10,000 transactions simultaneously and it needs to survive crashes without losing client state. How will you do it? You could install an expensive, finicky database, or you could write some Java code that happens to work well because Linux has a scheduler and a filesystem that can handle it. Oh wait, we don't have the second one yet, but maybe we soon will.

I will not claim that stupidly fast and scalable fsync is the main reason that somebody should want Tux3, however, the lack of a high performance fsync was in fact used as a means of spreading FUD about Tux3, so I had some fun going way beyond the call of duty to answer that. By the way, I am still waiting for the original source of the FUD to concede the point politely, but maybe he is waiting for the code to land, which it still has not as of today, so I guess that is fair. Note that it would have landed quite some time ago if Tux3 was already merged.

Quoted 10 lines

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

Anyway... you have brand new filesystem. Of course it should be faster/better/nicer than the existing filesystems. So don't be too harsh with XFS people.

Quoted 11 lines · Pavel Machek

Quoted 10 lines

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Quoted 3 lines

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

That's why I hedged my claim with "similar or identical". The difference in media speed seems to be a relatively small effect compared to extra seeks. It seems that XFS puts big spaces between new directories, and suffers a lot of extra seeks because of it. I propose to batch new directories together initially, then change the allocation goal to a new, relatively empty area if a big batch of files lands on a directory in a crowded region. The "big" gaps would be on the order of delta size, so not really very big.

Quoted 16 lines · Daniel Phillips

Quoted 11 lines · Pavel Machek

Quoted 10 lines

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Quoted 3 lines

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

Quoted 8 lines

That's why I hedged my claim with "similar or identical". The difference in media speed seems to be a relatively small effect compared to extra seeks. It seems that XFS puts big spaces between new directories, and suffers a lot of extra seeks because of it. I propose to batch new directories together initially, then change the allocation goal to a new, relatively empty area if a big batch of files lands on a directory in a crowded region. The "big" gaps would be on the order of delta size, so not really very big.

This is an interesting idea, but what happens if the files don't arrive as a big batch, but rather trickle in over time (think a logserver that if putting files into a bunch of directories at a fairly modest rate per directory)

And when you then decide that you have to move the directory/file info, doesn't that create a potentially large amount of unexpected IO that could end up interfering with what the user is trying to do?

Hi David,

Quoted 23 lines · David Lang

Quoted 13 lines · Daniel Phillips

Quoted 4 lines · Pavel Machek

Quoted 4 lines

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

Quoted 8 lines

Quoted 5 lines

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

Quoted 8 lines

That's why I hedged my claim with "similar or identical". The difference in media speed seems to be a relatively small effect compared to extra seeks. It seems that XFS puts big spaces between new directories, and suffers a lot of extra seeks because of it. I propose to batch new directories together initially, then change the allocation goal to a new, relatively empty area if a big batch of files lands on a directory in a crowded region. The "big" gaps would be on the order of delta size, so not really very big.

Quoted 3 lines

This is an interesting idea, but what happens if the files don't arrive as a big batch, but rather trickle in over time (think a logserver that if putting files into a bunch of directories at a fairly modest rate per directory)

If files are trickling in then we can afford to spend a lot more time finding nice places to tuck them in. Log server files are an especially irksome problem for a redirect-on-write filesystem because the final block tends to be rewritten many times and we must move it to a new location each time, so every extent ends up as one block. Oh well. If we just make sure to have some free space at the end of the file that only that file can use (until everywhere else is full) then the long term result will be slightly ravelled blocks that nonetheless tend to be on the same track or flash block as their logically contiguous neighbours. There will be just zero or one empty data blocks mixed into the file tail as we commit the tail block over and over with the same allocation goal. Sometimes there will be a block or two of metadata as well, which will eventually bake themselves into the middle of contiguous data and stop moving around.

Putting this together, we have:

* At delta flush, break out all the log type files
* Dedicate some block groups to append type files
* Leave lots of space between files in those block groups
* Peek at the last block of the file to set the allocation goal

Something like that. What we don't want is to throw those files into the middle of a lot of rewrite-all files, messing up both kinds of file. We don't care much about keeping these files near the parent directory because one big seek per log file in a grep is acceptable, we just need to avoid thousands of big seeks within the file, and not dribble single blocks all over the disk.

It would also be nice to merge together extents somehow as the final block is rewritten. One idea is to retain the final block dirty until the next delta, and write it again into a contiguous position, so the final block is always flushed twice. We already have the opportunistic merge logic, but the redirty behavior and making sure it only happens to log files would be a bit fiddly.

We will also play the incremental defragmentation card at some point, but first we should try hard to control fragmentation in the first place. Tux3 is well suited to online defragmentation because the delta commit model makes it easy to move things around efficiently and safely, but it does generate extra IO, so as a basic mechanism it is not ideal. When we get to piling on features, that will be high on the list, because it is relatively easy, and having that fallback gives a certain sense of security.

Quoted 3 lines

And when you then decide that you have to move the directory/file info, doesn't that create a potentially large amount of unexpected IO that could end up interfering with what the user is trying to do?

Right, we don't like that and don't plan to rely on it. What we hope for is behavior that, when you slowly stir the pot, tends to improve the layout just as often as it degrades it. It may indeed become harder to find ideal places to put things as time goes by, but we also gain more information to base decisions on.

Regards,

Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Pavel,

Quoted 11 lines · Pavel Machek

Quoted 10 lines

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Quoted 3 lines

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

That's why I hedged my claim with "similar or identical". The difference in media speed seems to be a relatively small effect

Quoted 14 lines · Daniel Phillips

Hi Pavel,

Quoted 11 lines · Pavel Machek

Quoted 10 lines

Quoted 4 lines

It is a fact of life that when you change one aspect of an intimately interconnected system, something else will change as well. You have naive/nonexistent free space management now; when you design something workable there it is going to impact everything else you've already done. It's an easy bet that the impact will be negative, the only question is to what degree.

You might lose that bet. For example, suppose we do strictly linear allocation each delta, and just leave nice big gaps between the deltas for future expansion. Clearly, we run at similar or identical speed to the current naive strategy until we must start filling in the gaps, and at that point our layout is not any worse than XFS, which started bad and stayed that way.

Quoted 6 lines

Quoted 3 lines

Umm, are you sure. If "some areas of disk are faster than others" is still true on todays harddrives, the gaps will decrease the performance (as you'll "use up" the fast areas more quickly).

That's why I hedged my claim with "similar or identical". The difference in media speed seems to be a relatively small effect

When you knew it can't be identical? That's rather confusing, right?

Perhaps you should post more details how your benchmark is structured next time, so we can see you did not make any trivial mistakes...?

Or just clean the code up so that it can get merged, so that we can benchmark ourselves...

Let's say our delta size is 400MB (typical under load) and we leave a "nice big gap" of 112 MB after flushing each one. Let's say we do two thousand of those before deciding that we have enough information available to switch to some smarter strategy. We used one GB of a a 4TB disk, say. The media transfer rate decreased by a factor of:

    (1 - 2/1000) = .2%.

The performance deficit in question and the difference in media rate are three orders of magnitude apart, does that justify the term "similar or identical?".

Third possibility: build from our repository, as Mike did.

Quoted 1 line · Daniel Phillips

Third possibility: build from our repository, as Mike did.

Sorry about that folks. I've lost all interest, it won't happen again.

Quoted 20 lines · Mike Galbraith

Quoted 18 lines · Daniel Phillips

Quoted 8 lines · Mike Galbraith

Quoted 2 lines · Daniel Phillips

Lovely sounding argument, but it is wrong because Tux3 still beats XFS even with seek time factored out of the equation.

Hm. Do you have big-storage comparison numbers to back that? I'm no storage guy (waiting for holographic crystal arrays to obsolete all this crap;), but Dave's big-storage guy words made sense to me.

This has nothing to do with big storage. The proposition was that seek time is the reason for Tux3's fsync performance. That claim was easily falsified by removing the seek time.

Dave's big storage words are there to draw attention away from the fact that XFS ran the Git tests four times slower than Tux3 and three times slower than Ext4. Whatever the big storage excuse is for that, the fact is, XFS obviously sucks at little storage.

Quoted 3 lines

If you allocate spanning the disk from start of life, you're going to eat seeks that others don't until later. That seemed rather obvious and straight forward.

It is a logical falacy. It mixes a grain of truth (spreading all over the disk causes extra seeks) with an obvious falsehood (it is not necessarily the only possible way to avoid long term fragmentation).

Quoted 3 lines

He flat stated that xfs has passable performance on
single bit of rust, and openly explained why. I see no misdirection, only some evidence of bad blood between you two.

Raising the spectre of theoretical fragmentation issues when we have not even begun that work is a straw man and intellectually dishonest. You have to wonder why he does it. It is destructive to our community image and harmful to progress.

Well ok, let's forget bad blood, straw men... and answering my question too I suppose. Not having any sexy IO gizmos in my little desktop box, I don't care deeply which stomps the other flat on beastly boxen.

[dbench bakeoff]

With dbench v4.00, tux3 seems to be king of the max_latency hill, but btrfs took throughput on my box. With v3.04, tux3 took 1st place at splashing about in pagecache, but last place at dbench -S.

Hohum, curiosity satisfied.

Quoted 7 lines · Mike Galbraith

[dbench bakeoff]

With dbench v4.00, tux3 seems to be king of the max_latency hill, but btrfs took throughput on my box. With v3.04, tux3 took 1st place at splashing about in pagecache, but last place at dbench -S.

Quoted 1 line

Hohum, curiosity satisfied.

Thanks for that. Please keep in mind, that was our B team, it does a full fs sync for every fsync. Maybe a rematch when the shiny new one lands? Also, hardware? It looks like a single 7200 RPM disk, but it would be nice to know. And it seems, not all dbench 4.0 are equal. Mine doesn't have a -B option.

Yeah, I also want to know hardware. Also, what size of partition? And each test was done by fresh FS (i.e. after mkfs), or same FS was used through all tests?

Quoted 8 lines · Mike Galbraith

[dbench bakeoff]

With dbench v4.00, tux3 seems to be king of the max_latency hill, but btrfs took throughput on my box. With v3.04, tux3 took 1st place at splashing about in pagecache, but last place at dbench -S.

Hohum, curiosity satisfied.

Hi Mike,

Thanks for that. Please keep in mind, that was our B team, it does a full fs sync for every fsync. Maybe a rematch when the shiny new one lands? Also, hardware? It looks like a single 7200 RPM disk, but it would be nice to know. And it seems, not all dbench 4.0 are equal. Mine doesn't have a -B option.

That order of magnitude latency difference is striking. It sounds good, but what does it mean? I see a smaller difference here, maybe because of running under KVM.

Quoted 4 lines · Daniel Phillips

That order of magnitude latency difference is striking. It sounds good, but what does it mean? I see a smaller difference here, maybe because of running under KVM.

That max_latency thing is flush.

Greetings,

This post is dedicated to Ted, who raised doubts a while back about whether Tux3 can ever have a fast fsync:

https://lkml.org/lkml/2013/5/11/128
"Re: Tux3 Report: Faster than tmpfs, what?"

I measured fsync performance using a 7200 RPM disk as a virtual drive under KVM, configured with cache=none so that asynchronous writes are cached and synchronous writes translate into direct writes to the block device.

To focus purely on fsync, I wrote a
small utility (at the end of this post) that forks a number of tasks, each of which continuously appends to and fsyncs its own file. For a single task doing 1,000 fsyncs of 1K each, we have:

   Ext4:  34.34s
   XFS:   23.63s
   Btrfs: 34.84s
   Tux3:  17.24s

Things get more interesting with parallel fsyncs. In this test, each task does ten fsyncs and task count scales from ten to ten thousand. We see that all tested filesystems are able to combine fsyncs into group commits, with varying degrees of success:

   Tasks:   10      100    1,000    10,000
   Ext4:   0.79s   0.98s    4.62s    61.45s
   XFS:    0.75s   1.68s   20.97s   238.23s
   Btrfs   0.53s   0.78s    3.80s    84.34s
   Tux3:   0.27s   0.34s    1.00s     6.86s

Is there any practical use for fast parallel fsync of tens of thousands of tasks? This could be useful for a scalable transaction server that sits directly on the filesystem instead of a database, as is the fashion for big data these days. It certainly can't hurt to know that if you need that kind of scaling, Tux3 will do it.

Of course, a pure fsync load could be viewed as somewhat unnatural. We also need to know what happens under a realistic load with buffered operations mixed with fsyncs. We turn to an old friend, dbench:

Dbench -t10

   Tasks:       8           16           32
   Ext4:    35.32 MB/s   34.08 MB/s   39.71 MB/s
   XFS:     32.12 MB/s   25.08 MB/s   30.12 MB/s
   Btrfs:   54.40 MB/s   75.09 MB/s  102.81 MB/s
   Tux3:    85.82 MB/s  133.69 MB/s  159.78 MB/s
                  (higher is better)

Dbench -t10 -s (all file operations synchronous)

   Tasks:       8           16           32
   Ext4:     4.51 MB/s    6.25 MB/s    7.72 MB/s
   XFS:      4.24 MB/s    4.77 MB/s    5.15 MB/s
   Btrfs:    7.98 MB/s   13.87 MB/s   22.87 MB/s
   Tux3:    15.41 MB/s   25.56 MB/s   39.15 MB/s
                  (higher is better)

In the full disclosure department, Tux3 is still not properly optimized in some areas. One of them is fragmentation: it is not very hard to make Tux3 slow down by running long tests. Our current

Quoted 4 lines

I measured fsync performance using a 7200 RPM disk as a virtual drive under KVM, configured with cache=none so that asynchronous writes are cached and synchronous writes translate into direct writes to the block device.

Yup, a slow single spindle, so fsync performance is determined by seek latency of the filesystem. Hence the filesystem that "wins" will be the filesystem that minimises fsync seek latency above all other considerations.

http://www.spinics.net/lists/kernel/msg1978216.html

So, to demonstrate, I'll run the same tests but using a 256GB samsung 840 EVO SSD and show how much the picture changes.

I didn't test tux3, you don't make it easy to get or build.

Quoted 9 lines

To focus purely on fsync, I wrote a
small utility (at the end of this post) that forks a number of tasks, each of which continuously appends to and fsyncs its own file. For a single task doing 1,000 fsyncs of 1K each, we have:

   Ext4:  34.34s
   XFS:   23.63s
   Btrfs: 34.84s
   Tux3:  17.24s

   Ext4:   1.94s
   XFS:    2.06s
   Btrfs:  2.06s

All equally fast, so I can't see how tux3 would be much faster here.

   Tasks:   10      100    1,000    10,000
   Ext4:   0.05s   0.12s    0.48s     3.99s
   XFS:    0.25s   0.41s    0.96s     4.07s
   Btrfs   0.22s   0.50s    2.86s   161.04s
             (lower is better)

Ext4 and XFS are fast and show similar performance. Tux3 *can't* be very much faster as most of the elapsed time in the test is from forking the processes that do the IO and fsyncs.

FWIW, btrfs shows it's horrible fsync implementation here, burning huge amounts of CPU to do bugger all IO. i.e. it burnt all 16p for 2 and a half minutes in that 10000 fork test so wasn't IO bound at all.

Quoted 5 lines

Is there any practical use for fast parallel fsync of tens of thousands of tasks? This could be useful for a scalable transaction server that sits directly on the filesystem instead of a database, as is the fashion for big data these days. It certainly can't hurt to know that if you need that kind of scaling, Tux3 will do it.

Ext4 and XFS already do that just fine, too, when you use storage suited to such a workload and you have a sane interface for submitting tens of thousands of concurrent fsync operations. e.g

http://oss.sgi.com/archives/xfs/2014-06/msg00214.html

On a SSD (256GB samsung 840 EVO), running 4.0.0:

   Tasks:       8           16           32
   Ext4:    598.27 MB/s    981.13 MB/s 1233.77 MB/s
   XFS:     884.62 MB/s   1328.21 MB/s 1373.66 MB/s
   Btrfs:   201.64 MB/s    137.55 MB/s  108.56 MB/s

dbench looks *very different* when there is no seek latency, doesn't it?

Quoted 8 lines

Dbench -t10 -s (all file operations synchronous)

   Tasks:       8           16           32
   Ext4:     4.51 MB/s    6.25 MB/s    7.72 MB/s
   XFS:      4.24 MB/s    4.77 MB/s    5.15 MB/s
   Btrfs:    7.98 MB/s   13.87 MB/s   22.87 MB/s
   Tux3:    15.41 MB/s   25.56 MB/s   39.15 MB/s
                  (higher is better)

    Ext4:   173.54 MB/s  294.41 MB/s  424.11 MB/s
    XFS:    172.98 MB/s  342.78 MB/s  458.87 MB/s
    Btrfs:   36.92 MB/s   34.52 MB/s   55.19 MB/s

Again, the numbers are completely the other way around on a SSD, with the conventional filesystems being 5-10x faster than the WA/COW style filesystem.

Quoted 3 lines

In the full disclosure department, Tux3 is still not properly optimized in some areas. One of them is fragmentation: it is not very hard to make Tux3 slow down by running long tests. Our current

Oh, that still hasn't been fixed?

Until you sort of how you are going to scale allocation to tens of TB and not fragment free space over time, fsync performance of the filesystem is pretty much irrelevant. Changing the allocation algorithms will fundamentally alter the IO patterns and so all these benchmarks are essentially meaningless.

Quoted 2 lines · Daniel Phillips

It looks like Btrfs hit a bug, not a huge surprise. Btrfs hit an assert for me earlier this evening. It is rare but it happens.

Hi Daniel,

Would you mind reporting (to [email protected]) the bug/assertion you hit during your tests with btrfs?

Quoted 9 lines · Filipe Manana

Quoted 2 lines · Daniel Phillips

It looks like Btrfs hit a bug, not a huge surprise. Btrfs hit an assert for me earlier this evening. It is rare but it happens.

Hi Daniel,

Would you mind reporting (to [email protected]) the bug/assertion you hit during your tests with btrfs?

Kernel 3.19.0 under KVM with BTRFS mounted on a file in /tmp, see the KVM command below. I believe I was running the 10,000 task test using the "sync" program below: "syncs foo 10 10000".

    346 ------------[ cut here ]------------
    347 kernel BUG at fs/btrfs/extent_io.c:4548!
    348 invalid opcode: 0000 [#1] PREEMPT SMP
    349 Modules linked in:
    350 CPU: 2 PID: 5754 Comm: sync6 Not tainted 3.19.0-56544-g65cf1a5 #756
    351 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
    352 task: ec3c0ea0 ti: ec3ea000 task.ti: ec3ea000
    353 EIP: 0060:[<c1301a30>] EFLAGS: 00010202 CPU: 2
    354 EIP is at btrfs_release_extent_buffer_page+0xf0/0x100
    355 EAX: 00000001 EBX: f47198f0 ECX: 00000000 EDX: 00000001
    356 ESI: f47198f0 EDI: f61f1808 EBP: ec3ebbac ESP: ec3ebb9c
    357  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
    358 CR0: 8005003b CR2: b756a356 CR3: 2c3ce000 CR4: 000006d0
    359 Stack:
    360  00000005 f47198f0 f61f1000 f61f1808 ec3ebbc0 c1301a7f f47198f0 00000000
    361  f6a3d940 ec3ebbcc c1301ee5 d9a6c770 ec3ebbdc c12b436d fff92000 da136b20
    362  ec3ebc74 c12e42b6 00000c00 00000000 00000000 00001000 00000000 00000000
    363 Call Trace:
    364  [<c1301a7f>] release_extent_buffer+0x3f/0xb0
    365  [<c1301ee5>] free_extent_buffer+0x45/0x80
    366  [<c12b436d>] btrfs_release_path+0x2d/0x90
    367  [<c12e42b6>] cow_file_range_inline+0x466/0x600
    368  [<c12e495e>] cow_file_range+0x50e/0x640
    369  [<c12fdde1>] ? find_lock_delalloc_range.constprop.42+0x2e1/0x320
    370  [<c12e5af9>] run_delalloc_range+0x419/0x450
    371  [<c12fdf6b>] writepage_delalloc.isra.32+0x14b/0x1d0
    372  [<c12ff20e>] __extent_writepage+0xde/0x2b0
    373  [<c11208fd>] ? find_get_pages_tag+0xad/0x120
    374  [<c130135c>] extent_writepages+0x29c/0x350
    375  [<c12e1530>] ? btrfs_direct_IO+0x300/0x300
    376  [<c12e009f>] btrfs_writepages+0x1f/0x30
    377  [<c11299e5>] do_writepages+0x15/0x40
    378  [<c112199f>] __filemap_fdatawrite_range+0x4f/0x60
    379  [<c1121aa2>] filemap_fdatawrite_range+0x22/0x30
    380  [<c12f4768>] btrfs_fdatawrite_range+0x28/0x70
    381  [<c12f47d1>] start_ordered_ops+0x21/0x30
    382  [<c12f4823>] btrfs_sync_file+0x43/0x370
    383  [<c115c3e5>] ? vfs_write+0x135/0x1c0
    384  [<c12f47e0>] ? start_ordered_ops+0x30/0x30
    385  [<c1183e27>] do_fsync+0x47/0x70
    386  [<c118403d>] SyS_fsync+0xd/0x10
    387  [<c15bd8ae>] syscall_call+0x7/0x7
    388 Code: 8b 03 f6 c4 20 75 26 f0 80 63 01 f7 c7 43 1c 00 00 00 00 89 d8 e8 61 94 e2 ff eb c3 8d
b4 26 00 00 00 00 83 c4 04 5b 5e 5f 5d c3 <0f> 0b 0f 0b     388 0f 0b 0f 0b 90 8d b4 26 00 00 00 00
55 89 e5 57 56
    389 EIP: [<c1301a30>] btrfs_release_extent_buffer_page+0xf0/0x100 SS:ESP 0068:ec3ebb9c
    390 ---[ end trace 12b9bbe75d9541a3 ]---

KVM command:

mkfs.btrfs -f /tmp/disk.img && kvm -kernel /src/linux-tux3/arch/x86/boot/bzImage -append "root=/dev/sda1 console=ttyS0 console=tty0 oops=panic tux3.tux3_trace=0" -serial file:serial.txt -hda /more/kvm/hdd.img -hdb /tmp/disk.img -net nic -net user,hostfwd=tcp::1234-:22 -smp 4 -m 2000

Source code:

/*
* syncs.c
*
* D.R. Phillips, 2015
*
* To build: c99 -Wall syncs.c -o syncs
* To run: ./syncs [<filename> [<syncs> [<tasks>]]] */

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/wait.h>
#include <errno.h>
#include <sys/stat.h>

char text[1024] = { "hello world!\n" };

int main(int argc, const char *argv[]) {
        const char *basename = argc < 1 ? "foo" : argv[1];
        char name[100];
        int steps = argc < 3 ? 1 : atoi(argv[2]);
        int tasks = argc < 4 ? 1 : atoi(argv[3]);
        int err, fd;

        for (int t = 0; t < tasks; t++) {
                snprintf(name, sizeof name, "%s%i", basename, t);
                if (!fork())
                        goto child;
        }
        for (int t = 0; t < tasks; t++)
                wait(&err);
        return 0;

child:

        fd = creat(name, S_IRWXU);
        for (int i = 0; i < steps; i++) {
                write(fd, text, sizeof text);
                fsync(fd);
        }
        return 0;
}

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

There's a very recent patch to fix that issue:
https://patchwork.kernel.org/patch/6261421/
It's not really specific to fsync.

Thanks for reporting it.

Quoted 10 lines · Dave Chinner

Quoted 4 lines

I measured fsync performance using a 7200 RPM disk as a virtual drive under KVM, configured with cache=none so that asynchronous writes are cached and synchronous writes translate into direct writes to the block device.

Yup, a slow single spindle, so fsync performance is determined by seek latency of the filesystem. Hence the filesystem that "wins" will be the filesystem that minimises fsync seek latency above all other considerations.

Quoted 1 line

http://www.spinics.net/lists/kernel/msg1978216.html

If you want to declare that XFS only works well on solid state disks and big storage arrays, that is your business. But if you do, you can no longer call XFS a general purpose filesystem. And

Quoted 2 lines

So, to demonstrate, I'll run the same tests but using a 256GB samsung 840 EVO SSD and show how much the picture changes.

I will go you one better, I ran a series of fsync tests using tmpfs, and I now have a very clear picture of how the picture changes. The executive summary is: Tux3 is still way faster, and still scales way better to large numbers of tasks. I have every confidence that the same is true of SSD.

Quoted 1 line

I didn't test tux3, you don't make it easy to get or build.

There is no need to apologize for not testing Tux3, however, it is unseemly to throw mud at the same time. Remember, you are the

Quoted 4 lines

Quoted 4 lines

To focus purely on fsync, I wrote a
small utility (at the end of this post) that forks a number of tasks, each of which continuously appends to and fsyncs its own file. For a single task doing 1,000 fsyncs of 1K each, we have:

Quoted 1 line

All equally fast, so I can't see how tux3 would be much faster here.

Running the same thing on tmpfs, Tux3 is significantly faster:

    Ext4:   1.40s
    XFS:    1.10s
    Btrfs:  1.56s
    Tux3:   1.07s

Quoted 9 lines

Tasks: 10 100 1,000 10,000

  Ext4:   0.05s   0.12s    0.48s     3.99s
  XFS:    0.25s   0.41s    0.96s     4.07s
  Btrfs   0.22s   0.50s    2.86s   161.04s

(lower is better)

Ext4 and XFS are fast and show similar performance. Tux3 *can't* be very much faster as most of the elapsed time in the test is from forking the processes that do the IO and fsyncs.

You wish. In fact, Tux3 is a lot faster.

triple checked and reproducible:

   Tasks:   10      100    1,000    10,000
   Ext4:   0.05     0.14    1.53     26.56
   XFS:    0.05     0.16    2.10     29.76
   Btrfs:  0.08     0.37    3.18     34.54
   Tux3:   0.02     0.05    0.18      2.16

Note: you should recheck your final number for Btrfs. I have seen Btrfs fall off the rails and take wildly longer on some tests just like that.

One easily reproducible one is a denial of service
during the 10,000 task test where it takes multiple seconds to cat small files. I saw XFS do this on both spinning disk and tmpfs, and I have seen it hang for minutes trying to list a directory. I looked a bit into it, and I see that you are blocking for aeons trying to acquire a lock in open.

You and I both know the truth: Ext4 is the only really reliable general purpose filesystem on Linux at the moment.

Quoted 9 lines

On a SSD (256GB samsung 840 EVO), running 4.0.0:

Tasks: 8 16 32

  Ext4:    598.27 MB/s    981.13 MB/s 1233.77 MB/s
  XFS:     884.62 MB/s   1328.21 MB/s 1373.66 MB/s
  Btrfs:   201.64 MB/s    137.55 MB/s  108.56 MB/s

dbench looks *very different* when there is no seek latency, doesn't it?

It looks like Btrfs hit a bug, not a huge surprise. Btrfs hit an assert for me earlier this evening. It is rare but it happens. I rebooted and got sane numbers. Running dbench -t10 on tmpfs I get:

    Tasks:       8            16            32
    Ext4:    660.69 MB/s   708.81 MB/s   720.12 MB/s
    XFS:     692.01 MB/s   388.53 MB/s   134.84 MB/s
    Btrfs:   229.66 MB/s   341.27 MB/s   377.97 MB/s
    Tux3:   1147.12 MB/s  1401.61 MB/s  1283.74 MB/s

Looks like XFS hit a bump and fell off the cliff at 32 threads. I reran that one many times because I don't want to give you an inaccurate report.

Quoted 9 lines

Quoted 8 lines

Dbench -t10 -s (all file operations synchronous)

Tasks: 8 16 32

  Ext4:     4.51 MB/s    6.25 MB/s    7.72 MB/s
  XFS:      4.24 MB/s    4.77 MB/s    5.15 MB/s
  Btrfs:    7.98 MB/s   13.87 MB/s   22.87 MB/s

Tux3: 15.41 MB/s 25.56 MB/s 39.15 MB/s
(higher is better)

Quoted 7 lines

   Ext4:   173.54 MB/s  294.41 MB/s  424.11 MB/s
   XFS:    172.98 MB/s  342.78 MB/s  458.87 MB/s

Btrfs: 36.92 MB/s 34.52 MB/s 55.19 MB/s

Again, the numbers are completely the other way around on a SSD, with the conventional filesystems being 5-10x faster than the WA/COW style filesystem.

I wouldn't be so sure about that...

    Tasks:       8            16            32
    Ext4:     93.06 MB/s    98.67 MB/s   102.16 MB/s
    XFS:      81.10 MB/s    79.66 MB/s    73.27 MB/s
    Btrfs:    43.77 MB/s    64.81 MB/s    90.35 MB/s
    Tux3:    198.49 MB/s   279.00 MB/s   318.41 MB/s

Quoted 5 lines

Until you sort of how you are going to scale allocation to tens of TB and not fragment free space over time, fsync performance of the filesystem is pretty much irrelevant. Changing the allocation algorithms will fundamentally alter the IO patterns and so all these benchmarks are essentially meaningless.

Ahem, are you the same person for whom fsync was the most important issue in the world last time the topic came up, to the extent of spreading around FUD and entirely ignoring the great work we had accomplished for regular file operations?

I said then that when we
got around to a proper fsync it would be competitive. Now here it is, so you want to change the topic. I understand.

Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the domain of high performance storage...

Quoted 8 lines

Quoted 2 lines

So, to demonstrate, I'll run the same tests but using a 256GB samsung 840 EVO SSD and show how much the picture changes.

I will go you one better, I ran a series of fsync tests using tmpfs, and I now have a very clear picture of how the picture changes. The executive summary is: Tux3 is still way faster, and still scales way better to large numbers of tasks. I have every confidence that the same is true of SSD.

/dev/ramX can't be compared to an SSD. Yes, they both have low seek/IO latency but they have very different dispatch and IO concurrency models. One is synchronous, the other is fully asynchronous.

This is an important distinction, as we'll see later on....

These trees:

git://git.kernel.org/pub/scm/linux/kernel/git/daniel/linux-tux3.git
git://git.kernel.org/pub/scm/linux/kernel/git/daniel/linux-tux3-test.git

have not been updated for 11 months. I thought tux3 had died long ago.

You should keep them up to date, and send patches for xfstests to support tux3, and then you'll get a lot more people running, testing and breaking tux3....

Quoted 6 lines

Running the same thing on tmpfs, Tux3 is significantly faster:

    Ext4:   1.40s
    XFS:    1.10s
    Btrfs:  1.56s
    Tux3:   1.07s

3% is not "signficantly faster". It's within run to run variation!

Quoted 1 line

You wish. In fact, Tux3 is a lot faster. ...

Yes, it's easy to be fast when you have simple, naive algorithms and an empty filesystem.

Quoted 7 lines

triple checked and reproducible:

   Tasks:   10      100    1,000    10,000
   Ext4:   0.05     0.14    1.53     26.56
   XFS:    0.05     0.16    2.10     29.76
   Btrfs:  0.08     0.37    3.18     34.54
   Tux3:   0.02     0.05    0.18      2.16

Yet I can't reproduce those XFS or ext4 numbers you are quoting there. eg. XFS on a 4GB ram disk:

$ for i in 10 100 1000 10000; do rm /mnt/test/foo* ; time
./test-fsync /mnt/test/foo 10 $i; done

real    0m0.030s
user    0m0.000s
sys     0m0.014s

real    0m0.031s
user    0m0.008s
sys     0m0.157s

real    0m0.305s
user    0m0.029s
sys     0m1.555s

real    0m3.624s
user    0m0.219s
sys     0m17.631s

$

That's roughly 10x faster than your numbers. Can you describe your test setup in detail? e.g. post the full log from block device creation to benchmark completion so I can reproduce what you are doing exactly?

Quoted 3 lines

Note: you should recheck your final number for Btrfs. I have seen Btrfs fall off the rails and take wildly longer on some tests just like that.

Completely reproducable...

Quoted 6 lines

One easily reproducible one is a denial of service
during the 10,000 task test where it takes multiple seconds to cat small files. I saw XFS do this on both spinning disk and tmpfs, and I have seen it hang for minutes trying to list a directory. I looked a bit into it, and I see that you are blocking for aeons trying to acquire a lock in open.

Yes, that's the usual case when XFS is waiting on buffer readahead IO completion. The latency of which is completely determined by block layer queuing and scheduling behaviour. And the block device queue is being dominated by the 10,000 concurrent write processes you just ran.....

"Doctor, it hurts when I do this!"

Quoted 2 lines

You and I both know the truth: Ext4 is the only really reliable general purpose filesystem on Linux at the moment.

That's the funniest thing I've read in a long time :)

Quoted 6 lines

I wouldn't be so sure about that...

    Tasks:       8            16            32
    Ext4:     93.06 MB/s    98.67 MB/s   102.16 MB/s
    XFS:      81.10 MB/s    79.66 MB/s    73.27 MB/s
    Btrfs:    43.77 MB/s    64.81 MB/s    90.35 MB/s ...

     Ext4:     807.21 MB/s    1089.89 MB/s   867.55 MB/s
     XFS:      997.77 MB/s    1011.51 MB/s   876.49 MB/s
     Btrfs:     55.66 MB/s      56.77 MB/s    60.30 MB/s

Numbers are again very different for XFS and ext4 on /dev/ramX on my system. Need to work out why yours are so low....

Quoted 4 lines

Ahem, are you the same person for whom fsync was the most important issue in the world last time the topic came up, to the extent of spreading around FUD and entirely ignoring the great work we had accomplished for regular file operations? ...

Actually, I don't remember any discussions about fsync.

Things I remember that needed addressing are:
- the lack of ENOSPC detection
- the writeback integration issues
- the code cleanliness issues (ifdef mess, etc)
- the page forking design problems
- the lack of scalable inode and space allocation algorithms.

Those are the things I remember, and fsync performance pales in comparison to those.

Quoted 3 lines

I said then that when we
got around to a proper fsync it would be competitive. Now here it is, so you want to change the topic. I understand.

I haven't changed the topic, just the storage medium. The simple fact is that the world is moving away from slow sata storage at a pretty rapid pace and it's mostly going solid state. Spinning disks also changing - they are going to ZBC based SMR, which is a compeltely different problem space which doesn't even appear to be on the tux3 radar....

So where does tux3 fit into a storage future of byte addressable persistent memory and ZBC based SMR devices?

Quoted 5 lines · Dave Chinner

Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the domain of high performance storage...

OK. Well, Tux3 is general purpose and that means we care about single spinning disk and small systems.

Quoted 9 lines

Quoted 8 lines

Quoted 2 lines

So, to demonstrate, I'll run the same tests but using a 256GB samsung 840 EVO SSD and show how much the picture changes.

I will go you one better, I ran a series of fsync tests using tmpfs, and I now have a very clear picture of how the picture changes. The executive summary is: Tux3 is still way faster, and still scales way better to large numbers of tasks. I have every confidence that the same is true of SSD.

Quoted 4 lines

/dev/ramX can't be compared to an SSD. Yes, they both have low seek/IO latency but they have very different dispatch and IO concurrency models. One is synchronous, the other is fully asynchronous.

I had ram available and no SSD handy to abuse. I was interested in measuring the filesystem overhead with the device factored out. I mounted loopback on a tmpfs file, which seems to be about the same as /dev/ram, maybe slightly faster, but much easier to configure. I ran some tests on a ramdisk just now and was mortified to find that I have to reboot to empty the disk. It would take a compelling reason before I do that again.

Quoted 1 line

This is an important distinction, as we'll see later on....

I regard it as predictive of Tux3 performance on NVM.

Quoted 8 lines

Quoted 6 lines

Running the same thing on tmpfs, Tux3 is significantly faster:

    Ext4:   1.40s
    XFS:    1.10s
    Btrfs:  1.56s
    Tux3:   1.07s

3% is not "signficantly faster". It's within run to run variation!

You are right, XFS and Tux3 are within experimental error for single syncs on the ram disk, while Ext4 and Btrfs are way slower:

      Ext4:   1.59s
      XFS:    1.11s
      Btrfs:  1.70s
      Tux3:   1.11s

A distinct performance gap appears between Tux3 and XFS as parallel tasks increase.

Quoted 4 lines

Quoted 1 line

You wish. In fact, Tux3 is a lot faster. ...

Yes, it's easy to be fast when you have simple, naive algorithms and an empty filesystem.

No it isn't or the others would be fast too. In any case our algorithms are far from naive, except for allocation. You can rest assured that when allocation is brought up to a respectable standard in the fullness of time, it will be competitive and will not harm our clean filesystem performance at all.

There is no call for you to disparage our current achievements, which are significant. I do not mind some healthy skepticism about the allocation work, you know as well as anyone how hard it is. However your denial of our current result is irritating and creates the impression that you have an agenda. If you want to complain about something real, complain that our current code drop is not done yet. I will humbly apologize, and the same for enospc.

Quoted 4 lines

That's roughly 10x faster than your numbers. Can you describe your test setup in detail? e.g. post the full log from block device creation to benchmark completion so I can reproduce what you are doing exactly?

Mine is a lame i5 minitower with 4GB from Fry's. Yours is clearly way more substantial, so I can't compare my numbers directly to yours.

Clearly the curve is the same: your numbers increase 10x going from 100 to 1,000 tasks and 12x going from 1,000 to 10,000. The Tux3 curve is significantly flatter and starts from a lower base, so it ends with a really wide gap. You will need to take my word for that for now. I promise that the beer is on me should you not find that reproducible.

The repository delay is just about not bothering Hirofumi for a merge while he finishes up his inode table anti-fragmentation work.

Quoted 5 lines

Quoted 3 lines

Note: you should recheck your final number for Btrfs. I have seen Btrfs fall off the rails and take wildly longer on some tests just like that.

Completely reproducable...

I believe you. I found that Btrfs does that way too much. So does XFS from time to time, when it gets up into lots of tasks. Read starvation on XFS is much worse than Btrfs, and XFS also exhibits some very undesirable behavior with initial file create. Note: Ext4 and Tux3 have roughly zero read starvation in any of these tests, which pretty much proves it is not just a block scheduler thing. I don't think this is something you should dismiss.

Quoted 7 lines

Quoted 6 lines

I wouldn't be so sure about that...

    Tasks:       8            16            32
    Ext4:     93.06 MB/s    98.67 MB/s   102.16 MB/s
    XFS:      81.10 MB/s    79.66 MB/s    73.27 MB/s
    Btrfs:    43.77 MB/s    64.81 MB/s    90.35 MB/s ...

Quoted 6 lines

     Ext4:     807.21 MB/s    1089.89 MB/s   867.55 MB/s
     XFS:      997.77 MB/s    1011.51 MB/s   876.49 MB/s
     Btrfs:     55.66 MB/s      56.77 MB/s    60.30 MB/s

Numbers are again very different for XFS and ext4 on /dev/ramX on my system. Need to work out why yours are so low....

Your machine makes mine look like a PCjr.

Quoted 11 lines

Quoted 3 lines

I said then that when we
got around to a proper fsync it would be competitive. Now here it is, so you want to change the topic. I understand.

I haven't changed the topic, just the storage medium. The simple fact is that the world is moving away from slow sata storage at a pretty rapid pace and it's mostly going solid state. Spinning disks also changing - they are going to ZBC based SMR, which is a compeltely different problem space which doesn't even appear to be on the tux3 radar....

Quoted 2 lines

So where does tux3 fit into a storage future of byte addressable persistent memory and ZBC based SMR devices?

You won't convince us to abandon spinning rust, it's going to be around a lot longer than you think. Obviously, we care about SSD and I believe you will find that Tux3 is more than competitive there. We lay things out in a very erase block friendly way. We need to address the volume wrap issue of course, and that is in progress. This is much easier than spinning disk.

Tux3's redirect-on-write[1] is obviously a natural for SMR, however I will not get excited about it unless a vendor waves money.

Quoted 7 lines · Daniel Phillips

Quoted 5 lines · Dave Chinner

Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the domain of high performance storage...

Quoted 2 lines

OK. Well, Tux3 is general purpose and that means we care about single spinning disk and small systems.

keep in mind that if you optimize only for the small systems you may not scale as well to the larger ones.

per the ramdisk but, possibly not as relavent as you may think. This is why it's good to test on as many different systems as you can. As you run into different types of performance you can then pick ones to keep and test all the time.

Single spinning disk is interesting now, but will be less interesting later. multiple spinning disks in an array of some sort is going to remain very interesting for quite a while.

now, some things take a lot more work to test than others. Getting time on a system with a high performance, high capacity RAID is hard, but getting hold of an SSD from Fry's is much easier. If it's a budget item, ping me directly and I can donate one for testing (the cost of a drive is within my unallocated budget and using that to improve Linux is worthwhile)

As I'm reading Dave's comments, he isn't attacking you the way you seem to think he is. He is pointing ot that there are problems with your data, but he's also taking a lot of time to explain what's happening (and yes, some of this is probably because your simple tests with XFS made it look so bad)

the other filesystems don't use naive algortihms, they use something more complex, and while your current numbers are interesting, they are only preliminary until you add something to handle fragmentation. That can cause very significant problems.

Remember how fabulous btrfs looked in the initial reports? and then corner cases were found that caused real problems and as the algorithms have been changed to prevent those corner cases from being so easy to hit, the common case has suffered somewhat. This isn't an attack on Tux2 or btrfs, it's just a reality of programming. If you are not accounting for all the corner cases, everything is easier, and faster.

Quoted 2 lines

Mine is a lame i5 minitower with 4GB from Fry's. Yours is clearly way more substantial, so I can't compare my numbers directly to yours.

If you are doing tests with a 4G ramdisk on a machine with only 4G of RAM, it seems like you end up testing a lot more than just the filesystem. Testing in such low memory situations can indentify significant issues, but it is questionable as a 'which filesystem is better' benchmark.

Just a suggestion, but before you do a huge post about how great your filesystem is performing, making the code avaialble so that others can test it when prompted by your post is probably a very good idea. If it means that you have to send out your post a week later, it's a very small cost for the benefit of having other people able to easily try it on hardware that you don't have access to.

If there is a reason to post wihtout the code being in the main, publicised repo, then your post should point people at what code they can use to duplicate it.

but really, 11 months without updating the main repo?? This is Open Source development, publish early and often.

something to investigate, but I have seen probelms on ext* in the past. ext4 may have fixed this, or it may just have moved the point where it triggers.

Quoted 2 lines

Quoted 1 line

...

Your machine makes mine look like a PCjr. ...

The interesting thing here is that on the faster machine btrfs didn't speed up significantly while ext4 and xfs did. It will be interesting to see what the results are for tux3

and both of you need to remember that while servers are getting faster, we are also seeing much lower power, weaker servers showing up as well. And while these smaller servers are not trying to do teh 10000 thread fsync workload, they are using flash based storage more frequently than they are spinning rust (frequently through the bottleneck of a SD card) so continuing tests on low end devices is good.

what drives are available now? see if you can get a couple (either directly or donated)

Quoted 8 lines · Daniel Phillips

Quoted 7 lines · Dave Chinner

Quoted 5 lines

Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the domain of high performance storage...

Quoted 4 lines

Quoted 2 lines

OK. Well, Tux3 is general purpose and that means we care about single spinning disk and small systems.

keep in mind that if you optimize only for the small systems you may not scale as well to the larger ones.

Tux3 is designed to scale, and it will when the time comes. I look forward to putting Shardmap through its billion file test in due course. However, right now it would be wise to stay focused on basic functionality suited to a workstation because volunteer devs tend to have those. After that, phones are a natural direction, where hard core ACID commit and really smooth file ops are particularly attractive.

Quoted 1 line

per the ramdisk but, possibly not as relavent as you may think. This is why it's good to test on as many different systems as you can. As you run into different types of performance you can then pick ones to keep and test all the time.

I keep being surprised how well it works for things we never tested before.

Quoted 1 line

Single spinning disk is interesting now, but will be less interesting later. multiple spinning disks in an array of some sort is going to remain very interesting for quite a while.

The way to do md well is to integrate it into the block layer like Freebsd does (GEOM) and expose a richer interface for the filesystem. That is how I think Tux3 should work with big iron raid. I hope to be able to tackle that sometime before the stars start winking out.

Quoted 1 line

now, some things take a lot more work to test than others. Getting time on a system with a high performance, high capacity RAID is hard, but getting hold of an SSD from Fry's is much easier. If it's a budget item, ping me directly and I can donate one for testing (the cost of a drive is within my unallocated budget and using that to improve Linux is worthwhile)

Thanks.

Quoted 1 line

As I'm reading Dave's comments, he isn't attacking you the way you seem to think he is. He is pointing ot that there are problems with your data, but he's also taking a lot of time to explain what's happening (and yes, some of this is probably because your simple tests with XFS made it look so bad)

I hope the lightening up trend is a trend.

Quoted 1 line

the other filesystems don't use naive algortihms, they use something more complex, and while your current numbers are interesting, they are only preliminary until you add something to handle fragmentation. That can cause very significant problems.

Fsync is pretty much agnostic to fragmentation, so those results are unlikely to change substantially even if we happen to do a lousy job on allocation policy, which I naturally consider unlikely. In fact, Tux3 fsync is going to get faster over time for a couple of reasons: the minimum blocks per commit will be reduced, and we will get rid of most of the seeks to beginning of volume that we currently suffer per commit.

Quoted 6 lines

Remember how fabulous btrfs looked in the initial reports? and then corner cases were found that caused real problems and as the algorithms have been changed to prevent those corner cases from being so easy to hit, the common case has suffered somewhat. This isn't an attack on Tux2 or btrfs, it's just a reality of programming. If you are not accounting for all the corner cases, everything is easier, and faster.

Quoted 2 lines

Mine is a lame i5 minitower with 4GB from Fry's. Yours is clearly way more substantial, so I can't compare my numbers directly to yours.

If you are doing tests with a 4G ramdisk on a machine with only 4G of RAM, it seems like you end up testing a lot more than just the filesystem. Testing in such low memory situations can indentify significant issues, but it is questionable as a 'which filesystem is better' benchmark.

A 1.3 GB tmpfs, and sorry, it is 10 GB (the machine next to it is 4G). I am careful to ensure the test environment does not have spurious memory or cpu hogs. I will not claim that this is the most sterile test environment possible, but it is adequate for the task at hand. Nearly always, when I find big variations in the test numbers it turns out to be a quirk of one filesystem that is not exhibited by the others. Everything gets multiple runs and lands in a spreadsheet. Any fishy variance is investigated.

By the way, the low variance kings by far are Ext4 and Tux3, and of those two, guess which one is more consistent. XFS is usually steady, but can get "emotional" with lots of tasks, and Btrfs has regular wild mood swings whenever the stars change alignment. And while I'm making gross generalizations: XFS and Btrfs go OOM way before Ext4 and Tux3.

Quoted 1 line

Just a suggestion, but before you do a huge post about how great your filesystem is performing, making the code avaialble so that others can test it when prompted by your post is probably a very good idea. If it means that you have to send out your post a week later, it's a very small cost for the benefit of having other people able to easily try it on hardware that you don't have access to.

Next time. This time I wanted it off my plate as soon as possible so I could move on to enospc work. And this way is more involving, we get a little suspense before the rematch.

Quoted 1 line

If there is a reason to post wihtout the code being in the main, publicised repo, then your post should point people at what code they can use to duplicate it.

I could have included the patch in the post, it is small enough. If it still isn't in the repo in a few days then I will post it, to avoid giving the impression I'm desperately trying to fix obscure bugs in it, which isn't the case.

Quoted 1 line

but really, 11 months without updating the main repo?? This is Open Source development, publish early and often.

It's not as bad as that:

https://github.com/OGAWAHirofumi/linux-tux3/commits/hirofumi
https://github.com/OGAWAHirofumi/linux-tux3/commits/hirofumi-user

Quoted 1 line

something to investigate, but I have seen probelms on ext* in the past. ext4 may have fixed this, or it may just have moved the point where it triggers.

My spectrum of tests is small and I am not hunting for anomalies, only reporting what happened to come up. It is not very surprising that some
odd things happen with 10,000 tasks, there is probably not much test coverage there. On the whole I was surprised and impressed when all filesystems mostly just worked. I was expecting to hit scheduler issues for one thing, and nothing obvious came up. Also, not one oops on any filesystem (even Tux3) and only one assert, already reported upstream and turned out to be fixed a week or two ago.

Quoted 4 lines

Quoted 2 lines

Quoted 1 line

...

Your machine makes mine look like a PCjr. ...

The interesting thing here is that on the faster machine btrfs didn't speed up significantly while ext4 and xfs did. It will be interesting to see what the results are for tux3

The numbers are well into the something-is-really-wrong zone (and I should have flagged that earlier but it was a long day). That test is supposed to be -s, all synchronous, and his numbers are more typical of async. Needs double checking all round, including here. Anybody can replicate that test, it is only an apt-get install dbench away (hint hint).

Differences: my numbers are kvm with loopback mount on tmpfs. His are on ramdisk and probably native. I have to reboot to make a ramdisk big enough to run dbench and I would rather not right now.

How important is it to get to the bottom of the variance in test results running on RAM? Probably important in the long run, because storage devices are looking more like RAM all the time, but as of today, maybe not very urgent.

Also, I was half expecting somebody to question the wisdom of running benchmarks under KVM instead of native, but nobody did. Just for the record, I would respond: running virtual probably accounts for the majority of server instances today.

Quoted 1 line

and both of you need to remember that while servers are getting faster, we are also seeing much lower power, weaker servers showing up as well. And while these smaller servers are not trying to do teh 10000 thread fsync workload, they are using flash based storage more frequently than they are spinning rust (frequently through the bottleneck of a SD card) so continuing tests on low end devices is good.

Low end servers and embedded concerns me more, indeed.

Quoted 1 line

what drives are available now? see if you can get a couple (either directly or donated)

Right, time to hammer on flash.

Regards,

Daniel

On the 2nd of May 2015 12:26, Daniel Phillips wrote:

Aloha everybody

Quoted 21 lines · David Lang

Quoted 12 lines · Daniel Phillips

Quoted 10 lines · Dave Chinner

Quoted 6 lines

Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the domain of high performance storage...

Quoted 6 lines

Quoted 2 lines

OK. Well, Tux3 is general purpose and that means we care about single spinning disk and small systems.

keep in mind that if you optimize only for the small systems you may not scale as well to the larger ones.

Quoted 7 lines

Tux3 is designed to scale, and it will when the time comes. I look forward to putting Shardmap through its billion file test in due course. However, right now it would be wise to stay focused on basic functionality suited to a workstation because volunteer devs tend to have those. After that, phones are a natural direction, where hard core ACID commit and really smooth file ops are particularly attractive.

Has anybody else a deja vu?

Quoted 26 lines · Christian Stroetmann

On the 2nd of May 2015 12:26, Daniel Phillips wrote:

Aloha everybody

Quoted 22 lines · David Lang

Quoted 8 lines · Daniel Phillips

Quoted 7 lines · Dave Chinner

Quoted 5 lines

Well, yes - I never claimed XFS is a general purpose filesystem. It is a high performance filesystem. Is is also becoming more relevant to general purpose systems as low cost storage gains capabilities that used to be considered the domain of high performance storage...

Quoted 5 lines

Quoted 2 lines

OK. Well, Tux3 is general purpose and that means we care about single spinning disk and small systems.

keep in mind that if you optimize only for the small systems you may not scale as well to the larger ones.

Tux3 is designed to scale, and it will when the time comes. I look forward to putting Shardmap through its billion file test in due course. However, right now it would be wise to stay focused on basic functionality suited to a workstation because volunteer devs tend to have those. After that, phones are a natural direction, where hard core ACID commit and really smooth file ops are particularly attractive.

Quoted 1 line

Has anybody else a deja vu?

Yes, the onto-troll strikes again...

...For example, we currently
overestimate the cost of a rewrite because we would need to go poking around in btrees to do that more accurately. Fixing that will be quite a bit of work...

We identified the following quality metrics for this algorithm:

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

Quoted 6 lines

We identified the following quality metrics for this algorithm:

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

Hmm. Can you also overwrite existing data in files when a volume is full? I guess applications expect that to work..

Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures)
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Pavel Machek

Quoted 3 lines

We identified the following quality metrics for this algorithm:

Quoted 3 lines

Quoted 3 lines

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

Quoted 5 lines · Mosis Tembo

Quoted 3 lines · Pavel Machek

Quoted 3 lines

We identified the following quality metrics for this algorithm:

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

Because some extra space needs to be on the volume in order to do the atomic commit. Specifically, there must be enough extra space to keep both old and new copies of any changed metadata, plus enough space for new data or metadata. You are almost right: we can't support rm, rmdir or truncate _with atomic commit_ unless some space is available on the volume. So we keep a small reserve to handle those operations, which only those operations can access. We define the volume as "full" when only the reserve remains. The reserve is not included in "available" blocks reported to statfs, so the volume appears to be 100% full when only the reserve remains.

For Tux3, that reserve is variable - about 1% of free space, declining to a minimum of 10 blocks as free space runs out. Eventually, we will reduce the minimum a bit as we develop finer control over how free space is used in very low space conditions, but 10 blocks is not bad at all. With no journal and only 10 blocks of unusable space, we do pretty well with tiny volumes.

Quoted 25 lines · Daniel Phillips

Quoted 5 lines · Mosis Tembo

Quoted 3 lines · Pavel Machek

Quoted 3 lines

We identified the following quality metrics for this algorithm:

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

Because some extra space needs to be on the volume in order to do the atomic commit. Specifically, there must be enough extra space to keep both old and new copies of any changed metadata, plus enough space for new data or metadata. You are almost right: we can't support rm, rmdir or truncate _with atomic commit_ unless some space is available on the volume. So we keep a small reserve to handle those operations, which only those operations can access. We define the volume as "full" when only the reserve remains. The reserve is not included in "available" blocks reported to statfs, so the volume appears to be 100% full when only the reserve remains.

Quoted 6 lines

For Tux3, that reserve is variable - about 1% of free space, declining to a minimum of 10 blocks as free space runs out. Eventually, we will reduce the minimum a bit as we develop finer control over how free space is used in very low space conditions, but 10 blocks is not bad at all. With no journal and only 10 blocks of unusable space, we do pretty well with tiny volumes.

Yeah. Filesystem that could not do rm on full filesystem would be braindead.

Now, what about

1) writing to already-allocated space in existing files?

2) writing to already-allocated space in existing files using mmap?

Quoted 36 lines · Pavel Machek

Quoted 25 lines · Daniel Phillips

Quoted 5 lines · Mosis Tembo

Quoted 3 lines · Pavel Machek

Quoted 3 lines

We identified the following quality metrics for this algorithm:

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

Because some extra space needs to be on the volume in order to do the atomic commit. Specifically, there must be enough extra space to keep both old and new copies of any changed metadata, plus enough space for new data or metadata. You are almost right: we can't support rm, rmdir or truncate _with atomic commit_ unless some space is available on the volume. So we keep a small reserve to handle those operations, which only those operations can access. We define the volume as "full" when only the reserve remains. The reserve is not included in "available" blocks reported to statfs, so the volume appears to be 100% full when only the reserve remains.

Quoted 6 lines

For Tux3, that reserve is variable - about 1% of free space, declining to a minimum of 10 blocks as free space runs out. Eventually, we will reduce the minimum a bit as we develop finer control over how free space is used in very low space conditions, but 10 blocks is not bad at all. With no journal and only 10 blocks of unusable space, we do pretty well with tiny volumes.

Yeah. Filesystem that could not do rm on full filesystem would be braindead.

Quoted 3 lines

Now, what about

1) writing to already-allocated space in existing files?

I mentioned earlier, it seems to work pretty well in Tux3. But do user applications really expect it to work? I do not know of any, perhaps you do.

Quoted 5 lines

We identified the following quality metrics for this algorithm:

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

Hmm. Can you also overwrite existing data in files when a volume is full? I guess applications expect that to work..
Pavel

Quoted 2 lines · Pavel Machek

Quoted 2 lines

We identified the following quality metrics for this algorithm:

Quoted 3 lines

Quoted 3 lines

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

Quoted 8 lines · Mosis Tembo

Quoted 5 lines · Pavel Machek

Quoted 5 lines

We identified the following quality metrics for this algorithm:

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

Quoted 4 lines

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

I assume you are referring either to Tux3 specifically or COW filesystems in general,

because you very much _can_ do any of those on any of the non-COW filesystems in the Linux kernel

(I know from experience). Also, IIRC, it was mentioned somewhere that Tux3 keeps a small reserve of space on the volume for internal operations; and, I would assume that if that is the case, it reports the volume full when everything *except* that reserve of space is used, in which case rm, rmdir, and truncate should work fine when the volume is full.

Quoted 13 lines · Austin S Hemmelgarn

Quoted 12 lines · Mosis Tembo

Quoted 5 lines · Pavel Machek

Quoted 5 lines

We identified the following quality metrics for this algorithm:

1) Never fails to detect out of space in the front end.
2) Always fills a volume to 100% before reporting out of space.
3) Allows rm, rmdir and truncate even when a volume is full.

This is definitely nonsense. You can not rm, rmdir and truncate when the volume is full. You will need a free space on disk to perform such operations. Do you know why?

Quoted 2 lines

I assume you are referring either to Tux3 specifically or COW filesystems in general,

I am referring to modern file systems with transaction models and delayed actions.
Tux3 is not the case?

Quoted 3 lines

because you very much _can_ do any of those on any of the non-COW filesystems in the Linux kernel

It is simply incorrect. ReiserFS is a counterexample.

Quoted 7 lines

(I know from experience). Also, IIRC, it was mentioned somewhere that Tux3 keeps a small reserve of space on the volume for internal operations; and, I would assume that if that is the case, it reports the volume full when everything *except* that reserve of space is used, in which case rm, rmdir, and truncate should work fine when the volume is full.

Sorry, I prefer to not manipulate with rumors and assumptions when it comes to the review for kernel inclusion.

Hi Rik,

Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main part you may be interested in is rmap.c, which addresses the issues raised at the 2013 Linux Storage Filesystem and MM Summit 2015 in San Francisco.[1]

LSFMM: Page forking
http://lwn.net/Articles/548091/

This is just a FYI. An upcoming Tux3 report will be a tour of the page forking design and implementation. For now, this is just to give a general sense of what we have done. We heard there are concerns about how ptrace will work. I really am not familiar with the issue, could you please explain what you were thinking of there?

Quoted 8 lines · Daniel Phillips

Hi Rik,

Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main part you may be interested in is rmap.c, which addresses the issues raised at the 2013 Linux Storage Filesystem and MM Summit 2015 in San Francisco.[1]

Quoted 8 lines

LSFMM: Page forking
http://lwn.net/Articles/548091/

This is just a FYI. An upcoming Tux3 report will be a tour of the page forking design and implementation. For now, this is just to give a general sense of what we have done. We heard there are concerns about how ptrace will work. I really am not familiar with the issue, could you please explain what you were thinking of there?

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Hi Rik,

Added Mel, Andrea and Peterz to CC as interested parties. There are probably others, please just jump in.

Quoted 26 lines · Rik van Riel

Quoted 8 lines · Daniel Phillips

Hi Rik,

Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main part you may be interested in is rmap.c, which addresses the issues raised at the 2013 Linux Storage Filesystem and MM Summit 2015 in San Francisco.[1]

Quoted 8 lines

LSFMM: Page forking
http://lwn.net/Articles/548091/

This is just a FYI. An upcoming Tux3 report will be a tour of the page forking design and implementation. For now, this is just to give a general sense of what we have done. We heard there are concerns about how ptrace will work. I really am not familiar with the issue, could you please explain what you were thinking of there?

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

Quoted 4 lines

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Is this a bit like page migration?

Quoted 37 lines · Daniel Phillips

Hi Rik,

Added Mel, Andrea and Peterz to CC as interested parties. There are probably others, please just jump in.

Quoted 30 lines · Rik van Riel

Quoted 8 lines · Daniel Phillips

Hi Rik,

Our linux-tux3 tree currently currently carries this 652 line diff against core, to make Tux3 work. This is mainly by Hirofumi, except the fs-writeback.c hook, which is by me. The main part you may be interested in is rmap.c, which addresses the issues raised at the 2013 Linux Storage Filesystem and MM Summit 2015 in San Francisco.[1]

Quoted 8 lines

LSFMM: Page forking
http://lwn.net/Articles/548091/

This is just a FYI. An upcoming Tux3 report will be a tour of the page forking design and implementation. For now, this is just to give a general sense of what we have done. We heard there are concerns about how ptrace will work. I really am not familiar with the issue, could you please explain what you were thinking of there?

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Quoted 1 line

Is this a bit like page migration?

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

Quoted 16 lines · Rik van Riel

Quoted 14 lines · Daniel Phillips

Quoted 5 lines

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

Quoted 7 lines

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Is this a bit like page migration?

Quoted 3 lines

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

When I said it's not like page migration, I was referring to the fact that a COW on a pinned page for RDMA is a different problem to page migration. The COW of a pinned page can lead to lost writes or corruption depending on the ordering of events.

Page migration fails
when there are unexpected problems to avoid this class of issue which is fine for page migration but may be a critical failure in a filesystem depending on exactly why the copy is required.

Quoted 21 lines · Mel Gorman

Quoted 19 lines · Rik van Riel

Quoted 8 lines · Daniel Phillips

Quoted 8 lines

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

Quoted 6 lines

Quoted 4 lines

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Is this a bit like page migration?

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

Quoted 4 lines

When I said it's not like page migration, I was referring to the fact that a COW on a pinned page for RDMA is a different problem to page migration. The COW of a pinned page can lead to lost writes or corruption depending on the ordering of events.

I see the lost writes case, but not the corruption case,

Do you
mean corruption by changing a page already in writeout? If so, don't all filesystems have that problem?

If RDMA to a mmapped file races with write(2) to the same file, maybe it is reasonable and expected to lose some data.

Quoted 29 lines · Daniel Phillips

Quoted 25 lines · Mel Gorman

Quoted 14 lines · Rik van Riel

Quoted 13 lines · Daniel Phillips

Quoted 12 lines

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Quoted 5 lines

Quoted 1 line

Is this a bit like page migration?

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

When I said it's not like page migration, I was referring to the fact that a COW on a pinned page for RDMA is a different problem to page migration. The COW of a pinned page can lead to lost writes or corruption depending on the ordering of events.

Quoted 1 line

I see the lost writes case, but not the corruption case,

Data corruption can occur depending on the ordering of events and the applications expectations. If a process starts IO, RDMA pins the page for read and forks are combined with writes from another thread then when the IO completes the reads may not be visible. The application may take improper action at that point.

Users of RDMA are typically expected to use MADV_DONTFORK to avoid this class of problem.

You can choose to not define this as data corruption because thge kernel is not directly involved and that's your call.

Quoted 4 lines

Do you
mean corruption by changing a page already in writeout? If so, don't all filesystems have that problem?

No, the problem is different. Backing devices requiring stable pages will block the write until the IO is complete. For those that do not require stable pages it's ok to allow the write as long as the page is dirtied so that it'll be written out again and no data is lost.

Quoted 3 lines

If RDMA to a mmapped file races with write(2) to the same file, maybe it is reasonable and expected to lose some data.

In the RDMA case, there is at least application awareness to work around the problems. Normally it's ok to have both mapped and write() access to data although userspace might need a lock to co-ordinate updates and event ordering.

Quoted 30 lines · Mel Gorman

Quoted 29 lines · Daniel Phillips

Quoted 25 lines · Mel Gorman

Quoted 14 lines · Rik van Riel

Quoted 13 lines · Daniel Phillips

Quoted 12 lines

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Quoted 5 lines

Quoted 1 line

Is this a bit like page migration?

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

When I said it's not like page migration, I was referring to the fact that a COW on a pinned page for RDMA is a different problem to page migration. The COW of a pinned page can lead to lost writes or corruption depending on the ordering of events.

Quoted 7 lines

Quoted 1 line

I see the lost writes case, but not the corruption case,

Data corruption can occur depending on the ordering of events and the applications expectations. If a process starts IO, RDMA pins the page for read and forks are combined with writes from another thread then when the IO completes the reads may not be visible. The application may take improper action at that point.

if tux3 forks the page and writes the copy while the original page is being modified by other things, this means that some of the changes won't be in the version written (and this could catch partial writes with 'interesting' results if the forking happens at the wrong time)

But if the original page gets re-marked as needing to be written out when it's changed by one of the other things that are accessing it, there shouldn't be any long-term corruption.

As far as short-term corruption goes, any time you have a page mmapped it could get written out at any time, with only some of the application changes applied to it, so this sort of corruption could happen anyway couldn't it?

Quoted 11 lines

Users of RDMA are typically expected to use MADV_DONTFORK to avoid this class of problem.

You can choose to not define this as data corruption because thge kernel is not directly involved and that's your call.

Quoted 4 lines

Do you
mean corruption by changing a page already in writeout? If so, don't all filesystems have that problem?

Quoted 4 lines

No, the problem is different. Backing devices requiring stable pages will block the write until the IO is complete. For those that do not require stable pages it's ok to allow the write as long as the page is dirtied so that it'll be written out again and no data is lost.

so if tux3 is prevented from forking the page in cases where the write would be blocked, and will get forked again for follow-up writes if it's modified again otherwise, won't this be the same thing?

Quoted 8 lines · Daniel Phillips

Quoted 8 lines

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

Quoted 6 lines

Quoted 4 lines

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Is this a bit like page migration?

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

Only pages that have no extra refcount can be migrated.

Similarly, your cow code needs to fail if there is an extra reference count pinning the page. As long as the page has a user that you cannot migrate, you cannot move any of the other users over. They may rely on data written by the hidden-to-you user, and the hidden-to-you user may write to the page when you think it is a read only stable snapshot.

Quoted 22 lines · Rik van Riel

Quoted 8 lines · Daniel Phillips

Quoted 8 lines

The issue is that things like ptrace, AIO, infiniband RDMA, and other direct memory access subsystems can take a reference to page A, which Tux3 clones into a new page B when the process writes it.

However, while the process now points at page B, ptrace, AIO, infiniband, etc will still be pointing at page A.

Quoted 6 lines

Quoted 4 lines

This causes the process and the other subsystem to each look at a different page, instead of at shared state, causing ptrace to do nothing, AIO and RDMA data to be invisible (or corrupted), etc...

Is this a bit like page migration?

Yes. Page migration will fail if there is an "extra" reference to the page that is not accounted for by the migration code.

Only pages that have no extra refcount can be migrated.

Quoted 7 lines

Similarly, your cow code needs to fail if there is an extra reference count pinning the page. As long as the page has a user that you cannot migrate, you cannot move any of the other users over. They may rely on data written by the hidden-to-you user, and the hidden-to-you user may write to the page when you think it is a read only stable snapshot.

Please bear with me as I study these cases one by one.

First one is ptrace. Only for executable files, right? Maybe we don't need to fork pages in executable files,

Quoted 1 line · Daniel Phillips

On 05/14/2015 08:06 PM, Rik van Riel wrote: ...

Umm. Why do you think it is only issue for executable files?

I'm free to mmap() any file, and then execute from it.

/lib/ld-linux.so /path/to/binary

is known way to exec programs that do not have x bit set.

Quoted 4 lines · Pavel Machek

Quoted 1 line · Daniel Phillips

On 05/14/2015 08:06 PM, Rik van Riel wrote: ...

Umm. Why do you think it is only issue for executable files?

I meant: files with code in them, that will be executed. Please excuse me for colliding with the chmod sense. I will say "code files" to avoid ambiguity.

Quoted 2 lines

I'm free to mmap() any file, and then execute from it.

Quoted 3 lines

/lib/ld-linux.so /path/to/binary

is known way to exec programs that do not have x bit set.

So... why would I write to a code file at the same time as stepping through it with ptrace? Should I expect ptrace to work perfectly if I do that? What would "work perfectly" mean, if the code is changing at the same time as being traced?

Quoted 14 lines · Daniel Phillips

Quoted 4 lines · Pavel Machek

Quoted 1 line · Daniel Phillips

On 05/14/2015 08:06 PM, Rik van Riel wrote: ...

Umm. Why do you think it is only issue for executable files?

I meant: files with code in them, that will be executed. Please excuse me for colliding with the chmod sense. I will say "code files" to avoid ambiguity.

Quoted 4 lines

I'm free to mmap() any file, and then execute from it.

/lib/ld-linux.so /path/to/binary