ZFS with Linux: An Open Plea
Before I go on, let me appologise. I don't really know what I hope to accomplish, beyond trying to garner thoughts (and support?) for the topic.
Essentially: I want to use Linux and ZFS. I don't particularly care about licences or any of the rest of that nonsense. The code is there; it merely needs to be made to work with Linux. Done and done -- provided I can find some one to do this for me (I'd do it myself, but I haven't the foggiest notion how to go about such a feat).
By the way, forget about this FUSE business. I don't know why they're bothering: It's not real, it's slow and, in general, silly.
What are the thoughts of the Linux community?
I appologise right now for my intrusion. I am a Linux-nobody; I freely admit it. I haven't even subscribed to this list (so do CC me) because I don't want to be over-whelmed with the list's glorious posts. But, part of Linux is it's being a community. If a member of this community (that is, a user of Linux) can't ask the others their thoughts and opinions, then the community has failed in a large respect. Take this letter as you will.
Signature · 3 lines
—A watched bread-crumb never boils. —My hover-craft is full of eels. —[...]and that's the he and the she of it.
Quoted 5 lines · On Friday April 13, [email protected] wrote:
Before I go on, let me appologise. I don't really know what I hope to accomplish, beyond trying to garner thoughts (and support?) for the topic.
Essentially: I want to use Linux and ZFS. I don't particularly care about licences
You may not care about licenses, but a lot of people do. And they are a real issue.
But I'm more interested in what you really want.
What, specifically, is there in ZFS that you want to use in Linux? There is lots of interesting stuff in there, but which bit has caught your attention? It is possible that that functionality can be incorporated into Linux without trying to clone or copy ZFS.
NeilBrown
Quoted 7 lines · Neil Brown
Quoted 5 lines · On Friday April 13, [email protected] wrote:
Before I go on, let me appologise. I don't really know what I hope to accomplish, beyond trying to garner thoughts (and support?) for the topic.
Essentially: I want to use Linux and ZFS. I don't particularly care about licences
Quoted 8 lines
You may not care about licenses, but a lot of people do. And they are a real issue.
But I'm more interested in what you really want.
What, specifically, is there in ZFS that you want to use in Linux? There is lots of interesting stuff in there, but which bit has caught your attention? It is possible that that functionality can be incorporated into Linux without trying to clone or copy ZFS.
Linux definitly missed the hype value. So everyone please start blogs and do a of handwaving to get the mindspehere in web 2.0 back..
/me sighs
Quoted 7 lines · Neil Brown
Quoted 5 lines · On Friday April 13, [email protected] wrote:
Before I go on, let me appologise. I don't really know what I hope to accomplish, beyond trying to garner thoughts (and support?) for the topic.
Essentially: I want to use Linux and ZFS. I don't particularly care about licences
Quoted 8 lines
You may not care about licenses, but a lot of people do. And they are a real issue.
But I'm more interested in what you really want.
What, specifically, is there in ZFS that you want to use in Linux? There is lots of interesting stuff in there, but which bit has caught your attention? It is possible that that functionality can be incorporated into Linux without trying to clone or copy ZFS.
ZFS does have some powerful features but much of it depends on their broken layering of volume management. Embedding the equivalent of LVM into a filesystem _feels_ quite wrong.
That aside, the native snapshot capabilities of ZFS really stand out for me. The redirect on write semantics aren't exclusive to ZFS; NetApp's WAFL employs the same. But with both ZFS and WAFL they were designed to do snapshots extremely well from the ground up.
Unfortunately in order for Linux to incorporate such a feature I'd imagine a new filesystem would need to be developed with redirect on write at its core. Can't really see ext4 or any other existing Linux filesystem grafting such a feature into it. But even though I can't see it; do others?
Begin non-technical patent related babble:
As you noted licenses are important, as are patents. Interestingly I've learned that Sun and NetApp's lawyers had it out over the redirect on write capability of ZFS. When the dust settled Sun had enough patent protection to motivate a truce with NetApp.
The interesting side-effect is now ZFS is "open" and with that comes redirect on write in a file system other than WAFL. But ZFS's CDDL conflicts with the GPL so I'm not too sure how Linux could hit the ground running in this potentially patent mired area of filesystem development. The validity of NetApp having patented redirect on write aside; does the conflict between CDDL and GPL _really_ matter? Or did the CDDL release of ZFS somehow undermine NetApp's WAFL patent?
Mike
Quoted 5 lines · Mike Snitzer
ZFS does have some powerful features but much of it depends on their broken layering of volume management. Embedding the equivalent of LVM into a filesystem _feels_ quite wrong.
Quoted 7 lines
[...]
Unfortunately in order for Linux to incorporate such a feature I'd imagine a new filesystem would need to be developed with redirect on write at its core. Can't really see ext4 or any other existing Linux filesystem grafting such a feature into it. But even though I can't see it; do others?
FUSE-based filesystems. (That said, might wanna try through "ZFS-on-fuse" fs).
Jan
Quoted 9 lines · Jan Engelhardt
Quoted 7 lines · Mike Snitzer
ZFS does have some powerful features but much of it depends on their broken layering of volume management. Embedding the equivalent of LVM into a filesystem _feels_ quite wrong.
[...]
Quoted 8 lines
Quoted 5 lines
Unfortunately in order for Linux to incorporate such a feature I'd imagine a new filesystem would need to be developed with redirect on write at its core. Can't really see ext4 or any other existing Linux filesystem grafting such a feature into it. But even though I can't see it; do others?
FUSE-based filesystems. (That said, might wanna try through "ZFS-on-fuse" fs).
FUSE implemtation uses too many context switches on each I/O so on the same hardware it will never will be so fast as on Linux, BSD and MOX (yes ZFS is now avalaible on all this OSes .. and few people says Windows implemtation is very close). Current FUSE implemntation can't be comparable in aspects of speed and probably never will be on using threads (very simmilar case to ALSA and mixing in user space .. in both cases it is broken by design). Also ZFS on Solaris uses few other system abilities for example it interract with PSH.
So FUSE ZFS is IMO only kind of academic demo implemtation only but never will be usefull/ready for production (plain waste of time). If Linux licensing will not be relaxed probaly we can forget about ZFS on Linux (licensing problems in this case are obly on Linux side .. <ironically>in this case ZFS licensing is more oppened than Linux licensing </ironically>).
kloczek
PS. IIRC implemting ZFS on FUSE was some kind of workaround for licensing issue but FUSE fill uses vfs layer and probaly still thinking about licensing is actual.
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 14 lines
FUSE implemtation uses too many context switches on each I/O so on the same hardware it will never will be so fast as on Linux, BSD and MOX (yes ZFS is now avalaible on all this OSes .. and few people says Windows implemtation is very close). Current FUSE implemntation can't be comparable in aspects of speed and probably never will be on using threads (very simmilar case to ALSA and mixing in user space .. in both cases it is broken by design). Also ZFS on Solaris uses few other system abilities for example it interract with PSH.
So FUSE ZFS is IMO only kind of academic demo implemtation only but never will be usefull/ready for production (plain waste of time). If Linux licensing will not be relaxed probaly we can forget about ZFS on Linux (licensing problems in this case are obly on Linux side .. <ironically>in this case ZFS licensing is more oppened than Linux licensing </ironically>).
Quoted 4 lines
kloczek
PS. IIRC implemting ZFS on FUSE was some kind of workaround for licensing issue but FUSE fill uses vfs layer and probaly still thinking about licensing is actual.
I'm just a lowly proto-hacker trying to get into kernel development, but even I know a few things, and I'm getting very tired of hearing FUSE slammed as stupid and the Linux licencing policy as foolish. I can sum things up in two short passages.
FUSE is intended for end-users, not critical server applications. End-users have plenty of processor to spare at lower-than-ever prices. So you have a simple and flexible end-user API that lets you do anything with anything in any way you want, all the time, without needing to (or even needing to be able to) alter your kernel. Win.
Linux licencing basically says, "If you want in on this good thing, you have to play by the rules. That means no cheating, no scamming, no profiting at the expense of others, and no getting on a high horse and proclaiming to the rest of the world that you are in control of their destinies. Play nice, kiddies." If you want to do something, you do it open and you do it Free, and if you don't want to, well...write your own kernel!
There is a wonderful quotation on the GNU site comparing agencies like Microsoft to crack dealers: They appear more open in theory in order to get people hooked, then trigger the beartrap around your foot. Linux has rules--hard, fast, and immutable--and they protect everybody from their own stupidities. Open-community development takes a whole lot longer, but you don't have to sell your soul to do it.
Quoted 2 lines · Tomasz Kłoczko
Current FUSE implemntation can't be
comparable in aspects of speed and probably never will be on using threads
Did you measure this on a few hardwares and workloads?
Quoted 1 line
(very simmilar case to ALSA and mixing in user space ..
Audio is about guaranteed latency, not "speed".
Signature · 3 lines
Stefan Richter -=====-=-=== -=-- -==-= http://arcgraph.de/sr/
Quoted 5 lines · Stefan Richter
Quoted 2 lines · Tomasz Kłoczko
Current FUSE implemntation can't be
comparable in aspects of speed and probably never will be on using threadsDid you measure this on a few hardwares and workloads?
Before asking firs you must try look on current ZFS on FUSE discuss phorums/docu .. like on: http://groups.google.com/group/zfs-fuse/feed/rss_v2_0_msgs.xml
ZFS on Solaris provides for many workloads better speed than any Linux technology on the same hardware but ZFS ond FUSE in current form provides lower speed than now avalaible Linux technologies.
Example http://groups.google.com/group/zfs-fuse/msg/5b10c69707a46c07:
"According to bonnie, I see 125 MB/s reads on ext3+RAID5, 65 MB/s on ZFS+RAID5 (using Linux's software RAID) and 20 MB/s on ZFS+raidz (using the same raw drives). Writes are also proportionally slower. The real performance hit with ZFS-FUSE was random accesses for lots of small files. The bonnie++ results showed something like 75 random seeks for ZFS vs 470 for ext3 (..)"
On the same phorun you can find threads with discuss about utilize treading under FUSE.
Quoted 3 lines
Quoted 1 line
(very simmilar case to ALSA and mixing in user space ..
Audio is about guaranteed latency, not "speed".
You meam "guaranteed worser latency" ? excelent 8-)
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 3 lines · Tomasz K?oczko
ZFS on Solaris provides for many workloads better speed than any Linux technology on the same hardware but ZFS ond FUSE in current form provides lower speed than now avalaible Linux technologies.
Numbers, please. So far in all interesting benchmarks it actually was slower. But when they're faster than XFS somewhere I'd defintly be interesting in looking at why this is true and if possible and important enough fix it.
Quoted 9 lines · Christoph Hellwig
Quoted 3 lines · Tomasz K?oczko
ZFS on Solaris provides for many workloads better speed than any Linux technology on the same hardware but ZFS ond FUSE in current form provides lower speed than now avalaible Linux technologies.
Numbers, please. So far in all interesting benchmarks it actually was slower. But when they're faster than XFS somewhere I'd defintly be interesting in looking at why this is true and if possible and important enough fix it.
Google still is working but .. (from my bookmarks):
http://cmynhier.blogspot.com/2006/05/zfs-io-reordering-benchmark.html
http://cmynhier.blogspot.com/2006/05/zfs-benchmarking.html
also some other interestig numbers can be founnd on:
http://milek.blogspot.com/2006/08/hw-raid-vs-zfs-software-raid-part-ii.html
Probably more can be find on looking for zfs+benchmark on http://blogs.sun.com/
IMO real power ZFS shows on growing number if phisical devices so don't look so close on this kind of documents because usualy it was prepared on limited number of devices.
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 2 lines · Tomasz K?oczko
Probably more can be find on looking for zfs+benchmark on
http://blogs.sun.com/
If I was searching for sun marketing material I'd have fun the way to blogs.sun.com myself, thanks :)
Quoted 2 lines · Tomasz Kłoczko
also some other interestig numbers can be founnd on:
http://milek.blogspot.com/2006/08/hw-raid-vs-zfs-software-raid-part-ii.html
So software raid can be faster than HW raid. News at 11.
Quoted 6 lines · Diego Calleja
Quoted 2 lines · Tomasz Kłoczko
also some other interestig numbers can be founnd on:
http://milek.blogspot.com/2006/08/hw-raid-vs-zfs-software-raid-part-ii.htmlSo software raid can be faster than HW raid. News at 11.
Of cources it can be true in most cases (probably for some more advanced RAID controlers). Few weeks ago I perform some basic test on Dell 2950 with 8x73GB SAS disk .. just as for kill time (waiting for access to some bigger box ;). This small iron box have inside RAID controller (Dell uses in this box LSI Logic SAS MegaRAID based ctrl). Anykind combinations on controler level RAID was slower than using this as plain JBOD with LVM or MD+LVM. Diffrence between HW and soft RAID was not so big (1-6% depending on configuration) but allways HW produces worser results (don't ask me why). Finaly I decide using this disk as four RAID1 luns only because under Linux I can't read each phisical disk SMART data and protecting this by RAID on controller level and collecting SNMP traps from DRAC card was kind of worakaround for this (in my case it will be better constanlty monitor disk healt and collesting some SMART data for observe trends on for example zabbix graphs for try predict some faults using triggers). On top of this was configured diffrent types of volumes on LVM level (some with stripping some without, some with bigger some with smaller chunk size).
But .. in case ZFS diffrences can be better visable.
Q: why ?
A: because most of HW RAID controlers (nevermind is it small/simple internal HW controler for DAS sotorage or advances in storage processor in dedicated FC array) are _optimized_ for classic FS workloads but .. ZFS uses bunch of devices in completly diffrent way/characteristics. If you will see flashing LEDs on box with disk first time configurad for classic RAID<any_level> (nevermind soft or hw) and on second time configured for ZFS you will see kind of organoleptic diffrence. And yes .. ZFS may be kind of problem for some HW vendors ;)
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 9 lines · Tomasz K?oczko
Of cources it can be true in most cases (probably for some more advanced RAID controlers). Few weeks ago I perform some basic test on Dell 2950 with 8x73GB SAS disk .. just as for kill time (waiting for access to some bigger box ;). This small iron box have inside RAID controller (Dell uses in this box LSI Logic SAS MegaRAID based ctrl). Anykind combinations on controler level RAID was slower than using this as plain JBOD with LVM or MD+LVM. Diffrence between HW and soft RAID was not so big (1-6% depending on configuration) but allways HW produces worser results (don't ask me why). Finaly I decide using this disk as four RAID1 luns only because
Quoted 8 lines
under Linux I can't read each phisical disk SMART data and protecting this by RAID on controller level and collecting SNMP traps from DRAC card was kind of worakaround for this (in my case it will be better constanlty monitor disk healt and collesting some SMART data for observe trends on for example zabbix graphs for try predict some faults using triggers). On top of this was configured diffrent types of volumes on LVM level (some with stripping some without, some with bigger some with smaller chunk size).
Does it matter that google's recent report on disk failures indicated that SMART never predicted anything useful as far as they could tell? Certainly none of my drive failures ever had SMART make any kind of indication that anything was wrong.
I think the main benefit of MD raid, is that it is portable, doesn't lock you into a specific piece of hardware, and you can span multiple controllers, and it is likely easier to have bugs in MD raid fixed that in some raid controller's firmware if any were to be found. Performance advantages are a bonus of course.
--
Len Sorensen
Quoted 8 lines · Lennart Sorensen
Quoted 1 line · Tomasz K?oczko
Of cources it can be true in most cases (probably for some more advanced RAID controlers). Few weeks ago I perform some basic test on Dell 2950 with 8x73GB SAS disk .. just as for kill time (waiting for access to some bigger box ;). This small iron box have inside RAID controller (Dell uses in this box LSI Logic SAS MegaRAID based ctrl). Anykind combinations on controler level RAID was slower than using this as plain JBOD with LVM or MD+LVM. Diffrence between HW and soft RAID was not so big (1-6% depending on configuration) but allways HW produces worser results (don't ask me why). Finaly I decide using this disk as four RAID1 luns only because under Linux I can't read each phisical disk SMART data and protecting this by RAID on controller level and collecting SNMP traps from DRAC card was kind of worakaround for this (in my case it will be better constanlty monitor disk healt and collesting some SMART data for observe trends on for example zabbix graphs for try predict some faults using triggers). On top of this was configured diffrent types of volumes on LVM level (some with stripping some without, some with bigger some with smaller chunk size).
Does it matter that google's recent report on disk failures indicated that SMART never predicted anything useful as far as they could tell? Certainly none of my drive failures ever had SMART make any kind of indication that anything was wrong.
Quoted 5 lines
I think the main benefit of MD raid, is that it is portable, doesn't lock you into a specific piece of hardware, and you can span multiple controllers, and it is likely easier to have bugs in MD raid fixed that in some raid controller's firmware if any were to be found. Performance advantages are a bonus of course.
SMART largely depends on how you use it. Simply polling the current status will not give you all the benefits SMART provides. On the dedicated servers that I rent, running the extended test ('-t long') often finds problems before you start losing data, or deal with a drive death. Certainly not a huge sample size, but it backs up what I hear in the field. Running the SMART tests on a weekly basis seems most effective, though you'll want to stagger the tests if running in a RAID set.
Jeff
Quoted 5 lines · Lennart Sorensen
Does it matter that google's recent report on disk failures indicated that SMART never predicted anything useful as far as they could tell? Certainly none of my drive failures ever had SMART make any kind of indication that anything was wrong.
I saw that talk, and that's not what I got out of it. They found that SMART error reports _did_ correlate with drive failure. See page 8 of:
http://www.usenix.org/events/fast07/tech/full_papers/pinheiro/pinheiro.pdf
(If you're not a USENIX member, you may be able to find a free download copy elsewhere.)
However, they found that the correlation was not strong enough to make it economically feasible to replace disks reporting SMART failures, since something like 70% of disks were still working a year after the first failure report. Also, they found that some disks failed without any SMART error reports.
Now, Google keeps multiple copies (3 in GoogleFS, last I heard) of data, so for them, "economically feasible" means something different than for my personal laptop hard drive. I have twice had my laptop hard drive start spitting SMART errors and then die within a week. It is economically quite sensible for me to replace my laptop drive once it has an error, since I don't carry around 3 laptops everywhere I go.
-VAL
Quoted 9 lines · Valerie Henson
Quoted 5 lines · Lennart Sorensen
Does it matter that google's recent report on disk failures indicated that SMART never predicted anything useful as far as they could tell? Certainly none of my drive failures ever had SMART make any kind of indication that anything was wrong.
I saw that talk, and that's not what I got out of it. They found that SMART error reports _did_ correlate with drive failure.
In fact, a certain small set of SMART indicators were a very good sign that a drive would fail.
Quoted 5 lines
However, they found that the correlation was not strong enough to make it economically feasible to replace disks reporting SMART failures, since something like 70% of disks were still working a year after the first failure report. Also, they found that some disks failed without any SMART error reports.
Indeed, SMART registered no counts at all for most failures, so on the whole, it can't be said that SMART can predict failures.
So: not a good idea to base your backup scheme on SMART warnings, but not entirely useless.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 6 lines · Tomasz Kłoczko
Quoted 4 lines · Christoph Hellwig
Numbers, please. So far in all interesting benchmarks it actually was slower. But when they're faster than XFS somewhere I'd defintly be interesting in looking at why this is true and if possible and important enough fix it.
Christoph, could you show some numbers as well? While I usually trust your opinion, I have yet to see any substantial argument against ZFS from your side.
http://blogs.sun.com/bill/#zfs_vs_the_benchmark
If you read closely you may notice that ZFS had relatively little to do with read performance under heavy write load. ZFS simply has "some fancy I/O scheduling code" that in particular deals with deadlines. The Linux equivalent appears to be CONFIG_IOSCHED_DEADLINE. But the quoted benchmark does not mention which scheduler was used for Linux.
So unless the benchmark is redone and properly documented, its numbers are fairly worthless. Bummer.
"The company I work for would probably balk if I put that script here"
No publically available benchmark. So even if a third party wanted to, it couldn't recreate the benchmark. Again, fairly worthless.
So by my count, neither side has showed any worthwile numbers. Whether ZFS performance is better or worse is anyone's guess.
Jörn
Signature · 2 lines
Simplicity is prerequisite for reliability. -- Edsger W. Dijkstra
Quoted 8 lines · Tomasz Kłoczko
Quoted 6 lines · Stefan Richter
Quoted 3 lines · Tomasz Kłoczko
Current FUSE implemntation can't be
comparable in aspects of speed and probably never will be on using threadsDid you measure this on a few hardwares and workloads?
Quoted 7 lines
Before asking firs you must try look on current ZFS on FUSE discuss phorums/docu .. like on:
http://groups.google.com/group/zfs-fuse/feed/rss_v2_0_msgs.xmlZFS on Solaris provides for many workloads better speed than any Linux technology on the same hardware but ZFS ond FUSE in current form provides lower speed than now avalaible Linux technologies.
There isn't a lot of useful facts in this last sentence.
Quoted 1 line
Example http://groups.google.com/group/zfs-fuse/msg/5b10c69707a46c07:
That's much more like it...
Quoted 6 lines
"According to bonnie, I see 125 MB/s reads on ext3+RAID5, 65 MB/s on ZFS+RAID5 (using Linux's software RAID) and 20 MB/s on ZFS+raidz (using the same raw drives). Writes are also proportionally slower. The real performance hit with ZFS-FUSE was random accesses for lots of small files. The bonnie++ results showed something like 75 random seeks for ZFS vs 470 for ext3 (..)"
...although the tester doesn't say a lot about the test setup (e.g. no word on the used hardware).
Another thread in the forum links to
http://www.ntfs-3g.org/performance.html which shows that filesystems implemented on top of FUSE may actually yield performance in the same league as classic Linux filesystems. However that page doesn't say anything about the test method either, beyond that bonnie++ was used.
Quoted 2 lines
On the same phorun you can find threads with discuss about utilize treading under FUSE.
Could you point out what you meant by "not comparable... on using threads"? (Who is using which threads for what purposes; what are the pertaining issues?) Then I might be able to find those forum posts which explain why a FUSE fs can never work comparably well "with threads".
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
(very simmilar case to ALSA and mixing in user space ..
Audio is about guaranteed latency, not "speed".
You meam "guaranteed worser latency" ?
I meant that the central requirement on the design and implementation of audio subsystems is an (ideally guaranteed) bounded maximum of latencies; and that's exactly the major point where I heard that there are problems with ALSA driver components in userspace. You were talking about throughput of storage systems, for which latencies of the software part of the stack do not play such a central role. Therefore your comparison appeared off the mark to me.
Signature · 3 lines
Stefan Richter -=====-=-=== -=-- =---- http://arcgraph.de/sr/
Quoted 7 lines · In article <[email protected]> you wrote:
I meant that the central requirement on the design and implementation of audio subsystems is an (ideally guaranteed) bounded maximum of latencies; and that's exactly the major point where I heard that there are problems with ALSA driver components in userspace. You were talking about throughput of storage systems, for which latencies of the software part of the stack do not play such a central role. Therefore your comparison appeared off the mark to me.
Unfortunatelle Latency is critical for a number of critical applications like databases or file based transaction systems (mail, news) - mainly the users of fsync().
Gruss
Bernd
Quoted 9 lines · Bernd Eckenfels
Quoted 7 lines · In article <[email protected]> you wrote:
I meant that the central requirement on the design and implementation of audio subsystems is an (ideally guaranteed) bounded maximum of latencies; and that's exactly the major point where I heard that there are problems with ALSA driver components in userspace. You were talking about throughput of storage systems, for which latencies of the software part of the stack do not play such a central role. Therefore your comparison appeared off the mark to me.
Quoted 3 lines
Unfortunatelle Latency is critical for a number of critical applications like databases or file based transaction systems (mail, news) - mainly the users of
fsync().
Whether you mix audio in userspace or kernel does not impact latency - you still need to schedule the application playing audio every N milliseconds or there will be dropouts. I don't see where audio mixing issue has any relevance to this thread.
Lee
Quoted 8 lines · In article <[email protected]> you…
Quoted 3 lines
Unfortunatelle Latency is critical for a number of critical applications like databases or file based transaction systems (mail, news) - mainly the users of
fsync().Whether you mix audio in userspace or kernel does not impact latency - you still need to schedule the application playing audio every N milliseconds or there will be dropouts. I don't see where audio mixing issue has any relevance to this thread.
Well, I was not talking about Audio - I was saying that there are a class of applications which need low latency on commits (fsync or rename).
And for the audio mixer, the problem is that you have multiple reschedules and data ping pong if you do user mode mixing - i guess.
Anyway.. not important. I still think filesystem servers can be a good thing. And I really hope we will find some interesting numbers on ZFS advantages...
Gruss
Bernd
Quoted 4 lines · Bernd Eckenfels
Quoted 3 lines · In article <[email protected]> you wrote:
I meant that the central requirement on the design and implementation of audio subsystems is an (ideally guaranteed) bounded maximum of latencies;
[...]
Quoted 2 lines
Quoted 2 lines
You were talking about throughput of storage systems, for which latencies of the software part of the stack do not play such a central role.
[...]
Quoted 3 lines
Unfortunatelle Latency is critical for a number of critical applications like databases or file based transaction systems (mail, news) - mainly the users of
fsync().
Yes, I know that; and there may even be mass storage systems with actual realtime requirements. Note though that latencies in FUSE-based filesystems due to (what do I know) context switches and possibly CPU scheduling latencies are not necessarily relevant to the overall latencies in applications which you mentioned.
Anyway, that's not at all what the original poster alluded to when he made vague statements about FUSE vs. kernelspace filesystems, finishing in the odd comparison with ALSA.
(Please try not to delete Ccs.)
Signature · 3 lines
Stefan Richter -=====-=-=== -=-- =---- http://arcgraph.de/sr/
Quoted 3 lines · Mike Snitzer
ZFS does have some powerful features but much of it depends on their broken layering of volume management. Embedding the equivalent of LVM into a filesystem _feels_ quite wrong.
They have a clustering concept in their volume management that isn't expressable with something like LVM. That justifes their approach from what I can see.
Quoted 4 lines
That aside, the native snapshot capabilities of ZFS really stand out for me. The redirect on write semantics aren't exclusive to ZFS; NetApp's WAFL employs the same. But with both ZFS and WAFL they were designed to do snapshots extremely well from the ground up.
Write allocation for these kinds of system (especially when concerned with mirroring) is non-trivial.
Quoted 5 lines
Unfortunately in order for Linux to incorporate such a feature I'd imagine a new filesystem would need to be developed with redirect on write at its core. Can't really see ext4 or any other existing Linux filesystem grafting such a feature into it. But even though I can't see it; do others?
You also can't use the standard page cache to buffer all of the sophicated semantics of these systems and have to create your own.
Quoted 3 lines
I've learned that Sun and NetApp's lawyers had it out over the redirect on write capability of ZFS. When the dust settled Sun had enough patent protection to motivate a truce with NetApp.
I think they are still talking and it's far from over the last I heard. The creation of a new inode and decending indirect blocks is a fundamental concept behind WAFL. Also ZFS tends to be a heavy weight as far as metadata goes and quite possibly uneccessarily so which is likely to effect performance for things related to keep a relevant block allocation map in memory. ZFS is a complete pig compared to traditional file systems.
Quoted 7 lines
The interesting side-effect is now ZFS is "open" and with that comes redirect on write in a file system other than WAFL. But ZFS's CDDL conflicts with the GPL so I'm not too sure how Linux could hit the ground running in this potentially patent mired area of filesystem development. The validity of NetApp having patented redirect on write aside; does the conflict between CDDL and GPL _really_ matter? Or did the CDDL release of ZFS somehow undermine NetApp's WAFL patent?
That doesn't really matter. FUSE could be extended to handle this kind of stuff and still have it be in userspace. The BSD get around including Stephen Tweedy's (sp?) ext2 header file by making the user manually compile it. That's not a problem for Linux folks that can download a patch and compile a kernel.
FreeBSD already has a port of ZFS. Just for a kick, Google for that as a possible basis for a Linux kernel port.
bill
On Sat, 14 Apr 2007, Mike Snitzer wrote:
[..]
Quoted 3 lines
ZFS does have some powerful features but much of it depends on their broken layering of volume management. Embedding the equivalent of LVM into a filesystem _feels_ quite wrong.
Hundret years ago some people told "people can fly depends on broken idea".
LVM was never defined strictly and no one says current LVM "definition/abstracion" is finished and complet .. try to think about this and never use phrases like "broken idea" when you have real proof for "people can fly" because you can simple look on sky or "embedding the equivalent of LVM into a filesystem _feels_ quite wrong" when you can use real and working ZFS on some OSes .. specialy when this new "definition" of LVM provides better functionalities than current FS way of thinking about some OS aspects.
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
A message that is not in the archive (sent off-list or not yet received): [email protected]
On Mon, 16 Apr 2007, Mike Snitzer wrote:
[..]
Quoted 2 lines
The ZFS volume management piece could've easily been abstracted to be outside of the actual filesystem.
Sorry .. but can you poof this by show one example of this kind of problems ?
Quoted 1 line
In fact its more natural to maintain that separation.
Sorry but cause of developing ZFS was not braking classic/"natural" FS abstraction but try to solve things like limited I/O bandwich and statatistical number of errors on storing portion informations (it ~constant but capacity of used storage divices grows) now avalaible storage devices and simplify managing hundrets or thousands phisical devices. From this point of view it is state of art.
Quoted 1 line
It allows people to have choices about how they manage the underlying block layer rather than imposing one specialized solution. Look at the various technologies in Linux (LVM2, MD, Linux Multipath) to understand the flexibility Linux provides for managing storage.
Except ext4 all this all this technologies not solves some problems which are plugged by ZFS. Also adding checksuming to ext3 (aka ext4) it is not all what profides ZFS way ..
Quoted 1 line
Not to mention the availability of commercial solutions from the various storage vendors.
It is not matter comercial/non-commercial .. forget about this argument (don't practice this kind of FUDing).
ZFS was implemnted and is useable .. very useable like many commersial technologiess (do you remenber form where come NFS to Linux ?)
It is so useable it case many people are now ready for switch from Linux to Solaris.
Few days ago I'm swich two backup servers with few TB storage from Linux to Solaris .. only because client want use ZFS .. because ZFS is EXCELENT for this kind tasks (only because it allow save many thousands of <put_your_currency_name> because it allow better utilize the same storage) and trust me .. cases like this will be more .. much more and changing licensing Linux code will be MUCH more easier than reinventing wheel (wich will be reimplemnting ZFS under GPL).
Problem is not on technical area but on licensing and it is plain Linux word problem because neccessary in this case changes on CDDL side will make this code less oppended than now .. so you can (probably ?) forget about GPLing ZFS code (and ZFS it is not all what will good to have from Sol in Linux). IMO current Linux licensing less is importand than bringing in possible simpler way things like ZFS to Linux. So best/simpler way will be start change Linux licensing for save all GPL goodies and allow interract with code on license like CDDL. Licensing is for allow keep in best possible form Linux. If it can't do this in best possible way it must be change (must evolve .. like many othes things around).
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 8 lines · Tomasz Kłoczko
...
Few days ago I'm swich two backup servers with few TB storage from Linux to Solaris .. only because client want use ZFS .. because ZFS is EXCELENT for this kind tasks (only because it allow save many thousands of <put_your_currency_name> because it allow better utilize the same storage) and trust me .. cases like this will be more .. much more and changing licensing Linux code will be MUCH more easier than reinventing wheel (wich will be reimplemnting ZFS under GPL).
This is a technical mailing list, so let's start with technical arguments:
Why did this client want to use ZFS?
Because his boss was convinced by a marketing guy that ZFS was the best invention since sliced bread?
Or due to technical limitations in what Linux currently offers resulting in ZFS bringing him direct advantages on these servers?
If the latter, what exactly?
Quoted 10 lines
Problem is not on technical area but on licensing and it is plain Linux word problem because neccessary in this case changes on CDDL side will make this code less oppended than now .. so you can (probably ?) forget about GPLing ZFS code (and ZFS it is not all what will good to have from Sol in Linux).
IMO current Linux licensing less is importand than bringing in possible simpler way things like ZFS to Linux. So best/simpler way will be start change Linux licensing for save all GPL goodies and allow interract with code on license like CDDL.
Quoted 3 lines
Licensing is for allow keep in best possible form Linux. If it can't do this in best possible way it must be change (must evolve .. like many othes things around).
There are at about 10.000 people who contributed to the Linux kernel, some of them unreachable or even dead.
AFAIK the only way to relicense the Linux kernel that wouldn't be legally questionable in any jurisdiction would be to get each one of them to agree to the license change.
IOW, it's more or less practically impossible to relicense the kernel.
If the GPL is a problem for you, FreeBSD might be a good choice for you.
Quoted 1 line
kloczek
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 10 lines · Adrian Bunk
Quoted 8 lines · Tomasz Kłoczko
...
Few days ago I'm swich two backup servers with few TB storage from Linux to Solaris .. only because client want use ZFS .. because ZFS is EXCELENT for this kind tasks (only because it allow save many thousands of <put_your_currency_name> because it allow better utilize the same storage) and trust me .. cases like this will be more .. much more and changing licensing Linux code will be MUCH more easier than reinventing wheel (wich will be reimplemnting ZFS under GPL).
Quoted 4 lines
This is a technical mailing list, so let's start with technical arguments:
Why did this client want to use ZFS?
Because switching to Solaris was chipper than buying next faster FC/SCSI storage. Simple ?
Quoted 2 lines
Because his boss was convinced by a marketing guy that ZFS was the best invention since sliced bread?
In this scenario ther is no place for "marketing guy" .. try again .. (maybe it can occure in US or Germany but trust me .. not in Poland 8-)
Quoted 2 lines
Or due to technical limitations in what Linux currently offers resulting in ZFS bringing him direct advantages on these servers?
Yes .. it is Linux limitiations because it is very hard to provide simultanouse streams of backup data with threaded compression (using in this case pbzip2) with good CPUs utilization because most streams waits on I/Os and most of CPUs are not fully utilized. All this becase single stram of compressed data can't be easy dinamically switched to another (not busy) disk in JBOD. ZFS by two level allocation (on device and block level) will not wait for finish I/O but will try use another/not busy device in ZFS pool. This is *main* reason integrate in one layer VFS and LVM in case ZFS. By integrate this two layers you can make deciion where data will be written depending on *current* devices utilization. In all other "classic" ways you will break layered OS model .. so in ZFS case conclution like "we must integrate this two layers in one" it is not bug but feacture and was FUNDAMENTAL.
This is not all .. backup data must be safe in best possible form .. in time .. and it mean in this case that checksumming is NECCESSARY. Look .. ext4 for now only have plans for implemting checksuming (http://www.bullopensource.org/ext4/files/ext4.txt) and ATM on Linux there is no FS with this kind abilities .. so yes again: this was Linux technical limitation.
Quoted 10 lines
Quoted 10 lines
Problem is not on technical area but on licensing and it is plain Linux word problem because neccessary in this case changes on CDDL side will make this code less oppended than now .. so you can (probably ?) forget about GPLing ZFS code (and ZFS it is not all what will good to have from Sol in Linux).
IMO current Linux licensing less is importand than bringing in possible simpler way things like ZFS to Linux. So best/simpler way will be start change Linux licensing for save all GPL goodies and allow interract with code on license like CDDL.
Quoted 6 lines
Quoted 3 lines
Licensing is for allow keep in best possible form Linux. If it can't do this in best possible way it must be change (must evolve .. like many othes things around).
There are at about 10.000 people who contributed to the Linux kernel, some of them unreachable or even dead.
Do you know who was Paracelsus ? He was medic hundriet years ago. They discover and verbalise some kind fundamental (?) law for medicine which can be used not only on medicine area. He sayd "kills not subtance but dose of substance". So anything can kill you/animal/project .. you can kill someon also using oxigen (not only low level of oxigen kills but also to much can kill). Try to think on how this law on how many diffret ways can be trasformed/appied to this kind of arguments. Look on how many developers migrate to another unices in last few years (count only two for simplicity like Solaris and MOX). Try looking for public forums statistics for example Linux vs. Solaris and after this try to answer on "is it 10k is it realy big number in this case or not ?" (IIRC google provides very good tools for anyone who want this kind answers).
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 33 lines · Tomasz Kłoczko
Quoted 14 lines · Adrian Bunk
Quoted 4 lines · Tomasz Kłoczko
...
Few days ago I'm swich two backup servers with few TB storage from Linux to
Solaris .. only because client want use ZFS .. because ZFS is EXCELENTQuoted 8 lines
for
this kind tasks (only because it allow save many thousands of <put_your_currency_name> because it allow better utilize the same storage)
and trust me .. cases like this will be more .. much more and changing licensing Linux code will be MUCH more easier than reinventing wheel (wich
will be reimplemnting ZFS under GPL).Quoted 4 lines
This is a technical mailing list, so let's start with technical arguments:
Why did this client want to use ZFS?
Because switching to Solaris was chipper than buying next faster FC/SCSI storage. Simple ?
Quoted 2 lines
Because his boss was convinced by a marketing guy that ZFS was the best invention since sliced bread?
In this scenario ther is no place for "marketing guy" .. try again .. (maybe it can occure in US or Germany but trust me .. not in Poland 8-)
Quoted 2 lines
Or due to technical limitations in what Linux currently offers resulting in ZFS bringing him direct advantages on these servers?
Quoted 13 lines
Yes .. it is Linux limitiations because it is very hard to provide simultanouse streams of backup data with threaded compression (using in this case pbzip2) with good CPUs utilization because most streams waits on I/Os and most of CPUs are not fully utilized. All this becase single stram of compressed data can't be easy dinamically switched to another (not busy) disk in JBOD. ZFS by two level allocation (on device and block level) will not wait for finish I/O but will try use another/not busy device in ZFS pool. This is *main* reason integrate in one layer VFS and LVM in case ZFS. By integrate this two layers you can make deciion where data will be written depending on *current* devices utilization. In all other "classic" ways you will break layered OS model .. so in ZFS case conclution like "we must integrate this two layers in one" it is not bug but feacture and was FUNDAMENTAL.
You are having IO problems doing bzip2 ???
This sounds as if your application is doing something silly like e.g. using O_DIRECT or you are mounting your filesystems with "-o sync".
On my 1.8 GHz Athlon, it takes 2.5 minutes to compress a 250 MB kernel tar to 40 MB - and this is with the data cached in RAM, so no IO involved. That's 1.7 MB/s resp. 0.3 MB/s.
I'm even having problems to imagine IO problems with 4 or 8 CPU machines doing parallel bzip2 to a single disk - with any filesystem.
Quoted 6 lines
This is not all .. backup data must be safe in best possible form .. in time .. and it mean in this case that checksumming is NECCESSARY. Look .. ext4 for now only have plans for implemting checksuming
(http://www.bullopensource.org/ext4/files/ext4.txt) and ATM on Linux there is no FS with this kind abilities .. so yes again: this was Linux technical limitation.
bzip2 already checksums all data, so why do you need a second checksum at the filesystem level for your backups?
Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
Problem is not on technical area but on licensing and it is plain Linux word problem because neccessary in this case changes on CDDL side will make
this code less oppended than now .. so you can (probably ?) forget about GPLing ZFS code (and ZFS it is not all what will good to have from Sol in Linux).
IMO current Linux licensing less is importand than bringing in possible simpler way things like ZFS to Linux. So best/simpler way will be start change Linux licensing for save all GPL goodies and allow interract with code on license like CDDL.Quoted 7 lines
Quoted 4 lines
Licensing is for allow keep in best possible form Linux. If it can't do this in best possible way it must be change (must evolve .. like many othes
things around).There are at about 10.000 people who contributed to the Linux kernel, some of them unreachable or even dead.
Quoted 12 lines
Do you know who was Paracelsus ? He was medic hundriet years ago. They discover and verbalise some kind fundamental (?) law for medicine which can be used not only on medicine area. He sayd "kills not subtance but dose of substance". So anything can kill you/animal/project .. you can kill someon also using oxigen (not only low level of oxigen kills but also to much can kill). Try to think on how this law on how many diffret ways can be trasformed/appied to this kind of arguments. Look on how many developers migrate to another unices in last few years (count only two for simplicity like Solaris and MOX). Try looking for public forums statistics for example Linux vs. Solaris and after this try to answer on "is it 10k is it realy big number in this case or not ?" (IIRC google provides very good tools for anyone who want this kind answers).
I don't need Google for this.
It's easy to extract from git that patches that were applied to the Linux kernel during the last 2 years contained 3196 unique
Signed-off-by: lines.
Some people might have lines with different email addresses, but this still makes > 2000 contributors during the last 2 years alone. Plus all the other people who contributed during the first 14 years of Linux kernel development but didn't during the last 2 years.
You could argue whether there were really 10.000 people or "only" 5.000 people who contributed code to the Linux kernel, but that doesn't make a real difference.
Quoted 1 line
kloczek
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 4 lines · Neil Brown
What, specifically, is there in ZFS that you want to use in Linux? There is lots of interesting stuff in there, but which bit has caught your attention? It is possible that that functionality can be incorporated into Linux without trying to clone or copy ZFS.
Filesystem block hashes (checksums) and in-fs redundancy (using the hashes to determine which copy of data is correct in case of corruption). It would mean multi-device filesystem.
Signature · 1 line
Krzysztof Halasa
Quoted 10 lines · Krzysztof Halasa
Quoted 4 lines · Neil Brown
What, specifically, is there in ZFS that you want to use in Linux? There is lots of interesting stuff in there, but which bit has caught your attention? It is possible that that functionality can be incorporated into Linux without trying to clone or copy ZFS.
Filesystem block hashes (checksums) and in-fs redundancy (using the hashes to determine which copy of data is correct in case of corruption). It would mean multi-device filesystem.
Yup - multidevice support in XFS is something I'm working on at the moment. There's many nice features that multi-device filesystems can take advantage of (different geomteries, multiple journals, internal RAID, redundancy, partially online/offline filesystems, partial filesystem repair, etc).
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 11 lines · David R. Litwin
Before I go on, let me appologise. I don't really know what I hope to accomplish, beyond trying to garner thoughts (and support?) for the topic.
Essentially: I want to use Linux and ZFS. I don't particularly care about licences or any of the rest of that nonsense. The code is there; it merely needs to be made to work with Linux. Done and done -- provided I can find some one to do this for me (I'd do it myself, but I haven't the foggiest notion how to go about such a feat).
By the way, forget about this FUSE business. I don't know why they're
bothering: It's not real, it's slow and, in general, silly.
This seems to me to be a rather uninformed, arrogant, and quite stupid comment.
Quoted 11 lines
What are the thoughts of the Linux community?
I appologise right now for my intrusion. I am a Linux-nobody; I freely admit it. I haven't even subscribed to this list (so do CC me) because I don't want to be over-whelmed with the list's glorious posts. But, part of Linux is it's being a community. If a member of this community (that is, a user of Linux) can't ask the others their
thoughts and opinions, then the community has failed in a large respect. Take this letter as you will.
Quoted 5 lines · David R. Litwin
Before I go on, let me appologise. I don't really know what I hope to accomplish, beyond trying to garner thoughts (and support?) for the topic.
Essentially: I want to use Linux and ZFS. I don't particularly care about licences or any of the rest of that nonsense.
The license that protects the code we write is far from nonsense.
Quoted 5 lines
The code is there; it merely
needs to be made to work with Linux. Done and done -- provided I can find some one to do this for me (I'd do it myself, but I haven't the foggiest notion how to go about such a feat).
I suggest the first thing you do is contact the ZFS developers and convince them to release their code under a license that's GPL compatible, then we can start looking at a Linux port.
Quoted 3 lines
By the way, forget about this FUSE business. I don't know why they're
bothering: It's not real, it's slow and, in general, silly.
FUSE is nice for trying out new and interresting ideas in userspace - it has its uses.
Quoted 2 lines
What are the thoughts of the Linux community?
Can't tell you that, all I can tell you is my own oppinion. I believe ZFS for Linux would be interresting, if not for any other reason then for compatibility. I'd personally like to see it - but, settle the license issue first.
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
On Fri, 13 Apr 2007, David R. Litwin wrote:
[..]
I'm just found something new in filtered folder by "ZFS" word in RSS feed from blogs.sun.com and on firs look it may be some continuation of this thread:
http://blogs.sun.com/darren/entry/zfs_under_gplv2_already_exists
I'm not check completly this .. so don't beat me if it is not true :)
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 10 lines · On Wed, 2 May 2007 17:03:22 +0200 (CEST) Tomasz Kłoczko…
On Fri, 13 Apr 2007, David R. Litwin wrote:
[..]I'm just found something new in filtered folder by "ZFS" word in RSS feed from blogs.sun.com and on firs look it may be some continuation of this thread:
http://blogs.sun.com/darren/entry/zfs_under_gplv2_already_exists
I'm not check completly this .. so don't beat me if it is not true :)
Useful if accurate as the GPLv2 and Sun origin means youa re clear for any patents Sun used in that code but maybe not for any in the pure CDDL bits. Definitely positive.
Quoted 11 lines · Alan Cox
Quoted 7 lines
I'm just found something new in filtered folder by "ZFS" word in RSS feed from blogs.sun.com and on firs look it may be some continuation of this thread:
http://blogs.sun.com/darren/entry/zfs_under_gplv2_already_exists
I'm not check completly this .. so don't beat me if it is not true :)
Useful if accurate as the GPLv2 and Sun origin means youa re clear for any patents Sun used in that code but maybe not for any in the pure CDDL bits. Definitely positive.
Well, except remember that they only made sources available so that GRUB could boot OpenSolaris. Grub only requires read-only access to the filesystem, and what was made available under GPLv2 was only a distinct subset of the full ZFS sources.
- Ted