Skip to contentSkip to conversation

vfs: Filemash fs

RFC PATCH New Ram Pai 1 participant 1 message Apr 3, 2013

Subsystems (2)
lore
Share & tools
lore

Ram Pai

author
#0 lore raw

RFC PATCH 1/1 vfs: Filemash fs

The following patch implements a filesystem driver which provides the ability to mashup exisiting files in creative ways in-order to create new files.

Think of it as a way to union files; not filesystems.

Its a prototype idea with a prototype implementation. Tested and working on 3.0.9-rc1. I have included Documentation file which details the idea with examples and possible applications.

Any suggestions/ideas to make this useful and generally applicable is very much appreciated!

Documentation/filesystems/filemash.txt |  422 ++++++++++++++++++
fs/Kconfig                             |    1 
fs/Makefile                            |    1 
fs/filemashfs/Kconfig                  |    6 
fs/filemashfs/Makefile                 |    5 
fs/filemashfs/super.c                  |  743 +++++++++++++++++++++++++++++++++
6 files changed, 1178 insertions(+)
Documentation/filesystems/filemash.txt new file +422 -0
new file mode 100644
index 0000000..9ff93b9
@@ -0,0 +1,422 @@
+File Mash Filesystem
+---------------
+Contents:
+	1) Overview
+ 	2) How to use it?
+	3) Features
+	4) Format of mount option
+	5) Practical examples
+	6) Nice to have capabilities
+	7) FAQ
+
+
+1) Overview
+-----------
+
+Consider the following situation:
+
+a) You have a large file that needs to be expanded but you are running out of
+space on that filesystem. For muliple reasons you do not have the capability
+to expand the filesystem. You either have to move the file to a bigger
+filesystem or cleanup space on the current filesystem. What if you had the
+capability to expand the file to another filesystem?
+
+b) You have a large file whose contents change every day by a few bytes. An
+incremental backup of the filesystem; everyday using rsync, will end up making
+copies of the large file everytime. Instead; if your large file was mashedup
+using multiple smaller files, an incremental backup of the filesystem will
+just backup the smaller files that have changed; thus saving lots of backup
+space.
+
+c) Large files; imagine virtual disk images, that have some sections that
+hardly change, some sections that have significant i/o activity, and some
+sections that need redundancy. This need can be met if there is a way
+to mashup the file using smaller files residing on filesystem appropriate
+to the need.
+
+d) Imagine multiple files containing data which is mostly common. In such case
+storage space is unnecessary wasted.  What if there was a way to separate out
+the common data into a file and use that file to mashup each of the files?
+Data can be deduplicated saving lots of storage space.
+
+e) Imagine a script file which for whatever reason cannot be modified, but
+needs to be fixed to make it usable. What if there was way to temporarily
+patch that script without actually modifying the script?
+
+f) <fill in more applications>
+
+FileMash filesystem is an attempt to provide the ability to mash-up a new
+file using multiple existing files in creative ways.
+
+It is currently implemented as a filesystem driver that stacks on top of other
+filesystem drivers. It provides the capability to create the illusion of a new
+file by using the space provided by the sub-ordinate files residing on other
+filesystems.
+
+In other words, it provides unioning capability -- the capability to union
+files, not filesystems.
+
+
+2) How to use it?
+------------------
+
+Apply the patch.  This patch has been prototyped/tested on 3.0.9-rc1.
+
+$ make fs/filemashfs/filemashfs.ko
+$ modprobe  filemashfs.ko
+
+$ cat > file_even <<-!
+0000
+2222
+4444
+6666
+8888
+!
+
+$ cat > file_odd <<-!
+1111
+3333
+5555
+7777
+9999
+!
+
+# Example 1
+# create a new file named file_evenodd using contents of file 'file_odd' followed by
+# contents of file 'file_even'. Use the content in a striped fashion.
+$ touch file_evenodd
+$ mount -t filemashfs -ofile=file_even,file=file_odd,layout=stripe:5 s file_evenodd
+
+$ cat file_evenodd
+0000
+1111
+2222
+3333
+4444
+5555
+6666
+7777
+8888
+9999
+
+# Example 2
+# create a new file named file_evenodd using contents of file 'file_odd' followed by
+# contents of  file 'file_even'. Use the content in a striped fashion.
+$ touch file_oddeven
+$ mount -t filemashfs -ofile=file_odd,file=file_even,layout=stripe:5 s file_oddeven
+
+$ cat file_oddeven
+1111
+0000
+3333
+2222
+5555
+4444
+7777
+6666
+9999
+8888
+
+# Example 3:
+# create a new file named file_concat using entire contents of file 'file_odd' followed by
+# entire contents of  file 'file_even'.
+$ touch file_concat
+$ mount -t filemashfs -ofile=file_odd,file=file_even,layout=concat s file_concat
+$ cat file_concat
+1111
+3333
+5555
+7777
+9999
+0000
+2222
+4444
+6666
+8888
+
+# Example 4:
+# Here is a example demonstating how contents get fanned out to subordinate file when data is
+# written to the mashedup file.
+
+$ touch firstfile
+$ touch secondfile
+$ touch mashfile
+
+$ mount -t filemashfs -ofile=firstfile,file=secondfile,layout=stripe:10 s mashfile
+$ cat > mashfile <<!
+> 000000000
+> 111111111
+> 222222222
+> 333333333
+> 444444444
+> 555555555
+> 666666666
+> 777777777
+> 888888888
+> 999999999
+> !
+
+$ cat firstfile
+000000000
+222222222
+444444444
+666666666
+888888888
+
+$ cat secondfile 
+111111111
+333333333
+555555555
+777777777
+999999999
+
+
+
+2) Features
+-----------
+
+FileMash filesystem provides two features currently.
+
+a) Striping
+b) Concatenation
+
+If the data needs to striped evenly across all the subordinate files, striping
+option helps do that.  In the above example layout=stripe:5 is used. It tells
+the filesystem to use first 5 bytes from the first file, the second 5 bytes
+from the second file, the third 5 bytes from the first file, and so on an so
+forth till the first and the second file have no more data to offer.
+
+If the data needs to concatenated using all the subordinate files, concat
+option is your choice. In the above example layout=concat is used. It tells
+the filesystem to use all the available bytes in the first file and then
+proceed to use all the available bytes in the second file.
+
+The area of the subordinate file to be used can be specified on the command
+line. This will ensure that only the specified area is used by the mashed file
+and thus cannot cross the boundaries of that sub-ordinate file.
+
+$ mount -t filemashfs -ofile=file_odd:10:20,file=file_even,layout=concat s file_concat
+
+This command specifies that file_odd from byte 10 to the next 20bytes must
+be used. The sub-ordinate file cannot not be expanded by writing additional
+data to the mashedup file.
+
+
+3) Format of the mount option
+--------------------------
+
+ The format of the mount option is as follows
+
+ -o (file=filename:[offset]:[length],)*,layout=<stripe:size|concat>,.....
+ 
+  The file keyword specifies the name of the sub-ordinate file to be used
+  for mashup. The 'offset' specifies the location from which the file can
+  be used for read/write. The 'length' specifies the size of the file, starting
+  from the 'offset' location, to be used for read/write. If 'offset' is not
+  specified the file is used from its zeroth byte. if 'length' is not
+  specified the file is assumed to have infinite size, limited only by 
+  the size of the filesystem on which it resides.
+
+  Multile files can be specified. The order in which files are specified is
+  significant, since it dictates the order in which the mashed-up file will
+  map its contents to the underlying sub-ordinate files.
+
+  The layout keyword specifies the file mashup policy. Currently it supports
+  stripe or concat policy. The layout defaults to concat policy. The 'stripe'
+  keyword takes an addition option which specifies the length of the stripe.
+
+
+$ mount -t filemashfs -ofile=file_even,file=file_odd,layout=stripe:5  \
+                 s file_evenodd
+
+          file_even                           file_evenodd
+          ,----,                                ,----,
+          |0000| <----------------------------- |0000|
+          |2222| <--,                ,--------  |1111|
+          |4444| <-, `--------------/---------  |2222|
+          |6666|<-, \              / ,--------  |3333|
+          |8888|<~ \ '---------- -/-/---------  |4444|
+          '----'  \ \            / / ,--------  |5555|
+                   \ '-------- -/-/-/---------  |6666|
+                    \          / / /  ,------   |7777|
+          file_odd   '--------/-/-/--/ ------   |8888|
+          ,----,             / / /  /  /-----   |9999|
+          |1111|  <---------' / /  /  /         '----'
+          |3333|  <----------' /  /  /
+          |5555|  <-----------'  /  /
+          |7777|  <-------------'  /
+          |9999|  <---------------'
+          '----' 
+        
+$ mount -t filemashfs -ofile=file_even:5,file=file_odd,layout=stripe:5  \
+               s file_evenodd
+
+          file_even                           file_evenodd
+          ,----,                                ,----,
+          |0000|    ,-------------------------- |2222|
+          |2222| <-'                 ,--------  |1111|
+          |4444| <------------------/---------  |4444|
+          |6666| <--,              / ,--------  |3333|
+          |8888| <-, '---------- -/-/---------  |6666|
+          '----'    \            / / ,--------  |5555|
+                     '-------- -/-/-/---------  |8888|
+                               / / /  ,------   |7777|
+          file_odd            / / /  /  ,-----  |9999|
+          ,----,             / / /  /  /        '----'
+          |1111|  <---------' / /  /  /         
+          |3333|  <----------' /  /  /
+          |5555|  <-----------'  /  /
+          |7777|  <-------------'  /
+          |9999|  <---------------'
+          '----' 
+
+$ mount -t filemashfs -ofile=file_even:5:10,file=file_odd,layout=stripe:5  \
+                s file_evenodd
+
+          file_even                           file_evenodd
+          ,----,                                ,----,
+          |0000|    ,-------------------------- |2222|
+          |2222| <-'                 ,--------  |1111|
+          |4444| <------------------/---------  |4444|
+          |6666|                   / ,--------  |3333|
+          |8888|                  / / ,-------  |5555|
+          '----'                 / / /  ,-----  |7777|
+                                / / /  /  ,---  |9999|
+                               / / /  /  /      '----' 
+          file_odd            / / /  /  /
+          ,----,             / / /  /  /
+          |1111|  <---------' / /  /  /         
+          |3333|  <----------' /  /  /
+          |5555|  <-----------'  /  /
+          |7777|  <-------------'  /
+          |9999|  <---------------'
+          '----' 
+
+
+$ mount -t filemashfs -ofile=file_even:10:20,file=file_odd:5:,layout=concat \
+                s file_concat
+
+          file_even                           file_concat
+          ,----,                                ,----,
+          |0000|     ,------------------------- |4444|
+          |2222|    /                ,--------  |6666|
+          |4444| <-`                / ,-------  |3333|
+          |6666| <---------------- / / ,------  |5555|
+          |8888|                    / /  ,----  |7777|
+          '----'                   / /  /  ,--  |9999|
+                                  / /  /  /     '----'
+                                 / /  /  /
+          file_odd              / /  /  /
+          ,----,               / /  /  /
+          |1111|              / /  /  /         
+          |3333|  <----------' /  /  /
+          |5555|  <-----------'  /  /
+          |7777|  <-------------'  /
+          |9999|  <---------------'
+          '----' 
+        
+
+4) Practical examples
+----------------------
+
+a) how to create a large file spanning multiple filesystems?
+
+assuming that we have 5 differen filesystems mounted at /F1, /F2, /F3, /F4, /F5
+
+$ touch /F1/f1
+$ touch /F2/f2
+$ touch /F3/f3
+$ touch /F4/f4
+$ touch /F5/f5
+
+$ touch /mybigfile
+$ mount -t filemashfs -ofile=/F1/f1,file=/F2/f2,file=/F3/f3,file=/F4/f4,file=F5/f5,layout=stripe:10000 s /mybigfile
+
+start writing data to /mybigfile and it will continue to grow that file till
+all the space on all the filesystem is exhausted! I have not tested it yet.
+But that should work by design. Implementation; I promise, is currently buggy :).
+
+
+
+b) how do I patch a file?
+
+Assume you have a script named hello.sh
+
+$ cat hello.sh
+#/bin/bash
+echo Spanish:  hola?
+echo Mandarin: ni hao ma?
+echo English:  I don't know :(
+echo Hindi:    aap kaise hai?
+echo Kannada:  Nee Hege Iddiya?
+
+	we know that "I don't know :(" has to be "How are you?".
+	So create a file named "fix" with the correct string in it.
+
+#echo 'How are you?' > fix
+
+	And mashup a new "hello.sh" file using parts of "hello.sh" and parts of
+	"fix".
+
+#mount -t filemashfs -ofile=hello.sh:0:73,file=fix,file=hello.sh:89,layout=concat s hello.sh
+
+# cat hello.sh
+#/bin/bash
+echo Spanish:  hola?
+echo Mandarin: ni hao ma?
+echo English:  How are you?
+echo Hindi:    aap kaise hai?
+echo Kannada:  Nee Hege Iddiya?
+
+<hello.sh is now fixed!>
+
+
+c) How do I deduplicate files?
+
+	I leave this as an exercise to you :)
+
+
+6. Nice to have capabilities
+-------------------------
+
+a) Ability to add or remove a file dynamically from a mashed up file.
+
+b) ??
+
+
+7. FAQ
+-------
+
+a) Why is this not implemented using FUSE?
+
+	I do not have numbers to prove this. But I think it might be less
+	performant, given that reads/writes have to make a few trips
+	between userspace and kernel.
+
+
+b) btrfs has a feature which allows a filesystem to extended using files from
+	other filesystem using loop device. Why is this different?
+
+	That feature lets you extend filesystem using files. The proposed
+	feature lets us extend files using filesystems. The features are
+	complementary in nature.
+
+
+c) One of the application; you mentioned above, mentions deduplication. Does it
+	deduplicate automatically?
+
+	No. The current prototype code does support any
+	deduplication. It just provides the mechanism to deduplicate
+	data and consolidate filesystem space. The user has to
+	identify the files that have
+	common data, move that data into a single file and mashup
+	all the other files using this new file.
+	A userspace tool can be written to do so.
+
+
+
+Thanks for your interest till now!
+------------------------------------------------------------------------
+version 0.1  (created the initial document, Ram Pai [email protected])
+version 0.2  (added nice to have capabilities and FAQ, based on inputs from
+			Chandra and Malahal)
fs/Kconfig +1 -0
index 780725a..4ea2cfd 100644
@@ -67,6 +67,7 @@ source "fs/quota/Kconfig"
 
 source "fs/autofs4/Kconfig"
 source "fs/fuse/Kconfig"
+source "fs/filemashfs/Kconfig"
 
 config GENERIC_ACL
 	bool
fs/Makefile +1 -0
index 9d53192..296f3b2 100644
@@ -127,3 +127,4 @@ obj-$(CONFIG_F2FS_FS)		+= f2fs/
 obj-y				+= exofs/ # Multiple modules
 obj-$(CONFIG_CEPH_FS)		+= ceph/
 obj-$(CONFIG_PSTORE)		+= pstore/
+obj-$(CONFIG_FILEMASH_FS)	+= filemashfs/
fs/filemashfs/Kconfig new file +6 -0
new file mode 100644
index 0000000..eaf2ee7
@@ -0,0 +1,6 @@
+config FILEMASH_FS
+	tristate "FileMash file system (EXPERIMENTAL)"
+	help
+	  Add support for FileMash filesystem.
+
+	  If unsure, say N.
fs/filemashfs/Makefile new file +5 -0
new file mode 100644
index 0000000..73b82d7
@@ -0,0 +1,5 @@
+#
+# Makefile for FileMash filesystem
+#
+obj-$(CONFIG_FILEMASH_FS) += filemashfs.o
+filemashfs-objs	:= super.o
fs/filemashfs/super.c new file +743 -0

743 lines not on this page · raw message