Data deduplication for Linux : lessfs
For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com
I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.
For the time being lessfs is now stable and even usable as a blockdevice with the help of ietd, loop or nbd.
Mark Ruijter
Quoted 3 lines · Mark Ruijter
For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com
It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.
Quoted 3 lines
I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.
Hi Roy,
Quoted 6 lines
It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.
Almost all the traffic is on the forum - open discussion. Only one person posted to the mailing list. ;-)
Quoted 5 lines
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
Typically the meta data on production systems is approx 10%~20% of the deduplicated stored data.
Stored data is on my systems 40x less then the data written to the filesystem.
For example, from a real life backup server making dozens of backups each day:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/cciss/c0d0p3 9.7G 2.4G 6.9G 26% / /dev/cciss/c0d0p1 99M 23M 72M 24% /boot tmpfs 7.9G 0 7.9G 0% /dev/shm /dev/cciss/c0d0p4 246G 6.0G 241G 3% /meta /dev/cciss/c0d1p1 274G 73G 202G 27% /blockdata /dev/cciss/c1d0p1 4.1T 1.5T 2.7T 35% /data lessfs 4.1T 1.5T 2.7T 35% /pooldata [root@lessfssrv pooldata]# du . -s -h 31T . [root@lessfssrv pooldata]# ls -alh /data/current/ total 314G drwxr-xr-x 2 root root 26 Jun 1 00:12 . drwxr-xr-x 6 root root 59 Jun 1 00:12 .. -rw-r--r-- 1 root root 314G Jun 22 14:26 blockdata.tch [root@lessfssrv pooldata]# ls -alh /meta/current/ total 1.4G drwxr-xr-x 2 root root 63 Jun 1 00:12 . drwxr-xr-x 6 root root 59 Jun 1 00:12 .. -rw-r--r-- 1 root root 1.3G Jun 22 14:52 blockusage.tch -rw-r--r-- 1 root root 89M Jun 22 14:45 dirent.tcb -rw-r--r-- 1 root root 89M Jun 22 14:52 metadata.tcb
Mark.
Quoted 20 lines
roy
Signature · 16 lines
Roy Sigurd Karlsbakk (+47) 97542685 [email protected] http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk. _______________________________________________ linux-lvm mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 1 line · Roy Sigurd Karlsbakk
On 24. juni. 2009, at 17.12, Mark Ruijter wrote:
<snip>
Quoted 8 lines
Quoted 2 lines
I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
As of the 2.6.30 kernel a default mount option is "relatime". That is a relatively new mount option that drastically reduces the number of atime updates. ie. Not realtime, but relatime.
Greg
Signature · 10 lines
Greg Freemyer Head of EDD Tape Extraction and Processing team Litigation Triage Solutions Specialist http://www.linkedin.com/in/gregfreemyer First 99 Days Litigation White Paper - http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf The Norcross Group The Intersection of Evidence & Technology http://www.norcrossgroup.com
Quoted 7 lines · Mark Ruijter
Hi Roy,
Quoted 3 lines
It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.
Almost all the traffic is on the forum - open discussion. Only one person posted to the mailing list. ;-)
Why??
Mailing lists are so much easier to use. Instead of visiting a bunch of websites, they all sit in my mailbox.
Quoted 4 lines
Quoted 1 line
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
Typically the meta data on production systems is approx 10%~20% of the deduplicated stored data. Stored data is on my systems 40x less then the data written to the filesystem.
The problems with metadata is not that they take up a lot of space, but that they are updated so regularly. As Greg Freemyer pointed out, relatime will help a lot, but still, deduplicating metadata may take up a serious amount of time because of the frequent updates.
roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.
Quoted 6 lines · Roy Sigurd Karlsbakk
Quoted 3 lines · Mark Ruijter
For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com
Quoted 6 lines
It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.
Quoted 3 lines
I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.
Signature · 2 lines
Les Mikesell [email protected]
Quoted 5 lines · Les Mikesell
Quoted 4 lines · Roy Sigurd Karlsbakk
Quoted 3 lines · Mark Ruijter
For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com
Quoted 7 lines
Quoted 5 lines
It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.
Quoted 3 lines
I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.
My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".
roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.
Quoted 6 lines
Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.
Exactly. And this is why lessfs can easily write files with speeds up to 200MB~300MB/sec on modern hardware.
Mark.
Quoted 8 lines · Roy Sigurd Karlsbakk
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
I am thinking about starting to work on a data deduplicating
blockdevice, a kernel module called blockless.
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.
My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".
Why? How can it possibly make any difference? It's not likely that you'd have dupes in the metadata block, but if you do it doesn't matter that they are transparently mapped into one. You need a copy-on-write mechanism anyway since if you write to either they won't be dups any more.
Signature · 2 lines
Les Mikesell [email protected]
Quoted 13 lines · Les Mikesell [[email protected]] wrote:
Quoted 12 lines · Roy Sigurd Karlsbakk
Quoted 12 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
I am thinking about starting to work on a data deduplicating
blockdevice, a kernel module called blockless.
If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.
Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.
Quoted 7 lines
Quoted 2 lines
My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".
Why? How can it possibly make any difference? It's not likely that you'd have dupes in the metadata block, but if you do it doesn't matter that they are transparently mapped into one. You need a copy-on-write mechanism anyway since if you write to either they won't be dups any more.
Because some file systems create duplicate copies of metadata for recovery if there is some sectors go bad on the media. You really don't want to merge them!
Quoted 8 lines · [email protected] wrote:
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.
My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".
Why? How can it possibly make any difference? It's not likely that you'd have dupes in the metadata block, but if you do it doesn't matter that they are transparently mapped into one. You need a copy-on-write mechanism anyway since if you write to either they won't be dups any more.
Because some file systems create duplicate copies of metadata for recovery if there is some sectors go bad on the media. You really don't want to merge them!
My experience with disks is that if any part of them fails you don't want to trust data from any other part. So I'd consider this a big waste of time and generally keep data that matters on mirrored drives. Hmmm, I suppose you would want it to know not to de-dup the mirrored blocks..
Signature · 2 lines
Les Mikesell [email protected]