Skip to contentSkip to conversation

Data deduplication for Linux : lessfs

linux-lvm Mark Ruijter 5 participants 11 messages Jun 24, 2009

lore
Share & tools
lore
#0 lore raw

For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com
I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.
For the time being lessfs is now stable and even usable as a blockdevice with the help of ietd, loop or nbd.

Mark Ruijter

#1 lore raw
Quoted 3 lines · Mark Ruijter

For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com

It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.

Quoted 3 lines

I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.

#2 lore raw

Hi Roy,

Quoted 6 lines

It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.

Almost all the traffic is on the forum - open discussion. Only one person posted to the mailing list. ;-)

Quoted 5 lines

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

Typically the meta data on production systems is approx 10%~20% of the deduplicated stored data.
Stored data is on my systems 40x less then the data written to the filesystem.

For example, from a real life backup server making dozens of backups each day:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p3     9.7G  2.4G  6.9G  26% /
/dev/cciss/c0d0p1      99M   23M   72M  24% /boot
tmpfs                 7.9G     0  7.9G   0% /dev/shm
/dev/cciss/c0d0p4     246G  6.0G  241G   3% /meta
/dev/cciss/c0d1p1     274G   73G  202G  27% /blockdata
/dev/cciss/c1d0p1     4.1T  1.5T  2.7T  35% /data
lessfs                4.1T  1.5T  2.7T  35% /pooldata
[root@lessfssrv pooldata]# du . -s -h
31T     .
[root@lessfssrv pooldata]# ls -alh /data/current/
total 314G
drwxr-xr-x 2 root root   26 Jun  1 00:12 .
drwxr-xr-x 6 root root   59 Jun  1 00:12 ..
-rw-r--r-- 1 root root 314G Jun 22 14:26 blockdata.tch
[root@lessfssrv pooldata]# ls -alh /meta/current/
total 1.4G
drwxr-xr-x 2 root root   63 Jun  1 00:12 .
drwxr-xr-x 6 root root   59 Jun  1 00:12 ..
-rw-r--r-- 1 root root 1.3G Jun 22 14:52 blockusage.tch
-rw-r--r-- 1 root root  89M Jun 22 14:45 dirent.tcb
-rw-r--r-- 1 root root  89M Jun 22 14:52 metadata.tcb

Mark.

Quoted 20 lines

roy

Signature · 16 lines
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
-- 
I all pedagogikk er det essensielt at pensum presenteres
intelligibelt. Det er et element�rt imperativ for alle pedagoger �
unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de
fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.


_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
#4 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 1 line · Roy Sigurd Karlsbakk

On 24. juni. 2009, at 17.12, Mark Ruijter wrote:

<snip>

Quoted 8 lines
Quoted 2 lines

I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

As of the 2.6.30 kernel a default mount option is "relatime". That is a relatively new mount option that drastically reduces the number of atime updates. ie. Not realtime, but relatime.

Greg

Signature · 10 lines
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
#3 lore raw
In reply to Mark Ruijter
Quoted 7 lines · Mark Ruijter

Hi Roy,

Quoted 3 lines

It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.

Almost all the traffic is on the forum - open discussion. Only one person posted to the mailing list. ;-)

Why??
Mailing lists are so much easier to use. Instead of visiting a bunch of websites, they all sit in my mailbox.

Quoted 4 lines
Quoted 1 line

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

Typically the meta data on production systems is approx 10%~20% of the deduplicated stored data. Stored data is on my systems 40x less then the data written to the filesystem.

The problems with metadata is not that they take up a lot of space, but that they are updated so regularly. As Greg Freemyer pointed out, relatime will help a lot, but still, deduplicating metadata may take up a serious amount of time because of the frequent updates.

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.

#5 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 6 lines · Roy Sigurd Karlsbakk
Quoted 3 lines · Mark Ruijter

For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com

Quoted 6 lines

It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.

Quoted 3 lines

I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.

Signature · 2 lines
  Les Mikesell
   [email protected]
#6 lore raw
Quoted 5 lines · Les Mikesell
Quoted 4 lines · Roy Sigurd Karlsbakk
Quoted 3 lines · Mark Ruijter

For those who need OpenSource data deduplication today instead of tomorrow one might take a look at lessfs.
http://www.lessfs.com

Quoted 7 lines
Quoted 5 lines

It's a good idea, but given the current traffic on the lessfs mailing list, I'm not sure if much work is done. I have been a member of that list since June 1 and haven't received more than one message, which was the one I wrote myself.

Quoted 3 lines

I am thinking about starting to work on a data deduplicating blockdevice, a kernel module called blockless.

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.

My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
[email protected]
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element�rt imperativ for alle pedagoger � unng� eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p� norsk.

#10 lore raw
In reply to Les Mikesell
Quoted 6 lines

Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.

Exactly. And this is why lessfs can easily write files with speeds up to 200MB~300MB/sec on modern hardware.

Mark.

#7 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 8 lines · Roy Sigurd Karlsbakk
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line

I am thinking about starting to work on a data deduplicating

blockdevice, a kernel module called blockless.

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.

My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".

Why? How can it possibly make any difference? It's not likely that you'd have dupes in the metadata block, but if you do it doesn't matter that they are transparently mapped into one. You need a copy-on-write mechanism anyway since if you write to either they won't be dups any more.

Signature · 2 lines
  Les Mikesell
   [email protected]
#8 lore raw
Quoted 13 lines · Les Mikesell [[email protected]] wrote:
Quoted 12 lines · Roy Sigurd Karlsbakk
Quoted 12 lines
Quoted 7 lines
Quoted 2 lines
Quoted 1 line

I am thinking about starting to work on a data deduplicating

blockdevice, a kernel module called blockless.

If done smartly, this may perhaps be possible, but the problem is the filesystem's metadata. Is this going to be dedup'ed? How much will this take? A simple backup will update atime on all the files backed up, and although atime isn't always wanted or needed, the problem occurs elsewhere.

Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.

Quoted 7 lines
Quoted 2 lines

My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".

Why? How can it possibly make any difference? It's not likely that you'd have dupes in the metadata block, but if you do it doesn't matter that they are transparently mapped into one. You need a copy-on-write mechanism anyway since if you write to either they won't be dups any more.

Because some file systems create duplicate copies of metadata for recovery if there is some sectors go bad on the media. You really don't want to merge them!

#9 lore raw
Quoted 8 lines · [email protected] wrote:
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

Block level deduplication isn't going to know/care about the difference between file contents and metadata. It is either stored in blocks that match other blocks or not and the difference should not be visible to the filesystem living on top of the block device.

My point exactly. If dedup was to be done on the block layer, you'd need flag to say "do not dedup this".

Why? How can it possibly make any difference? It's not likely that you'd have dupes in the metadata block, but if you do it doesn't matter that they are transparently mapped into one. You need a copy-on-write mechanism anyway since if you write to either they won't be dups any more.

Because some file systems create duplicate copies of metadata for recovery if there is some sectors go bad on the media. You really don't want to merge them!

My experience with disks is that if any part of them fails you don't want to trust data from any other part. So I'd consider this a big waste of time and generally keep data that matters on mirrored drives. Hmmm, I suppose you would want it to know not to de-dup the mirrored blocks..

Signature · 2 lines
  Les Mikesell
   [email protected]