Skip to contentSkip to conversation

bdar: efficiently backup allocated bytes in file systems

DISCUSSION Zach Brown 6 participants 11 messages Mar 18, 2008 – Mar 20, 2008 VFS

lore
Share & tools
lore

Zach Brown

author
#0 lore raw

So, I had a fun time throwing together a utility last weekend. I thought I'd share it sooner rather than later.

I found myself wanting to backup a copy of an ancient ~75g ext3 file system. I got frustrated by of our utilities which don't saturate storage. I wanted dd line rates but I also only wanted to copy referenced data.

So I threw something together which does that. I made it work roughly like tar so that people have some idea what to expect. So you can do something like:

 $ bdar -cf - /dev/sda3 | gzip -c > /tmp/sda3-backup.bdar.gz
...
 $ zcat /tmp/sda3-backup.bdar.gz | bdar -xf - /dev/sda3

and it will do exactly what you would guess it would do after reading those command lines.

The bdar file format is just a header and then a series of regions of bytes described by their length and offset. To create a bdar file from a file system bdar needs to know enough to figure out what extents are referenced. Restoring a bdar is generic, though, it just stamps bytes into the target file.

I only taught it the most basic knowledge of ext[234]. Just enough to show that generating the bdar is ~4x faster than tar and ~2x faster than dump :). There's still some available disk bandwidth to consume with read-ahead, but it's pretty close. (single spindle, ~5g of kernel trees, beefy cpus.)

I'm going to continue hacking this into something which could be trusted with data but not on any rigorous schedule. I thought I would put it up for others to get a look at and, hopefully, contribute to. There's a lot of fun stuff we can do.

It's in a mercurial repo:

http://www.zabbo.net/hg/bdar

  $ hg clone http://www.zabbo.net/hg/bdar ; ls ./bdar

Let me know if you give it a try, I'm interested in all feedback.

- z

#1 lore raw
Quoted 16 lines · Zach Brown

So, I had a fun time throwing together a utility last weekend. I thought I'd share it sooner rather than later.

I found myself wanting to backup a copy of an ancient ~75g ext3 file system. I got frustrated by of our utilities which don't saturate storage. I wanted dd line rates but I also only wanted to copy referenced data.

So I threw something together which does that. I made it work roughly like tar so that people have some idea what to expect. So you can do something like:

 $ bdar -cf - /dev/sda3 | gzip -c > /tmp/sda3-backup.bdar.gz
...
 $ zcat /tmp/sda3-backup.bdar.gz | bdar -xf - /dev/sda3
Quoted 8 lines

and it will do exactly what you would guess it would do after reading those command lines.

The bdar file format is just a header and then a series of regions of bytes described by their length and offset. To create a bdar file from a file system bdar needs to know enough to figure out what extents are referenced. Restoring a bdar is generic, though, it just stamps bytes into the target file.

So the question is whether the ".bdar" file is specific to the filesystem being backed up, and if it only allows backing up the whole filesystem? Does it create a dense output file or a sparse one? Does it store the data as chunks of blocks in a full-device map or on a per file basis?

If you can't restore a .bdar backup file to a smaller device than the source device that makes it less useful than most of the other tools.

Quoted 5 lines

I only taught it the most basic knowledge of ext[234]. Just enough to show that generating the bdar is ~4x faster than tar and ~2x faster than dump :). There's still some available disk bandwidth to consume with read-ahead, but it's pretty close. (single spindle, ~5g of kernel trees, beefy cpus.)

The question is whether the 2x speed improvement is worth the lack of portability compared to even dump?

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

author
#2 lore raw
Quoted 5 lines

So the question is whether the ".bdar" file is specific to the filesystem being backed up, and if it only allows backing up the whole filesystem? Does it create a dense output file or a sparse one? Does it store the data as chunks of blocks in a full-device map or on a per file basis?

Let's see.. yes, yes, dense, full-device. It's a tiny little program, you could read it in 10 minutes :). You'll giggle at how incomplete its knowledge of ext* is.

Quoted 2 lines

If you can't restore a .bdar backup file to a smaller device than the source device that makes it less useful than most of the other tools.

It is different in a way that won't satisfy people who want to restore to smaller devices, yes.

Quoted 2 lines

The question is whether the 2x speed improvement is worth the lack of portability compared to even dump?

Indeed. For me, in my current situation, it is.

- z

#3 lore raw
In reply to Zach Brown
Quoted 16 lines · Zach Brown

So, I had a fun time throwing together a utility last weekend. I thought I'd share it sooner rather than later.

I found myself wanting to backup a copy of an ancient ~75g ext3 file system. I got frustrated by of our utilities which don't saturate storage. I wanted dd line rates but I also only wanted to copy referenced data.

So I threw something together which does that. I made it work roughly like tar so that people have some idea what to expect. So you can do something like:

 $ bdar -cf - /dev/sda3 | gzip -c > /tmp/sda3-backup.bdar.gz
...
 $ zcat /tmp/sda3-backup.bdar.gz | bdar -xf - /dev/sda3
Quoted 8 lines

and it will do exactly what you would guess it would do after reading those command lines.

The bdar file format is just a header and then a series of regions of bytes described by their length and offset. To create a bdar file from a file system bdar needs to know enough to figure out what extents are referenced. Restoring a bdar is generic, though, it just stamps bytes into the target file.

Neat, Zach. You should look at xfs_copy - it does pretty much this for XFS filesystems....

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
author
#4 lore raw
Quoted 3 lines

Neat, Zach. You should look at xfs_copy - it does pretty much this for XFS filesystems....

haha, yet another round of the -fsdevel XFS drinking game :)

Does xfs_copy tend to assert the XFS file format in the backup files it generates? One of the things I was hoping for with bdar was to have the resulting copy image be agnostic. It's just a sparse map with some checksumming, really.

That limits what we can do, of course. The current trivial format only has one address space which doesn't fit well with the plans file systems have of working with multiple addressable block ranges.

But I think I'm fine with that. The value:complexity ratio of this trivial version is refreshingly large.

- z

#5 lore raw
Quoted 5 lines · Zach Brown
Quoted 3 lines

Neat, Zach. You should look at xfs_copy - it does pretty much this for XFS filesystems....

haha, yet another round of the -fsdevel XFS drinking game :)

/me grins

Quoted 2 lines

Does xfs_copy tend to assert the XFS file format in the backup files it generates?

Yes. If the destination is a file, the resultant image is a sparse file that is a mountable XFS filesystem.

Quoted 3 lines

One of the things I was hoping for with bdar was to have the resulting copy image be agnostic. It's just a sparse map with some checksumming, really.

Yup - the use of a sparse file avoids the need for an internal map. But that does't really work for piping the output, though. xfs_metadump is probably more similar to bdar in that respect, but it doesn't copy data....

FWIW, xfs_copy is really for efficient duplication of one source disk to many destination disks in manufacturing, not so much as a filesystem backup tool...

Quoted 3 lines

That limits what we can do, of course. The current trivial format only has one address space which doesn't fit well with the plans file systems have of working with multiple addressable block ranges.

Can't do everything ;)

Quoted 2 lines

But I think I'm fine with that. The value:complexity ratio of this trivial version is refreshingly large.

Agreed.

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#6 lore raw
Quoted 4 lines · David Chinner

Yup - the use of a sparse file avoids the need for an internal map. But that does't really work for piping the output, though. xfs_metadump is probably more similar to bdar in that respect, but it doesn't copy data....

Ntfsclone also efficiently copies data or metadata to dense, mountable sparse or a device file since 2003.

It was an important factor to get reliable NTFS write support sooner because NTFS is slightly bigger than the other file systems (the size of the Microsoft NTFS driver is almost like the sum of the 60 Linux file systems altogether) and we needed to handle the heavily used (meta)data images efficiently during development and quality assurance.

Szaka

Signature · 1 line
NTFS-3G:  http://ntfs-3g.org
#7 lore raw
In reply to David Chinner
Quoted 6 lines · David Chinner
Quoted 2 lines · Zach Brown

Does xfs_copy tend to assert the XFS file format in the backup files it generates?

Yes. If the destination is a file, the resultant image is a sparse file that is a mountable XFS filesystem.

The "e2image" program also works in that manner.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#8 lore raw
In reply to Zach Brown
Quoted 10 lines · Zach Brown
Quoted 2 lines

Neat, Zach. You should look at xfs_copy - it does pretty much this for XFS filesystems....

haha, yet another round of the -fsdevel XFS drinking game :)

Does xfs_copy tend to assert the XFS file format in the backup files it generates? One of the things I was hoping for with bdar was to have the resulting copy image be agnostic. It's just a sparse map with some checksumming, really.

Quoted 8 lines

That limits what we can do, of course. The current trivial format only has one address space which doesn't fit well with the plans file systems have of working with multiple addressable block ranges.

But I think I'm fine with that. The value:complexity ratio of this trivial version is refreshingly large.

- z

About a year back, I was trying various ways to read every file on a fairly massive (reiserfs v3) file system (order of tens of millions of files).

I don't recall how close I came to native dd speed, but I could get a substantial win by grabbing a substantial chunk of files (say 5000), sort them by either inode number or creation time, and then read them in that order.

We had some good experience with this, but our use case has no sparse files and tends to have lots of little or medium sized files. This only did the read phase, but the basic assumption is that the file system will tend to allocate disk sectors in sequential order over time and this gave a fairly close approximation of that ;-)

ric

#9 lore raw
In reply to Zach Brown
Quoted 5 lines · Zach Brown

The bdar file format is just a header and then a series of regions of bytes described by their length and offset. To create a bdar file from a file system bdar needs to know enough to figure out what extents are referenced. Restoring a bdar is generic, though, it just stamps bytes into the target file.

Is it possible to also make it write files out in partimage (
http://www.partimage.org/Main_Page ) format or will this slow it down too much?

Signature · 1 line
Sitsofe | http://sucs.org/~sits/
author
#10 lore raw
Quoted 4 lines

Is it possible to also make it write files out in partimage (
http://www.partimage.org/Main_Page ) format or will this slow it down too much?

I'm not sure what the point would be.. why not just use partimage to write out partimage files?

- z