Skip to contentSkip to conversation

Miscellaneous e2fsprogs 64-bit patches - description

PATCH In review Nick Dokos 4 participants 6 messages Apr 8, 2009 – Apr 16, 2009

lore
Share & tools
lore

Nick Dokos

author
#0 lore raw

cover Miscellaneous e2fsprogs 64-bit patches - description

[I submitted the first three of these last week, but the first one, although the fix was correct (in the sense of fiddling with the correct bits), stylistically and from the point of code conventions and readability, was rather a botch. #2 and #3 are identical as patches but I fixed up parts of the commentary that were wrong or misleading. So I am reposting these as well as a couple of new ones. Please let me know if there are other problems of this sort. Thanks!]

The following bugs were found by running various commands on a 32TiB file system, containing a 16TiB file (maximum size).

---------------------------------------------------------------------------
1) ext2fs_block_alloc_stats2: fix size comparison for 64-bit compatibility.

Fixes a journal malformation that would not allow the fs to even be mounted.

2) Enable 64-bitness in e2image.c.

I needed to use it later.

3) blk_t -> blk64_t change in ext2fs_extent_get()/cast in extent_node_split().

e2image was reporting a corrupt extent header on the big file. I wrote a python script to dump the extents (since debugfs was not working - see 4) and determined that the on-disk extents were fine. Running e2image in gdb led to this.

4) debugfs.c extents display.

The inode of the 16TiB file was 13. I tried doing a "stat <13>" in debugfs and it produced wrong extent info (block numbers wrapped).

With this change, it seems to produce the right block numbers: I spot-checked against the extents that my script reported, but I have not done an exhaustive comparison.

5) Fix inode->i_blocks 32-bit wrap in e2fsck/pass1.c.

e2fsck was complaining about an i_blocks mismatch on inode <13> (among other things that I'm still working on).

This fixes the i_blocks mismatch issue.

---------------------------------------------------------------------------

Thanks,
Nick

P.S. Here is an interesting side issue:

Patch 3 mentions that e2image ran to completion after the patch was applied (btw, in addition to the 16TiB file, the fs contains a directory with about 10^7 zero length files):

# time e2image -r /dev/mapper/bigvg-bigvol /dev/null
e2image 1.41.4-shared-64bit (27-Jan-2009)

real 37m18.991s
user 15m21.148s
sys 17m57.151s

but there is an interesting catch-22: how do I save its output?

I can try the command line suggested in the manual page:

e2image -r <dev> - | bzip2 > image.bz2

but it takes forever: I started a run on Saturday and it was not done by Tuesday when I killed it - writing to the pipe at 4096 bytes a pop is very slow.

Or I can forego the compression and try to save to a file: it's sparse (I only used 7GiB before it failed), but its nominal size exceeded the maximum file size limit on ext4, at which point I start getting lseek failures.

#1 lore raw
Quoted 7 lines · Nick Dokos

[I submitted the first three of these last week, but the first one, although the fix was correct (in the sense of fiddling with the correct bits), stylistically and from the point of code conventions and readability, was rather a botch. #2 and #3 are identical as patches but I fixed up parts of the commentary that were wrong or misleading. So I am reposting these as well as a couple of new ones. Please let me know if there are other problems of this sort. Thanks!]

Hi Nick,

Thanks for the testing and patches! I apologize for the delay in replying; if it's any consolation, we were all at the Linux file systems workshop when you sent this email.

All the patches look good to me (and I will ACK them individually). I pulled them into my shared-64bit branch at:

http://git.kernel.org/?p=fs/ext2/val/e2fsprogs.git;a=summary

Quoted 19 lines

P.S. Here is an interesting side issue:

Patch 3 mentions that e2image ran to completion after the patch was applied (btw, in addition to the 16TiB file, the fs contains a directory with about 10^7 zero length files):

# time e2image -r /dev/mapper/bigvg-bigvol /dev/null
e2image 1.41.4-shared-64bit (27-Jan-2009)

real 37m18.991s
user 15m21.148s
sys 17m57.151s

but there is an interesting catch-22: how do I save its output?

I can try the command line suggested in the manual page:

e2image -r <dev> - | bzip2 > image.bz2

Quoted 8 lines

but it takes forever: I started a run on Saturday and it was not done by Tuesday when I killed it - writing to the pipe at 4096 bytes a pop is very slow.

Or I can forego the compression and try to save to a file: it's sparse (I only used 7GiB before it failed), but its nominal size exceeded the maximum file size limit on ext4, at which point I start getting lseek failures.

The 16TB limit on ext4 files is an enormous pain for testing 64-bit (>= 16TB) file systems. I keep intending to write some simple dm setup to concatenate two loopback files together, but instead I always install XFS and create a loopback file on an XFS partition.

-VAL

#2 lore raw
Quoted 9 lines · Valerie Aurora Henson
Quoted 6 lines · Nick Dokos

but there is an interesting catch-22: how do I save its output?

I can try the command line suggested in the manual page:

e2image -r <dev> - | bzip2 > image.bz2

Quoted 8 lines
Quoted 8 lines

but it takes forever: I started a run on Saturday and it was not done by Tuesday when I killed it - writing to the pipe at 4096 bytes a pop is very slow.

Or I can forego the compression and try to save to a file: it's sparse (I only used 7GiB before it failed), but its nominal size exceeded the maximum file size limit on ext4, at which point I start getting lseek failures.

We really need some e2image format which encodes the sparseness, I think...

Quoted 4 lines

The 16TB limit on ext4 files is an enormous pain for testing 64-bit (>= 16TB) file systems. I keep intending to write some simple dm setup to concatenate two loopback files together, but instead I always install XFS and create a loopback file on an XFS partition.

For testing a large device, say /dev/sdb1 is 10GB large:

# TERABYTES=expr 20 \* 1024 \* 1024 \* 1024 \* 2 # 20 TB in sectors

# echo "0 $ERABYTES zero" | dmsetup create zero1
# echo "0 $TERABYTES snapshot /dev/mapper/zero1 /dev/sdb1 p 128" | \
   dmsetup create sparse1

This will create a 20TB sparse device called /dev/mapper/sparse1 that has 10GB of actual storage space available. If more than 10GB of data is written to this device, it will start returning I/O errors.

This is from Documentation/device-mapper/zero.txt

-Eric

author
#5 lore raw
In reply to Valerie Aurora Henson
Quoted 9 lines · Valerie Aurora Henson

Thanks for the testing and patches! I apologize for the delay in replying; if it's any consolation, we were all at the Linux file systems workshop when you sent this email.

All the patches look good to me (and I will ACK them individually). I pulled them into my shared-64bit branch at:

http://git.kernel.org/?p=fs/ext2/val/e2fsprogs.git;a=summary

Val,

Thanks for the feedback (no problem about the delay: I really was not expecting a response last week).

I should add a caution that these were (mostly) problems that arose in testing and I tried to fix them (and only them: e.g., in a subsequent email you found a place where a blk_t should be a blk64_t: there are probably *many other* such places); in particular, there is no pretense of a complete and thorough code audit - that still needs to be done and I'm trying to do it, but I'm not there yet.

Thanks,
Nick

#3 lore raw
In reply to Eric Sandeen
Quoted 14 lines · Eric Sandeen
Quoted 13 lines · Valerie Aurora Henson
Quoted 10 lines · Nick Dokos

but there is an interesting catch-22: how do I save its output?

I can try the command line suggested in the manual page:

e2image -r <dev> - | bzip2 > image.bz2

but it takes forever: I started a run on Saturday and it was not done by Tuesday when I killed it - writing to the pipe at 4096 bytes a pop is very slow.

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

Or I can forego the compression and try to save to a file: it's sparse (I only used 7GiB before it failed), but its nominal size exceeded the maximum file size limit on ext4, at which point I start getting lseek failures.

We really need some e2image format which encodes the sparseness, I think...

That seems like a low priority to me. But it would make a great Google Summer of Code project...

Quoted 12 lines
Quoted 4 lines

The 16TB limit on ext4 files is an enormous pain for testing 64-bit (>= 16TB) file systems. I keep intending to write some simple dm setup to concatenate two loopback files together, but instead I always install XFS and create a loopback file on an XFS partition.

For testing a large device, say /dev/sdb1 is 10GB large:

# TERABYTES=expr 20 \* 1024 \* 1024 \* 1024 \* 2 # 20 TB in sectors

# echo "0 $ERABYTES zero" | dmsetup create zero1
# echo "0 $TERABYTES snapshot /dev/mapper/zero1 /dev/sdb1 p 128" | \
   dmsetup create sparse1
Quoted 5 lines

This will create a 20TB sparse device called /dev/mapper/sparse1 that has 10GB of actual storage space available. If more than 10GB of data is written to this device, it will start returning I/O errors.

This is from Documentation/device-mapper/zero.txt

Wow, that just rolls right off the tongue... Thanks for the pointer. I'm going to see if I can come up with something that I can memorize (although generally not hurting for a copy of the kernel source).

-VAL

#4 lore raw
In reply to Eric Sandeen
Quoted 7 lines · Eric Sandeen
Quoted 4 lines · Valerie Aurora Henson
Quoted 4 lines

Or I can forego the compression and try to save to a file: it's sparse (I only used 7GiB before it failed), but its nominal size exceeded the maximum file size limit on ext4, at which point I start getting lseek failures.

We really need some e2image format which encodes the sparseness, I think...

On my two do list for when I have copies amounts of free time is to take the qemu image format code, and retrofit it into the I/O manager code, and then teach e2image to use it, and then teach e2fsck, debugfs, dumpe2fs, to be able to specify that arbitrary I/O managers by name. That would allow dumpe2fs, debugfs, e2fsck, etc., to operate on an qemu-img file, where e2image would have a new option to create a meta-data only qemu-img format file.

Now, if I only had a minion at hand to carry out my evil plans. :-)

- Ted