Data corruption on large, multi-device filesystem

linux-lvm Jens Beyer 5 participants 12 messages Jan 18, 2005 – Apr 29, 2005 Device mapper

Share & tools

Jens Beyer

author

2005-01-18 09:11 UTC #0 lore raw

Hi,

I get severe data corruption using an logical volume larger then 2 TB. Finally I was able to track down device mappper or lvm as last suspects.

My first guess where problems with filesystems but recently I tried using md / RAID0 - and didnt have any errors of any kind. I would prefer using LVM since we want to use snapshots to simplify backup, but I have no clue how to further debug.

On a system with 3 devices each larger then 1 TB and a logical volume striped over all devices some data gets corrupted while written (or read ?) from disk. This shows up as md5 or crc sums changes on sequenced reads of files if filecache is not involved (by reading a lot data).
On ext2fs there are error while writing data (kernel: EXT2-fs error (device dm-0): ext2_new_block: Allocating block in system zone -
block = 722239884), on other filesystems successive fsck/repairs
shows corrupted metadata.

The system setup is
- Three 29160B Adaptec scsi-controller each with one ATA-Disk Raid sized 1240 GB, (dual PIII, HP DL360 G2, 2 GB Ram)
- Volume group over all three devices, logical volume stripped full size (3.7 TB)
- Filesystem either ext2fs/ext3fs (1.34), reiserfs (3.6.13) or xfs (2.6.25)

- host:~ # lvm version
LVM version: 2.00.33 (2005-01-07)
Library version: 1.00.21-ioctl (2005-01-07)
Driver version: 4.3.0
- 2.6.10 vanilla + 2.6.10-udm1 patches

The problems where initially discovered on 2.6.8, tracked on 2.6.9-udm and also occurs if only 2 devices (sum 2.4 TB) are used.

For a limited time I will be able to further debug the system though it takes some time to generate more then 2 TB of data (max seq read/write rate is ~80 MB/s).

Jens

Signature · 1 line

Nur tote Fische schwimmen mit dem Strom

[email protected]

2005-01-19 22:14 UTC #1 lore raw

I have recently run into this problem also. I have seen it happen on SuSe 9.2, Fedora Core 2 and 3, and vanilla kernels 2.6.8.1, 2.6.9, and 2.6.10. All of my tests were using xfs.

It happens whenever 2 or more devices are striped together with a total volume size greater than 2TB. I have played with a single 4TB raid (12x 400GB RAID5) and did not see any corruption (but I did not fill the disk either).

I initially saw the problem running video files over samba. But have recreated the problem by simply copying some large (5GB+) files and then checking md5sums.

I don't see any corruption on the files unless I specify the -i option to lvcreate. I usually see data corruption within an hour using my current tests.

Let me know if I can be of any assistance.
Joe

Quoting Jens Beyer <[email protected]>:

Quoted 54 lines

Hi,
I get severe data corruption using an logical volume larger then 2 TB. Finally I was able to track down device mappper or lvm as last suspects.
My first guess where problems with filesystems but recently I tried using md / RAID0 - and didnt have any errors of any kind. I would prefer using LVM since we want to use snapshots to simplify backup, but I have no clue how to further debug.
On a system with 3 devices each larger then 1 TB and a logical volume striped over all devices some data gets corrupted while written (or read ?) from disk. This shows up as md5 or crc sums changes on sequenced reads of files if filecache is not involved (by reading a lot data).
On ext2fs there are error while writing data (kernel: EXT2-fs error (device dm-0): ext2_new_block: Allocating block in system zone -
block = 722239884), on other filesystems successive fsck/repairs
shows corrupted metadata.
The system setup is
- Three 29160B Adaptec scsi-controller each with one ATA-Disk Raid sized 1240 GB, (dual PIII, HP DL360 G2, 2 GB Ram)
- Volume group over all three devices, logical volume stripped full size (3.7 TB)
- Filesystem either ext2fs/ext3fs (1.34), reiserfs (3.6.13) or xfs (2.6.25)
- host:~ # lvm version
LVM version: 2.00.33 (2005-01-07)
Library version: 1.00.21-ioctl (2005-01-07)
Driver version: 4.3.0
- 2.6.10 vanilla + 2.6.10-udm1 patches
The problems where initially discovered on 2.6.8, tracked on 2.6.9-udm and also occurs if only 2 devices (sum 2.4 TB) are used.
For a limited time I will be able to further debug the system though it takes some time to generate more then 2 TB of data (max seq read/write rate is ~80 MB/s).
Jens
--
Nur tote Fische schwimmen mit dem Strom
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Randall A. Jones

2005-01-20 14:06 UTC #2 lore raw

Quoted 8 lines · [email protected] wrote:

I have recently run into this problem also. I have seen it happen on SuSe 9.2, Fedora Core 2 and 3, and vanilla kernels 2.6.8.1, 2.6.9, and 2.6.10. All of my tests were using xfs.
It happens whenever 2 or more devices are striped together with a total volume size greater than 2TB. I have played with a single 4TB raid (12x 400GB RAID5) and did not see any corruption (but I did not fill the disk either).

Quoted 9 lines

I initially saw the problem running video files over samba. But have recreated the problem by simply copying some large (5GB+) files and then checking md5sums.
I don't see any corruption on the files unless I specify the -i option to lvcreate. I usually see data corruption within an hour using my current tests.

To verify, this corruption you are seeing only happens when you have a LV larger than 2TB and when you use striping specifically with lvcreate -i.
Has anyone experienced data corruption with >2TB LV and no striping?

Randall
-

Quoted 57 lines

Let me know if I can be of any assistance.
Joe
Quoting Jens Beyer <[email protected]>:
Quoted 49 lines
Hi,
I get severe data corruption using an logical volume larger then 2 TB. Finally I was able to track down device mappper or lvm as last suspects.
My first guess where problems with filesystems but recently I tried using md / RAID0 - and didnt have any errors of any kind. I would prefer using LVM since we want to use snapshots to simplify backup, but I have no clue how to further debug.
On a system with 3 devices each larger then 1 TB and a logical volume striped over all devices some data gets corrupted while written (or read ?) from disk. This shows up as md5 or crc sums changes on sequenced reads of files if filecache is not involved (by reading a lot data).
On ext2fs there are error while writing data (kernel: EXT2-fs error (device dm-0): ext2_new_block: Allocating block in system zone - block = 722239884), on other filesystems successive fsck/repairs shows corrupted metadata.
The system setup is
- Three 29160B Adaptec scsi-controller each with one ATA-Disk Raid sized 1240 GB, (dual PIII, HP DL360 G2, 2 GB Ram)
- Volume group over all three devices, logical volume stripped full size (3.7 TB)
- Filesystem either ext2fs/ext3fs (1.34), reiserfs (3.6.13) or xfs (2.6.25)
- host:~ # lvm version
LVM version: 2.00.33 (2005-01-07)
Library version: 1.00.21-ioctl (2005-01-07)
Driver version: 4.3.0
- 2.6.10 vanilla + 2.6.10-udm1 patches
The problems where initially discovered on 2.6.8, tracked on 2.6.9-udm and also occurs if only 2 devices (sum 2.4 TB) are used.
For a limited time I will be able to further debug the system though it takes some time to generate more then 2 TB of data (max seq read/write rate is ~80 MB/s).
Jens
--
Nur tote Fische schwimmen mit dem Strom

Signature · 5 lines

..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
[email protected]      Code 610.3      301-286-2239

Alasdair G Kergon

maintainer

2005-01-20 17:59 UTC #3 lore raw

Quoted 4 lines · Randall A. Jones

To verify, this corruption you are seeing only happens when you have a LV larger than 2TB
and when you use striping specifically with lvcreate -i. Has anyone experienced data corruption with >2TB LV and no striping?

Does this patch help?

Missing cast causing data corruption on devices with stripes > ~1TB.

drivers/md/dm-stripe.c +1 -1

@@ -179,7 +179,7 @@
 
 	bio->bi_bdev = sc->stripe[stripe].dev->bdev;
 	bio->bi_sector = sc->stripe[stripe].physical_start +
-	    (chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
+	    ((sector_t) chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
 	return 1;
 }

Alasdair

Signature · 1 line

[email protected]

Jens Beyer

author

2005-01-21 15:09 UTC #4 lore raw

Hi,

Quoted 8 lines · Alasdair G Kergon

Quoted 4 lines · Randall A. Jones
To verify, this corruption you are seeing only happens when you have a LV larger than 2TB
and when you use striping specifically with lvcreate -i. Has anyone experienced data corruption with >2TB LV and no striping?
Does this patch help?

No, it doesn't. I just checked multiple times in different setups. To address the 'chunksize' Problem mentioned i used lvcreate -i 3 -I 256 .

Jens

Quoted 6 lines

Missing cast causing data corruption on devices with stripes > ~1TB.

drivers/md/dm-stripe.c line 185

Quoted 8 lines

drivers/md/dm-stripe.c +1 -1

@@ -179,7 +179,7 @@
 
 	bio->bi_bdev = sc->stripe[stripe].dev->bdev;
 	bio->bi_sector = sc->stripe[stripe].physical_start +
-	    (chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
+	    ((sector_t) chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
 	return 1;
 }

--
Nur tote Fische schwimmen mit dem Strom

[email protected]

2005-01-25 17:37 UTC #5 lore raw

I just thought I would give a quick update on what I have been doing in case anyone can give me more/better direction.

I incorporated the (sector_t) cast and I also changed stripe_width to sector_t since my individual devices are bigger than 2TB.

I took lvm out of the equation by using dmsetup and striping together my devices. Then I created and mounted and XFS filesystem directly on /dev/dm-0 This still caused corruption.

I have compiled hashdd with largefile support and am now letting it run directly on /dev/dm-0. The write finished last night and I am running the read pass now (man it takes a long time to to write and verify 4+TB)

I have also been putting some debug statements in stripe_map() but this hasn't shown me anything useful yet.

Joe

Quoting Jens Beyer <[email protected]>:

Quoted 38 lines

Hi,
Quoted 8 lines · Alasdair G Kergon
Quoted 4 lines · Randall A. Jones
To verify, this corruption you are seeing only happens when you have a LV larger than 2TB
and when you use striping specifically with lvcreate -i. Has anyone experienced data corruption with >2TB LV and no striping?
Does this patch help?
No, it doesn't. I just checked multiple times in different setups. To address the 'chunksize' Problem mentioned i used lvcreate -i 3 -I 256 .
Jens
Quoted 6 lines
Missing cast causing data corruption on devices with stripes > ~1TB.
drivers/md/dm-stripe.c line 185
Quoted 8 lines
drivers/md/dm-stripe.c +1 -1
@@ -179,7 +179,7 @@
 
 	bio->bi_bdev = sc->stripe[stripe].dev->bdev;
 	bio->bi_sector = sc->stripe[stripe].physical_start +
-	    (chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
+	    ((sector_t) chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
 	return 1;
 }
 
--
Nur tote Fische schwimmen mit dem Strom

Quoted 6 lines

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Jens Beyer

author

2005-01-28 09:23 UTC #6 lore raw

Hi,

since the two patches (stripe_width and sector_t cast) didnt help, I started to put printk in the dm-stripe code.

At a certain point stripe_map() starts to get called with zero data (eg bio->bi_sector = 0 or 2^32, 2^31 ). This lead me to put some debugging into dm.c especially into max_io_len() where an inline function dm_round_up() is used. Interpreting my printk's this funktion doesnt work well with offsets larger then 2^32. I did some (very) ugly hack to calculate the boundary in that case and it seems that the error is gone (though the full check takes a few days).

Maybe someone with insight on what should happen in dm.c can have a look at max_io_len and the inlined function?

Jens

Signature · 7 lines

Nur tote Fische schwimmen mit dem Strom

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Alasdair G Kergon

maintainer

2005-01-28 14:01 UTC #7 lore raw

Quoted 1 line · Jens Beyer

since the two patches (stripe_width and sector_t cast) didnt help,

Indeed - you need a version of Kevin's patch too; but I've widened the scope of this to looking into why these functions are being called. (Now down to three 64/32 divisions.)

Alasdair

Signature · 7 lines

[email protected]

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Jens Beyer

author

2005-01-21 15:09 UTC #8 lore raw

In reply to Alasdair G Kergon

Hi,

Quoted 8 lines · Alasdair G Kergon

Quoted 4 lines · Randall A. Jones
To verify, this corruption you are seeing only happens when you have a LV larger than 2TB
and when you use striping specifically with lvcreate -i. Has anyone experienced data corruption with >2TB LV and no striping?
Does this patch help?

No, it doesn't. I just checked multiple times in different setups. To address the 'chunksize' Problem mentioned i used lvcreate -i 3 -I 256 .

Jens

Quoted 6 lines

Missing cast causing data corruption on devices with stripes > ~1TB.

drivers/md/dm-stripe.c line 185

Quoted 8 lines

drivers/md/dm-stripe.c +1 -1

@@ -179,7 +179,7 @@
 
 	bio->bi_bdev = sc->stripe[stripe].dev->bdev;
 	bio->bi_sector = sc->stripe[stripe].physical_start +
-	    (chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
+	    ((sector_t) chunk << sc->chunk_shift) + (offset & sc->chunk_mask);
 	return 1;
 }

--
Nur tote Fische schwimmen mit dem Strom

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

Alasdair G Kergon

maintainer

2005-01-20 16:53 UTC #9 lore raw

In reply to [email protected]

Quoted 2 lines · On Wed, Jan 19, 2005 at 02:14:43PM -0800, [email protected] wrote:

I don't see any corruption on the files unless I specify the -i option to lvcreate.

That's a useful observation: different code gets run when -i is used, and I've taken a look and there are two 32-bit limits in that code.

One appears to be intentional for performance reasons and can be worked around by increasing the chunk size (default 64K), though it really should detect when the limit is exceeded and tell you!

The other unintentional limit (which is the one affecting you I think) leads to wrap-around at striped device offsets beyond ~2^32 sectors i.e. 2^39 bytes into a PV, or ~2^40 bytes (2TB) into an LV in your case of 2 stripes. The fix for that is a basic exercise in C. [Ref. stripe_map() in drivers/md/dm-stripe.c]

Alasdair

Signature · 1 line

[email protected]

[email protected]

2005-01-21 02:38 UTC #10 lore raw

Thanks, I'll give it(the patch you sent) a try and let you know what I see.

When you say "chunk size" what lvcreate arg does that map to? the stripe size(-I) or the chunk size(-c) that goes along with snapshots?

Joe

Quoting Alasdair G Kergon <[email protected]>:

Quoted 27 lines · On Wed, Jan 19, 2005 at 02:14:43PM -0800, [email protected] wrote:

Quoted 2 lines
I don't see any corruption on the files unless I specify the -i option to lvcreate.
That's a useful observation: different code gets run when -i is used, and I've taken a look and there are two 32-bit limits in that code.
One appears to be intentional for performance reasons and can be worked around by increasing the chunk size (default 64K), though it really should detect when the limit is exceeded and tell you!
The other unintentional limit (which is the one affecting you I think) leads to wrap-around at striped device offsets beyond ~2^32 sectors i.e. 2^39 bytes into a PV, or ~2^40 bytes (2TB) into an LV in your case of 2 stripes. The fix for that is a basic exercise in C. [Ref. stripe_map() in drivers/md/dm-stripe.c]
Alasdair
--
[email protected]
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Dan Pritts

2005-04-29 16:00 UTC #11 lore raw

Hi folks - I'm new to the list and to the Linux LVM.

I believe I've just run into the same problem described in this thread from back in January.

x86_64 dell, redhat AS 4, kernel 2.6.9 (redhat source, self-compiled to add xfs) with two ~5TB RAIDs that I'm striping with LVM. (it's really the same RAID presenting two LUNs over two parallel fibre channel busses.)

About 4TB into a "dd if=/dev/zero of=testfile" the filesystem (XFS) started complaining bitterly of errors.

On the other hand I can write a full 5TB of zero into an XFS created on one of the raw devices with no errors.

Both XFS & the dm code in the kernel seem to have had some non-trivial updates between 2.6.9 from redhat and 2.6.11.7, so i'm compiling that now.

Is it believed that this problem should be fixed in the mainline kernel or are there other LVM patches I should apply?

Also, seems to be a FAQ on this list but my search of the archives didn't show any definitive answer - is it true that there are no large device limits with LVM2/device mapper?

thanks
danno

Quoted 38 lines · On Thu, Jan 20, 2005 at 06:38:41PM -0800, [email protected] wrote:

Thanks, I'll give it(the patch you sent) a try and let you know what I see.
When you say "chunk size" what lvcreate arg does that map to? the stripe size(-I) or the chunk size(-c) that goes along with snapshots?
Joe
Quoting Alasdair G Kergon <[email protected]>:
Quoted 27 lines · On Wed, Jan 19, 2005 at 02:14:43PM -0800, [email protected] wrote:
Quoted 2 lines
I don't see any corruption on the files unless I specify the -i option to lvcreate.
That's a useful observation: different code gets run when -i is used, and I've taken a look and there are two 32-bit limits in that code.
One appears to be intentional for performance reasons and can be worked around by increasing the chunk size (default 64K), though it really should detect when the limit is exceeded and tell you!
The other unintentional limit (which is the one affecting you I think) leads to wrap-around at striped device offsets beyond ~2^32 sectors i.e. 2^39 bytes into a PV, or ~2^40 bytes (2TB) into an LV in your case of 2 stripes. The fix for that is a basic exercise in C. [Ref. stripe_map() in drivers/md/dm-stripe.c]
Alasdair
--
[email protected]
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

danno
--
dan pritts - systems administrator - internet2
734/352-4953 office 734/834-7224 mobile