RAID10 Layouts
Hello list,
Researching RAID10, trying to learn the most advanced system for a 2 SATA drive system. Have two WD 2TB drives for a media computer, and the most important requirement is data redundancy. I realize that RAID is no substitute for backups, but this is a backup for the backups and the purpose here is data safety. The secondary goal is speed enhancement. It appears that RAID10 can give both.
First question is on layout of RAID10. In studying the man pages it seems that Far mode gives 95% of the speed of RAID0, but with increased seek for writes. And that Offset retains much of this benefit while increasing efficiency of writes. What should be the preference, Far or Offset? Are they equally as robust?
How safe is the data in Far or Offset mode? If a drive fails, will a complete, usable, bootable system exist on the other drive? (These two are the only drives in the system, which is Debian Testing, Debian kernel 2.6.30-5) Need I make any special Grub settings?
What about this Intel firmware 'RAID'? Would this assist in any way? How does it relate (if it does) to the linux md system? Should I set in BIOS to RAID, or leave it at ACPI?
How does this look:
# mdadm --create /dev/md0 --level=raid10 --layout=o2 --metadata=1.2 --chunk=64 --raid-disks=2 missing /dev/sdb1
Quoted 9 lines · [email protected] writes:
Hello list,
Researching RAID10, trying to learn the most advanced system for a 2 SATA drive system. Have two WD 2TB drives for a media computer, and the most important requirement is data redundancy. I realize that RAID is no substitute for backups, but this is a backup for the backups and the purpose here is data safety. The secondary goal is speed enhancement. It appears that RAID10 can give both.
Quoted 5 lines
First question is on layout of RAID10. In studying the man pages it seems that Far mode gives 95% of the speed of RAID0, but with increased seek for writes. And that Offset retains much of this benefit while increasing efficiency of writes. What should be the preference, Far or Offset? Are they equally as robust?
All raid10 layouts offer the same robustness. Which layout is best for you really depends on your use case. Probably the biggest factor will be the average file size. My experience is that with large files the far copies do not cost noticeable write speed while being twice as fast reading as raid1.
Quoted 5 lines
How safe is the data in Far or Offset mode? If a drive fails, will a complete, usable, bootable system exist on the other drive? (These two are the only drives in the system, which is Debian Testing, Debian kernel 2.6.30-5) Need I make any special Grub settings?
I don't think lilo or grub1 can boot from raid10 at all with offset or far copies. With near copies you are identical to a simple raid1 so that would boot.
So to be bootable even with a failed drive you should partition the disk. Create a small raid1 for the system and a large raid10 for the data.
Quoted 3 lines
What about this Intel firmware 'RAID'? Would this assist in any way? How does it relate (if it does) to the linux md system? Should I set in BIOS to RAID, or leave it at ACPI?
I would stay away from any half baked bios stuff. It will be no better than linux software raid but will tie you to the specific bios. If your mainboard fails and the next one has a different bios you can't boot your disks.
Quoted 2 lines
How does this look:
# mdadm --create /dev/md0 --level=raid10 --layout=o2 --metadata=1.2 --chunk=64 --raid-disks=2 missing /dev/sdb1
On partitions it is save to use 1.1 format. Saves you 4k. Jupey.
You should play with the chunksize though and try with and without bitmap and different bitmap sizes. Bitmap costs some write performance but it greatly speeds up resyncs after a crash or temporary drive failure.
MfG
Goswin
Thank you Goswin.
Quoted 7 lines · On Friday 21 August 2009 09:43:28 Goswin von Brederlow wrote:
I don't think lilo or grub1 can boot from raid10 at all with offset or far copies. With near copies you are identical to a simple raid1 so that would boot.
So to be bootable even with a failed drive you should partition the disk. Create a small raid1 for the system and a large raid10 for the data.
Uh oh, already set all 3 parts for RAID10, but haven't switched over yet.
As it happens my / is on sda1 and /home is sda3 (swap is sda2), so it'll be pretty easy to just make / RAID1. Do I need to make swap RAID1 and not 10?
Quoted 4 lines
I would stay away from any half baked bios stuff. It will be no better than linux software raid but will tie you to the specific bios. If your mainboard fails and the next one has a different bios you can't boot your disks.
Thank you.
Quoted 4 lines
Quoted 2 lines
How does this look:
# mdadm --create /dev/md0 --level=raid10 --layout=o2 --metadata=1.2 --chunk=64 --raid-disks=2 missing /dev/sdb1On partitions it is save to use 1.1 format. Saves you 4k. Jupey.
4k of what? One time only, or on every cluster? Any additional benefit to 1.2?
My system records mpeg4 from DishNetwork satellite (R5000-HD), so it handles mostly files over 1GB. However its most rigorous duty is scanning those videos for commercials, and marking locations in a mysql database. The disk light is constantly on and system response is sluggish when this is being done. I don't understand how an advanced drive like this can be so bogged down, but I hope RAID10 will speed things up. Maybe there is a way to increase disk cache size?
Quoted 4 lines
You should play with the chunksize though and try with and without bitmap and different bitmap sizes. Bitmap costs some write performance but it greatly speeds up resyncs after a crash or temporary drive failure.
My partitions and data are so enormous that I can't really do any experimenting. Definitely will use write-intent log.
Re: Help
My God, the command is not working. I need to remove sdb1 from md0 so I can change it from a RAID10 to RAID1, and it simply ignores my command:
# mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md2 : active raid10 sdb3[1]
1868560128 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U]
bitmap: 94/446 pages [376KB], 2048KB chunkmd1 : active raid10 sdb2[1]
6297344 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U]
bitmap: 0/25 pages [0KB], 128KB chunkmd0 : active raid10 sdb1[1]
78654080 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U]
bitmap: 76/151 pages [304KB], 256KB chunkunused devices: <none>
#
My system is half-converted and is now unbootable. What am I going to do?
Re: Help
Quoted 8 lines · On 21/08/2009 20:20, [email protected] wrote:
My God, the command is not working. I need to remove sdb1 from md0 so I can change it from a RAID10 to RAID1, and it simply ignores my command:
# mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1 # cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] md2 : active raid10 sdb3[1] 1868560128 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 94/446 pages [376KB], 2048KB chunk
Quoted 7 lines
md1 : active raid10 sdb2[1]
6297344 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 0/25 pages [0KB], 128KB chunkmd0 : active raid10 sdb1[1]
78654080 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 76/151 pages [304KB], 256KB chunk
Well, it won't let you remove the only thing keeping the array active. Stop the array first with mdadm --stop /dev/md0. After that I think you can just create your new RAID-1 array without doing anything else.
Cheers,
John.
Quoted 17 lines · Goswin von Brederlow
Quoted 9 lines · [email protected] writes:
Hello list,
Researching RAID10, trying to learn the most advanced system for a 2 SATA drive system. Have two WD 2TB drives for a media computer, and the most important requirement is data redundancy. I realize that RAID is no substitute for backups, but this is a backup for the backups and the purpose here is data safety. The secondary goal is speed enhancement. It appears that RAID10 can give both.
Quoted 5 lines
First question is on layout of RAID10. In studying the man pages it seems that Far mode gives 95% of the speed of RAID0, but with increased seek for writes. And that Offset retains much of this benefit while increasing efficiency of writes. What should be the preference, Far or Offset? Are they equally as robust?
Quoted 5 lines
All raid10 layouts offer the same robustness. Which layout is best for you really depends on your use case. Probably the biggest factor will be the average file size. My experience is that with large files the far copies do not cost noticeable write speed while being twice as fast reading as raid1.
The file system elevator makes up for the Far write head movement.
Quoted 9 lines
Quoted 5 lines
How safe is the data in Far or Offset mode? If a drive fails, will a complete, usable, bootable system exist on the other drive? (These two are the only drives in the system, which is Debian Testing, Debian kernel 2.6.30-5) Need I make any special Grub settings?
I don't think lilo or grub1 can boot from raid10 at all with offset or far copies. With near copies you are identical to a simple raid1 so that would boot.
there is a howto on setting up a system, that can continue runnig, if one disk fails at
http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk
Quoted 9 lines
Quoted 2 lines
How does this look:
# mdadm --create /dev/md0 --level=raid10 --layout=o2 --metadata=1.2 --chunk=64 --raid-disks=2 missing /dev/sdb1On partitions it is save to use 1.1 format. Saves you 4k. Jupey.
You should play with the chunksize though and try with and without bitmap and different bitmap sizes. Bitmap costs some write performance but it greatly speeds up resyncs after a crash or temporary drive failure.
I would recommend a bigger chunk size. at least 256 kiB.
Best regards
keld
Re: Help
In reply to John RobinsonQuoted 3 lines · On Friday 21 August 2009 12:38:00 John Robinson wrote:
Well, it won't let you remove the only thing keeping the array active. Stop the array first with
mdadm --stop /dev/md0. After that I think you can just create your new RAID-1 array without doing anything else.
WHEW, thank you.
Quoted 3 lines · On Friday 21 August 2009 13:42:34 Keld Jørn Simonsen wrote:
there is a howto on setting up a system, that can continue runnig, if one disk fails at
http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_disk
...
Quoted 1 line
I would recommend a bigger chunk size. at least 256 kiB.
Very good, thank you Keld.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 50 lines · Keld Jørn Simonsen
Quoted 18 lines · Goswin von Brederlow
Quoted 9 lines · [email protected] writes:
Hello list,
Researching RAID10, trying to learn the most advanced system for a 2 SATA drive system. Have two WD 2TB drives for a media computer, and the most important requirement is data redundancy. I realize that RAID is no substitute for backups, but this is a backup for the backups and the purpose here is data safety. The secondary goal is speed enhancement. It appears that RAID10 can give both.
Quoted 6 lines
First question is on layout of RAID10. In studying the man pages it seems that Far mode gives 95% of the speed of RAID0, but with increased seek for writes. And that Offset retains much of this benefit while increasing efficiency of writes. What should be the preference, Far or Offset? Are they equally as robust?
Quoted 6 lines
All raid10 layouts offer the same robustness. Which layout is best for you really depends on your use case. Probably the biggest factor will be the average file size. My experience is that with large files the far copies do not cost noticeable write speed while being twice as fast reading as raid1.
The file system elevator makes up for the Far write head movement.
Quoted 10 lines
Quoted 6 lines
How safe is the data in Far or Offset mode? If a drive fails, will a complete, usable, bootable system exist on the other drive? (These two are the only drives in the system, which is Debian Testing, Debian kernel 2.6.30-5) Need I make any special Grub settings?
I don't think lilo or grub1 can boot from raid10 at all with offset or far copies. With near copies you are identical to a simple raid1 so that would boot.
there is a howto on setting up a system, that can continue runnig, if one disk fails at
http://linux-raid.osdl.org/index.php/Preventing_against_a_failing_diskQuoted 5 lines
Quoted 3 lines
How does this look:
# mdadm --create /dev/md0 --level=raid10 --layout=o2 --metadata=1.2 --chunk=64 --raid-disks=2 missing /dev/sdb1On partitions it is save to use 1.1 format. Saves you 4k. Jupey.
Quoted 8 lines
Quoted 5 lines
You should play with the chunksize though and try with and without bitmap and different bitmap sizes. Bitmap costs some write performance but it greatly speeds up resyncs after a crash or temporary drive failure.
I would recommend a bigger chunk size. at least 256 kiB.
You really want to look at stripe-size and stride-size creating an ext[234] filesystem on top of raid, good things to happen there.
Signature · 13 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc "You are disgraced professional losers. And by the way, give us our money back." - Representative Earl Pomeroy, Democrat of North Dakota on the A.I.G. executives who were paid bonuses after a federal bailout. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help
In reply to [email protected]Not able to boot to my RAID devices. md0 is / and ext3 RAID1, but md1 and md2 are swap and JFS respectively, RAID10 created like this:
mdadm --create /dev/md1 --level=raid10 --layout=o2 --metadata=1.2 --chunk=256 --raid-disks=2 missing /dev/sdb2
It gives the initial kernel boot message but then says invalid raid superblock magic on sdb2
invalid raid superblock magic on sdb3
... and halts progress. I have to hard-reset to continue. Why isn't the error more specific?
I've tried setting the metadata to 1.1, and tried adjusting mdadm.conf from /dev/md/1 to /dev/md1, but neither helped. The parts are set to raid autodetect and the kernel parameter is set to md_autodetect. What could be wrong?
Quoted 24 lines · On Friday 21 August 2009 12:20:28 [email protected] wrote:
My God, the command is not working. I need to remove sdb1 from md0 so I can change it from a RAID10 to RAID1, and it simply ignores my command:
# mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1 # cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] md2 : active raid10 sdb3[1] 1868560128 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 94/446 pages [376KB], 2048KB chunkmd1 : active raid10 sdb2[1]
6297344 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 0/25 pages [0KB], 128KB chunkmd0 : active raid10 sdb1[1]
78654080 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 76/151 pages [304KB], 256KB chunkunused devices: <none>
#My system is half-converted and is now unbootable. What am I going to do?
Quoted 8 lines · [email protected] writes:
Thank you Goswin.
Quoted 4 lines · On Friday 21 August 2009 09:43:28 Goswin von Brederlow wrote:
I don't think lilo or grub1 can boot from raid10 at all with offset or far copies. With near copies you are identical to a simple raid1 so that would boot.
Quoted 7 lines
Quoted 3 lines
So to be bootable even with a failed drive you should partition the disk. Create a small raid1 for the system and a large raid10 for the data.
Uh oh, already set all 3 parts for RAID10, but haven't switched over yet.
As it happens my / is on sda1 and /home is sda3 (swap is sda2), so it'll be pretty easy to just make / RAID1. Do I need to make swap RAID1 and not 10?
Need? no. Want? No idea. My experience is that if you need swap (as in swap in/out, not just it being used for garbage) then you have lost anyway. Half or twice the speed on swap doesn't matter, the system will crawl anyway.
Quoted 8 lines
Quoted 4 lines
I would stay away from any half baked bios stuff. It will be no better than linux software raid but will tie you to the specific bios. If your mainboard fails and the next one has a different bios you can't boot your disks.
Thank you.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
How does this look:
# mdadm --create /dev/md0 --level=raid10 --layout=o2 --metadata=1.2 --chunk=64 --raid-disks=2 missing /dev/sdb1On partitions it is save to use 1.1 format. Saves you 4k. Jupey.
4k of what? One time only, or on every cluster? Any additional benefit to 1.2?
4k of space overall. The 1.2 format leaves the first 4k of the device free for use of a bootloader/MBR.
Quoted 8 lines
My system records mpeg4 from DishNetwork satellite (R5000-HD), so it handles mostly files over 1GB. However its most rigorous duty is scanning those videos for commercials, and marking locations in a mysql database. The disk light is constantly on and system response is sluggish when this is being done. I don't understand how an advanced drive like this can be so bogged down, but I hope RAID10 will speed things up. Maybe there is a way to increase disk cache size?
man blockdev
Raid1 and the different Raid10 layouts work good for different access patterns. The plain raid1 allows 2 streams to read from the drive in parallel. If you have multiple streams that will reduce seeks. On the other hand the raid10 far layout double sequential read speed. And so on. Every layout behaves differently.
For scanning your videos raid10 with far layout is probably best with a large read ahead. For your database a simple raid1 is probably better. It might be benefitial to have the two on seperate partitions with different raid mode.
Or it might be benefitial to have 2 partitions (sdX3 + sdX4) both with simple raid1 but flag sdb3 and sda4 as --write-mostly. That way the DB would always read from sda while the videos read from sdb.
As said before a lot of this depends on the usage pattern and that means trying different things with the work load you will have productively.
Quoted 7 lines
Quoted 4 lines
You should play with the chunksize though and try with and without bitmap and different bitmap sizes. Bitmap costs some write performance but it greatly speeds up resyncs after a crash or temporary drive failure.
My partitions and data are so enormous that I can't really do any experimenting. Definitely will use write-intent log.
That is always a problem.
MfG
Goswin
Re: Help
In reply to [email protected]Quoted 6 lines · On Sat, August 22, 2009 4:14 pm, [email protected] wrote:
Not able to boot to my RAID devices. md0 is / and ext3 RAID1, but md1 and md2 are swap and JFS respectively, RAID10 created like this:
mdadm --create/dev/md1--level=raid10 --layout=o2 --metadata=1.2 --chunk=256 --raid-disks=2 missing/dev/sdb2
Quoted 6 lines
It gives the initial kernel boot message but then says invalid raid superblock magic on sdb2
invalid raid superblock magic on sdb3... and halts progress. I have to hard-reset to continue. Why isn't the error more specific?
You say md0 is raid1 but mdstat shows it to be raid10, so that won't boot.
'raid autodetect' only works for 0.90 metadata, and you are using 1.x. You should not use 'raid autodetect' partitions. Rather the initrd should use mdadm to assemble the arrays. Most distros seem to get this right these days. Maybe you just need to rebuild your initrd...
NeilBrown
Quoted 42 lines
I've tried setting the metadata to 1.1, and tried adjusting mdadm.conf from
/dev/md/1to/dev/md1, but neither helped. The parts are set to raid autodetect and the kernel parameter is set tomd_autodetect. What could be wrong?Quoted 19 lines · On Friday 21 August 2009 12:20:28 [email protected] wrote:
My God, the command is not working. I need to remove sdb1 from md0 so I can change it from a RAID10 to RAID1, and it simply ignores my command:
# mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1 # cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] md2 : active raid10 sdb3[1] 1868560128 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 94/446 pages [376KB], 2048KB chunkmd1 : active raid10 sdb2[1]
6297344 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 0/25 pages [0KB], 128KB chunkmd0 : active raid10 sdb1[1]
78654080 blocks super 1.2 64K chunks 2 offset-copies [2/1] [_U] bitmap: 76/151 pages [304KB], 256KB chunkQuoted 8 lines
unused devices: <none>
#My system is half-converted and is now unbootable. What am I going to do?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help
Quoted 1 line · On Saturday 22 August 2009 02:34:12 NeilBrown wrote:
You say md0 is raid1 but mdstat shows it to be raid10, so that won't boot.
Thanks Neil. However that was an early attempt before I knew RAID10 won't boot.
Quoted 5 lines
'raid autodetect' only works for 0.90 metadata, and you are using 1.x. You should not use 'raid autodetect' partitions. Rather the initrd should use mdadm to assemble the arrays. Most distros seem to get this right these days. Maybe you just need to rebuild your initrd...
I am not using an initrd. Have all the RAID and disk drivers built into the (custom-compiled) kernel. It uses mdadm to assemble the arrays? Maybe this is the problem.
I am using this procedure to build a RAID array from a live system:
http://www.howtoforge.com/software-raid1-grub-boot-debian-etch
It is very lucid and clear, however I am slightly modifying it to use RAID10 on my second and third partitions. When I come to update-initramfs -u
... the only initrd it updates is for an old stock kernel. It doesn't build one for any of my compiled kernels.
What partition type should I use rather than raid autodetect? Or should I revert to 0.90 metadata?
Looking at dmesg it does say that md1 & 2 do not have a valid ==v.90== superblock. There is no other linux raid partition type, so I guess it's got to be v.090. Why do they make 1.1 and 1.2 then, if they do not work?
Re: Help
On 22/08/2009 13:56, [email protected] wrote:
[...]
Quoted 2 lines
It is very lucid and clear, however I am slightly modifying it to use RAID10 on my second and third partitions. When I come to update-initramfs -u ... the only initrd it updates is for an old stock kernel. It doesn't build one for any of my compiled kernels.
You should have mkinitrd (that's what it is on Fedora/RHEL/CentOS) or something similar with which you can build initramfs images for any kernel.
Quoted 1 line
What partition type should I use rather than raid autodetect? Or should I revert to 0.90 metadata?
Probably type DA, Non-FS data, though type FD will be fine even if they're not auto-detected.
Quoted 1 line
Looking at dmesg it does say that md1 & 2 do not have a valid ==v.90== superblock. There is no other linux raid partition type, so I guess it's got to be v.090. Why do they make 1.1 and 1.2 then, if they do not work?
The newer metadata types have their benefits. Auto-detection is being deprecated, I think it's because things which are only for boot-up time are being pushed out of the permanently-loaded kernel into initramfs, so they don't hang around wasting space on a running system. For example, CentOS 5 uses autodetection, Fedora 10 automatically puts mdadm in the initramfs and runs it at the right time.
Cheers,
John.
Re: Help
Quoted 2 lines · On Saturday 22 August 2009 09:47:48 John Robinson wrote:
You should have mkinitrd (that's what it is on Fedora/RHEL/CentOS) or something similar with which you can build initramfs images for any kernel.
OK once I changed the version to 0.90 it stopped just at the kernel banner on boot and hung. I was about to give up on RAID when your message came through, and I created the initrd.img file. I always compile my own kernels and don't depend on an initrd, but it now seems to be necessary. So in Debian:
# update-initramfs -o /boot/initrd.img-2.6.30-5 ... reboot, and voila it did what it was supposed to, for a change. I'm now resyncing my 2TB drives, which will take a good while.
Quoted 4 lines
Quoted 1 line
What partition type should I use rather than raid autodetect? Or should I revert to 0.90 metadata?
Probably type DA, Non-FS data, though type FD will be fine even if they're not auto-detected.
It simply found 'bad magick' with FD, so that doesn't work with the newer versions. I tried to use both newer versions, but it's not possible. You sound not quite sure of the partition type, so I'll stick with FD and 0.90. Thanks though John.
Goswin says, "For scanning your videos raid10 with far layout is probably best with
a large read ahead." I have the RAID10 blocksize set to 1024 for the video partition, but any idea how to set readahead?
Re: Help
Quoted 2 lines · On Saturday 22 August 2009 11:12:35 [email protected] wrote:
Goswin says, "For scanning your videos raid10 with far layout is probably best with
a large read ahead." I have the RAID10 blocksize set to 1024 for the video partition, but any idea how to set readahead?
My gosh, it turns out this setting is astounding. You test your drive speed with some large file, as such:
# time dd if={somelarge}.iso of=/dev/null bs=256k... and check your drive's default readahead setting:
# blockdev --getra /dev/sda 256
... then test with various settings like 1024, 1536, 2048, 4096, 8192, and maybe 16384:
# blockdev --setra 4096 /dev/sda
Here are the results for my laptop. I can't test the HTPC with the array yet, as it's still syncing.
256 40.4 MB/s
1024 123 MB/s
1536 2.7 GB/s
2048 2.4 GB/s
4096 2.4 GB/s
8192 2.4 GB/s
16384 2.5 GB/s
I suspect it's best to use the minimum readahead for the best speed (in my case 1536), for two reasons:
- To save memory;
- So there isn't such a performance impact when the blocks are not sequential.
RE: Help
} -----Original Message-----
} From: [email protected] [mailto:linux-raid-
} [email protected]] On Behalf Of [email protected]
} Sent: Saturday, August 22, 2009 4:45 PM
} To: [email protected]
} Subject: Re: Help
}
} On Saturday 22 August 2009 11:12:35 [email protected] wrote:
} > Goswin says, "For scanning your videos raid10 with far layout is } probably best with
} > a large read ahead." I have the RAID10 blocksize set to 1024 for the } video partition, but any idea how to set readahead?
}
} My gosh, it turns out this setting is astounding. You test your drive } speed with some large file, as such:
} # time dd if={somelarge}.iso of=/dev/null bs=256k
}
} ... and check your drive's default readahead setting:
} # blockdev --getra /dev/sda
} 256
}
} ... then test with various settings like 1024, 1536, 2048, 4096, 8192, and } maybe 16384:
} # blockdev --setra 4096 /dev/sda
}
} Here are the results for my laptop. I can't test the HTPC with the array } yet, as it's still syncing.
} 256 40.4 MB/s
} 1024 123 MB/s } 1536 2.7 GB/s } 2048 2.4 GB/s } 4096 2.4 GB/s } 8192 2.4 GB/s
} 16384 2.5 GB/s
}
} I suspect it's best to use the minimum readahead for the best speed (in my } case 1536), for two reasons:
} - To save memory;
} - So there isn't such a performance impact when the blocks are not } sequential.
The disk cache is being used. You should reboot between each test, or use a file much bigger than the amount of RAM you have. Or use a different file each time.
Re: Help
In reply to [email protected]Quoted 1 line · On 22/08/2009 19:12, [email protected] wrote:
On Saturday 22 August 2009 09:47:48 John Robinson wrote:
[...]
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
What partition type should I use rather than raid autodetect? Or should I revert to 0.90 metadata?
Probably type DA, Non-FS data, though type FD will be fine even if they're not auto-detected.
It simply found 'bad magick' with FD, so that doesn't work with the newer versions. I tried to use both newer versions, but it's not possible. You sound not quite sure of the partition type, so I'll stick with FD and 0.90. Thanks though John.
I said "probably" DA because that's what's been suggested by others previously on this list. Others have simply used 83, but that's not ideal because if the partitions appear to have filesystems on (e.g. the metadata's not at the beginning), they might get auto-mounted without md RAID. I'm sure FD will work fine with later metadata versions as long as you have mdadm in your initramfs, and while as you've noted there'll be a whinge in the boot log about it not being version 0.90, it's not going to cause the kernel to lock up or anything like that.
Cheers,
John.
Re: Help
In reply to a message not in the archiveThe sync has finally finished, but something's wrong with the first partition set; Only sda1 is a member of md0. In dmesg I find:
[ 4.756365] md: kicking non-fresh sdb1 from array!
Huh? If it's not fresh, why doesn't it sync it? What should I do about this? How did it happen on a new array?
Re: Help
Quoted 7 lines · On Tue, August 25, 2009 9:08 am, [email protected] wrote:
The sync has finally finished, but something's wrong with the first partition set; Only sda1 is a member of md0. In dmesg I find:
[ 4.756365] md: kicking non-fresh sdb1 from array!Huh? If it's not fresh, why doesn't it sync it? What should I do about this? How did it happen on a new array?
You'll need to provide a lot more information, starting with kernel log at all relevant times (and don't use 'grep', just cut out of contiguous section of the log including a few lines before and after anything that might be relevant).
And "mdadm -E" of any relevant device.
"Kicking non-free sdb1 from arrays" is a message that you get when assembling an array if the metadata on sdb1 is older than the others. This can happen if it was evicted from the array due to failure or if the array was assembled without sdb1 for some reason. There are probably other scenarios.
That is why I need to see recent history, including anything from the last time the array was active.
NeilBrown
Re: Help
Quoted 5 lines · On Monday 24 August 2009 16:38:41 you wrote:
You'll need to provide a lot more information, starting with kernel log at all relevant times (and don't use 'grep', just cut out of contiguous section of the log including a few lines before and after anything that might be relevant).
And "mdadm -E" of any relevant device.
# mdadm -E /dev/sdb mdadm: No md superblock detected on /dev/sdb. #
Aug 22 21:17:17 localhost kernel: [ 3.048020] ata4: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.048037] ata6: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.048045] ata5: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.048054] ata3: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.201035] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.201044] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.204330] ata2.00: ATA-8: WDC WD20EADS-00S2B0, 04.05G04, max UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.204333] ata2.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32) Aug 22 21:17:17 localhost kernel: [ 3.206097] ata1.00: ATA-8: WDC WD20EADS-00R6B0, 01.00A01, max UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.206100] ata1.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32) Aug 22 21:17:17 localhost kernel: [ 3.207345] ata2.00: configured for UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.211122] ata1.00: configured for UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.211203] scsi 0:0:0:0: Direct-Access ATA WDC WD20EADS-00R 01.0 PQ: 0 ANSI: 5 Aug 22 21:17:17 localhost kernel: [ 3.211406] sd 0:0:0:0: [sda] 3907029168 512-byte hardware sectors: (2.00 TB/1.81 TiB) Aug 22 21:17:17 localhost kernel: [ 3.211417] sd 0:0:0:0: [sda] Write Protect is off Aug 22 21:17:17 localhost kernel: [ 3.211435] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 22 21:17:17 localhost kernel: [ 3.211495] sda:<5>sd 0:0:0:0: Attached scsi generic sg0 type 0 Aug 22 21:17:17 localhost kernel: [ 3.211568] scsi 1:0:0:0: Direct-Access ATA WDC WD20EADS-00S 04.0 PQ: 0 ANSI: 5 Aug 22 21:17:17 localhost kernel: [ 3.211719] sd 1:0:0:0: [sdb] 3907029168 512-byte hardware sectors: (2.00 TB/1.81 TiB) Aug 22 21:17:17 localhost kernel: [ 3.211728] sd 1:0:0:0: [sdb] Write Protect is off Aug 22 21:17:17 localhost kernel: [ 3.211746] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 22 21:17:17 localhost kernel: [ 3.211798] sdb:<5>sd 1:0:0:0: Attached scsi generic sg1 type 0 Aug 22 21:17:17 localhost kernel: [ 3.222483] sdb1 sdb2 sdb3 Aug 22 21:17:17 localhost kernel: [ 3.222744] sd 1:0:0:0: [sdb] Attached SCSI disk Aug 22 21:17:17 localhost kernel: [ 3.223107] sda1 sda2 sda3 Aug 22 21:17:17 localhost kernel: [ 3.223321] sd 0:0:0:0: [sda] Attached SCSI disk
...
Aug 22 21:17:17 localhost kernel: [ 4.719904] md: md0 stopped. Aug 22 21:17:17 localhost kernel: [ 4.756229] md: bind<sdb1> Aug 22 21:17:17 localhost kernel: [ 4.756348] md: bind<sda1> Aug 22 21:17:17 localhost kernel: [ 4.756365] md: kicking non-fresh sdb1 from array! Aug 22 21:17:17 localhost kernel: [ 4.756370] md: unbind<sdb1> Aug 22 21:17:17 localhost kernel: [ 4.761035] md: export_rdev(sdb1) Aug 22 21:17:17 localhost kernel: [ 4.762357] raid1: raid set md0 active with 1 out of 2 mirrors Aug 22 21:17:17 localhost kernel: [ 4.768650] md0: bitmap initialized from disk: read 10/10 pages, set 198 bits Aug 22 21:17:17 localhost kernel: [ 4.768653] created bitmap (151 pages) for device md0 Aug 22 21:17:17 localhost kernel: [ 4.777530] md: md1 stopped. Aug 22 21:17:17 localhost kernel: [ 4.777616] md0: unknown partition table Aug 22 21:17:17 localhost kernel: [ 4.781705] md: bind<sdb2> Aug 22 21:17:17 localhost kernel: [ 4.781820] md: bind<sda2> Aug 22 21:17:17 localhost kernel: [ 4.783078] raid10: raid set md1 active with 2 out of 2 devices Aug 22 21:17:17 localhost kernel: [ 4.791063] md1: bitmap initialized from disk: read 13/13 pages, set 0 bits Aug 22 21:17:17 localhost kernel: [ 4.791066] created bitmap (193 pages) for device md1 Aug 22 21:17:17 localhost kernel: [ 4.827200] md: md2 stopped. Aug 22 21:17:17 localhost kernel: [ 4.827294] md1: unknown partition table Aug 22 21:17:17 localhost kernel: [ 4.835293] md: bind<sdb3> Aug 22 21:17:17 localhost kernel: [ 4.835413] md: bind<sda3> Aug 22 21:17:17 localhost kernel: [ 4.846525] raid10: raid set md2 active with 2 out of 2 devices Aug 22 21:17:17 localhost kernel: [ 4.862129] md2: bitmap initialized from disk: read 14/14 pages, set 0 bits Aug 22 21:17:17 localhost kernel: [ 4.862132] created bitmap (223 pages) for device md2 Aug 22 21:17:17 localhost kernel: [ 4.898461] md2: unknown partition table
...
Hm, how would the superblock have been destroyed? This is a little disturbing.
Re: Help
OK I think I've resolved this, so don't worry about me anymore.
Quoted 72 lines · On Tuesday 25 August 2009 06:18:38 [email protected] wrote:
Quoted 5 lines · On Monday 24 August 2009 16:38:41 you wrote:
You'll need to provide a lot more information, starting with kernel log at all relevant times (and don't use 'grep', just cut out of contiguous section of the log including a few lines before and after anything that might be relevant).
And "mdadm -E" of any relevant device.# mdadm -E /dev/sdb mdadm: No md superblock detected on /dev/sdb. #Aug 22 21:17:17 localhost kernel: [ 3.048020] ata4: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.048037] ata6: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.048045] ata5: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.048054] ata3: SATA link down (SStatus 0 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.201035] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.201044] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Aug 22 21:17:17 localhost kernel: [ 3.204330] ata2.00: ATA-8: WDC WD20EADS-00S2B0, 04.05G04, max UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.204333] ata2.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32) Aug 22 21:17:17 localhost kernel: [ 3.206097] ata1.00: ATA-8: WDC WD20EADS-00R6B0, 01.00A01, max UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.206100] ata1.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32) Aug 22 21:17:17 localhost kernel: [ 3.207345] ata2.00: configured for UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.211122] ata1.00: configured for UDMA/133 Aug 22 21:17:17 localhost kernel: [ 3.211203] scsi 0:0:0:0: Direct-Access ATA WDC WD20EADS-00R 01.0 PQ: 0 ANSI: 5 Aug 22 21:17:17 localhost kernel: [ 3.211406] sd 0:0:0:0: [sda] 3907029168 512-byte hardware sectors: (2.00 TB/1.81 TiB) Aug 22 21:17:17 localhost kernel: [ 3.211417] sd 0:0:0:0: [sda] Write Protect is off Aug 22 21:17:17 localhost kernel: [ 3.211435] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 22 21:17:17 localhost kernel: [ 3.211495] sda:<5>sd 0:0:0:0: Attached scsi generic sg0 type 0 Aug 22 21:17:17 localhost kernel: [ 3.211568] scsi 1:0:0:0: Direct-Access ATA WDC WD20EADS-00S 04.0 PQ: 0 ANSI: 5 Aug 22 21:17:17 localhost kernel: [ 3.211719] sd 1:0:0:0: [sdb] 3907029168 512-byte hardware sectors: (2.00 TB/1.81 TiB) Aug 22 21:17:17 localhost kernel: [ 3.211728] sd 1:0:0:0: [sdb] Write Protect is off Aug 22 21:17:17 localhost kernel: [ 3.211746] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Aug 22 21:17:17 localhost kernel: [ 3.211798] sdb:<5>sd 1:0:0:0: Attached scsi generic sg1 type 0 Aug 22 21:17:17 localhost kernel: [ 3.222483] sdb1 sdb2 sdb3 Aug 22 21:17:17 localhost kernel: [ 3.222744] sd 1:0:0:0: [sdb] Attached SCSI disk Aug 22 21:17:17 localhost kernel: [ 3.223107] sda1 sda2 sda3 Aug 22 21:17:17 localhost kernel: [ 3.223321] sd 0:0:0:0: [sda] Attached SCSI disk...
Aug 22 21:17:17 localhost kernel: [ 4.719904] md: md0 stopped. Aug 22 21:17:17 localhost kernel: [ 4.756229] md: bind<sdb1> Aug 22 21:17:17 localhost kernel: [ 4.756348] md: bind<sda1> Aug 22 21:17:17 localhost kernel: [ 4.756365] md: kicking non-fresh sdb1 from array! Aug 22 21:17:17 localhost kernel: [ 4.756370] md: unbind<sdb1> Aug 22 21:17:17 localhost kernel: [ 4.761035] md: export_rdev(sdb1) Aug 22 21:17:17 localhost kernel: [ 4.762357] raid1: raid set md0 active with 1 out of 2 mirrors Aug 22 21:17:17 localhost kernel: [ 4.768650] md0: bitmap initialized from disk: read 10/10 pages, set 198 bits Aug 22 21:17:17 localhost kernel: [ 4.768653] created bitmap (151 pages) for device md0 Aug 22 21:17:17 localhost kernel: [ 4.777530] md: md1 stopped. Aug 22 21:17:17 localhost kernel: [ 4.777616] md0: unknown partition table Aug 22 21:17:17 localhost kernel: [ 4.781705] md: bind<sdb2> Aug 22 21:17:17 localhost kernel: [ 4.781820] md: bind<sda2> Aug 22 21:17:17 localhost kernel: [ 4.783078] raid10: raid set md1 active with 2 out of 2 devices Aug 22 21:17:17 localhost kernel: [ 4.791063] md1: bitmap initialized from disk: read 13/13 pages, set 0 bits Aug 22 21:17:17 localhost kernel: [ 4.791066] created bitmap (193 pages) for device md1 Aug 22 21:17:17 localhost kernel: [ 4.827200] md: md2 stopped. Aug 22 21:17:17 localhost kernel: [ 4.827294] md1: unknown partition table Aug 22 21:17:17 localhost kernel: [ 4.835293] md: bind<sdb3> Aug 22 21:17:17 localhost kernel: [ 4.835413] md: bind<sda3> Aug 22 21:17:17 localhost kernel: [ 4.846525] raid10: raid set md2 active with 2 out of 2 devices Aug 22 21:17:17 localhost kernel: [ 4.862129] md2: bitmap initialized from disk: read 14/14 pages, set 0 bits Aug 22 21:17:17 localhost kernel: [ 4.862132] created bitmap (223 pages) for device md2 Aug 22 21:17:17 localhost kernel: [ 4.898461] md2: unknown partition table...
Hm, how would the superblock have been destroyed? This is a little disturbing.