Skip to contentSkip to conversation

Odd --examine output

DISCUSSION Vanhorn, Mike 7 participants 13 messages Apr 12, 2013 – Apr 18, 2013 1 message not in the archive MD RAID

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

On 4/12/13 10:52 AM, "Phil Turmel" <[email protected]> wrote:

[snip]

Quoted 3 lines

As noted above, the partition tables aren't wiped. Just the device nodes are missing. You could try a "blockdev --rereadpt /dev/sdX" on affected drives to see if it is a transient issue.

That did it! I was able to run blockdev for all of the drives that had missing devices for the partitions, and then was able to

mdadm --assemble --force /dev/md0 /dev/sd[cdefghi]1

and it assembled using all of the disks except, for some reason, sde1 and sdf1. I think sde1 got left out because it had been dropped before the raid actually stopped, and I think I could have added it back in with

mdadm /dev/md0 --re-add /dev/sde1

(since /dev/sde actually seems to be fine). However, once I got the filesystem mounted, my first priority was to get the data off, so I didn't try to re-add that disk.

I don't know why sdf1 got left out.

[snip]

Quoted 2 lines

If the partition is *not* aligned, each large chunk written will have at least two R-M-W cycles.

I snipped most of that explanation, but thank you for it; it really helps me understand what was going on with my partitions.

Quoted 2 lines

I guess "lsdrv" didn't work for you. I'm naturally curious how it failed....

I don't have an lsdrv command, so I did the 'ls -l' that you suggested.

Quoted 6 lines

Anyways, your detailed smartctl reports show big problems:

1) You have multiple drives with many dozens of pending relocations. This suggests that your regular scrubs are not happening on schedule. A "check" scrub turns pending relocations into either real relocations, or no error at all (successful rewrite). Typically the latter.

I've got a raid-check script that runs from cron.weekly. I really did think it was working, because every week I would check and the array was re-building.

Quoted 3 lines

2) All of your self-test log entries show "short offline". That isn't rigorous enough. You need "long offline" self-tests occasionally, too. Or just use the long self-test every time.

I will take this into account, and being using the long test.

Quoted 3 lines

3) You have a drive that entirely failed its SMART assessment {WD-WMAUR0381532 ==> /dev/sdj} due to excessive actual relocations. Replace this drive immediately.

I will. I have a spare disk on the shelf ready to go, once I feel safe that the data is copied.

[snip]

Quoted 4 lines

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight (8) sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

That is exactly what I'm going to do. I feel like an idiot that there seems to have been so many things wrong and I didn't realize it. Now, thanks to your help, and I am much more enlightened.

Thanks!

---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
[email protected]
http://www.cecs.wright.edu/~mvanhorn/

#2 lore raw
Quoted 1 line · Mike

On 4/12/13 10:52 AM, "Phil Turmel" <[email protected]> wrote:

[snip /]

Quoted 4 lines
Quoted 2 lines

I guess "lsdrv" didn't work for you. I'm naturally curious how it failed....

I don't have an lsdrv command, so I did the 'ls -l' that you suggested.

Ah. You missed the footnote in the original suggestion. "lsdrv" is a script I published on github based on encouragement from others on linux-raid. It summarizes a system's storage layout, annotated with drive, subsystem, and filesystem serial numbers wherever possible.

http://github.com/pturmel/lsdrv

I occasionally poke at it to keep it usable on a variety of distros, including older ones. Bug reports welcome.

Phil

#3 lore raw
In reply to a message not in the archive
Quoted 6 lines · Phil Turmel

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight (8) sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

So, if I start the partition at sector 64 (rather than 63), that's better, right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

---
Mike VanHorn
Senior Computer Systems Administrator
College of Engineering and Computer Science
Wright State University
265 Russ Engineering Center
937-775-5157
[email protected]
http://www.cecs.wright.edu/~mvanhorn/

#4 lore raw
Quoted 9 lines · Mike
Quoted 6 lines · Phil Turmel

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight (8) sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

Quoted 3 lines

So, if I start the partition at sector 64 (rather than 63), that's better, right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

No, you're right. Sector 64 is a reasonable choice for spinning rust disks. The more modern default of sector 2048 (1MB) is intended to cover future advances, including solid state drives' erase blocks. (Although it seems that the best SSDs handle that factor on their own.)

Phil

#5 lore raw
In reply to Vanhorn, Mike
Quoted 8 lines
Quoted 7 lines

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight
(8)
sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

Quoted 4 lines

So, if I start the partition at sector 64 (rather than 63), that's better,
right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

I still don't understand why people use partitions for RAID when the whole drive is used anyway. Partitions were invented to partition things up and are of no use if you want to spend the whole drive's space for RAID use (or otherwise).

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line

"Roy" == Roy Sigurd Karlsbakk <[email protected]> writes:

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight
(8)
sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

Quoted 4 lines
Quoted 4 lines

So, if I start the partition at sector 64 (rather than 63), that's better,
right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

Roy> I still don't understand why people use partitions for RAID when Roy> the whole drive is used anyway. Partitions were invented to Roy> partition things up and are of no use if you want to spend the Roy> whole drive's space for RAID use (or otherwise).

Because if I take a 2tb disk a I put a partition on there which is a bit smaller than the full disk, if I then add a new 2tb (or any other size) disk which says it's 2tb, but it's really a bit smaller, then I'm not screwed. I've had it happen.

John

#7 lore raw
Quoted 17 lines · John Stoffel
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

"Roy" == Roy Sigurd Karlsbakk <[email protected]> writes:

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight
(8)
sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

Quoted 4 lines
Quoted 4 lines

So, if I start the partition at sector 64 (rather than 63), that's better,
right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

Roy> I still don't understand why people use partitions for RAID when Roy> the whole drive is used anyway. Partitions were invented to Roy> partition things up and are of no use if you want to spend the Roy> whole drive's space for RAID use (or otherwise).

Quoted 4 lines

Because if I take a 2tb disk a I put a partition on there which is a bit smaller than the full disk, if I then add a new 2tb (or any other size) disk which says it's 2tb, but it's really a bit smaller, then I'm not screwed. I've had it happen.

Or if you use them for your OS, and have several raids for /, /usr, /var etc.
Or you buy 2 or 3 batches of disks, all varying sizes (1TB != 1TB) and having partitions allows you to at least align them all to the same size. And its even possible to buy a replacement disk, that's actually larger but cheaper as prices came down.
A strange use case could be that 1 disk split into 2 partitions could be a hot-spare for 2 arrays.

When using 1MiB offsets for partitions, I don't think there's any performance loss at all (due to alignment) nor will it slow down anything because md is on a partition rather then a disk. Wasting 1 MiB per disk is really not significant.

And when using raid1/10 it allows you in theory to wipe the superblock and make the partition point tot he real data; though there's very little use for that imo.

I personally, don't see why you want to use the entire disk, is there any advantage?

Quoted 6 lines

John
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw
In reply to John Stoffel
Quoted 20 lines · John Stoffel
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 1 line
Quoted 1 line

"Roy" == Roy Sigurd Karlsbakk <[email protected]> writes:

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight
(8)
sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

Quoted 4 lines
Quoted 4 lines

So, if I start the partition at sector 64 (rather than 63), that's better,
right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

Roy> I still don't understand why people use partitions for RAID when Roy> the whole drive is used anyway. Partitions were invented to Roy> partition things up and are of no use if you want to spend the Roy> whole drive's space for RAID use (or otherwise).

Quoted 5 lines

Because if I take a 2tb disk a I put a partition on there which is a bit smaller than the full disk, if I then add a new 2tb (or any other size) disk which says it's 2tb, but it's really a bit smaller, then I'm not screwed. I've had it happen.

Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use standardised sizes (i.e. any 2TB disk will present exactly the same number of sectors). I recall reading that all the manufacturers agreed to do this to prevent just this sort of issue, though I can't find a reference to it now.

Cheers,
Robin

Signature · 4 lines
     ___        
    ( ' }     |       Robin Hill        <[email protected]> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |
#9 lore raw
Quoted 12 lines
Quoted 6 lines

Roy> I still don't understand why people use partitions for RAID when
Roy> the whole drive is used anyway. Partitions were invented to Roy> partition things up and are of no use if you want to spend the Roy> whole drive's space for RAID use (or otherwise).

Quoted 6 lines

Because if I take a 2tb disk a I put a partition on there which is a bit smaller than the full disk, if I then add a new 2tb (or any other
size) disk which says it's 2tb, but it's really a bit smaller, then I'm not screwed. I've had it happen.

Quoted 5 lines

Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use standardised sizes (i.e. any 2TB disk will present exactly the same number of sectors). I recall reading that all the manufacturers agreed to do this to prevent just this sort of issue, though I can't find a reference to it now.

Indeed - you have to go back to smaller drives than those in current production to find varying sizes for equally labelled drive sizes.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

Roy> I still don't understand why people use partitions for RAID when
Roy> the whole drive is used anyway. Partitions were invented to Roy> partition things up and are of no use if you want to spend the Roy> whole drive's space for RAID use (or otherwise).

Because if I take a 2tb disk a I put a partition on there which is a bit smaller than the full disk, if I then add a new 2tb (or any other
size) disk which says it's 2tb, but it's really a bit smaller, then I'm not screwed. I've had it happen.

Quoted 5 lines
Quoted 5 lines

Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use standardised sizes (i.e. any 2TB disk will present exactly the same number of sectors). I recall reading that all the manufacturers agreed to do this to prevent just this sort of issue, though I can't find a reference to it now.

Roy> Indeed - you have to go back to smaller drives than those in Roy> current production to find varying sizes for equally labelled Roy> drive sizes.

Who says it might never happen again, that drive makers go back to varying block counts for disks? Or SSDs? Better safe than sorry, and a gig of wasted space is cheap insurance. Plus, it gives you a place to put boot blocks for GRUB, etc.

John

#11 lore raw
Quoted 8 lines

Roy> Indeed - you have to go back to smaller drives than those in Roy> current production to find varying sizes for equally labelled Roy> drive sizes.

Who says it might never happen again, that drive makers go back to varying block counts for disks? Or SSDs? Better safe than sorry, and a gig of wasted space is cheap insurance. Plus, it gives you a place to put boot blocks for GRUB, etc.

Well, do as you please. When disk vendors standardise, they tend to keep those standards. As I said, the new drive was *smaller* than the older one, so even if I had used partitions, problems would arise. If you're paranoid, though, cut the drive size by 1-5%. Still, all the [124]TB drives I've seen so far have had the same amount of sectors, from Hitachi, Seagate and WD.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw
In reply to Robin Hill
Quoted 12 lines
Quoted 6 lines

Roy> I still don't understand why people use partitions for RAID when
Roy> the whole drive is used anyway. Partitions were invented to Roy> partition things up and are of no use if you want to spend the Roy> whole drive's space for RAID use (or otherwise).

Quoted 6 lines

Because if I take a 2tb disk a I put a partition on there which is a bit smaller than the full disk, if I then add a new 2tb (or any other
size) disk which says it's 2tb, but it's really a bit smaller, then I'm not screwed. I've had it happen.

Quoted 5 lines

Recently? AFAIK, all modern drives (everything over 320G-ish IIRC) use standardised sizes (i.e. any 2TB disk will present exactly the same number of sectors). I recall reading that all the manufacturers agreed to do this to prevent just this sort of issue, though I can't find a reference to it now.

Oh, and btw, last time I had an issue with varying drive sizes, I had two drives in a MD mirror on a Sun^WOracle box (X2100 M2), and the new drive that arrived from Sun^WOracle, was *smaller* than the other one, even though one of the argument I've heard for buying overpriced drives from Sun^WOracle, was that they tuned their firmware to cap their drives to always expose the same amount of sectors. I guess that was just marketing BS. I ended up reinstalling the box…

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw
In reply to Roy Sigurd Karlsbakk
Quoted 6 lines · Roy Sigurd Karlsbakk
Quoted 6 lines
Quoted 5 lines

NOT a guess. Back up what you can, while you can, and start over. Use "fdisk -u" so you can ensure partitions start on multiples of eight
(8)
sectors. (Modern fdisk uses 1MB alignment by default. Highly recommended.)

Quoted 7 lines
Quoted 4 lines

So, if I start the partition at sector 64 (rather than 63), that's better,
right (since 64 is a multiple of 8)? Or is there more math to do and I'm still not getting it?

I still don't understand why people use partitions for RAID when the whole drive is used anyway. Partitions were invented to partition things up and are of no use if you want to spend the whole drive's space for RAID use (or otherwise).

I partition devices because it makes it much simpler to know what is on that device. Granted there are other ways to do this but if I put a partition on it, during some random installer process it won't assume the drive is empty and happily write a partition table or otherwise screw over the device without my knowledge...

I tried going partition-less at one point, but it wasn't worth it to save 1MB of space while considering the previous issue and removing the flexibility to utilize any additional space due to uneven sized drives to, for example, hold a "boot" array that is on the same drives but is a raid1 instead of a raid6...

Sam