Skip to contentSkip to conversation

Any way to Increase MD_SB_DISKS=27 ? I need 31 devices

DISCUSSION Tyler 3 participants 11 messages May 2, 2005 – May 3, 2005 MD RAID

lore
Share & tools
lore

Tyler

author
#0 lore raw

I'd like to build a Raid-6 array using 31 drives, is there a patch available or an easy way to safely increase the *MD_SB_DISKS*=27 to a higher number? My attempts at patching the source myself were unsuccessful, more than likely because i don't know what/why the limit is 27, and whether that effects other parts of the kernel.

Thanks,
Tyler.

PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, using the --metadata 1.0 option on the create line, and still get:

# mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
mdadm: invalid number of raid devices: 31
# mdadm -V
mdadm - v2.0-devel-1 - DEVELOPMENT VERSION NOT FOR REGULAR USE - 18 February 2005
# uname -a
Linux pink 2.6.12-rc3-mm2 #1 SMP Mon May 2 02:15:28 CEST 2005 i686 GNU/Linux
# dmesg |grep md
md: raid5 personality registered as nr 4
md: raid6 personality registered as nr 8
md: md driver 0.90.2 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 3.38
#1 lore raw
Quoted 5 lines · Tyler

I'd like to build a Raid-6 array using 31 drives, is there a patch available or an easy way to safely increase the *MD_SB_DISKS*=27 to a higher number? My attempts at patching the source myself were unsuccessful, more than likely because i don't know what/why the limit is 27, and whether that effects other parts of the kernel.

It's 27 because that's how much space there is in some of the structs.

Look in include/linux/raid. md_p.h, for example.

   #define MD_SB_DISKS                     27

and it calculates how much space that needs:

   #define MD_SB_DISKS_WORDS (MD_SB_DISKS*MD_SB_DESCRIPTOR_WORDS)
   #define MD_SB_DESCRIPTOR_WORDS          32

and all that is part of what must fit in a "mdp_superblock_s" struct.

           /*
            * Disks information
            */
           mdp_disk_t disks[MD_SB_DISKS];

with the space at the tail of the struct calculated ...

        /*
         * Reserved
         */
        __u32 reserved[MD_SB_RESERVED_WORDS];

... by figuring how much space was used and subtracting from 1K:

       #define MD_SB_RESERVED_WORDS            (1024 - MD_SB_GENERIC_WORDS - MD_SB_PERSONALITY_WORDS - MD_SB_DISKS_WORDS - MD_SB_DESCRIPTOR_WORDS)

and the space used includes the MD_SB_DESCRIPTOR_WORDS conisting of those 27 32-byte array elements.

To make this number bigger, you would first have to make more room in the struct, by changing the 1K in the calculation to 4096 (thus extending the struct by extending its reserved section), then you could use more of that room by increasing the 27. I presume the max would now be 3*32+27 = 123.

There might be another such struct in the includes. I haven't looked
...

Peter

author
#2 lore raw

Peter, after looking at the code after i sent the email, I more or less figured out what you put below, i'm wondering though, will changing the structures here affect anything else? other than the mdadm tool.. what about mkfs.ext3/xfs and possibly mount?

Thanks,
Tyler.

Quoted 60 lines · Peter T. Breuer
Quoted 3 lines · Tyler

I'd like to build a Raid-6 array using 31 drives, is there a patch available or an easy way to safely increase the *MD_SB_DISKS*=27 to a higher number? My attempts at patching the source myself were unsuccessful, more than likely because i don't know what/why the limit is 27, and whether that effects other parts of the kernel.

It's 27 because that's how much space there is in some of the structs.

Look in include/linux/raid. md_p.h, for example.

  #define MD_SB_DISKS                     27

and it calculates how much space that needs:

  #define MD_SB_DISKS_WORDS (MD_SB_DISKS*MD_SB_DESCRIPTOR_WORDS)
  #define MD_SB_DESCRIPTOR_WORDS          32

and all that is part of what must fit in a "mdp_superblock_s" struct.

          /*
           * Disks information
           */
          mdp_disk_t disks[MD_SB_DISKS];

with the space at the tail of the struct calculated ...

       /*
        * Reserved
        */
       __u32 reserved[MD_SB_RESERVED_WORDS];

... by figuring how much space was used and subtracting from 1K:

      #define MD_SB_RESERVED_WORDS            (1024 - MD_SB_GENERIC_WORDS - MD_SB_PERSONALITY_WORDS - MD_SB_DISKS_WORDS - MD_SB_DESCRIPTOR_WORDS)

and the space used includes the MD_SB_DESCRIPTOR_WORDS conisting of those 27 32-byte array elements.

To make this number bigger, you would first have to make more room in the struct, by changing the 1K in the calculation to 4096 (thus extending the struct by extending its reserved section), then you could use more of that room by increasing the 27. I presume the max would now be 3*32+27 = 123.

There might be another such struct in the includes. I haven't looked
...

Peter

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 2 lines · Tyler

figured out what you put below, i'm wondering though, will changing the structures here affect anything else? other than the mdadm tool.. what

I can't think of anything, offhand. I would first check through the kernel cde looking for associated gotchas before worrying about trivia like applications, however!

Quoted 1 line

about mkfs.ext3/xfs and possibly mount?

Not affected at all. They have no knowledge of such things.

You want to look for anything that uses md_p.h. I assume that is "nothing", since applications will use md_u.h.

Peter

#4 lore raw
Quoted 7 lines · Peter T. Breuer
Quoted 2 lines · Tyler

figured out what you put below, i'm wondering though, will changing the structures here affect anything else? other than the mdadm tool.. what

I can't think of anything, offhand. I would first check through the kernel cde looking for associated gotchas before worrying about trivia like applications, however!

mdp_super_t (the size of which you modified) is used throughout md.c only. You want to check if it is mapped to disk or not. I think it is (offhand) ...

mdp_super_t *sb;

...

  bdevname(rdev->bdev, b);
  sb = (mdp_super_t*)page_address(rdev->sb_page);

So it seems to me to be intended to be 1KB of (sb) info on disk. If the blocksize is 1K, then reading/writing from disk may only deliver 1K.
...

     if (!sync_page_io(rdev->bdev, rdev->sb_offset<<1, MD_SB_BYTES, rdev->sb_page, READ))
              goto fail;

Hmm ... MD_SB_BYTES. But ....

  #define MD_SB_BYTES                     4096

So we read 4K off disk at a time! Phew!

Arrgh, but the undimensioned counts in md_p.h are in units of 4B, so when it says "1024 - ...", it really means "4KB - ...", and hence if you are going to icrease the size of the superblock, what you should do is

1) increase MD_SB_BYTES to 8KB.
2) change "1024 - ..." to "2048 - ..." in md_p.h.
3) keep looking. Can one get 2 pages at once off disk with sync_page_io?

Where is that rdev->sb_offset set? Great, in calc_dev_sboffset:

        /*
         * Calculate the position of the superblock,
         * it's at the end of the disk.
         *
         * It also happens to be a multiple of 4Kb.
         */
        sb_offset = calc_dev_sboffset(rdev->bdev);
        rdev->sb_offset = sb_offset;

Yes, but if we are going to increase the size to 8KB, we'd better check that calc ...

inline static sector_t calc_dev_sboffset(struct block_device *bdev)

{
        sector_t size = bdev->bd_inode->i_size >> BLOCK_SIZE_BITS;
        return MD_NEW_SIZE_BLOCKS(size);

}

Uh huhh... it counts the blocks in the device (I think those are 1KB units, not device-dependent, but I may be wrong). Then it chooses the index of a smaller number to return - 1 past the available array size:

  #define MD_NEW_SIZE_BLOCKS(x)           ((x & ~(MD_RESERVED_BLOCKS - 1)) - MD_RESERVED_BLOCKS)

Ummmm ... that appears to be an attempt to calculate (x / y) * y - y, where y is a power of two ...

   #define MD_RESERVED_BLOCKS      (MD_RESERVED_BYTES / BLOCK_SIZE)

and the clunker ...

   #define MD_RESERVED_BYTES       (64 * 1024)

OK. So AT LEAST 64KB are reserved at the end of the disk. We can only count on 64KB, however. The good news is that it looks like the first 4KB of that is the superblock, and that we might be able to use the rest for a bigger superblock (I suspect bitmap info will be kept there eventually). Provided you can arrange to read 8KB of sb, you might be OK.

Peter

Peter

author
#5 lore raw

Hi Peter,

Thanks for all the great info, I had about half of it figured out :) I'm starting to wonder though, isn't the version 1 superblock supposed to be able to use many more drives than 27? .. If so, should this not already be patched in the kernel source, and in the mdadm source? What am I missing? .. it seems like I am trying to do something that's already supposedly done.. am I not?

Thanks,
Tyler.

Quoted 111 lines · Peter T. Breuer
Quoted 9 lines · Peter T. Breuer
Quoted 3 lines · Tyler

figured out what you put below, i'm wondering though, will changing the structures here affect anything else? other than the mdadm tool.. what

Quoted 5 lines

I can't think of anything, offhand. I would first check through the kernel cde looking for associated gotchas before worrying about trivia like applications, however!

mdp_super_t (the size of which you modified) is used throughout md.c only. You want to check if it is mapped to disk or not. I think it is (offhand) ...

mdp_super_t *sb;

...

bdevname(rdev->bdev, b);
sb = (mdp_super_t*)page_address(rdev->sb_page);

So it seems to me to be intended to be 1KB of (sb) info on disk. If the blocksize is 1K, then reading/writing from disk may only deliver 1K.
...

    if (!sync_page_io(rdev->bdev, rdev->sb_offset<<1, MD_SB_BYTES, rdev->sb_page, READ))
             goto fail;

Hmm ... MD_SB_BYTES. But ....

#define MD_SB_BYTES 4096

So we read 4K off disk at a time! Phew!

Arrgh, but the undimensioned counts in md_p.h are in units of 4B, so when it says "1024 - ...", it really means "4KB - ...", and hence if you are going to icrease the size of the superblock, what you should do is

1) increase MD_SB_BYTES to 8KB.
2) change "1024 - ..." to "2048 - ..." in md_p.h.
3) keep looking. Can one get 2 pages at once off disk with sync_page_io?

Where is that rdev->sb_offset set? Great, in calc_dev_sboffset:

       /*
        * Calculate the position of the superblock,
        * it's at the end of the disk.
        *
        * It also happens to be a multiple of 4Kb.
        */
       sb_offset = calc_dev_sboffset(rdev->bdev);
       rdev->sb_offset = sb_offset;

Yes, but if we are going to increase the size to 8KB, we'd better check that calc ...

inline static sector_t calc_dev_sboffset(struct block_device *bdev)

{
       sector_t size = bdev->bd_inode->i_size >> BLOCK_SIZE_BITS;
       return MD_NEW_SIZE_BLOCKS(size);
}

Uh huhh... it counts the blocks in the device (I think those are 1KB units, not device-dependent, but I may be wrong). Then it chooses the index of a smaller number to return - 1 past the available array size:

#define MD_NEW_SIZE_BLOCKS(x) ((x & ~(MD_RESERVED_BLOCKS - 1)) - MD_RESERVED_BLOCKS)

Ummmm ... that appears to be an attempt to calculate (x / y) * y - y, where y is a power of two ...

  #define MD_RESERVED_BLOCKS      (MD_RESERVED_BYTES / BLOCK_SIZE)

and the clunker ...

  #define MD_RESERVED_BYTES       (64 * 1024)

OK. So AT LEAST 64KB are reserved at the end of the disk. We can only count on 64KB, however. The good news is that it looks like the first 4KB of that is the superblock, and that we might be able to use the rest for a bigger superblock (I suspect bitmap info will be kept there eventually). Provided you can arrange to read 8KB of sb, you might be OK.

Peter

Peter

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
Quoted 3 lines · Tyler

Thanks for all the great info, I had about half of it figured out :) I'm starting to wonder though, isn't the version 1 superblock supposed to be able to use many more drives than 27? .. If so, should this not

I don't know (nor do I know anything about superblocks, i.e. which format you should be using). All I can see is that the struct you wanted to modify (the one with the "27" in its standard definition) is read and written to disk in the first 4K of the 64K just after the array contents.

Maybe you don't want to modify that struct but something else. Maybe something slightly different is written to disk. That's impossible to tell without documetation or experiment!

According to Paul, all should just work hunky-dory if you just suppress the check in mdadm itself!

Quoted 3 lines

already be patched in the kernel source, and in the mdadm source? What am I missing? .. it seems like I am trying to do something that's already supposedly done.. am I not?

It most definitely is not the case that a particular one of the structs in md_p.h can take more than 27 entries in the array that appears in it. What significance that has requires some strategic inverse engineering to discover. The most obvious interpretation is that it is a real limit, but maybe not one that comes into play because we are using a different ondis struct.

Peter

#7 lore raw
In reply to Tyler
Quoted 4 lines · Tyler

PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, using the --metadata 1.0 option on the create line, and still get:

# mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
mdadm: invalid number of raid devices: 31

As far as I can tell, mdadm just shouldn't give this error when you're using a version 1 superblock. If you disable that check in mdadm.c, I think everything will work. Version 1 superblocks allow something like 2000 disks in an array.

--
Paul

author
#8 lore raw

Hi Paul,

Isn't the superblock still limited to 27 drives though? .. is there maybe a patch for version 1 superblocks that i don't have in 2.6.12-rc3-mm2, that changes the max drives in the md_p.h and md.c driver? There is only 4096 bytes earmarked for superblocks, and within that amount, there isn't enough room to list more than 27 drives that may be part of the same array.

Do I need some more patches for this to work, that aren't in the -mm tree yet? I was under the impression (from the mdadm v2.0 announce file, that anything *after* 2.6.11-rc3 in the -mm tree would have the required patches.

Also, where can I disable the check in mdadm?

Thanks,
Tyler.

Quoted 13 lines · Paul Clements
Quoted 5 lines · Tyler

PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, using the --metadata 1.0 option on the create line, and still get:

# mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
mdadm: invalid number of raid devices: 31

As far as I can tell, mdadm just shouldn't give this error when you're using a version 1 superblock. If you disable that check in mdadm.c, I think everything will work. Version 1 superblocks allow something like 2000 disks in an array.

Signature · 1 line
Paul
#9 lore raw
Quoted 1 line · Tyler

Isn't the superblock still limited to 27 drives though? .. is there

The version 0.90 superblock is limited to 27 drives. Version 1 superblock is limited to about 2000 drives, because Neil shrunk the amount of per-drive data stored in the superblock drastically. The check in mdadm, is in the file mdadm.c, just grep for the message that you got ("invalid number of raid devices"). Comment out that code and recompile mdadm. Everything should work after that. The kernel you have already has support for version 1 superblocks.

--
Paul

Quoted 33 lines

maybe a patch for version 1 superblocks that i don't have in 2.6.12-rc3-mm2, that changes the max drives in the md_p.h and md.c driver? There is only 4096 bytes earmarked for superblocks, and within that amount, there isn't enough room to list more than 27 drives that may be part of the same array.

Do I need some more patches for this to work, that aren't in the -mm tree yet? I was under the impression (from the mdadm v2.0 announce file, that anything *after* 2.6.11-rc3 in the -mm tree would have the required patches.

Also, where can I disable the check in mdadm?

Thanks,
Tyler.

Quoted 6 lines · Paul Clements
Quoted 1 line · Tyler

PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, using the --metadata 1.0 option on the create line, and still get:

Quoted 9 lines
Quoted 2 lines
# mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
mdadm: invalid number of raid devices: 31

As far as I can tell, mdadm just shouldn't give this error when you're using a version 1 superblock. If you disable that check in mdadm.c, I think everything will work. Version 1 superblocks allow something like 2000 disks in an array.

Signature · 1 line
Paul

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#10 lore raw

Okay, I will give that a try paul... but i've actually tryed using mdadm v2.0-devel to create an array with a version 1 superblock, using only 3 drives in a raid5 array. It hasn't worked so far, read my 2 bugreports to the mailing list, to see whats happening.. and maybe draw some insight from them? I'm at a loss as to why mdadm is failing right now.. even without going over the 27 drive limit. Neil hasn't responded to any emails I've sent regarding this either..

Thanks,
Tyler.

PS - Can you see any kind of order to the "applied" patches on Neil's site? .. I can't tell which ones are for the 2.0-devel version and which are for 1.9.x .. and one says 1.8 ... etc..

Quoted 46 lines · Paul Clements
Quoted 1 line · Tyler

Isn't the superblock still limited to 27 drives though? .. is there

The version 0.90 superblock is limited to 27 drives. Version 1 superblock is limited to about 2000 drives, because Neil shrunk the amount of per-drive data stored in the superblock drastically. The check in mdadm, is in the file mdadm.c, just grep for the message that you got ("invalid number of raid devices"). Comment out that code and recompile mdadm. Everything should work after that. The kernel you have already has support for version 1 superblocks.

Signature · 1 line
Paul
Quoted 36 lines

maybe a patch for version 1 superblocks that i don't have in 2.6.12-rc3-mm2, that changes the max drives in the md_p.h and md.c driver? There is only 4096 bytes earmarked for superblocks, and within that amount, there isn't enough room to list more than 27 drives that may be part of the same array.

Do I need some more patches for this to work, that aren't in the -mm tree yet? I was under the impression (from the mdadm v2.0 announce file, that anything *after* 2.6.11-rc3 in the -mm tree would have the required patches.

Also, where can I disable the check in mdadm?

Thanks,
Tyler.

Quoted 7 lines · Paul Clements
Quoted 1 line · Tyler

PS - I've tryed the latest 2.6.12-rc3-mm2 kernel, with mdadm v2.0-devel, using the --metadata 1.0 option on the create line, and still get:

Quoted 10 lines
Quoted 2 lines
# mdadm -C -e 1.0 -l 6 -n 31 /dev/md0 /dev/hdb /dev/hdc /dev/hdd /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz /dev/sdaa /dev/sdab
mdadm: invalid number of raid devices: 31

As far as I can tell, mdadm just shouldn't give this error when you're using a version 1 superblock. If you disable that check in mdadm.c, I think everything will work. Version 1 superblocks allow something like 2000 disks in an array.

Signature · 1 line
Paul

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html