Proposed Enhancements to MD
(I already posted this to LKML a few hours ago but forgot to post it over here)
All,
Adaptec has been looking at the MD driver for a foundation for their Open-Source software RAID stack. This will help us provide full and open support for current and future Adaptec RAID products (as opposed to the limited support through closed drivers that we have now).
While MD is fairly functional and clean, there are a number of enhancements to it that we have been working on for a while and would like to push out to the community for review and integration. These
include:
- partition support for md devices: MD does not support the concept of fdisk partitions; the only way to approximate this right now is by creating multiple arrays on the same media. Fixing this is required for not only feature-completeness, but to allow our BIOS to recognise the partitions on an array and properly boot them as it would boot a normal disk.
- generic device arrival notification mechanism: This is needed to support device hot-plug, and allow arrays to be automatically configured regardless of when the md module is loaded or initialized. RedHat EL3 has a scaled down version of this already, but it is specific to MD and only works if MD is statically compiled into the kernel. A general mechanism will benefit MD as well as any other storage system that wants hot-arrival notices.
- RAID-0 fixes: The MD RAID-0 personality is unable to perform I/O that spans a chunk boundary. Modifications are needed so that it can take a request and break it up into 1 or more per-disk requests.
- Metadata abstraction: We intend to support multiple on-disk metadata formats, along with the 'native MD' format. To do this, specific knowledge of MD on-disk structures must be abstracted out of the core and personalities modules.
- DDF Metadata support: Future products will use the 'DDF' on-disk metadata scheme. These products will be bootable by the BIOS, but must have DDF support in the OS. This will plug into the abstraction mentioned above.
I'm going to push these changes out in phases in order to keep the risk and churn to a minimum. The attached patch is for the partition support. It was originally from Ingo Molnar, but has changed quite a bit due to the radical changes in the disk/block layer in 2.6. The 2.4 version works quite well, while the 2.6 version is fairly fresh. One problem that I have with it is that the created partitions show up in /proc/partitions after running fdisk, but not after a reboot.
Scott
On 2004-01-12T20:41:54,
Scott Long <[email protected]> said:
Hi Scott, this is good to see!
Quoted 6 lines
- partition support for md devices: MD does not support the concept of fdisk partitions; the only way to approximate this right now is by creating multiple arrays on the same media. Fixing this is required for not only feature-completeness, but to allow our BIOS to recognise the partitions on an array and properly boot them as it would boot a normal disk.
I'm not too excited about this, because Device Mapping on top of md is much more flexible, but I see that users want it, and it should be pretty easy to add.
Quoted 7 lines
- generic device arrival notification mechanism: This is needed to support device hot-plug, and allow arrays to be automatically configured regardless of when the md module is loaded or initialized. RedHat EL3 has a scaled down version of this already, but it is specific to MD and only works if MD is statically compiled into the kernel. A general mechanism will benefit MD as well as any other storage system that wants hot-arrival notices.
Yes. Is anything missing from the 2.6 & hotplug & udev solution which you require?
Quoted 3 lines
- RAID-0 fixes: The MD RAID-0 personality is unable to perform I/O that spans a chunk boundary. Modifications are needed so that it can take a request and break it up into 1 or more per-disk requests.
Agreed.
Quoted 4 lines
- Metadata abstraction: We intend to support multiple on-disk metadata formats, along with the 'native MD' format. To do this, specific knowledge of MD on-disk structures must be abstracted out of the core and personalities modules.
This can get difficult, of course, and needs to be implemented in a way which doesn't slow us down too much.
Quoted 4 lines
- DDF Metadata support: Future products will use the 'DDF' on-disk metadata scheme. These products will be bootable by the BIOS, but must have DDF support in the OS. This will plug into the abstraction mentioned above.
OK. How does the DDF metadata differ from the current md data? Is it merely the layout, or are there functional differences?
In particular, I'm wondering whether partitions using the new activity logging features of md will still be bootable, or whether the boot partitions need to be 'md classic'.
Quoted 2 lines
bit due to the radical changes in the disk/block layer in 2.6. The 2.4 version works quite well, while the 2.6 version is fairly fresh.
I'd be reluctant doing any of the work for 2.4, but this is of course upto you.
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 8 lines
High Availability & Clustering \ ever tried. ever failed. no matter. SUSE Labs | try again. fail again. fail better. Research & Development, SUSE LINUX AG \ -- Samuel Beckett - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Lars Marowsky-Bree
On 2004-01-12T20:41:54,
Scott Long <[email protected]> said:Hi Scott, this is good to see!
Quoted 6 lines
- partition support for md devices: MD does not support the concept of fdisk partitions; the only way to approximate this right now is by creating multiple arrays on the same media. Fixing this is required for not only feature-completeness, but to allow our BIOS to recognise the partitions on an array and properly boot them as it would boot a normal disk.
Quoted 4 lines
I'm not too excited about this, because Device Mapping on top of md is much more flexible, but I see that users want it, and it should be pretty easy to add.
The biggest issue here is that a real fdisk table needs to exist on the array in order for our BIOS to recognise it as a boot device. While Device Mapper can probably do a good job at creating logical storage extends out of a single md device, it doesn't get us any closer to being able to boot off of an MD array.
Quoted 10 lines
Quoted 8 lines
- generic device arrival notification mechanism: This is needed to support device hot-plug, and allow arrays to be automatically configured regardless of when the md module is loaded or initialized. RedHat EL3 has a scaled down version of this already, but it is specific to MD and only works if MD is statically compiled into the kernel. A general mechanism will benefit MD as well as any other storage system that wants hot-arrival notices.
Quoted 3 lines
Yes. Is anything missing from the 2.6 & hotplug & udev solution which you require?
I'll admit that I'm not as familiar with 2.6 as I should be. Does a disk arrival mechanism already exist?
Quoted 9 lines
Quoted 4 lines
- RAID-0 fixes: The MD RAID-0 personality is unable to perform I/O that spans a chunk boundary. Modifications are needed so that it can take a request and break it up into 1 or more per-disk requests.
Agreed.
Quoted 9 lines
Quoted 4 lines
- Metadata abstraction: We intend to support multiple on-disk metadata formats, along with the 'native MD' format. To do this, specific knowledge of MD on-disk structures must be abstracted out of the core and personalities modules.
This can get difficult, of course, and needs to be implemented in a way which doesn't slow us down too much.
Normal I/O doesn't touch the metadata. Only during error recovery and
configuration would this be touched. Instead of the core and personality modules directly manipulating the metadata, a set of metadata-specific function pointers will be called through to handle
changing the on-disk metadata. So, no significant operational overhead
is introduced.
Quoted 10 lines
Quoted 5 lines
- DDF Metadata support: Future products will use the 'DDF' on-disk metadata scheme. These products will be bootable by the BIOS, but must have DDF support in the OS. This will plug into the abstraction mentioned above.
OK. How does the DDF metadata differ from the current md data? Is it merely the layout, or are there functional differences?
I'm not sure if the DDF spec has been officially published yet. It defines a set of data structures and their location on the disk that allows disk to be uniquely identified, logical extents to be grouped into arrays, recording of disk and array state, and event logging. It is completely different from the metadata that is used for classic MD. However, it is still compatible with the high-level striping and mirroring operations of MD.
Quoted 3 lines
In particular, I'm wondering whether partitions using the new activity logging features of md will still be bootable, or whether the boot partitions need to be 'md classic'.
Our products will only recognise and boot off of DDF arrays. They have no concept of classic MD metadata.
The goal of the abstraction is to allow new metadata personalities to be plugged in and 'Just Work', while not inhibiting the choice of using whatever metadata is most suitable for existing arrays. If you need to boot off of a DDF-aware controller, but use classic MD for secondary arrays, that will work.
Quoted 8 lines
Quoted 4 lines
bit due to the radical changes in the disk/block layer in 2.6. The 2.4 version works quite well, while the 2.6 version is fairly fresh.
I'd be reluctant doing any of the work for 2.4, but this is of course upto you.
This work was originally started on 2.4. With the closing of 2.4 and release of 2.6, we are porting are work forward. It would be nice to integrate the changes into 2.4 also, but we recognise the need for 2.4 to remain as stable as possible.
Scott
Quoted 2 lines · Scott Long
The biggest issue here is that a real fdisk table needs to exist on the array in order for our BIOS to recognise it as a boot device.
Hm, ok.
Quoted 5 lines
Quoted 2 lines
Yes. Is anything missing from the 2.6 & hotplug & udev solution which you require?
I'll admit that I'm not as familiar with 2.6 as I should be. Does a disk arrival mechanism already exist?
Yes. hotplug already will get you events when new disks arrive.
Quoted 6 lines
Quoted 3 lines
In particular, I'm wondering whether partitions using the new activity logging features of md will still be bootable, or whether the boot partitions need to be 'md classic'.
Our products will only recognise and boot off of DDF arrays. They have no concept of classic MD metadata.
OK. The question was meant differently. In 2.6, we have the ability to log resyncs and journal updates (see the discussions on linux-raid). I was just wondering whether DDF would allow this, or whether it is a simple minded "this disk good, that disk bad", and thus the boot drive might not be able to use the new md features with the DDF metadata.
Quoted 4 lines
This work was originally started on 2.4. With the closing of 2.4 and release of 2.6, we are porting are work forward. It would be nice to integrate the changes into 2.4 also, but we recognise the need for 2.4 to remain as stable as possible.
2.4 is dead and shouldn't see new features.
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 8 lines
High Availability & Clustering \ ever tried. ever failed. no matter. SUSE Labs | try again. fail again. fail better. Research & Development, SUSE LINUX AG \ -- Samuel Beckett - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Scott Long
- DDF Metadata support: Future products will use the 'DDF' on-disk metadata scheme. These products will be bootable by the BIOS, but must have DDF support in the OS. This will plug into the abstraction mentioned above.
For those unfamiliar with DDF (Disk Data Format), it is a Storage Networking Industry Association (SNIA) project ("Common RAID DDF TWG"), designed to provide a single metadata format to be used by all the RAID vendors (hardware and software alike). It removes vendor lock-in by having a metadata format that all can use, thus in theory you could move disks from an Adaptec hardware RAID controller to an LSI software RAID solution without reformatting the disks or touching your file systems in any way. Dell has been championing the DDF concept for quite a while, and is driving vendors from which we purchase RAID solutions to use DDF instead of their own individual metadata formats.
I haven't seen the spec yet myself, but I'm lead to believe that DDF allows for multiple logical drives to be created across a single set of disks (e.g. a 10GB RAID1 LD and a 140GB RAID0 LD together on two 80GB spindles), as well as whole disks be used. It has a mechanism to support reconstruction checkpointing, so you don't have to restart a reconstruct from the beginning after a reboot, but from where you left off. And other useful features too that you'd expect in a common RAID solution.
DDF is quickly becoming important to RAID and system vendors, and I welcome Adaptec's work to implement DDF support on Linux.
Thanks,
Matt
Signature · 4 lines
Matt Domsch Sr. Software Engineer, Lead Engineer Dell Linux Solutions www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com
Quoted 18 lines · Matt Domsch
Quoted 4 lines · Scott Long
- DDF Metadata support: Future products will use the 'DDF' on-disk metadata scheme. These products will be bootable by the BIOS, but must have DDF support in the OS. This will plug into the abstraction mentioned above.
For those unfamiliar with DDF (Disk Data Format), it is a Storage Networking Industry Association (SNIA) project ("Common RAID DDF TWG"), designed to provide a single metadata format to be used by all the RAID vendors (hardware and software alike). It removes vendor lock-in by having a metadata format that all can use, thus in theory you could move disks from an Adaptec hardware RAID controller to an LSI software RAID solution without reformatting the disks or touching your file systems in any way. Dell has been championing the DDF concept for quite a while, and is driving vendors from which we purchase RAID solutions to use DDF instead of their own individual metadata formats.
Quoted 8 lines
I haven't seen the spec yet myself, but I'm lead to believe that DDF allows for multiple logical drives to be created across a single set of disks (e.g. a 10GB RAID1 LD and a 140GB RAID0 LD together on two 80GB spindles), as well as whole disks be used. It has a mechanism to support reconstruction checkpointing, so you don't have to restart a reconstruct from the beginning after a reboot, but from where you left off. And other useful features too that you'd expect in a common RAID solution.
So, why not use EVMS and/or Device Mapper to read the DDF metadata and set up the mappings that way?
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 2 lines · Andreas Dilger
So, why not use EVMS and/or Device Mapper to read the DDF metadata and set up the mappings that way?
PS - outgoing email was delayed, this has already been covered. Sorry.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 2 lines · Matt Domsch
DDF is quickly becoming important to RAID and system vendors, and I welcome Adaptec's work to implement DDF support on Linux.
Fully agreed, the days of vendor-specific metadata formats need to be numbered (with a small number). Speaking a customer with a CMD FC-to-SCSI RAID controller, which used to be dual-redundant but is now single (because of a dead unit), we are not looking forward to the day when the remaining controller dies and we lose all the data on the array due to a forced metadata format change.
However, given that this will not likely be 2.6 material until after it's built and tested in 2.7 and then backported, it doesn't seem to make any sense to me to build any of this on top of the MD subsystem at all (see other replies about using DM instead). Additionally, it also does not seem to make any sense to build any of the DDF reading/writing/management in the kernel _at all_. There is no advantage to it being there once initramfs is a standard part of the boot process, so all of this should be done is userspace and just communicated into the kernel to tell it what logical devices to construct using which DM modules.
Quoted 4 lines · Matt Domsch
I haven't seen the spec yet myself, but I'm lead to believe that DDF allows for multiple logical drives to be created across a single set of disks (e.g. a 10GB RAID1 LD and a 140GB RAID0 LD together on two 80GB spindles), as well as whole disks be used. It has a
Me either. Any idea if there will be a public comment period, or is the spec "locked" into 1.0 when it's released in a month or so?
Jeff
Quoted 6 lines · Jeff Garzik
Quoted 4 lines · Matt Domsch
I haven't seen the spec yet myself, but I'm lead to believe that DDF allows for multiple logical drives to be created across a single set of disks (e.g. a 10GB RAID1 LD and a 140GB RAID0 LD together on two 80GB spindles), as well as whole disks be used. It has a
How is this different than the 20GB RAID0 and 6 15BB RAID1s that I've got on two 100GB spindles right now?
I think its on 2.4, might even be 2.2.
Chris
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 2 lines · Chris Friesen
How is this different than the 20GB RAID0 and 6 15BB RAID1s that I've got on two 100GB spindles right now?
Indeed, md does this with partitions on the disks today, so it is analogous; DDF does this with disk extents, which has the same functionality as partitions, but without an MSDOS partition table to define the partitions, but an on-disk metadata format (yes, partition tables are metadata too...).
The solution needs partitions/extents in two places.
1) below the logical drive, from which logical drives are created.
2) above the logical drive, on which multiple file systems are created.
md provides 1) today, and as discussed today patches exist to do 2) that have not yet been merged.
Signature · 4 lines
Matt Domsch Sr. Software Engineer, Lead Engineer Dell Linux Solutions www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com
Quoted 9 lines · Jeff Garzik
Quoted 4 lines · Matt Domsch
I haven't seen the spec yet myself, but I'm lead to believe that DDF allows for multiple logical drives to be created across a single set of disks (e.g. a 10GB RAID1 LD and a 140GB RAID0 LD together on two 80GB spindles), as well as whole disks be used. It has a
Me either. Any idea if there will be a public comment period, or is the spec "locked" into 1.0 when it's released in a month or so?
As it happens, Bill Dawkins of Dell is the DDF committee chair at SNIA. Here's what he's told me:
The current draft of the DDF specification is available for review to any member of SNIA. This is a "Work in Progress" draft. Anyone in a member company can go to www.snia.org and sign up for web access. They will then have to sign up for the DDF Technical Working Group. Acceptance to the DDF TWG is automatic and the current documents are available there.
(As Dell is a member, I signed up for the DDF TWG as an observer. Other companies are also on the member list, including Sistina, so Jeff you may be able to get a Sistina collegue to mail you a copy.
http://www.snia.org/about/member_list has the list of member companies. - Matt)
For people and companies who are not members of SNIA, I am writing to the SNIA Technical Director to see if I can release copies of the draft spec now. I'll let you know when I get a response.
As for the timeline, we have a face to face meeting of the DDF TWG next Tuesday and it is our intent to vote on releasing the specification as a "Trial Use" specification for public review and comment. If the vote is affirmative, the SNIA Technical Council will have to meet to determine when and if to release the "Trial Use" specification. This may take a few months, so we are probably looking at March for full release. Feel free to share this information with your Linux contacts.
So, for now, if you're in SNIA, you can get access to the draft spec, and in a few months the draft spec should be publicly available.
Thanks,
Matt
Signature · 4 lines
Matt Domsch Sr. Software Engineer, Lead Engineer Dell Linux Solutions www.dell.com/linux Linux on Dell mailing lists @ http://lists.us.dell.com