Skip to contentSkip to conversation

Lustre Lite 1.0 beta 1

ANNOUNCE Peter Braam 3 participants 6 messages Mar 12, 2003 – Mar 17, 2003 VFS

lore
Share & tools
lore
#0 lore raw

Subject: Lustre Lite 0.6 released (1.0 beta 1)

Summary
-------

We're pleased to announce that the first Lustre Lite beta (0.6) has been tagged and released. Seven months have passed since our last major release, and Lustre Lite is quickly approaching the goal of being stable, consistent, and fast on clusters up to 1,000 nodes.

Over the last few months we've spent thousands of hours improving and testing the file system, and now it's ready for a wider audience of early adopters. In particular, Lustre users on ia32 and ia64 Linux systems running 2.4.19 and Red Hat 2.4.18-based kernels. Lustre may work on other Linux platforms, but has not been extensively tested, and may require some additional porting effort.

We expect that you will find many bugs that we are unable to provoke in our testing, and we hope that you will take the time to report them to our bug system (see Reporting Bugs below).

Features
--------

Lustre Lite 0.6:

- has been tested extensively on ia32 and ia64 Linux platforms
- supports TCP/IP and Quadrics Elan3 interconnects
- supports multiple Object Storage Targets (OSTs) for file data storage
- supports multiple Metadata Servers (MDSs) in an active/passive failover configuration (requires shared storage between MDS nodes). Automatic failover requires an external failover package such as Red Hat's clumanager.
- provides a nearly POSIX-compliant filesystem interface (some areas remain non-compliant; for example, we do not synchronize atimes)
- aims to recover from any single failure without loss of data or application errors
- scales well; we have tested with as many as 1,100 clients and 128 OSTs
- is Free Software, released under the terms and conditions of the GNU General Public License

Risks
-----

As with any beta software, but especially kernel modules, Lustre Lite carries the real risk of data loss or system crashes. It is very likely that users will test situations which we have not, and provoke bugs which crash the system. We must insist that you

BACKUP YOUR DATA

prior to installing Lustre, and that you understand that

we make NO GUARANTEES about Lustre.

Please read the COPYING file included with the distribution for more information about the licensing of Lustre.

Known Bugs
----------

Although Lustre is for the most part stable, there are some known bugs with this current version that you should be particularly aware of:

- Some high-load situations involving multiple clients have been known to provoke a client crash in the lock manager (bug 984)

- Failover support is incomplete; some access patterns will not recover correctly

- Recovery does not gracefully handle multiple services present on the same node

- Failures can lead to unrecoverable states, which require the system to be umounted and remounted (and, in some case, nodes may require a reboot)

- Unmounting a client while an MDS is failed may hang the "umount" command, which will need to be "kill"ed manually (bug 978)

- Metadata recovery will time out and abort if there are clients which hold uncommitted requests, but which do not detect the death and failover of the MDS. Running a metadata operation on quiescent clients will cause them to join recovery. (bug 957)

Getting Started
---------------

<https://projects.clusterfs.com/lustre/LustreHowto> contains instructions for downloading, building, configuring, and running Lustre. If you encounter problems, you can seek help from others in the lustre-discuss mailing list (see below).

Reporting Bugs
--------------

We are eager to hear about new bugs, especially if you can tell us how to reproduce them. Please visit <http://bugzilla.lustre.org/> to report problems.

The closer that you can come to the ideal described in
<https://projects.clusterfs.com/lustre/BugFiling>, the better.

Mailing Lists
-------------

See <http://www.lustre.org/lists.html> for links to the various Lustre mailing lists.

Acknowledgement
---------------

The US government has funded much of the Lustre effort through the National Laboratories. In addition to money they have provided invaluable experience and fantastic help with testing both in terms of equipment and people. We thank them all, but in particular Mark Seager, Bill Boas and Terry Heidelberg's team at LLNL which went far beyond the call of duty, Lee Ward (Sandia) and Gary Grider (LANL), Scott Studham (PNNL). We have also had the benefit of partnerships with UCSC, HP, Intel, BlueArc and DDN and we are grateful to them.

---

Thank you for your interest in and testing of Lustre. We appreciate your effort, patience, and bug reports as we work towards Lustre Lite 1.0.

The Cluster File Systems team

Peter J. Braam     <[email protected]>
Phil Schwan        <[email protected]>
Andreas Dilger     <[email protected]>
Robert Read        <[email protected]>
Eric Barton        <[email protected]>
Radhika Vullikanti <[email protected]>
Mike Shaver        <[email protected]>
Eric Mei           <[email protected]>
Zach Brown         <[email protected]>
Chris Cooper       <[email protected]>
#1 lore raw
Quoted 5 lines · Peter Braam

Features
--------

Lustre Lite 0.6:

Quoted 18 lines

- has been tested extensively on ia32 and ia64 Linux platforms
- supports TCP/IP and Quadrics Elan3 interconnects
- supports multiple Object Storage Targets (OSTs) for file data storage
- supports multiple Metadata Servers (MDSs) in an active/passive failover configuration (requires shared storage between MDS nodes). Automatic failover requires an external failover package such as Red Hat's clumanager.
- provides a nearly POSIX-compliant filesystem interface (some areas remain non-compliant; for example, we do not synchronize atimes)
- aims to recover from any single failure without loss of data or application errors
- scales well; we have tested with as many as 1,100 clients and 128 OSTs
- is Free Software, released under the terms and conditions of the GNU General Public License

Quite interesting. How is it different from gfs and other cluster file systems? Is it a SAN/Shared-SCSI disk sharing file system or some cluster file system which ride on something like gnbd/nbd/iSCSI pool ? Not quite sensible to run cluster file systems data channels over TCP/IP as it is too slow, as the price of fibre-channel storage keep dropping as well as those for implement cluster file systems in a production environment are usually affordable for fibre-channel storage. The name "OST" seems interesting but I don't really like the iSCSI or pool of block devices spreading across multiple machines as it is very hard to manage and likely to fail easily. How is lustre handling these senarios?

regards,
David Chow

#2 lore raw
Quoted 20 lines · David Chow
Quoted 4 lines · Peter Braam

Features
--------
Lustre Lite 0.6:

Quoted 14 lines

- has been tested extensively on ia32 and ia64 Linux platforms
- supports TCP/IP and Quadrics Elan3 interconnects
- supports multiple Object Storage Targets (OSTs) for file data storage
- supports multiple Metadata Servers (MDSs) in an active/passive failover configuration (requires shared storage between MDS nodes). Automatic failover requires an external failover package such as Red Hat's clumanager.
- provides a nearly POSIX-compliant filesystem interface (some areas remain non-compliant; for example, we do not synchronize atimes)
- aims to recover from any single failure without loss of data or application errors
- scales well; we have tested with as many as 1,100 clients and 128 OSTs
- is Free Software, released under the terms and conditions of the GNU General Public License

Quoted 2 lines

Quite interesting. How is it different from gfs and other cluster file systems?

One primary difference between Lustre and other cluster file systems is that Lustre is designed with enormous scalability in mind. It can already (in the first "Lite" version) handle over 1000 clients, multi GB/s aggregate IO rates, and many tens or hundreds of TB of storage. Future plans scale up to tens of thousands of clients, and PB of storage.

Quoted 2 lines

Is it a SAN/Shared-SCSI disk sharing file system or some cluster file system which ride on something like gnbd/nbd/iSCSI pool ?

Not really either of these. Primarily it is a multiple-server based distributed file system, with (currently) one server doing all of the metadata work, and (some configurable number between 1 and hundreds of) storage servers (OSTs). The servers are independent (generally Linux) boxes with attached disks. There is the possibility of a "SAN" interface from the clients to the storage servers, but this isn't the common usage scenario.

Quoted 4 lines

Not quite sensible to run cluster file systems data channels over TCP/IP as it is too slow, as the price of fibre-channel storage keep dropping as well as those for implement cluster file systems in a production environment are usually affordable for fibre-channel storage.

Well, TCP/IP is one of the network types we support (via Portals Network Abstraction Layers, NALs) mostly for development/testing, along with Quadrics Elan (our large clusters use these, VERY FAST), and in-development Myrinet NAL.

The reason that FC interconnect isn't the primary interconnect is that:
1) FC is still more expensive than GigE (although less expensive than Elan)
2) you can't scale FC to thousands of nodes
3) more people have TCP interconnect than anything else (even if it is TCP-over-FC)
4) none of our customers actually have asked for FC yet

Quoted 3 lines

The name "OST" seems interesting but I don't really like the iSCSI or pool of block devices spreading across multiple machines as it is very hard to manage and likely to fail easily. How is lustre handling these senarios?

Well, each OST is primarily (from the Lustre sense) the network interface protocol, and the internal implementation is opaque to the outside world. Each OST is independent of the others, although the clients end up allocating files on all of them.

For Linux OSTs we use ext3 on regular block devices (raw disk, MD RAID, LOV, whatever you want to use) for the actual data storage, and the filesystem is journaled/managed totally independent from all of the other OSTs. We have also used reiserfs for OST storage at times (and concievably you could use XFS/JFS), and there are also 3rd party vendors who are building OST boxes with their own non-Linux internals.

Since this is just regular disk attached to regular Linux boxes, it is also possible to do storage server failover (already being implemented) without clients even being aware of a problem. Single disk failure is expected to be handled by RAID of some kind.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#3 lore raw
Quoted 14 lines

Well, each OST is primarily (from the Lustre sense) the network interface protocol, and the internal implementation is opaque to the outside world. Each OST is independent of the others, although the clients end up allocating files on all of them.

For Linux OSTs we use ext3 on regular block devices (raw disk, MD RAID, LOV, whatever you want to use) for the actual data storage, and the filesystem is journaled/managed totally independent from all of the other OSTs. We have also used reiserfs for OST storage at times (and concievably you could use XFS/JFS), and there are also 3rd party vendors who are building OST boxes with their own non-Linux internals.

Quoted 8 lines

Since this is just regular disk attached to regular Linux boxes, it is also possible to do storage server failover (already being implemented) without clients even being aware of a problem. Single disk failure is expected to be handled by RAID of some kind.

Cheers, Andreas

Andreas,

Thanks for your lengthly explanation. The design looked like Coda with OST as you refer to the actual data storage. In fact, it is a stacked file cache or your store data in files persistently on existing file systesms. However, how can it handle a disconnected storage server? Where this is the most diffcult problem for any cluster file systems that support disconnection. It is obviously not allow disconnection for system like having thousands of nodes is bad. The chance of node failure is very high in those cases. As file allocation is still allowed to be done across multiple storage servers. The answer to resolving data conflicts transparently after disconnection is impossible! I would really like to hear this from Lustre as it already played around with 1000 nodes. When I came down to design a distributed file system end up blowing my head about this. Thanks for comments or may yo give some directions for me as I am very interested in this topic.

regards,
David Chow

#4 lore raw
Quoted 17 lines · David Chow
Quoted 12 lines

Well, each OST is primarily (from the Lustre sense) the network interface protocol, and the internal implementation is opaque to the outside world. Each OST is independent of the others, although the clients end up allocating files on all of them.

For Linux OSTs we use ext3 on regular block devices (raw disk, MD RAID, LOV, whatever you want to use) for the actual data storage, and the filesystem is journaled/managed totally independent from all of the other OSTs. We have also used reiserfs for OST storage at times (and concievably you could use XFS/JFS), and there are also 3rd party vendors who are building OST boxes with their own non-Linux internals.

Quoted 4 lines

Since this is just regular disk attached to regular Linux boxes, it is also possible to do storage server failover (already being implemented) without clients even being aware of a problem. Single disk failure is expected to be handled by RAID of some kind.

Quoted 4 lines

Thanks for your lengthly explanation. The design looked like Coda with OST as you refer to the actual data storage. In fact, it is a stacked file cache or your store data in files persistently on existing file systems.

Correct, although there is not any aspect of a file "cache" around like Coda/InterMezzo. The clients are only clients and do not store any persistent data locally. They can cache data in memory (page cache), and there is a distributed lock manager to keep this data coherent across the cluster.

Quoted 1 line

However, how can it handle a disconnected storage server?

There is no concept of a "disconnected storage server" in the sense that Coda/InterMezzo support disconnected operation. You can have storage server failover, so the clients still have access to the same (shared) disk storage, and we also handle storage server failure (data on that server is inaccessible (-EIO), but you are still able to read data from other servers for which a file is striped across, and new files are not striped over the failed server.

Quoted 4 lines

Where this is the most diffcult problem for any cluster file systems that support disconnection. It is obviously not allow disconnection for system like having thousands of nodes is bad. The chance of node failure is very high in those cases.

Since clients don't have any persistent state (this is purely a client/server architecture and not a peer-peer relationship), we don't care about client failure. The servers will notice the client has failed, or it is not responding in a timely manner, and the server will revoke all of the locks that client is holding and boot it out of the cluster. If it tries to connect back, it will do so as a "new" client, and any existing open files will get -EIO if they are accessed.

Quoted 4 lines

I would really like to hear this from Lustre as it already played around with 1000 nodes. When I came down to design a distributed file system end up blowing my head about this. Thanks for comments or may yo give some directions for me as I am very interested in this topic.

I would really suggest that you read the Lustre book, which is available online at the lustre.org web site, or as part of the CVS repository.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#5 lore raw
In reply to David Chow
Quoted 17 lines

Andreas,

Thanks for your lengthly explanation. The design looked like Coda with OST as you refer to the actual data storage. In fact, it is a stacked file cache or your store data in files persistently on existing file systesms. However, how can it handle a disconnected storage server? Where this is the most diffcult problem for any cluster file systems that support disconnection. It is obviously not allow disconnection for system like having thousands of nodes is bad. The chance of node failure is very high in those cases. As file allocation is still allowed to be done across multiple storage servers. The answer to resolving data conflicts transparently after disconnection is impossible! I would really like to hear this from Lustre as it already played around with 1000 nodes. When I came down to design a distributed file system end up blowing my head about this. Thanks for comments or may yo give some directions for me as I am very interested in this topic.

Quoted 2 lines

regards,
David Chow

Hi David,

Lustre recovers from client failures and clients recover from server failures, but does not allow disconnected operations. Disconnected operation referes to the ability to make updates when the clients are not connected to the servers.

The Lustre architecture does allow for modular extensions that enable disconnected operation, but no customers have asked for it yet.

We have designed a very simple, automatic algortithm for handling conflicts arising during disconnected operations for InterMezzo, see the paper on www.inter-mezzo.org. Again, this could be implemented for Lustre, but we are waiting for a contract before we will do this.

Disconnected operation would involve a client cache, which will be many many times slower than the distributed network infrastructure when the file sizes exceed what can be cached in memory on the
client. This is unusual but quite important for supercomputing and
some industrial applications.

- Peter -