LVM and Debian
Hello, all.
Just joined in on the list - of course, to show proper etiquette, I
read the archives first, to catch up on the topics discussed. Something I didn't see, however, were comments on the stablility of the LVM code under Debian Potato.
Some background - I'm an AIX guy, working for a consulting company that
specializes in storage management. Wow, that sound WAY less fun than it actually is - basically, I'm a 23-year-old, fifth-year linux geek who managed to land a job doing unix-ish things.
Some more background - in our house, we're running just under 200G of
disk - the shoutcast "BeNOW" (http://www.benow.org) is running in the server room, right off my bedroom. I'm not the originator of it, but I'm a full partner and am the administrator of the servers it runs on.
And even more - we have friends, here in Calgary, that have the
reclaimant contracts on used gear. Basically, they pick up truckloads of Unix hardware from large corporations, give them tax credits, and sell it all off. They've got exclusive contracts with a lot of places, so they're an incredible resource. They're not out to turn a profit, just make a living - so incredible deals.
The deal I'm referring mostly to is the "9G fullheight Seagate SCSI
drive, 1M cache, 5500 RPM, 9ms access" for $50 CAD. We bought eight. LVM comes into play here.
So I did manage to get 0.7 to compile happily under Debian - using
2.2.10-ac12. This was difficult, and I've discovered why - either Debian or LVM isn't properly referencing where the includes are stored. My fix was:
cd /root/LVM/0.7/tools
ln -s /usr/src/linux/includes/linux linux
make
This worked.
I plugged in four of the 9G drives - would have been eight, but I blew
one of the two power supplies in the "cube" (a four-beside-four fullheight enclosure) when I didn't set the drives to delayed powerup. I created a 33G volume group called "rootvg" (sue me, I learned LVM on AIX) and created a 33G lv called lv01 inside that.
My problems begin here - it worked happily! I shared it out to the
others via Samba, and they happily drug and dropped mass quantities of .MP3's to it. After a while, it crashed.
I'm sort of hypothesizing that it crashed as it hit the end of the
first drive - I'm only doing a linear setup, no stripes. Roughly at the time that it WOULD have hit 9G, it locked the system solid, requiring a power-down. Unfortunately, the machine doesn't have a head most of the time, so no errors were noted.
Since then, I've compiled 2.2.11-ac3, and have been working on getting
the same setup running again - with one exception. I tried to put a fifth drive in the machine, INSIDE the actual desktop case - and it seems I don't own a single 50-pin ribbon cable that is clean enough to avoid timeouts. At 2am, I powered it off and left it for my bed - had to be up at 6 for work.
To give a long story another paragraph, I'll be continuing the struggle
tonight. Has anyone had any problems with stability under Debian, and/or tracked down what those might stem from? I'm running the machine now with it's own monitor (a rarity in the server room - nearing 20 machines now) and keyboard, constantly keeping an eye on it. I'll be setting it up again tonight with the same setup, 33G, but most likely striped.
Looking through the list, I don't see many people detailing what they
ARE doing with LVM, just what they CAN'T do, and asking for help. I apologize if I've offended anyone with the length of this post - but this is what I like to read, and I'd love to see what other people are doing with this software. If it's to be included in kernel 2.4, we'd damn well better push it to it's limits! :)
Cheers,
- Drew.
Quoted 4 lines · Drew Smith
So I did manage to get 0.7 to compile happily under Debian - using
2.2.10-ac12. This was difficult, and I've discovered why - either Debian or LVM isn't properly referencing where the includes are stored. My fix was:
I haven't actually compiled the tools under debian myself. Debian does not have /usr/include/linux matching up with the kernel release, so any patches you might do to the kernel won't show up.
Quoted 3 lines
My problems begin here - it worked happily! I shared it out to the
others via Samba, and they happily drug and dropped mass quantities of .MP3's to it. After a while, it crashed.
Was there anything recorded by syslog? How did it crash?
Quoted 2 lines
I'm sort of hypothesizing that it crashed as it hit the end of the
first drive - I'm only doing a linear setup, no stripes. Roughly at the
I've got about 40GB of mix'n'match drives -- 12GB IDE's, .5GB SCSIs, full height, half height, a real mix of everything, so striping wouldn't help that much. If you are using ext2fs on the lv, you'll hit all over the drives, so there won't be a "just before the second drive". ext2fs will space out the files all over the lv.
Quoted 3 lines
time that it WOULD have hit 9G, it locked the system solid, requiring a power-down. Unfortunately, the machine doesn't have a head most of the time, so no errors were noted.
Do you have magic sys-rq compiled in? This could be useful...
Quoted 2 lines
Since then, I've compiled 2.2.11-ac3, and have been working on getting
the same setup running again - with one exception. I tried to put a
I've been using 2.2.10-ac12 and 2.2.11-ac1, althought -ac1 has known TCP problems...
Quoted 2 lines
seems I don't own a single 50-pin ribbon cable that is clean enough to avoid timeouts. At 2am, I powered it off and left it for my bed - had
Yes, that can be a problem. It's also a problem when one drive dies and you aren't running raid on it at any level. When using older full height SCSI drives in stripe/linear, if you don't add raid to it somewhere, you are only as strong as the weakest drive. I dunno if I'd trust heavily used drives nearing the end of their life...
Quoted 2 lines
tonight. Has anyone had any problems with stability under Debian, and/or tracked down what those might stem from? I'm running the machine now
I'm running debian potato on one machine, but it isn't running LVM. The LVM system is a Slackware 4.0 system, still on libc5. Maybe it is libc?
If you are running the latest from potato (2.1.x) it may not be
compatible with the tools as is, I dunno if anyone has looked into it yet.
Hiya Ryan,
Quoted 3 lines · Ryan Murray
I haven't actually compiled the tools under debian myself. Debian does not have
/usr/include/linuxmatching up with the kernel release, so any patches you might do to the kernel won't show up.
Zuh? Lost me here - I patch the kernel all the time. LVM is actually
looking for lvm.h in /usr/src/linux/include/linux - and that's placed there properly by the Alan Cox patch... compiling the tools, it just wasn't finding the include directory happily. I symlinked it in, and it compiled fine.
Quoted 5 lines
Quoted 3 lines
My problems begin here - it worked happily! I shared it out to the
others via Samba, and they happily drug and dropped mass quantities of .MP3's to it. After a while, it crashed.Was there anything recorded by syslog? How did it crash?
*sigh* - I wish. No errors whatsoever, and the roommate rebooted it
before I had a chance to throw a graphics head on it and see what was onscreen at the time. It has a monitor now, specifically for that reason.
Quoted 8 lines
Quoted 2 lines
I'm sort of hypothesizing that it crashed as it hit the end of the
first drive - I'm only doing a linear setup, no stripes. Roughly at theI've got about 40GB of mix'n'match drives -- 12GB IDE's, .5GB SCSIs, full height, half height, a real mix of everything, so striping wouldn't help that much. If you are using ext2fs on the lv, you'll hit all over the drives, so there won't be a "just before the second drive". ext2fs will space out the files all over the lv.
Ah - this is news to me - I really don't know all that much about the
way filesystems work. This is why I'm jumping on the Linux LVM wagon early. :)
Quoted 5 lines
Quoted 3 lines
time that it WOULD have hit 9G, it locked the system solid, requiring a power-down. Unfortunately, the machine doesn't have a head most of the time, so no errors were noted.
Do you have magic sys-rq compiled in? This could be useful...
Nopers, and I'm afraid I've never actually used it - I know what a
kernel oops is, and that you can trace it - but not how. :)
Quoted 5 lines
Quoted 2 lines
Since then, I've compiled 2.2.11-ac3, and have been working on getting
the same setup running again - with one exception. I tried to put aI've been using 2.2.10-ac12 and 2.2.11-ac1, althought -ac1 has known TCP problems...
Hurm, the problems were with 2.2.10-ac12 - hopefully 2.2.11-ac3 will be
better.
Quoted 8 lines
Quoted 2 lines
seems I don't own a single 50-pin ribbon cable that is clean enough to avoid timeouts. At 2am, I powered it off and left it for my bed - had
Yes, that can be a problem. It's also a problem when one drive dies and you aren't running raid on it at any level. When using older full height SCSI drives in stripe/linear, if you don't add raid to it somewhere, you are only as strong as the weakest drive. I dunno if I'd trust heavily used drives nearing the end of their life...
*nod*, that's an issue - however, all of the data is static, and will
be on high-speed tape by the end of the week (unless I can find some optical media for my unpopulated Artecon library...) - so if a drive dies, that's about 1 hour's work, then another 8 hours unattended. Almost worth the risk rather than losing space to parity. Your thoughts? Ideas on maximizing security while maintaining speed and capacity?
Quoted 8 lines
Quoted 2 lines
tonight. Has anyone had any problems with stability under Debian, and/or tracked down what those might stem from? I'm running the machine now
I'm running debian potato on one machine, but it isn't running LVM. The LVM system is a Slackware 4.0 system, still on libc5. Maybe it is libc?
If you are running the latest from potato (2.1.x) it may not be
compatible with the tools as is, I dunno if anyone has looked into it yet.
Hurm, never even considered LibC. I'll look into it tonight at home -
can't remember the version, and the machine's down. Woo! Crash-test weekend!
Cheers,
- Drew.
Quoted 10 lines · Ryan Murray
Quoted 4 lines · Drew Smith
So I did manage to get 0.7 to compile happily under Debian - using
2.2.10-ac12. This was difficult, and I've discovered why - either Debian or LVM isn't properly referencing where the includes are stored. My fix was:I haven't actually compiled the tools under debian myself. Debian does not have
/usr/include/linuxmatching up with the kernel release, so any patches you might do to the kernel won't show up.
Actualy you are supposed to symlink /usr/include/linux to /usr/src/linux/include/linux. I'm told by the head honchos at Debian that this method has been semi-officialy endorsed as the proper method instead of installing the .h files in /usr/include/linux. If you do it this way you find all the patches include files as well.
Quoted 19 lines
Quoted 3 lines
My problems begin here - it worked happily! I shared it out to the
others via Samba, and they happily drug and dropped mass quantities of .MP3's to it. After a while, it crashed.Was there anything recorded by syslog? How did it crash?
Quoted 2 lines
I'm sort of hypothesizing that it crashed as it hit the end of the
first drive - I'm only doing a linear setup, no stripes. Roughly at theI've got about 40GB of mix'n'match drives -- 12GB IDE's, .5GB SCSIs, full height, half height, a real mix of everything, so striping wouldn't help that much. If you are using ext2fs on the lv, you'll hit all over the drives, so there won't be a "just before the second drive". ext2fs will space out the files all over the lv.
Quoted 3 lines
time that it WOULD have hit 9G, it locked the system solid, requiring a power-down. Unfortunately, the machine doesn't have a head most of the time, so no errors were noted.
Quoted 8 lines
Do you have magic sys-rq compiled in? This could be useful...
Quoted 2 lines
Since then, I've compiled 2.2.11-ac3, and have been working on getting
the same setup running again - with one exception. I tried to put aI've been using 2.2.10-ac12 and 2.2.11-ac1, althought -ac1 has known TCP problems...
Running vanilla 2.2.10+lvm-0.7 here on a production server that gets pounded every night round 2am. :) It's serving as a primary mirror for Debian http and ftp distribution. I get occasional scsi errors out of my ancient aha2940-pci and lvm handles these very nicely so far.
Quoted 9 lines
Quoted 2 lines
seems I don't own a single 50-pin ribbon cable that is clean enough to avoid timeouts. At 2am, I powered it off and left it for my bed - had
Yes, that can be a problem. It's also a problem when one drive dies and you aren't running raid on it at any level. When using older full height SCSI drives in stripe/linear, if you don't add raid to it somewhere, you are only as strong as the weakest drive. I dunno if I'd trust heavily used drives nearing the end of their life...
Quoted 8 lines
Quoted 2 lines
tonight. Has anyone had any problems with stability under Debian, and/or tracked down what those might stem from? I'm running the machine now
I'm running debian potato on one machine, but it isn't running LVM. The LVM system is a Slackware 4.0 system, still on libc5. Maybe it is libc?
If you are running the latest from potato (2.1.x) it may not be
compatible with the tools as is, I dunno if anyone has looked into it yet.
As noted above, Debian Potato( 2.2 ) with vanilla 2.2.10+lvm-0.7. Dual pIII 450/512MB 9g,5x4.5g scsi.
yamato.terrabox.com:
9:11am up 10 days, 11:14, 2 users, load average: 0.52, 0.33, 0.18
Host: scsi0 Channel: 00 Id: 00 Lun: 00 main.html
Vendor: SEAGATE Model: ST410800N Rev: 0025
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 01 Lun: 00
Vendor: SEAGATE Model: ST15150N Rev: 0017
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 02 Lun: 00
Vendor: SEAGATE Model: ST15150N Rev: 1005
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 03 Lun: 00
Vendor: SEAGATE Model: ST15150N Rev: 0020
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 10 Lun: 00
Vendor: MICROP Model: 4345WS Rev: x43h
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 11 Lun: 00
Vendor: MICROP Model: 4345WS Rev: X502
Type: Direct-Access ANSI SCSI revision: 02
Linux yamato 2.2.10 #2 SMP Mon Aug 9 03:59:02 CDT 1999 i686 unknown
LVM version 0.7 by Heinz Mauelshagen (12/07/1999)
lvm -- Driver successfully initialized
(scsi0) <Adaptec AHA-294X SCSI host adapter> found at PCI 17/0 (scsi0) Narrow Channel, SCSI ID=7, 16/255 SCBs
(scsi0) Downloading sequencer code... 406 instructions downloaded (scsi1) <Adaptec AHA-294X Ultra SCSI host adapter> found at PCI 18/0 (scsi1) Wide Channel, SCSI ID=7, 16/255 SCBs
(scsi1) Downloading sequencer code... 413 instructions downloaded scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.17/3.2.4
<Adaptec AHA-294X SCSI host adapter>
scsi1 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.17/3.2.4
<Adaptec AHA-294X Ultra SCSI host adapter>
scsi : 2 hosts.
Here are the scsi errros i've logged since i booted 10 days ago. Everything seems to be ok and functional...
SCSI disk error : host 0 channel 0 id 2 lun 0 return code = 2 scsidisk I/O error: dev 08:21, sector 5976322
(scsi0:0:2:0) Performing Domain validation.
(scsi0:0:2:0) Successfully completed Domain validation.
nmap uses obsolete (PF_INET,SOCK_PACKET)
SCSI disk error : host 0 channel 0 id 2 lun 0 return code = 2 scsidisk I/O error: dev 08:21, sector 8287666
(scsi0:0:2:0) Performing Domain validation.
(scsi0:0:2:0) Successfully completed Domain validation.
SCSI disk error : host 0 channel 0 id 2 lun 0 return code = 2 scsidisk I/O error: dev 08:21, sector 5478418
(scsi0:0:2:0) Performing Domain validation.
(scsi0:0:2:0) reducing SCSI transfer speed due to Domain validation failure. (scsi0:0:2:0) Performing Domain validation.
(scsi0:0:2:0) Successfully completed Domain validation.
Hi,
Quoted 10 lines
Quoted 4 lines
I haven't actually compiled the tools under debian myself. Debian does not have
/usr/include/linuxmatching up with the kernel release, so any patches you might do to the kernel won't show up.
=20Actualy you are supposed to symlink
/usr/include/linuxto /usr/src/linux/i=
nclude/linux. I'm told by the head honchos at Debian that this method has b= een semi-officialy endorsed as the proper method instead of installing the = .h files in/usr/include/linux. If you do it this way you find all the patc= hes include files as well.
here is the rationale of the /usr/include/linux vs. /usr/src/linux/include/linux issue.
(sorry, I don't have any reference to this, so I have to include
all the text here)
Basically the kernel headers match the kernel. The /usr/include headers match the program development tools. This allows to separate kernel and libc development.
Here is the story:
-------------------------------------------------------------------
The headers were included in libc5-dev after a rash of very buggy alpha kernel releases (1.3.7* or something like that) that proceeded to break compilations, etc. Kernel versions are changed far more rapidly than libc is, and there are higer chances that people install a custom kernel than they install custom libc.
libc6 includes it's own version of /usr/include/linux and
friends form the beginning (that is, this is no longer a Debian only feature, the upstream version has moved to this scheme as well).
Quoted 1 line
Quoted 1 line
"Linus" == Linus Torvalds said on Wed, 22 Jan 1997:
Linus> The kernel headers used to make sense exporting to user space, Linus> but the user space thing has grown so much that it's really not Linus> practical any more. The problem with Debian is just that they Linus> are different, not that they are doing anything wrong. That Linus> leads to differences between the distributions, and that in Linus> turn obviously can result in subtle problems.
Linus> As of glibc, the kernel headers will really be _kernel_ Linus> headers, and user level includes are user level Linus> includes. Matthias Ulrich did that partly because I've asked Linus> him to, but mainly just because it is no longer possible to try Linus> to synchronize the libc and the kernel the way it used to Linus> be. The symlinks have been a bad idea for at least a year now, Linus> and the problem is just how to get rid of them Linus> gracefully. Personally, I'm counting on glibc, which we are Linus> already using on alpha.
Linus> Just to give you some idea of exactly why the includes really Linus> can't be handled by simple symlinks: the main problem is Linus> version skew. Lots of people want to upgrade their library Linus> without affecting the kernel, and probably even more people Linus> want to be able to upgrade their kernel without affecting their Linus> compilation environment. Right now doing that has been Linus> extremely fragile.
Linus> Just to give _one_ example of why the symlinks are bad: NR_OPEN Linus> and "fd_set". I have had no end of problems making NR_OPEN Linus> larger in the kernel, exactly _because_ of the damn Linus> sym-links. If I just make NR_OPEN larger (the right thing to Linus> do), the problem is that people with old libraries will now Linus> compile against a header file that doesn't match the library Linus> any more. And when the library internally uses another NR_OPEN Linus> than the new program does, "interesting" things happen.
Linus> In contrast, with separate header files, this doesn't make any Linus> difference. If I change NR_OPEN in the kernel, the compilation Linus> environment won't notice UNTIL the library and associated Linus> header files are changed: thus the user will contine to compile Linus> with the old values, but because we'll still be binary Linus> compatible, the worst thing that happens is that new programs Linus> won't take advantage of new features unless the developer has Linus> upgraded his library. Compare that to breaking subtly.
Linus> NR_OPEN is just _one_ example, and actually it's one of the Linus> easier ones to handle (because the only thing that really makes Linus> much of a difference when it comes to NR_OPEN is the fd_set Linus> size - but it certainly bit some people). Another major problem Linus> is name-space pollution: the POSIX/ANSI/XOpen rules are not Linus> only complex, but they are actually contradictory too. And the Linus> kernel header files really can't reasonably support all of the Linus> intricacies very cleanly.
Linus> One specific example of why we want separate header files for Linus> libraries and kernel is offered by glibc: Matthias wanted to Linus> have a "sigset_t" that will suffice for the future when the Linus> POSIX.1b realtime signals are implemented. But at the same time Linus> he obviously wants to be able to support programmin on Linus> Linux-2.0 and the current 2.1 that do not have that support.
Linus> The _only_ reasonably clean way to handle these kinds of Linus> problems is to have separate header files: user programs see a Linus> larger sigset_t, and then the library interaction with the Linus> kernel doesn't necessarily use all of the bits, for Linus> example. Then later, when the kernel support is actually there, Linus> it's just a matter of getting a new shared library, and voila, Linus> all the realtime signals work.
Linus> The symlink approach simply wouldn't work for the above: that Linus> would have required everybody who uses the library to have a Linus> recent enough kernel that whatever magic all the above entails Linus> would be available in the kernel header files. But not only Linus> don't I want to pollute the kernel header files with user level Linus> decisions, it's actually possible that somebody wants to run Linus> glibc on a 1.2.x kernel, for example. We _definitely_ do not Linus> want him to get a 32-bit sigset_t just because he is happy with Linus> an old kernel.
Linus> Anyway, this email got longer than intended, but I just wanted Linus> to make clear that the symlinks will eventually be going away Linus> even in non-Debian distributions. Debian just happened to do it Linus> first - probably because Debian seems to be more interested in Linus> technical reasons than any old traditions. And technically, the Linus> symlinks really aren't very good.
Linus> The _only_ reason for the symlinks is to immediately give Linus> access to new features in the kernel when those happen. New Linus> ioctl numbers etc etc. That was an overriding concern early on:
Linus> the kernel interfaces expanded so rapidly even in "normal" Linus> areas that having the synchronization that symlinks offered was Linus> a good thing.
Linus> However, the kernel interfaces aren't really supposed to change Linus> all that quickly any more, and more importantly: the technical Linus> users know how to fix things any way they want, so if they want Linus> a new ioctl number to show up they can actually edit the header Linus> files themselves, for example. But having separation is good Linus> for the non-technical user, because there are less surprises Linus> and package dependencies.
Linus> Anyway, something like the patch that David suggested will Linus> certainly go in, although I suspect I'll wait for it to become Linus> "standard" and the glibc first real release to take place.
Add to that the fact that few programs really need the more
volatile elements of the header files (that is, things that really change from kernel version to kernel version), [before you reject this, consider: programs compiled on one kernel version usually work on other kernels].
So, it makes sense that a set of headers be provided from a
known good kernel version, and that is sufficient for compiling most programs, (it also makes the compile time environments for programs on debian machines a well known one, easing the process of dealing with problem reports), the few programs that really depend on cutting edge kernel data structures may just use -I/usr/src/linux/include (provided that kernel-headers or kernel-source exists on the system).
Most programs, even if they include <linux/something.h>, do
not really depend on the version of the kernel, as long as the kernel versions are not too far off, they will work. And the headers provided in libc5-dev (and libc6-dev) are just that.
libc5-dev is uploaded frequently enough that it never lags too
far behind the latest released kernel. libc6 has totally disconnected the included headers from kernel headers.
There are two different capabilities which are the issue, and
the kernel-packages and libc{5,6}-dev address different ones:
a) The kernel packages try tp provide a stable, well behaved kernel and modules, and may be upgraded whenever there are significant advances in those directions (bug fixes, more/better module support, etc). These, however, may not have include files that are non-broken as far as non-kernel programs are concerned, and the quality of the development/compilation environment is not the kernel packages priority (Also, please note that the kernel packages are tied together, so kernel-source, headers, and image are produced in sync)
b) Quality of the development/compilation environment is the priority of libc{5,6}-dev package, and it tries to ensure that the headers it provides would be stable and not break non-kernel programs. This assertion may fail for alpha kernels, which may otherwise be perfectly stable, hence the need for a different set of known-good kernel include files.