Skip to contentSkip to conversation

Linux should better cope with power failure

DISCUSSION Otto Wyss 8 participants 11 messages Mar 19, 2001

lore
Share & tools
lore

Otto Wyss

author
#0 lore raw

Lately I had an USB failure, leaving me without any access to my system since I only use an USB-keyboard/-mouse. All I could do in that situation was switching power off and on after a few minutes of inactivity. From the impression I got during the following startup, I assume Linux (2.4.2, EXT2-filesystem) is not very suited to any power failiure or manually switching it off. Not even if there wasn't any activity going on.

Shouldn't a good system allways try to be on the save side? Shouldn't Linux try to be more fail save? There is currently much work done in getting high performance during high activity but it seems there is no work done at all in getting a save system during low/no activity. I think this is a major drawback and should be addressed as fast as possible. Bringing a system to save state should allway have a high priority.

How could this be accomplished:
1. Flush any dirty cache pages as soon as possible. There may not be any dirty cache after a certain amount of idle time.
2. Keep open files in a state where it doesn't matter if they where improperly closed (if possible).
3. Swap may not contain anything which can't be discarded. Otherwise swap has to be treated as ordinary disk space.

These actions are not filesystem dependant. It might be that certain filesystem cope better with power failiure than others but still it's much better not to have errors instead to fix them.

Don't we tell children never go close to any abyss or doesn't have alpinist a saying "never go to the limits"? So why is this simple rule always broken with computers?

O. Wyss

#1 lore raw
Quoted 7 lines · Otto Wyss

Lately I had an USB failure, leaving me without any access to my system since I only use an USB-keyboard/-mouse. All I could do in that situation was switching power off and on after a few minutes of inactivity. From the impression I got during the following startup, I assume Linux (2.4.2, EXT2-filesystem) is not very suited to any power failiure or manually switching it off. Not even if there wasn't any activity going on.

You're not using the filesystem the way you should, if you expect to be able to kill the power and not lose data.

Quoted 3 lines

How could this be accomplished:
1. Flush any dirty cache pages as soon as possible. There may not be any dirty cache after a certain amount of idle time.

Mount the filesystem sychronously if you want this.

Quoted 2 lines

2. Keep open files in a state where it doesn't matter if they where improperly closed (if possible).

Mount the filesystem read-only if you want this.

Quoted 2 lines

3. Swap may not contain anything which can't be discarded. Otherwise swap has to be treated as ordinary disk space.

The kernel doesn't care about what's in swap. Fix your applications if they do.

Charles

Signature · 5 lines
-----------------------------------------------------------------------
Charles Cazabon                            <[email protected]>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
Any opinions expressed are just that -- my opinions.
-----------------------------------------------------------------------
#2 lore raw
In reply to Otto Wyss
Quoted 23 lines · Otto Wyss

Lately I had an USB failure, leaving me without any access to my system since I only use an USB-keyboard/-mouse. All I could do in that situation was switching power off and on after a few minutes of inactivity. From the impression I got during the following startup, I assume Linux (2.4.2, EXT2-filesystem) is not very suited to any power failiure or manually switching it off. Not even if there wasn't any activity going on.

Shouldn't a good system allways try to be on the save side? Shouldn't Linux try to be more fail save? There is currently much work done in getting high performance during high activity but it seems there is no work done at all in getting a save system during low/no activity. I think this is a major drawback and should be addressed as fast as possible. Bringing a system to save state should allway have a high priority.

How could this be accomplished:
1. Flush any dirty cache pages as soon as possible. There may not be any dirty cache after a certain amount of idle time.
2. Keep open files in a state where it doesn't matter if they where improperly closed (if possible).
3. Swap may not contain anything which can't be discarded. Otherwise swap has to be treated as ordinary disk space.

Quoted 8 lines

These actions are not filesystem dependant. It might be that certain filesystem cope better with power failiure than others but still it's much better not to have errors instead to fix them.

Don't we tell children never go close to any abyss or doesn't have alpinist a saying "never go to the limits"? So why is this simple rule always broken with computers?

Unix and other such variants have what's called a Virtual File System (VFS). The idea behind this is to keep as much recently-used file stuff in memory so that the system can be as fast as if you used a RAM disk instead of real physical (slow) hard disks. If you can't cope with this, use DOS. Even Windows tries to emulate Unix as far as VFS is concerned. However Windows never reports any errors. By design, Windows keeps trashing along until you must reinstall it because there is nothing left of the file-system.

If you want file-system trashing errors hidden, use Windows. Unix and its variants provide enough information in their file-systems to recover the file-system, although not necessaily a particular file, upon startup, if you have just switched the system off. It uses fsck for this.

If as you say; "bringing the system to a save state should always have a high priority...", then mount the disks sync.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of course you get it all back when you reboot..."; Actual explanation obtained from the Micro$oft help desk.

#3 lore raw
Quoted 35 lines · Richard B. Johnson
Quoted 23 lines · Otto Wyss

Lately I had an USB failure, leaving me without any access to my system since I only use an USB-keyboard/-mouse. All I could do in that situation was switching power off and on after a few minutes of inactivity. From the impression I got during the following startup, I assume Linux (2.4.2, EXT2-filesystem) is not very suited to any power failiure or manually switching it off. Not even if there wasn't any activity going on.

Shouldn't a good system allways try to be on the save side? Shouldn't Linux try to be more fail save? There is currently much work done in getting high performance during high activity but it seems there is no work done at all in getting a save system during low/no activity. I think this is a major drawback and should be addressed as fast as possible. Bringing a system to save state should allway have a high priority.

How could this be accomplished:
1. Flush any dirty cache pages as soon as possible. There may not be any dirty cache after a certain amount of idle time.
2. Keep open files in a state where it doesn't matter if they where improperly closed (if possible).
3. Swap may not contain anything which can't be discarded. Otherwise swap has to be treated as ordinary disk space.

Quoted 8 lines

These actions are not filesystem dependant. It might be that certain filesystem cope better with power failiure than others but still it's much better not to have errors instead to fix them.

Don't we tell children never go close to any abyss or doesn't have alpinist a saying "never go to the limits"? So why is this simple rule always broken with computers?

Quoted 5 lines

Unix and other such variants have what's called a Virtual File System (VFS). The idea behind this is to keep as much recently-used file stuff in memory so that the system can be as fast as if you used a RAM disk instead of real physical (slow) hard disks. If you can't cope with this, use DOS.

At the very least the disk should be consistent with memory. If the dirty pages aren't written back to the disk (but not necessarily removed from memory) after a reasonable idle period, then there is room for improvement.

--

Brian Gerst

#4 lore raw

On Mon, 19 Mar 2001, Brian Gerst wrote:
[SNIPPED...]

Quoted 6 lines

At the very least the disk should be consistent with memory. If the dirty pages aren't written back to the disk (but not necessarily removed from memory) after a reasonable idle period, then there is room for improvement.

Hmmm. Now think about it a minute. You have a database operation with a few hundred files open, most of which will be deleted after a sort/merge completes. At the same time, you've got a few thousand directories with their ATIME being updated. Also, you have thousands of temporary files being created in /tmp during a compile that didn't use "-pipe".

If you periodically write everything to disk, you don't have many CPU cycles available to do anything useful.

It is up to the application to decide if anything is 'precious'. If you've got some database running, it's got to be checkpointed with some "commitable" file-system so it can be interrupted at any time.

If you make your file-systems up of "slices", you can mount the executable stuff read/only. Currently, only /var and /tmp need to be writable for normal use, plus any user "slices", of course.
-- Yes I know you need to modify /etc/stuff occasionally (startup
and shutdown, plus an occasional password change). I proposed a long time ago that /etc/mtab get moved to /var.

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of course you get it all back when you reboot..."; Actual explanation obtained from the Micro$oft help desk.

#5 lore raw
Quoted 18 lines · Richard B. Johnson

On Mon, 19 Mar 2001, Brian Gerst wrote:
[SNIPPED...]

Quoted 6 lines

At the very least the disk should be consistent with memory. If the dirty pages aren't written back to the disk (but not necessarily removed from memory) after a reasonable idle period, then there is room for improvement.

Hmmm. Now think about it a minute. You have a database operation with a few hundred files open, most of which will be deleted after a sort/merge completes. At the same time, you've got a few thousand directories with their ATIME being updated. Also, you have thousands of temporary files being created in /tmp during a compile that didn't use "-pipe".

Quoted 2 lines

If you periodically write everything to disk, you don't have many CPU cycles available to do anything useful.

Note the key words "reasonable idle period". It was stated elsewhere that this is the case already so it is a moot point.

--

Brian Gerst

#6 lore raw
In reply to Richard B. Johnson
Quoted 24 lines · Richard B. Johnson

On Mon, 19 Mar 2001, Brian Gerst wrote:
[SNIPPED...]

Quoted 6 lines

At the very least the disk should be consistent with memory. If the dirty pages aren't written back to the disk (but not necessarily removed from memory) after a reasonable idle period, then there is room for improvement.

Hmmm. Now think about it a minute. You have a database operation with a few hundred files open, most of which will be deleted after a sort/merge completes. At the same time, you've got a few thousand directories with their ATIME being updated. Also, you have thousands of temporary files being created in /tmp during a compile that didn't use "-pipe".

If you periodically write everything to disk, you don't have many CPU cycles available to do anything useful.

It is up to the application to decide if anything is 'precious'. If you've got some database running, it's got to be checkpointed with some "commitable" file-system so it can be interrupted at any time.

Quoted 6 lines

If you make your file-systems up of "slices", you can mount the executable stuff read/only. Currently, only /var and /tmp need to be writable for normal use, plus any user "slices", of course.
-- Yes I know you need to modify /etc/stuff occasionally (startup
and shutdown, plus an occasional password change). I proposed a long time ago that /etc/mtab get moved to /var.

so how does mount update /var/mtab when mounting var? he he.

Actually, I think /etc/mtab is not needed at all. Originally, UNIX
used to put as much onto the disk (and not in "core") as possible. so much state information related only to one boot-cycle was taken out of kernel and stored on disk. /var/run/utmp, /etc/mtab, , rmtab, and many others. all are invalidated by a reboot, and are yet stored
in non-volatile storage. kernel memory is not swappable, so they manually separated out the minimum needed in core.

Linux currently has a lot of this info in core, and maintains the disk files for backwards compatibility. in the case of /etc/mtab, I believe /proc/mounts
has the same info. It appears to be in the same format as /etc/mtab, so most of the groundwork has already been done.
i've considered trying just changing /etc/mtab to /proc/mounts in some utilities, to remove the need for read-write root. This (and other cases) would guarantee consistency (look at /etc/mtab after restart in single user more - ugh)

I wonder if embedded folks would like to at least keep the old behaviour as a compile-time option; they're in almost the same boat as early (64k core memory) unix folks.

My favorite compromise between journaling and performance is the compaq smart array controllers, with a battery-backed sram
write cache; they can do (fast)lazy writes and still be perfectly reliable. plus they keep *everything* reliable, not just metadata.

I find this a fascinating topic... the ultimate would be to use the nvram (it's mostly empty if using LinuxBIOS) to store a clean shutdown flag, and/or a system heartbeat timestamp (like syslogd's)... only this timestamp would let the hdd spin down (not hit every 20 minutes or so with a timestamp log entry) and not wear out a flash disk based system.

Regards,

Jeremy

#7 lore raw
In reply to Richard B. Johnson
Quoted 9 lines

Actually, I think /etc/mtab is not needed at all. Originally, UNIX
used to put as much onto the disk (and not in "core") as possible. so much state information related only to one boot-cycle was taken out of kernel and stored on disk. /var/run/utmp, /etc/mtab, , rmtab, and many others. all are invalidated by a reboot, and are yet stored
in non-volatile storage. kernel memory is not swappable, so they manually separated out the minimum needed in core.

Quoted 9 lines

Linux currently has a lot of this info in core, and maintains the disk files for backwards compatibility. in the case of /etc/mtab, I believe /proc/mounts
has the same info. It appears to be in the same format as /etc/mtab, so most of the groundwork has already been done.
i've considered trying just changing /etc/mtab to /proc/mounts in some utilities, to remove the need for read-write root. This (and other cases) would guarantee consistency (look at /etc/mtab after restart in single user more - ugh)

It has been suggested to ln -sf /proc/mounts /etc/mtab. Linus has said this, I believe.

-b

#8 lore raw
In reply to Richard B. Johnson
Quoted 42 lines · Brian Gerst
Quoted 35 lines · Richard B. Johnson
Quoted 23 lines · Otto Wyss

Lately I had an USB failure, leaving me without any access to my system since I only use an USB-keyboard/-mouse. All I could do in that situation was switching power off and on after a few minutes of inactivity. From the impression I got during the following startup, I assume Linux (2.4.2, EXT2-filesystem) is not very suited to any power failiure or manually switching it off. Not even if there wasn't any activity going on.

Shouldn't a good system allways try to be on the save side? Shouldn't Linux try to be more fail save? There is currently much work done in getting high performance during high activity but it seems there is no work done at all in getting a save system during low/no activity. I think this is a major drawback and should be addressed as fast as possible. Bringing a system to save state should allway have a high priority.

How could this be accomplished:
1. Flush any dirty cache pages as soon as possible. There may not be any dirty cache after a certain amount of idle time.
2. Keep open files in a state where it doesn't matter if they where improperly closed (if possible).
3. Swap may not contain anything which can't be discarded. Otherwise swap has to be treated as ordinary disk space.

Quoted 8 lines

These actions are not filesystem dependant. It might be that certain filesystem cope better with power failiure than others but still it's much better not to have errors instead to fix them.

Don't we tell children never go close to any abyss or doesn't have alpinist a saying "never go to the limits"? So why is this simple rule always broken with computers?

Quoted 5 lines

Unix and other such variants have what's called a Virtual File System (VFS). The idea behind this is to keep as much recently-used file stuff in memory so that the system can be as fast as if you used a RAM disk instead of real physical (slow) hard disks. If you can't cope with this, use DOS.

Quoted 4 lines

At the very least the disk should be consistent with memory. If the dirty pages aren't written back to the disk (but not necessarily removed from memory) after a reasonable idle period, then there is room for improvement.

They are. If you leave your machine one for a minute or so (probably less is ok,
but I don't know) the kernel will flush dirty buffers... fsck will complain, but the file's
*data* blocks will be on the disk. There are way more reasons that this is a silly and annoying thread. You should read more about things like asynchronous/synchronous filesystems,
lazy-write cacheing, etc, etc,. If you know how to write software and/or configure your system,
you can avoid all of these problems. Or use a journaling filesystem ext3/xfs, etc. But I tire of this...

#9 lore raw
In reply to Otto Wyss
Quoted 4 lines · Otto Wyss

inactivity. From the impression I got during the following startup, I assume Linux (2.4.2, EXT2-filesystem) is not very suited to any power failiure or manually switching it off. Not even if there wasn't any activity going on.

What data, if any, did you lose?

While fsck complains loudly when the system comes back up, 9 times in 10 no data is actually lost during a power loss. e2fsck is really good at recovering damaged filesystems.

Quoted 3 lines

How could this be accomplished:
1. Flush any dirty cache pages as soon as possible. There may not be any dirty cache after a certain amount of idle time.

Mount the filesystem synchronously to accomplish this. It will prevent the kernel from using a write cache basically. It will ensure that if a write operation completes, then the data will be physically on the disk afterward.

Quoted 2 lines

2. Keep open files in a state where it doesn't matter if they where improperly closed (if possible).

The way to do this is to use a highly reliable filesystem, such as ext3fs, Tux or ReiserFS. These filesystems guarantee that metadata is consistent at all times.

Quoted 2 lines

3. Swap may not contain anything which can't be discarded. Otherwise swap has to be treated as ordinary disk space.

I can't think of a case where the contents of swap matter in any way for recovering from a power failure.

#10 lore raw
In reply to Otto Wyss
Quoted 2 lines · Richard B. Johnson

Unix and other such variants have what's called a Virtual File System (VFS).

Correct, but hardly relevant here, except possibly that this enables you to use a different, perhaps more resilient file system.

Quoted 3 lines

The idea behind this is to keep as much recently-used file stuff in memory so that the system can be as fast as if you used a RAM disk instead of real physical (slow) hard disks.

Correct, but does not require VFS.

Nice try, though.

- Werner

Signature · 3 lines
  _________________________________________________________________________
 / Werner Almesberger, ICA, EPFL, CH           [email protected] /
/_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/