Skip to contentSkip to conversation

Elastic Quota File System (EQFS)

DISCUSSION Amit Gud 9 participants 17 messages Jun 25, 2004 – Jun 28, 2004

lore
Share & tools
lore

Amit Gud

author
#0 lore raw

Hi all,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

On one school server, theres 10MB quota. (Okay, its admins are BOFHs^H^H^H^H^HSISAL). Everyone tries to run mozilla there (because its installed as default!), and immediately fills his/her quota with cache files, leading to failed login next time (gnome just will not start if it can't write to ~).

Imagine mozilla automatically marking cache files "elastic".

That would solve the problem -- mozilla caches would go away when disk space was demanded, still mozilla's on-disk caching would be effective when there is enough disk space.

Also this is applicable to mailboxes - automize the marking of old mails of the mailing list as elastic, whenever the threshold is reached, you might either want to put those mails as compressed archive or simply delete it. This has two advantages:
- No email bounces for the reason of 'mailbox full' and
- Optimized utlization of the mailbox

Yes, this can be done using scripts too, but what if you are to use other users' space that they are not using? Of course script cannot do that! You need to have some FS support or a libquota hack.

Quoted 12 lines
Quoted 4 lines

How does Mozilla (or any process) react when its files are deleted from under it? Would the file remain until all the open processes close the file or would it just "disappear"?

Of course, if mozilla marked them "elastic" it should better be prepared for they disappearance. I'd disappear them with simple unlink(), so they'd physically survive as long as someone held them open.

Quoted 2 lines

Would it delete entire directories or
just some of the files? How does it choose? (First up against the

delete

Quoted 6 lines
Quoted 1 line

when the drive space fills...)

Probably just some of the files... Or you could delete directory, too, if it was marked "elastic". What to delete first... probably file with oldest access time? Or random file, with chance of being selected proportional to file size?

One can either even mark the whole directory as elastic. When it comes to taking action, I think I'II opt for random deletion of the files for not being unfair to any particular user.

AG

#1 lore raw
Quoted 25 lines · Amit Gud

Hi all,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

On one school server, theres 10MB quota. (Okay, its admins are BOFHs^H^H^H^H^HSISAL). Everyone tries to run mozilla there (because its installed as default!), and immediately fills his/her quota with cache files, leading to failed login next time (gnome just will not start if it can't write to ~).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Imagine mozilla automatically marking cache files "elastic".

That would solve the problem -- mozilla caches would go away when disk space was demanded, still mozilla's on-disk caching would be effective when there is enough disk space.

Also this is applicable to mailboxes - automize the marking of old mails of the mailing list as elastic, whenever the threshold is reached, you might either want to put those mails as compressed archive or simply delete it. This has two advantages:
- No email bounces for the reason of 'mailbox full' and
- Optimized utlization of the mailbox

Quoted 5 lines

Yes, this can be done using scripts too, but what if you are to use other users' space that they are not using? Of course script cannot do that! You need to have some FS support or a libquota hack.

What you're suggesting is not something that could be controlled by the file system because there are far too many methods for storing email. Consider, for instance, if all email messages are stored in a single database file. Unless you include parsing code for all email storage methods, there's no possible way for your design to work. As you should be able to see, this "feature" should be implemented in the email server code; *not* the file system.

Personally, I would rather not use this feature, at all; but, I'm also entitled to leaving the elastic bit turned off on all my messages.

Sean

#2 lore raw
In reply to Amit Gud

"Amit Gud" <[email protected]> said:

[...]

Quoted 3 lines

Also this is applicable to mailboxes - automize the marking of old mails of the mailing list as elastic, whenever the threshold is reached, you might either want to put those mails as compressed archive or simply delete it.

Right. And which to do is a policy decision, left to the individual owner of this particular file or collection of files. Democracy at work ;-)

Quoted 2 lines

This has two advantages:
- No email bounces for the reason of 'mailbox full' and

It will fill up eventually...

Quoted 3 lines

- Optimized utlization of the mailbox

Yes, this can be done using scripts too,

Having the system fool around with my mail because you get too much junk and can't be bothered to delete it is what I'd consider an hostile act.

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#3 lore raw

Hi!

Quoted 2 lines

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

But... there's no reasonable way to do this in userspace.

Two pieces of kernel support are needed:

1) some way to indicate "this file is elastic" (okay perhaps xattrs can do this already)

and either

2a) file selection/deletion in kernel

or

2b) assume that disk does not fill up faster than 1GB/sec, allways keep 1GB free, make "deleting" daemon poll each second [ugly, unreliable]

or

2c) just before returning -ENOSPC, synchronously tell userspace to free space, and retry the operation.

BTW 2c) would be also usefull for undelete. Unfortunately 2c looks very complex, too; it might be easier to do 2a than 2c.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#4 lore raw
Quoted 22 lines · Pavel Machek

Hi!

Quoted 2 lines

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

But... there's no reasonable way to do this in userspace.

Two pieces of kernel support are needed:

1) some way to indicate "this file is elastic" (okay perhaps xattrs can do this already)

and either

2a) file selection/deletion in kernel

or

2b) assume that disk does not fill up faster than 1GB/sec, allways keep 1GB free, make "deleting" daemon poll each second [ugly, unreliable]

Quoted 7 lines

or

2c) just before returning -ENOSPC, synchronously tell userspace to free space, and retry the operation.

BTW 2c) would be also usefull for undelete. Unfortunately 2c looks very complex, too; it might be easier to do 2a than 2c.

Why does the kernel have to get involved with file deletion?

All it needs is to run at sufficient privs.

If you are overflowing drives that easily, it is time to buy a bigger drive. It is not the time to start deleting stuff at random. Data is usually put on a drive for a reason. Having a human decide what to delete is *much* better than letting some automated process do it in background.

This sounds like a hack to get around a badly designed system with too few resources.

Windows has an option to delete files "that are not needed". It tends to delete things that you wanted, but had not thought about in a while.

This really strikes me as a bad idea. It has lots of "special" things that programs will have to deal with for this particular case. This makes things much more complex in userspace for a problem that needs to be dealt with in meatspace.

Signature · 3 lines
"Ye have locked yerselves up in cages of fear--and, behold, do ye now
complain that ye lack FREEDOM!"
  - Lord Omar in THE EPISTLE TO THE PARANOIDS Chaper 1 Verse 1
author
#5 lore raw
Quoted 33 lines · Alan
Quoted 30 lines · Pavel Machek

Hi!

Quoted 4 lines

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

But... there's no reasonable way to do this in userspace.

Two pieces of kernel support are needed:

1) some way to indicate "this file is elastic" (okay perhaps xattrs can do this already)

and either

2a) file selection/deletion in kernel

or

2b) assume that disk does not fill up faster than 1GB/sec, allways keep 1GB free, make "deleting" daemon poll each second [ugly, unreliable]

or

2c) just before returning -ENOSPC, synchronously tell userspace to free space, and retry the operation.

Quoted 9 lines
Quoted 4 lines

BTW 2c) would be also usefull for undelete. Unfortunately 2c looks very complex, too; it might be easier to do 2a than 2c.

Why does the kernel have to get involved with file deletion?

Involvement of the kernel depends on how we are treating the elastic files:

- when files are made elastic, are we increasing the user's quota?
if so kernel becomes neccessary for if
- elastic file is deleted: decrement the user's quota
- elastic file is chowned: do the needfull (decrement previous
owners and increment new owners quota)

- when files are made elastic, we are just not accounting the file
size for the user's quota. In this case I dont see any involvement of the kernel, but will need some other hacks.

Quoted 10 lines

All it needs is to run at sufficient privs.

If you are overflowing drives that easily, it is time to buy a bigger drive. It is not the time to start deleting stuff at random. Data is usually put on a drive for a reason. Having a human decide what to delete is *much* better than letting some automated process do it in background.

Admin assigns quotas to the user with using some case studies, like he won't be allocating 500 Mb for a C language course students. We are trying to harness some user behavior here. I wish I had some Gartner or Forrestor figures to show you that how often _all_ the users use up their 100% quota at any given point of time. I'm assuming that this *does* happen very rarely and I also assume that my assumption is very close to correct. The point is their is some 'free' space available for the users who use up all their quotas. Of course this free space is likely to be someday fall short for the over-spilling users, but then its time to increase quotas of all the users, and its then admin's duty to look at this. Perhaps this could become a good indication for the admin for increasing the quota of all the users.

Quoted 12 lines

This sounds like a hack to get around a badly designed system with too few resources.

Windows has an option to delete files "that are not needed". It tends to delete things that you wanted, but had not thought about in a while.

This really strikes me as a bad idea. It has lots of "special" things
that programs will have to deal with for this particular case. This makes things much more complex in userspace for a problem that needs to be dealt with in meatspace.

I do believe that this system can be done in userspace, but it has its own flaws then. Suppose theres a daemon, call it eqfsd. It forks parent listens a char device, child watches the disk space usage. A kernel module reports the file deletes, chowns to the char device, parent does the needfull. Child 'periodically' checks that the threshold is not reached. Here what can be done is suppose a user can transfer data with say 5 Mbps speed to his account....then we can easily get the minimum time required to fill up the remaining free space. Child sleeps for this much time minus some value for safety. i.e. child sleeps for: ( (D - Ui) / 5 ) - safety_value seconds.

I'm sure this gotta be slow and *will* slow down the system, so I still insist that this should be in kspace.

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

AG

#6 lore raw
Quoted 1 line · Amit Gud

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

I personally see no use whatsoever. Why not just allocate 100% of the file system to everybody and ignore quota's, entirely? Each user will use whatever he/she requires and when space starts to run out, users will manually clean up what they don't need.

I am totally against the automatic deletion of files and believe that all users will _eventually_ walk in on a Monday morning to find out that the OS took it upon itself to delete a file that was flagged as elastic, that shouldn't have been. I also tend to believe that the exact time/date that the file was removed could conceivably occur six months prior to that Monday morning, without the users knowledge. Now the burden will again be placed on to system administrators. This time, to locate and recover the lost file(s) by sorting through months of archives. Personally, I prefer setting quota's on an individual bases, to finding a needle in a haystack

In my mind, you either have a quota or you don't; there's no in between.

Sean

#7 lore raw

Not to argue whether its a good idea but, one use for this would be undeleting. rm becomes a set EQFS bit. You keep all your old files that way until space is really needed and then they get reclaimed. You might not want that to count against your quota.

Quoted 32 lines · Sean
Quoted 3 lines · Amit Gud

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

I personally see no use whatsoever. Why not just allocate 100% of the file system to everybody and ignore quota's, entirely? Each user will use whatever he/she requires and when space starts to run out, users will manually clean up what they don't need.

I am totally against the automatic deletion of files and believe that all users will _eventually_ walk in on a Monday morning to find out that the OS took it upon itself to delete a file that was flagged as elastic, that shouldn't have been. I also tend to believe that the exact time/date that the file was removed could conceivably occur six months prior to that Monday morning, without the users knowledge. Now the burden will again be placed on to system administrators. This time, to locate and recover the lost file(s) by sorting through months of archives. Personally, I prefer setting quota's on an individual bases, to finding a needle in a haystack

In my mind, you either have a quota or you don't; there's no in between.

Sean

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

author
#8 lore raw
In reply to Fao, Sean
Quoted 8 lines · Sean
Quoted 1 line · Amit Gud

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

I personally see no use whatsoever. Why not just allocate 100% of the file system to everybody and ignore quota's, entirely? Each user will use whatever he/she requires and when space starts to run out, users will manually clean up what they don't need.

We should get our basics right first. We _do_ need quotas!! Without any quota system how are we going to avoid a malicious user from taking away all the space to keep other people starving? In EQFS also this can happen, but we are giving *controlled flexibility* to the user. He is having some stretching power but not beyond a certain limit. And do you think users are sincere enough to clean up there files when they are done?

Quoted 1 line

I am totally against the automatic deletion of files and believe that all users will _eventually_ walk in on a Monday morning to find out that the OS took it upon itself to delete a file that was flagged as elastic, that shouldn't have been.

User is the king, he decides what files should be elastic and what not. This can always be controlled.

AG

#9 lore raw
Quoted 4 lines · Amit Gud
Quoted 2 lines · Sean

Amit Gud wrote:

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

I personally see no use whatsoever. Why not just allocate 100% of the file system to everybody and ignore quota's, entirely? Each user will use whatever he/she requires and when space starts to run out, users will manually clean up what they don't need.

We should get our basics right first. We _do_ need quotas!! Without any quota system how are we going to avoid a malicious user from taking away all the space to keep other people starving? In EQFS also this can happen, but we are giving *controlled flexibility* to the user. He is having some stretching power but not beyond a certain limit. And do you think users are sincere enough to clean up there files when they are done?

And I suppose you think that users will be sincere enough to mark files as elastic? I, for one, already said that I absolutely would *not* mark a single file as elastic. If I'm using 110 MB and you need an additional 10 MB for storage, you won't be getting it from me because I don't want to come in some morning to find that a file has disappeared.

The system that you're asking for is a system without quotas. Think about what you're saying.

Quoted 6 lines
Quoted 2 lines

I am totally against the automatic deletion of files and believe that all users will _eventually_ walk in on a Monday morning to find out that the OS took it upon itself to delete a file that was flagged as elastic, that shouldn't have been.

User is the king, he decides what files should be elastic and what not. This can always be controlled.

Controlled how? Who is anybody to inform me of what files I need/don't need?

Sean

#10 lore raw
Quoted 8 lines · Sean
Quoted 6 lines · Amit Gud
Quoted 4 lines · Sean
Quoted 1 line · Amit Gud

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

I personally see no use whatsoever. Why not just allocate 100% of the file system to everybody and ignore quota's, entirely? Each user will use whatever he/she requires and when space starts to run out, users will manually clean up what they don't need.

We should get our basics right first. We _do_ need quotas!! Without any quota system how are we going to avoid a malicious user from taking away all the space to keep other people starving? In EQFS also this can happen, but we are giving *controlled flexibility* to the user. He is having some stretching power but not beyond a certain limit. And do you think users are sincere enough to clean up there files when they are done?

And I suppose you think that users will be sincere enough to mark files as elastic? I, for one, already said that I absolutely would *not* mark a single file as elastic. If I'm using 110 MB and you need an additional 10 MB for storage, you won't be getting it from me because I don't want to come in some morning to find that a file has disappeared.

The system that you're asking for is a system without quotas. Think about what you're saying.

I think you missed one of the main points - you don't get any extra space until you mark some of your files as elastic.
You're right - under this system, nobody would get any space from deletion of your files because you would use the system as a normal hard quota system - you would mark no files as elastic, and would therefore be limited to your quota (in the example you gave, you would not be using 110M, because your quota would have limited you to 100M). If you were so kind as to mark something as elastic (say, that recently doneloaded install tarball of the Gimp), then you would remove the storage taken by those files from your quota usage and would have more space available, with the risk that the elastic files might not stick around.

Under no circumstance would you lose any file that fits under your quota.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

I am totally against the automatic deletion of files and believe that all users will _eventually_ walk in on a Monday morning to find out that the OS took it upon itself to delete a file that was flagged as elastic, that shouldn't have been.

User is the king, he decides what files should be elastic and what not. This can always be controlled.

Controlled how? Who is anybody to inform me of what files I need/don't need?

Controlled by you using one of the methods that have been suggested:
a .elastic file/directory structure
/scratch/ space usage
a filesystem that can keep track of these things, and a program like chmod xattrs and other userspace tools

etc.

- Steve

#11 lore raw
Quoted 17 lines · Stephen Wille Padnos
Quoted 15 lines · Sean
Quoted 11 lines · Amit Gud
Quoted 6 lines · Sean
Quoted 1 line · Amit Gud

It cannot be denied that there _are_ applications for such a system that we already discussed and theres a class of users who will find the system useful.

I personally see no use whatsoever. Why not just allocate 100% of the file system to everybody and ignore quota's, entirely? Each user will use whatever he/she requires and when space starts to run out, users will manually clean up what they don't need.

We should get our basics right first. We _do_ need quotas!! Without any quota system how are we going to avoid a malicious user from taking away all the space to keep other people starving? In EQFS also this can happen, but we are giving *controlled flexibility* to the user. He is having some stretching power but not beyond a certain limit. And do you think users are sincere enough to clean up there files when they are done?

Quoted 7 lines
Quoted 3 lines

And I suppose you think that users will be sincere enough to mark files as elastic? I, for one, already said that I absolutely would *not* mark a single file as elastic. If I'm using 110 MB and you need an additional 10 MB for storage, you won't be getting it from me because I don't want to come in some morning to find that a file has disappeared.

The system that you're asking for is a system without quotas. Think about what you're saying.

I think you missed one of the main points - you don't get any extra space until you mark some of your files as elastic.
You're right - under this system, nobody would get any space from deletion of your files because you would use the system as a normal hard quota system - you would mark no files as elastic, and would therefore be limited to your quota (in the example you gave, you would not be using 110M, because your quota would have limited you to 100M). If you were so kind as to mark something as elastic (say, that recently doneloaded install tarball of the Gimp), then you would remove the storage taken by those files from your quota usage and would have more space available, with the risk that the elastic files might not stick around.

You're right; I did missunderstand one of the main points. I was under the impression that I could take whatever I needed, as long as it were available. This does change my view on the subject, somewhat.

Thanks for clearing that up,

Sean

#12 lore raw
In reply to Stephen Wille Padnos
Quoted 13 lines · Stephen Wille Padnos

I think you missed one of the main points - you don't get any extra space until you mark some of your files as elastic.
You're right - under this system, nobody would get any space from deletion of your files because you would use the system as a normal hard quota system - you would mark no files as elastic, and would therefore be limited to your quota (in the example you gave, you would not be using 110M, because your quota would have limited you to 100M). If you were so kind as to mark something as elastic (say, that recently doneloaded install tarball of the Gimp), then you would remove the storage taken by those files from your quota usage and would have more space available, with the risk that the elastic files might not stick around.

Quoted 1 line

Under no circumstance would you lose any file that fits under your quota.

-snip-

Quoted 9 lines

Controlled by you using one of the methods that have been suggested:
a .elastic file/directory structure
/scratch/ space usage
a filesystem that can keep track of these things, and a program like chmod xattrs and other userspace tools

etc.

- Steve

It looks (to my untrained eyes) like a user-driven caching "algorithm", where I can keep these KDE tarballs around next to the kernel sources, and a few shiny new slackware ISOs, and all are of course replaceable, but I mark them elastic or put them in /scratch/... to recover my space at the cost of an increased probability that I'll have to download some of them again. I like it.

Signature · 3 lines
Rob Couto [[email protected]]
computer safety tip: use only a non-conducting, static-free hammer.
--
#13 lore raw
In reply to Amit Gud

Hi!

Quoted 9 lines
Quoted 1 line

This sounds like a hack to get around a badly designed system with too

I do believe that this system can be done in userspace, but it has its own flaws then. Suppose theres a daemon, call it eqfsd. It forks parent listens a char device, child watches the disk space usage. A kernel module reports the file deletes, chowns to the char device, parent does the needfull. Child 'periodically' checks that the threshold is not reached. Here what can be done is suppose a user can transfer data with say 5 Mbps speed to his account....then we can easily get the minimum

I believe I can write data at 1GB/sec at common hardware (~RAM speed). You'll either need a *lot* of free space, or very fast polling, and it will still break when the daemon is swapped out, when scheduler decides writer is more important etc.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#14 lore raw
In reply to Pavel Machek
Quoted 6 lines · Pavel Machek

Hi!

Quoted 2 lines

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

But... there's no reasonable way to do this in userspace.

Let's see...

Quoted 4 lines

Two pieces of kernel support are needed:

1) some way to indicate "this file is elastic" (okay perhaps xattrs can do this already)

Or just a list of elastic files in ~/.elastic. Even better, could mark them as "Just delete", "compress"; could go as far as giving (fallback?) globs to select files for each treatment ("If space gets tight, delete *~ files, and compress *.tex that haven't been read in a week"). Sounds like a fun Perl project...

Quoted 3 lines

and either

2a) file selection/deletion in kernel

A daemon or cron job running as root can do that just fine. Or you can set it up for your own files.

Quoted 5 lines

or

2b) assume that disk does not fill up faster than 1GB/sec, allways keep 1GB free, make "deleting" daemon poll each second [ugly, unreliable]

Buy a larger disk. Make sure sum of all hard quotas is less than filesystem size. Need that anyway; so it reduces to a one-user problem with per-user solutions.

Quoted 4 lines

or

2c) just before returning -ENOSPC, synchronously tell userspace to free space, and retry the operation.

Ugly.

Quoted 2 lines

BTW 2c) would be also usefull for undelete. Unfortunately 2c looks very complex, too; it might be easier to do 2a than 2c.

As said, all this buys very little for a lot of hairy code in the kernel, which will be rarely used (and whose bugs will show up when it is badly needed to work right). Besides, I strongly oppose automatic file disposal. I love the Unix way exactly because I decide what to do (no "I know better than you what to do", "you dumb user aren't supposed to do that, so you have no right to know", no "undelete deleted files, but only sometimes; it just might still be around, but if space got tight it isn't" (this is even worse...))

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
author
#15 lore raw
Quoted 26 lines · Horst von Brand
Quoted 11 lines · Pavel Machek

Hi!

Quoted 4 lines

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

But... there's no reasonable way to do this in userspace.

Let's see...

Quoted 6 lines

Two pieces of kernel support are needed:

1) some way to indicate "this file is elastic" (okay perhaps xattrs can do this already)

Quoted 8 lines

Or just a list of elastic files in ~/.elastic. Even better, could mark them as "Just delete", "compress"; could go as far as giving (fallback?) globs to select files for each treatment ("If space gets tight, delete *~ files, and compress *.tex that haven't been read in a week"). Sounds like a fun Perl project...

.elastic is a file or directory? If its file, daemon has to do the ugly insertion deletion of the file enteries. If its a directory, daemon has to do the updating of the files in case of mv, plus follow the links for unlink, chown if we are storing the files as symlinks and we will be wasting the inodes.

Just think of the load on the system if we run a daemon, which sleeps for a time depending on the data transfer rate and the amount of free space left, and if the free space left is very close to the margin... the system might even freeze. The primary intention of taking filesystems into kernel is the speed and the security that we get, otherwise we do have userspace filesystems!

What we need is an application-transparent system which of course should be plugable within a system and which also shouldn't hamper the system throughput in a major way.

I really don't see user space implementation in the picture.

AG

#16 lore raw
In reply to Horst von Brand

Hi!

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Case closed, anyway. It belongs in the kernel only if there is no reasonable way to do it in userspace.

But... there's no reasonable way to do this in userspace.

Let's see...

Quoted 2 lines

Two pieces of kernel support are needed:

Quoted 8 lines
Quoted 2 lines

1) some way to indicate "this file is elastic" (okay perhaps xattrs can do this already)

Or just a list of elastic files in ~/.elastic. Even better, could mark them as "Just delete", "compress"; could go as far as giving (fallback?) globs to select files for each treatment ("If space gets tight, delete *~ files, and compress *.tex that haven't been read in a week"). Sounds like a fun Perl project...

.elastic is ugly but okay.

Quoted 6 lines
Quoted 3 lines

and either

2a) file selection/deletion in kernel

A daemon or cron job running as root can do that just fine. Or you can set it up for your own files.

If I make it cron job once an hour, users will still get -ENOSPC for up-to hour. You can make it down to second, but you'll still get -ENOSPC from time to time. If you want to eliminate -ENOSPC altogether, you'll delete file from kernel just before returning -ENOSPC and retry operation.

Quoted 7 lines
Quoted 3 lines

2b) assume that disk does not fill up faster than 1GB/sec, allways keep 1GB free, make "deleting" daemon poll each second [ugly, unreliable]

Buy a larger disk. Make sure sum of all hard quotas is less than filesystem size. Need that anyway; so it reduces to a one-user problem with per-user solutions.

Well, having sum of all hard quotas > filesystem size was point of this excersize...

Quoted 6 lines
Quoted 2 lines

BTW 2c) would be also usefull for undelete. Unfortunately 2c looks very complex, too; it might be easier to do 2a than 2c.

As said, all this buys very little for a lot of hairy code in the kernel, which will be rarely used (and whose bugs will show up when it is badly needed to work right). Besides, I strongly oppose automatic file

It may be little gain for lots of effort. I'm just trying to say that it is not complete nonsense.

I guess we are basically agreeing with each other... It may make nice student project, and if patch is very non-intrusive, I guess its okay. If it turns to be hairy, its not worth bothering.

Quoted 3 lines

that, so you have no right to know", no "undelete deleted files, but only sometimes; it just might still be around, but if space got tight it isn't" (this is even worse...))

Well, few times I wished unix had undelete... It actually *has* one, and its called power button if you are realize your mistake within 5 seconds.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!