Is there a "blackhole" /dev/null directory?
Hello Linux Kernel Hackers,
/dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work.
What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.
Here is how it could work:
mount -t nulldir nulldir /dev/nulldir
Now if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.
The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.
Is there already existing a possibility to create a null directory?
Signature · 6 lines
Best regards, Mika Lawando -- E-Mail sent with anti-spam site TrashMail.net! Free disposable email addresses: http://www.trashmail.net/
Quoted 8 lines · rzryyvzy
/dev/nullis often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so/dev/nullcould no more work.What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.Here is how it could work:
mount -t nulldir nulldir/dev/nulldir
Quoted 8 lines
Now if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.
Is there already existing a possibility to create a null directory?
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
-j
Quoted 4 lines · Jasper Bryant-Greene
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
Q.: how much work would fuse have to do until the user file system server could decide to ignore the data?
A.: pretty much all of a cached write including all the copies and context switches.
That is because FUSE has to first hand all the data to the server until it can decide to do nothing and that's pretty much all (and then some more) of the cost of a cached write.
So if you want any performance benefit from this (I'm a little sceptical) you should exactly not use FUSE.
The basic problem with the idea is that programs who create temporary files usually want to read them back at some point too. So if you throw everything away things break.
If you just don't want the write to not (usually) hit disk you can use tmpfs, although I believe at least ext2 (not ext3 unfortunately) is also reasonably good at not writing out very short lived files.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Jasper Bryant-Greene
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
There is a much more interesting 'problem' with a "/dev/null directory".
Q: Why would you need such a directory?
A: To temporarily fool a program into believing it wrote something.
Q: Should all files disappear? (e.g. "unlink after open")
A: Maybe not, programs may stat() the file right afterwards and
get confused by the "inexistence".
Q: What if a program attempts to mkdir /dev/nullmnt/foo to just
create a file /dev/nullmnt/foo/barfile?
A: /dev/nullmnt/foo must continue to exist or be accepted for a while,
or perhaps for eternity.
Been there, done that, -
http://dev.computergmbh.de/wsvn/misc_kernel/nullfs/trunk/nullfs.c - and hit that wall of unanswerable questions.
Am Thu, 14 Feb 2008 16:00:06 +0100 (CET)
schrieb Jan Engelhardt <[email protected]>:
Quoted 14 lines · Andi Kleen
Quoted 5 lines
Quoted 4 lines · Jasper Bryant-Greene
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
There is a much more interesting 'problem' with a "
/dev/nulldirectory".Q: Why would you need such a directory?
A: To temporarily fool a program into believing it wrote something.
Quoted 8 lines
Q: Should all files disappear? (e.g. "unlink after open")
A: Maybe not, programs maystat()the file right afterwards and
get confused by the "inexistence".Q: What if a program attempts to mkdir
/dev/nullmnt/footo just
create a file/dev/nullmnt/foo/barfile?
A:/dev/nullmnt/foomust continue to exist or be accepted for a while,
or perhaps for eternity.
Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded.
To be honest, I still cannot see many sensible usecases for that...
Thanks,
Hans
Quoted 6 lines · Hans-Jürgen Koch
Quoted 5 lines
Q: What if a program attempts to mkdir
/dev/nullmnt/footo just
create a file/dev/nullmnt/foo/barfile?
A:/dev/nullmnt/foomust continue to exist or be accepted for a while,
or perhaps for eternity.
Quoted 6 lines
Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to
/dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded.
Not even that. Suppose a userspace program (whose output you'd like to discard) does:
int main(void)
{
int fd = open("/nullmnt/foo.txt", O_WRONLY | O_CREAT | O_EXCL);
/* write lots of nonsensical data that we don't need anyway */
write(fd, "Hello Wor(l)d", 13);
if (lseek(fd, 0, SEEK_SET) < 0) {
/* should not happen */
fprintf(stderr, "Huh, did we write to a pipe or cdev?\n");
abort();
}
/* verify */
char buf[13];
read(fd, buf, 13);
if (memcmp(buf, "Hello Wor(l)d", 13) != 0)
fprintf(stderr, "Aïe, disk corruption!\n");
}Quoted 1 line
To be honest, I still cannot see many sensible usecases for that...
I agree.
Am Thu, 14 Feb 2008 16:23:37 +0100 (CET)
schrieb Jan Engelhardt <[email protected]>:
Quoted 15 lines · Hans-Jürgen Koch
Quoted 6 lines
Quoted 5 lines
Q: What if a program attempts to mkdir
/dev/nullmnt/footo just
create a file/dev/nullmnt/foo/barfile?
A:/dev/nullmnt/foomust continue to exist or be accepted for a while, or perhaps for eternity.Quoted 6 lines
Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to
/dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded.
Quoted 2 lines
Not even that. Suppose a userspace program (whose output you'd like to discard) does:
[...]
Well, if an application wants to read back written data, you can never use such a thing, not even in simple cases where the existing /dev/null would be enough.
Quoted 5 lines
}
Quoted 1 line
To be honest, I still cannot see many sensible usecases for that...
I agree.
Good :-)
Hans
Quoted 10 lines · Jan Engelhardt
Quoted 4 lines · Andi Kleen
Quoted 3 lines · Jasper Bryant-Greene
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
There is a much more interesting 'problem' with a "
/dev/nulldirectory".Q: Why would you need such a directory?
A: To temporarily fool a program into believing it wrote something.
Also: to let a program believe it was creating files which are used to hold the written data. Otherwise /dev/null would probably be the solution.
Quoted 4 lines
Q: Should all files disappear? (e.g. "unlink after open")
A: Maybe not, programs maystat()the file right afterwards and
get confused by the "inexistence".
I think what is going to happen is that files created behave as if they are the result of a mknod resulting in a /dev/null clone.
Quoted 5 lines
Q: What if a program attempts to mkdir
/dev/nullmnt/footo just
create a file/dev/nullmnt/foo/barfile?
A:/dev/nullmnt/foomust continue to exist or be accepted for a while,
or perhaps for eternity.
The directory structure can persist, it's the writing of data which can be avoided.
Real example:
A program which reads log files and prepares a whole raft of reports in a directory specified. If you just want to see the summary (stdout) and exception notices (stderr) having a nulldir would avoid the disk space and i/o load if you were just looking at the critical output rather than the analysis.
Yes, if this was an original program requirement it would or should have been a feature. Real world cases sometimes use tools in creative ways.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot
Quoted 15 lines · Jasper Bryant-Greene
Quoted 13 lines · rzryyvzy
/dev/nullis often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so/dev/nullcould no more work.What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.Here is how it could work:
mount -t nulldir nulldir/dev/nulldirNow if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.
Quoted 9 lines
Quoted 4 lines
The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.
Is there already existing a possibility to create a null directory?
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl.
Signature · 2 lines
Best regards, Mika
Quoted 21 lines · Mika Lawando
Quoted 20 lines · Jasper Bryant-Greene
Quoted 17 lines · rzryyvzy
/dev/nullis often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so/dev/nullcould no more work.What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.Here is how it could work:
mount -t nulldir nulldir/dev/nulldirNow if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.
Is there already existing a possibility to create a null directory?
Quoted 13 lines
Quoted 4 lines
This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.
Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl.
Signature · 3 lines
Best regards, Mika --
Creating a null directory wouldn't work because a directory is just a link to find a file. The actual file gets written using the file-descriptor, without any reference whosoever to the path. If you have root privileges, you can use mknod tempfile c 1 3 to create a null file with any name you want. Unfortunately, somebody decided that you need root privileges to execute mknod so ordinary users can not do this.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.28 BogoMips). My book : http://www.AbominableFirebug.com/
_
****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.
Thank you.