Skip to contentSkip to conversation

Is there a "blackhole" /dev/null directory?

DISCUSSION rzryyvzy 7 participants 10 messages Feb 14, 2008 – Feb 15, 2008

lore
Share & tools
lore

rzryyvzy

author
#0 lore raw

Hello Linux Kernel Hackers,

/dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work.

What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.

Here is how it could work:
mount -t nulldir nulldir /dev/nulldir

Now if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.

The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.

Is there already existing a possibility to create a null directory?

Signature · 6 lines
Best regards,
Mika Lawando

-- 
E-Mail sent with anti-spam site TrashMail.net!
Free disposable email addresses: http://www.trashmail.net/
#1 lore raw
Quoted 8 lines · rzryyvzy

/dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work.

What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.

Here is how it could work:
mount -t nulldir nulldir /dev/nulldir

Quoted 8 lines

Now if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.

The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.

Is there already existing a possibility to create a null directory?

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

-j

#2 lore raw
Quoted 4 lines · Jasper Bryant-Greene

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

Q.: how much work would fuse have to do until the user file system server could decide to ignore the data?
A.: pretty much all of a cached write including all the copies and context switches.

That is because FUSE has to first hand all the data to the server until it can decide to do nothing and that's pretty much all (and then some more) of the cost of a cached write.

So if you want any performance benefit from this (I'm a little sceptical) you should exactly not use FUSE.

The basic problem with the idea is that programs who create temporary files usually want to read them back at some point too. So if you throw everything away things break.

If you just don't want the write to not (usually) hit disk you can use tmpfs, although I believe at least ext2 (not ext3 unfortunately) is also reasonably good at not writing out very short lived files.

-Andi

#3 lore raw
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Jasper Bryant-Greene

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

There is a much more interesting 'problem' with a "/dev/null directory".

Q: Why would you need such a directory?
A: To temporarily fool a program into believing it wrote something.

Q: Should all files disappear? (e.g. "unlink after open")
A: Maybe not, programs may stat() the file right afterwards and
get confused by the "inexistence".

Q: What if a program attempts to mkdir /dev/nullmnt/foo to just
create a file /dev/nullmnt/foo/barfile?
A: /dev/nullmnt/foo must continue to exist or be accepted for a while,
or perhaps for eternity.

Been there, done that, -
http://dev.computergmbh.de/wsvn/misc_kernel/nullfs/trunk/nullfs.c - and hit that wall of unanswerable questions.

#4 lore raw

Am Thu, 14 Feb 2008 16:00:06 +0100 (CET)
schrieb Jan Engelhardt <[email protected]>:

Quoted 14 lines · Andi Kleen
Quoted 5 lines
Quoted 4 lines · Jasper Bryant-Greene

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

There is a much more interesting 'problem' with a "/dev/null directory".

Q: Why would you need such a directory?
A: To temporarily fool a program into believing it wrote something.

Quoted 8 lines

Q: Should all files disappear? (e.g. "unlink after open")
A: Maybe not, programs may stat() the file right afterwards and
get confused by the "inexistence".

Q: What if a program attempts to mkdir /dev/nullmnt/foo to just
create a file /dev/nullmnt/foo/barfile?
A: /dev/nullmnt/foo must continue to exist or be accepted for a while,
or perhaps for eternity.

Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded.
To be honest, I still cannot see many sensible usecases for that...

Thanks,
Hans

#5 lore raw
Quoted 6 lines · Hans-Jürgen Koch
Quoted 5 lines

Q: What if a program attempts to mkdir /dev/nullmnt/foo to just
create a file /dev/nullmnt/foo/barfile?
A: /dev/nullmnt/foo must continue to exist or be accepted for a while,
or perhaps for eternity.

Quoted 6 lines

Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded.

Not even that. Suppose a userspace program (whose output you'd like to discard) does:

int main(void)

{
	int fd = open("/nullmnt/foo.txt", O_WRONLY | O_CREAT | O_EXCL);

	/* write lots of nonsensical data that we don't need anyway */
	write(fd, "Hello Wor(l)d", 13);

	if (lseek(fd, 0, SEEK_SET) < 0) {
		/* should not happen */
		fprintf(stderr, "Huh, did we write to a pipe or cdev?\n");
		abort();
	}

	/* verify */
	char buf[13];
	read(fd, buf, 13);
	if (memcmp(buf, "Hello Wor(l)d", 13) != 0)
		fprintf(stderr, "Aïe, disk corruption!\n");
}
Quoted 1 line

To be honest, I still cannot see many sensible usecases for that...

I agree.

#6 lore raw

Am Thu, 14 Feb 2008 16:23:37 +0100 (CET)
schrieb Jan Engelhardt <[email protected]>:

Quoted 15 lines · Hans-Jürgen Koch
Quoted 6 lines
Quoted 5 lines

Q: What if a program attempts to mkdir /dev/nullmnt/foo to just
create a file /dev/nullmnt/foo/barfile?
A: /dev/nullmnt/foo must continue to exist or be accepted for a while, or perhaps for eternity.

Quoted 6 lines

Well, the problem seems to be that a "directory" is not just data but also contains metadata. While it's easy to write data to /dev/null, you cannot simply discard metadata associated with a directory. So, such a "/dev/null-directory" would have to remember metadata (at least all created filenames including subdirectories) in the same way as other filesystems do. Only file _content_ can be discarded.

Quoted 2 lines

Not even that. Suppose a userspace program (whose output you'd like to discard) does:

[...]

Well, if an application wants to read back written data, you can never use such a thing, not even in simple cases where the existing /dev/null would be enough.

Quoted 5 lines

}

Quoted 1 line

To be honest, I still cannot see many sensible usecases for that...

I agree.

Good :-)

Hans

#7 lore raw
In reply to Jan Engelhardt
Quoted 10 lines · Jan Engelhardt
Quoted 4 lines · Andi Kleen
Quoted 3 lines · Jasper Bryant-Greene

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

There is a much more interesting 'problem' with a "/dev/null directory".

Q: Why would you need such a directory?
A: To temporarily fool a program into believing it wrote something.

Also: to let a program believe it was creating files which are used to hold the written data. Otherwise /dev/null would probably be the solution.

Quoted 4 lines

Q: Should all files disappear? (e.g. "unlink after open")
A: Maybe not, programs may stat() the file right afterwards and
get confused by the "inexistence".

I think what is going to happen is that files created behave as if they are the result of a mknod resulting in a /dev/null clone.

Quoted 5 lines

Q: What if a program attempts to mkdir /dev/nullmnt/foo to just
create a file /dev/nullmnt/foo/barfile?
A: /dev/nullmnt/foo must continue to exist or be accepted for a while,
or perhaps for eternity.

The directory structure can persist, it's the writing of data which can be avoided.

Real example:

A program which reads log files and prepares a whole raft of reports in a directory specified. If you just want to see the summary (stdout) and exception notices (stderr) having a nulldir would avoid the disk space and i/o load if you were just looking at the critical output rather than the analysis.

Yes, if this was an original program requirement it would or should have been a feature. Real world cases sometimes use tools in creative ways.

Signature · 3 lines
Bill Davidsen <[email protected]>
  "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot
#8 lore raw
In reply to Jasper Bryant-Greene
Quoted 15 lines · Jasper Bryant-Greene
Quoted 13 lines · rzryyvzy

/dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work.

What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.

Here is how it could work:
mount -t nulldir nulldir /dev/nulldir

Now if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.

Quoted 9 lines
Quoted 4 lines

The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.

Is there already existing a possibility to create a null directory?

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl.

Signature · 2 lines
Best regards,
Mika
#9 lore raw
Quoted 21 lines · Mika Lawando
Quoted 20 lines · Jasper Bryant-Greene
Quoted 17 lines · rzryyvzy

/dev/null is often very useful, specially if programs force to save data in some file. But some programs like to creates different temporary file names, so /dev/null could no more work.

What is with a "/dev/null"-directory?
I mean a "blackhole pseudo directory" which eats every write to null.

Here is how it could work:
mount -t nulldir nulldir /dev/nulldir

Now if a program does a create(2),
it creates in the memory the file with its fd.
Then if a program does a write(2) to the fd, it eats the writes and give out fakely it has written the number of bytes. When the program calls does a close(2) of the fd, then the complete inode is deleted in the memory.

The directory should be permanently empty except for the inodes with open file descriptors. So only inode information would be temporary saved in this "nulldir tmpfs" directory.

Is there already existing a possibility to create a null directory?

Quoted 13 lines
Quoted 4 lines

This could be done fairly trivially with FUSE, and IMHO is a good use for FUSE because since you're just throwing most data away, performance is not a concern.

Unfortunately performance is a concern because if not I would write on the hard disk the files, and then remove them with a cronjob. But from the point of view of the time of developpment, FUSE is a good idea, because its possible to write a filesystem quickly in Perl.

Signature · 3 lines
Best regards,
Mika
--

Creating a null directory wouldn't work because a directory is just a link to find a file. The actual file gets written using the file-descriptor, without any reference whosoever to the path. If you have root privileges, you can use mknod tempfile c 1 3 to create a null file with any name you want. Unfortunately, somebody decided that you need root privileges to execute mknod so ordinary users can not do this.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.22.1 on an i686 machine (5588.28 BogoMips). My book : http://www.AbominableFirebug.com/
_

****************************************************************
The information transmitted in this message is confidential and may be privileged. Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited. If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to [email protected] - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.