Read from file fails
Hi,
can anybody help me with reading from file? I've got this code in my module for 2.6.5-mm6:
char buffer[4096];ssize_t read;
file *f;
f = filp_open("/some/file",O_RDONLY | O_LARGEFILE,0); f->f_pos = 0;
read = vfs_read(f,(char __user *) buffer,4096,&f->f_pos);
but here read value is "-14" :-((( any hints?
It seems that file is opened OK (I've tested:
if (f->f_op->read) {
read = f->f_op->read(file, buf, count, pos);
}and result was the same - so I assume that file is opened OK and structure "file f" is filled correctly.
I need to copy files (yes - I know that kernel shouldn't do this but I REALLY need) and there is nothing like "sys_copy" and "sys_sendfile" is not exported (which seems strange to me but there is nothing like EXPORT_SYMBOL(sys_sendfile) in fs/read_write.c
Thanks,
Libor Vanek
Quoted 11 lines · Libor Vanek
Hi,
can anybody help me with reading from file? I've got this code in my module for 2.6.5-mm6:char buffer[4096];
ssize_tread;
file *f;
f =filp_open("/some/file",O_RDONLY | O_LARGEFILE,0);f->f_pos= 0;
read =vfs_read(f,(char __user *) buffer,4096,&f->f_pos);
Quoted 2 lines
but here read value is "-14" :-((( any hints?
Sure it's -EFAULT. In user-space, you would get a core-dump.
Quoted 6 lines
It seems that file is opened OK (I've tested:
if (f->f_op->read) { read = f->f_op->read(file, buf, count, pos); }and result was the same - so I assume that file is opened OK and structure "file f" is filled correctly.
This has become a FAQ and NO If you know what you are doing, you NEVER need to read files from within the kernel. PERIOD. There are Hacks that __sometimes__ allow code to get away with it, but they are only going to work --- sometimes ---.
Quoted 3 lines
I need to copy files (yes - I know that kernel shouldn't do this but I REALLY need) and there is nothing like "
sys_copy" and "sys_sendfile" is not exported (which seems strange to me but there is nothing likeEXPORT_SYMBOL(sys_sendfile)infs/read_write.c
I will explain for the Nth time. The kernel is a bunch of code that performs functions on behalf of a process. It does that in the context of that process. A file descriptor is a number that belongs to a process. It is worthless without a process with which to associate. If the kernel were to open a file, who owns the file descriptor? The kernel isn't a process, it's a big LIBRARY! So, if you are lucky, you can steal the context from another unknowing process. This works unless there was a context-switch or if the process was asleep so it didn't access any files of its own. It leaves the kernel in a bad state because there is a wild file-descriptor that nobody owns that will never be closed because when your cludge closes that file-descriptor, you probably end up closing STDOUT_FILENO of some important task that you haven't discovered yet.
The cludges that work involve creating a "kernel thread" which has a process context. That thread does the file I/O. However, it's certainly easier to create a user-mode task (program) that opens your module and, using an ioctl(), sends it anything from anywhere in the known universe. You can get data from a file or from anywhere.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 6 lines
I will explain for the Nth time. The kernel is a bunch of code that performs functions on behalf of a process. It does that in the context of that process. A file descriptor is a number that belongs to a process. It is worthless without a process with which to associate. If the kernel were to open a file, who owns the file descriptor? The kernel isn't a process, it's a big LIBRARY!
OK - speaking of this I just want call one public (exported) function of library from another.
Quoted 9 lines
So, if you are lucky, you can steal the context from another unknowing process. This works unless there was a context-switch or if the process was asleep so it didn't access any files of its own. It leaves the kernel in a bad state because there is a wild file-descriptor that nobody owns that will never be closed because when your cludge closes that file-descriptor, you probably end up closing
STDOUT_FILENOof some important task that you haven't discovered yet.
Quoted 6 lines
The cludges that work involve creating a "kernel thread" which has a process context. That thread does the file I/O. However, it's certainly easier to create a user-mode task (program) that opens your module and, using an
ioctl(), sends it anything from anywhere in the known universe. You can get data from a file or from anywhere.
OK - so speaking of kernel thread - what function should I use? Or do you think that only the difference of having process context and pid will cause read to work? (I'm willing to believe it ;-) - I'll try it later)
I think there are reasons (speed, speed, speed...) why some things should be done kernel-space. And I think that this is one of them. I know that I'll never want to "send data from anywhere in the known universe" (famous last words?) - I just need to create backup copy of file.
Libor
Quoted 10 lines · Libor Vanek
Quoted 6 lines
I will explain for the Nth time. The kernel is a bunch of code that performs functions on behalf of a process. It does that in the context of that process. A file descriptor is a number that belongs to a process. It is worthless without a process with which to associate. If the kernel were to open a file, who owns the file descriptor? The kernel isn't a process, it's a big LIBRARY!
OK - speaking of this I just want call one public (exported) function of library from another.
No, you are corrupting the library by inserting a "man in the middle". The man in the middle needs resources and can only steal them because he doesn't have his own context.
Quoted 16 lines
Quoted 9 lines
So, if you are lucky, you can steal the context from another unknowing process. This works unless there was a context-switch or if the process was asleep so it didn't access any files of its own. It leaves the kernel in a bad state because there is a wild file-descriptor that nobody owns that will never be closed because when your cludge closes that file-descriptor, you probably end up closing
STDOUT_FILENOof some important task that you haven't discovered yet.Quoted 6 lines
The cludges that work involve creating a "kernel thread" which has a process context. That thread does the file I/O. However, it's certainly easier to create a user-mode task (program) that opens your module and, using an
ioctl(), sends it anything from anywhere in the known universe. You can get data from a file or from anywhere.
Quoted 5 lines
OK - so speaking of kernel thread - what function should I use? Or do you think that only the difference of having process context and pid will cause read to work? (I'm willing to believe it ;-) - I'll try it later)
Search existing drivers for "kernel_thread"
.../linux/drivers/net/8139too.c is a start.
And yes, kernel threads work. Pay particular attention to the way to run one down before you remove your module. They can do everything user-mode code can do and have the priviliges to totally screw up anything and everything in the kernel.
Quoted 1 line
I think there are reasons (speed, speed, speed...) why some things
Wrong. The kernel was designed to perform services on behalf of a user-mode task. That's how it will function the fastest. There is no "speed advantage" to kernel mode, only the lack of memory and instruction protection. None of these kernel mode instructions have anything to do with speed (except the ability to STOP everything) with cli;hlt. Then you have no speed at all.
Quoted 7 lines
should be done kernel-space. And I think that this is one of them. I know that I'll never want to "send data from anywhere in the known universe" (famous last words?) - I just need to create backup copy of file.
Libor
Common problem when persons first see the kernel code. They say; "Oh it's just 'C'! I can write this. I know 'C'! Next thing you know, they are trying to read/write files inside the kernel. Before you write kernel code, you need to understand what the kernel DOES. Bottom line, it does the stuff that user-mode code isn't allowed to do on its own. For instance, user mode code can't be allowed to read/write files on its own because
(1) it doesn't know how to do it, and (2) it could screw up the entire file-system either accidentally or on purpose. Therefore the kernel does it for the user task, but on behalf of the user.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 13 lines
Quoted 7 lines
Quoted 6 lines
The cludges that work involve creating a "kernel thread" which has a process context. That thread does the file I/O. However, it's certainly easier to create a user-mode task (program) that opens your module and, using an
ioctl(), sends it anything from anywhere in the known universe. You can get data from a file or from anywhere.Quoted 5 lines
OK - so speaking of kernel thread - what function should I use? Or do you think that only the difference of having process context and pid will cause read to work? (I'm willing to believe it ;-) - I'll try it later)
Quoted 7 lines
Search existing drivers for "
kernel_thread".../linux/drivers/net/8139too.c is a start.And yes, kernel threads work. Pay particular attention to the way to run one down before you remove your module. They can do everything user-mode code can do and have the priviliges to totally screw up anything and everything in the kernel.
I know that kernel threads work. My question was more like: "I'd like to know, whether writing my module as kernel thread will make it able to read/write files".
Quoted 10 lines
Quoted 1 line
I think there are reasons (speed, speed, speed...) why some things
Wrong. The kernel was designed to perform services on behalf of a user-mode task. That's how it will function the fastest. There is no "speed advantage" to kernel mode, only the lack of memory and instruction protection. None of these kernel mode instructions have anything to do with speed (except the ability to STOP everything) with cli;hlt. Then you have no speed at all.
From what I know about kernel (which doesn't say I'm correct!) is:
Using kernel module:
- user space process wants to change some file which is in "snapshoted" dir
- my module catches this request, holds it, creates copy of original file and allows original request to proceed
Using user-space approach my kernel module must call my user space program which causes context switching (which I think is quite expensive and should be avoid). It surely won't matter on some desktop machine but when you need to do snapshots you need it to do it on large file-server with 10s/100s of active users.
Another counter-example I think is LUFS (Linux Userspace FS) - at least last time I've heard about it it was too slow for serious usage.
Quoted 10 lines
Quoted 7 lines
should be done kernel-space. And I think that this is one of them. I know that I'll never want to "send data from anywhere in the known universe" (famous last words?) - I just need to create backup copy of file.
Libor
Common problem when persons first see the kernel code. They say; "Oh it's just 'C'! I can write this. I know 'C'! Next thing you
I HATE C - can't a man write a kernel module in PHP please? ;-)
Quoted 3 lines
know, they are trying to read/write files inside the kernel. Before you write kernel code, you need to understand what the kernel DOES. Bottom line, it does the stuff that user-mode
Is there any other way how to undestand what kernel DOES except starting hacking it and asking stupid questions like "how can I read file"? :-) I've read 10s papers about kernel design in general and comparison between Linux/Windows NT/HURD/... kernels and taken 2 semesters of "Operating systems" so I got some rought idea how kernel works - now I'm confrontating my idea with reality.
Quoted 6 lines
code isn't allowed to do on its own. For instance, user mode code can't be allowed to read/write files on its own because
(1) it doesn't know how to do it, and (2) it could screw up the entire file-system either accidentally or on purpose. Therefore the kernel does it for the user task, but on behalf of the user.
There something surely true in your approach. But this approach leads to micro-kernel solutions - "let's make kernel only as small as possible and all else be user-space". Where is line between "too overloaded" and "too small" let's Linus decide ;-)
Signature · 1 line
Libor
On Tue, 4 May 2004, Libor Vanek wrote:
[SNIPPED...all]
Did you catch this information? If all you want to do is to make a new version of a file, owned by the person who accesses the file (like VAX/VMS), then you trap open with O_RDWR or O_CREAT or O_TRUNC and make a copy with a numeral appended like: filename.typ;2.
You do this by making a shared object that does what you want. The total affect upon user-mode code is a delayed (slow) open().
Quoted 1 line
From [email protected] Tue May 4 09:21:52 2004
To: Jan-Benedict Glaw <[email protected]>
From: Pascal Schmidt <[email protected]>
Cc: [email protected]
Subject: Re: Reading from file in module fails
On Mon, 03 May 2004 14:50:10 +0200, you wrote in linux.kernel:
Quoted 7 lines
That can all be done in userspace.
$ export LD_PRELOAD=3Dlibcopyfilesbeforemodify.soYou just need to program a library that provides all functions that modify files (eg. write, open with
O_CREAT, ...) and you're done - 100% in userspace.
This won't catch asm programs that do syscalls by hand or statically linked programs. If you really need to catch all write accesses, it needs to be done in the kernel, probably as an LSM hook or something.
Signature · 31 lines
Ciao,
Pascal
Now, if you need to squirrel the file away to some secret location
owned by root, then you might want to use a kernel thread. It will
take the same time and delay the open the same amount.
Kernel mode is all about privilige, not about speed. A user-mode
program daemon that operates as root, could also perform the
same function by having the LD_PRELOAD code pipe information to
it. One needs to make sure that the daemon as finished copying
the file before the open() returns of it would be possible for
the original caller to trash the file before it was copied.
If I were given the task of; "Make sure that an idiot can't
delete his files in such a way they can't be restored....".
I'd use a daemon, simply because it's easier and more
interesting. Also, the daemon is configurable. It can read
some configuration file and the password file to find out
where to stash the "wastebaskets". You end up with an extensible
solution. Kernel mode programming is the last thing you want
to do, not the first. You can't access any of the 'C' runtime
library functions although there a few cloned for kernel
programming. It's a bitch to write your own string functions
in such a way that they catch all the corner cases that can
trash the kernel.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
Note 96.31% of all statistics are fiction.Quoted 8 lines · Richard B. Johnson
On Tue, 4 May 2004, Libor Vanek wrote:
[SNIPPED...all]
Did you catch this information? If all you want to do is to make a new version of a file, owned by the person who accesses the file (like VAX/VMS), then you trap open with
O_RDWRorO_CREATorO_TRUNCand make a copy with a numeral appended like: filename.typ;2.
Something like this (only make this copy in another directory)
Quoted 2 lines
You do this by making a shared object that does what you want. The total affect upon user-mode code is a delayed (slow)
open().
Yeah - I'm now studying how to communicate between kernel and user space (can't find out how to send data FROM ks TO us). What did you mean by "shared object"?
Signature · 1 line
Libor Vanek
Quoted 3 lines · Libor Vanek
I know that kernel threads work. My question was more like: "I'd like to know, whether writing my module as kernel thread will make it able to read/write files".
[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I think there are reasons (speed, speed, speed...) why some things should be done kernel-space.
Using a kernel thread won't improve speed, because to do anything you will have to context-switch to the thread. For the stuff you want to do you are probably better off having a tiny kernel module to intercept the events that you're interested in, notifying a userspace process to do the real work. Yes, it will be slower than in kernel space, but only slightly. Especially if you use sendfile from the userspace process. And it's also good to remember that Linux is optimized for running user space processes as fast as possible. :)
--Bart
Quoted 4 lines · Bart Samwel
Quoted 3 lines · Libor Vanek
I know that kernel threads work. My question was more like: "I'd like to know, whether writing my module as kernel thread will make it able to read/write files".
Quoted 8 lines
[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I think there are reasons (speed, speed, speed...) why some things should be done kernel-space.
Using a kernel thread won't improve speed, because to do anything you will have to context-switch to the thread. For the stuff you want to do you are probably better off having a tiny kernel module to intercept the events that you're interested in, notifying a userspace process to do
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
Quoted 6 lines
the real work. Yes, it will be slower than in kernel space, but only slightly. Especially if you use sendfile from the userspace process. And it's also good to remember that Linux is optimized for running user space processes as fast as possible. :)
--Bart
Quoted 4 lines · Libor Vanek
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
My first thoughts are to make it a blocking call.
--Bart
Quoted 6 lines
Quoted 4 lines · Libor Vanek
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
My first thoughts are to make it a blocking call.
You mean like:
- send signal to user-space process
- wait until user-space process pick ups data (filename etc.), creates copy of file (or whatever) and calls another system call that he's finished
- let kernel to continue syscall I blocked
?
Libor
Quoted 7 lines · Libor Vanek
Quoted 7 lines
Quoted 4 lines · Libor Vanek
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
Quoted 7 lines
Quoted 1 line
My first thoughts are to make it a blocking call.
You mean like:
- send signal to user-space process
- wait until user-space process pick ups data (filename etc.), creates copy of file (or whatever) and calls another system call that he's finished
- let kernel to continue syscall I blocked
?
No, more like:
- user-space process calls syscall, which blocks.
- kernel captures a file write event, puts the info in some kind of queue, wakes up the user-space process and then waits for some kind of acknowledgement to be returned so that it may continue.
- user-space process wakes up, the syscall completes, and passes a filename etc. to user-space. Copies the file, and calls a syscall to signify "hey, I'm done with that file". This syscall wakes up the kernel stuff that was waiting for this acknowledgement.
- file write event continues
- repeat from start
--Bart
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
Quoted 8 lines
Quoted 1 line
My first thoughts are to make it a blocking call.
You mean like:
- send signal to user-space process
- wait until user-space process pick ups data (filename etc.), creates copy of file (or whatever) and calls another system call that he's finished
- let kernel to continue syscall I blocked
?
Quoted 11 lines
No, more like:
- user-space process calls syscall, which blocks.
- kernel captures a file write event, puts the info in some kind of queue, wakes up the user-space process and then waits for some kind of acknowledgement to be returned so that it may continue.
- user-space process wakes up, the syscall completes, and passes a filename etc. to user-space. Copies the file, and calls a syscall to signify "hey, I'm done with that file". This syscall wakes up the kernel stuff that was waiting for this acknowledgement.
- file write event continues
- repeat from start
OK - I'm thinking of using semaphores to "block" system call - is there something why this is not a good idea?
Thanks,
Libor
Quoted 23 lines · Libor Vanek
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
Quoted 8 lines
Quoted 1 line
My first thoughts are to make it a blocking call.
You mean like:
- send signal to user-space process
- wait until user-space process pick ups data (filename etc.), creates copy of file (or whatever) and calls another system call that he's finished
- let kernel to continue syscall I blocked
?Quoted 8 lines
No, more like:
- user-space process calls syscall, which blocks.
- kernel captures a file write event, puts the info in some kind of
queue, wakes up the user-space process and then waits for some kind of acknowledgement to be returned so that it may continue.
- user-space process wakes up, the syscall completes, and passes a
filename etc. to user-space. Copies the file, and calls a syscall to signify "hey, I'm done with that file". This syscall wakes up the kernel stuff that was waiting for this acknowledgement.
- file write event continues
- repeat from start
Quoted 1 line
OK - I'm thinking of using semaphores to "block" system call - is there something why this is not a good idea?
Semaphores are useful to protect a shared resource (and you might want to use those to protect your queue of filenames to copy), but not for transferring control between threads. I think you might want to look at wait_queue, wait_event, wake_up and things like that.
--Bart
Quoted 6 lines · Libor Vanek
Quoted 6 lines
Quoted 4 lines · Libor Vanek
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
Quoted 8 lines
Quoted 1 line
My first thoughts are to make it a blocking call.
You mean like:
- send signal to user-space process
- wait until user-space process pick ups data (filename etc.), creates copy of file (or whatever) and calls another system call that he's finished - let kernel to continue syscall I blocked
?
I think he meant that userspace daemon should do a blocking syscall (a read for example). When that returns, daemon knows he has something to do.
--
vda
Quoted 19 lines · Denis Vlasenko
Quoted 7 lines · Libor Vanek
Quoted 7 lines
Quoted 4 lines · Libor Vanek
OK - how can I "notify" userspace process? Signals are "weak" - I need to send some data (filename etc.) to process. One solution is "on this signal call this syscall and result of this syscall will be data you need" - but I'd prefer to handle this in one "action".
Quoted 8 lines
Quoted 1 line
My first thoughts are to make it a blocking call.
You mean like:
- send signal to user-space process
- wait until user-space process pick ups data (filename etc.), creates copy of file (or whatever) and calls another system call that he's finished - let kernel to continue syscall I blocked
?
Quoted 3 lines
I think he meant that userspace daemon should do a blocking syscall (a read for example). When that returns, daemon knows he has something to do.
Much like coda already does IIRC - kernel wakes userspace blocking on a read to your special device, userspace 'writes' result back to special device. This was an idea for a generic userspace upcall mechanism originated by Alan Cox with his psdev circa 2.0/2.2 ?? which formed the basis of the coda filesystem which does close to what you would want.
I've written a userspace block device driver interface using this mechanism also (unpublished as of today, not wanting to compete with nbd and enbd - it is unlike enbd which blocks on an ioctl and far far simpler.
http://gort.metaparadigm.com/userblk/
This way to do zero-copy by using mmap on your special device (which I plan to do for my userspace block device interface).
~mc
Quoted 3 lines · Libor Vanek
Using kernel module:
- user space process wants to change some file which is in "snapshoted" dir
- my module catches this request, holds it, creates copy of original file and allows original request to proceed
Did you take a look at LVM snapshots?
http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html
Maybe your problem is already solved...
Anyway, you really shouldn't worry about the time it takes to make a context switch when you want to copy a file on modify ;)
Signature · 2 lines
Paulo Marques - www.grupopie.com "In a world without walls and fences who needs windows and gates?"
Quoted 8 lines · Paulo Marques
Quoted 4 lines · Libor Vanek
Using kernel module:
- user space process wants to change some file which is in "snapshoted" dir
- my module catches this request, holds it, creates copy of original file and allows original request to proceed
Quoted 5 lines
Did you take a look at LVM snapshots?
http://tldp.org/HOWTO/LVM-HOWTO/snapshotintro.html
Maybe your problem is already solved...
No - this has several disadvantages - mainly you can snapshot only whole "logical volume" and have preserved space for it (okay - you can add more disks) and it's not very flexible. My approach can even snapshot files to the NFS volume and if you loose primary volume you can still recover some data.
Quoted 2 lines
Anyway, you really shouldn't worry about the time it takes to make a context switch when you want to copy a file on modify ;)
Yeah - I'm now convinced that user space daemon is "The Only Right Thing (tm)". I'm now looking how can kernel comunicate with running user-space process.
--
Libor Vanek