Skip to contentSkip to conversation

Mapping a piece of one process' addrspace to another?

DISCUSSION Jeremy Elson 6 participants 16 messages Mar 7, 2001 – Mar 8, 2001

lore
Share & tools
lore
#0 lore raw

Greetings,

Is there some way to map a piece of process X's address space into process Y, without X's knowledge or cooperation? (The non-cooperating nature of process X is why I can't use plain old shared memory.)

Put another way, I need to grant Process Y permission to write into a private buffer owned by process X. X doesn't know this is happening, and I don't know where X's buffer even lives (X's stack, heap, etc.). X just passes a pointer to the kernel via a system call (e.g., like sys_read).

In the system I currently have implemented, Y writes a buffer to the kernel, and the kernel relays it to X on behalf of Y. But, for performance reasons, I want to try to get Y to write its data directly to X instead. X might be any process calling read() (e.g., "cat"), but Y is in collusion with the kernel.

If this is possible at all, is it also possible on a granularity down to one byte (as opposed to an entire page)?

Sorry if this seems like a strange thing to do - but, I hope to document and release my code soon, in which case its purpose should be clearer :-).

Regards,
Jeremy

#1 lore raw

You are reinventing the wheel.
man ptrace (see PTRACE_{PEEK,POKE}{TEXT,DATA} and PTRACE_{ATTACH,CONT,DETACH})

Cheers,
Al

#2 lore raw
Quoted 4 lines · Alexander Viro

You are reinventing the wheel.
man ptrace (see PTRACE_{PEEK,POKE}{TEXT,DATA} and PTRACE_{ATTACH,CONT,DETACH})

With ptrace data will be copied twice. As far as I understood, Jeremy wants to avoid that.

#3 lore raw
Quoted 6 lines · Marcelo Tosatti
Quoted 2 lines · Alexander Viro

You are reinventing the wheel.
man ptrace (see PTRACE_{PEEK,POKE}{TEXT,DATA} and PTRACE_{ATTACH,CONT,DETACH})

With ptrace data will be copied twice. As far as I understood, Jeremy wants to avoid that.

Yes - I've been looking at the sys_ptrace code and it seems that it does two copies through the kernel (but, using access_process_vm instead of copy_from_user -- I'm not sure how they differ). I'm trying to reduce the number of copies by giving one process a pointer into another process's address space.

Right now, my code looks something like this: (it might make more sense if you know that I've written a framework for writing user-space device drivers... I'm going to be releasing it soon, hopefully after I resolve this performance problem. Or maybe before, if it's hard.)

   Process X: (any arbitrary process)
        read(fd, my_local_buffer);
   Kernel:
        [notifies process Y that X needs data]
   Process Y: (the userspace device driver)
        memcpy(my_local_buffer, data_destined_for_process_x);
        write(fd, my_local_buffer);
   Kernel (in sys_write:)
        copy_from_user(kernel_buffer, Y's my_local_buffer);
        copy_to_user(X's my_local_buffer, kernel_buffer);     

Now, this all works fine, but it's slower than I'd like. I used SGI's kernprof (oss.sgi.com/projects/kernprof) and found that the kernel was spending most of its time (like 70%) in copy_to_user and copy_from_user. So, I am hoping to make this all go faster, while hopefully preserving correctness, by changing to something like this:

Process X:

        read(fd, my_local_buffer);
   Kernel:
        [maps X's my_local_buffer to Y's address space]
        [notifies process Y that X needs data, and sends along a
	 pointer to write data to, or a handle usable by shmat,
         or something]
   Process Y:
         memcpy(X's my_local_buffer, data_destined_for_process_x);

If I'm lucky we can avoid both copy_to_user and copy_from_user.

This would be easy if X was a "cooperative" process because we could just set up a shared memory segment between the two processes. The problem is that X is not cooperative -- it can be any existing program.

Regards,
Jeremy

#4 lore raw
Quoted 4 lines · Jeremy Elson

Right now, my code looks something like this: (it might make more sense if you know that I've written a framework for writing user-space device drivers... I'm going to be releasing it soon, hopefully after I resolve this performance problem. Or maybe before, if it's hard.)

Ugh. Why not make that a named pipe and use zerocopy stuff for pipes? I.e. why bother with making it look like a character device rather than a FIFO?
Cheers,
Al

#5 lore raw
Quoted 10 lines · Alexander Viro
Quoted 4 lines · Jeremy Elson

Right now, my code looks something like this: (it might make more sense if you know that I've written a framework for writing user-space device drivers... I'm going to be releasing it soon, hopefully after I resolve this performance problem. Or maybe before, if it's hard.)

Ugh. Why not make that a named pipe and use zerocopy stuff for pipes? I.e. why bother with making it look like a character device rather than a FIFO?

Well, because it's a character device :-). i.e,. the framework allows you to write a userspace program that services callbacks for character devices. Inside the kernel, all open()/release()/ioctl()/etc calls for the device are proxied out to userspace where a library calls a userspace callback, and the result goes back to the kernel where it is then returned to the calling process.

The problem is just that to return data (instead of just a retval), as is needed for read and some ioctls, it leads to 3 copies as I described earlier.

BTW, where are the zerocopy patches for pipes? Maybe I'm missing something but it seems that pipes inside the kernel are still implememented by copying into the kernel and then copying out. Whatever method the zerocopy pipes use is probably what I'm looking for though.

-Jer

#6 lore raw
Quoted 6 lines · Jeremy Elson

Well, because it's a character device :-). i.e,. the framework allows you to write a userspace program that services callbacks for character devices. Inside the kernel, all open()/release()/ioctl()/etc calls for the device are proxied out to userspace where a library calls a userspace callback, and the result goes back to the kernel where it is then returned to the calling process.

Double ugh. Why bother with ioctl() when you can just have a second channel and do read()/write() on it?

Quoted 9 lines

The problem is just that to return data (instead of just a retval), as is needed for read and some ioctls, it leads to 3 copies as I described earlier.

BTW, where are the zerocopy patches for pipes? Maybe I'm missing something but it seems that pipes inside the kernel are still implememented by copying into the kernel and then copying out. Whatever method the zerocopy pipes use is probably what I'm looking for though.

Ask DaveM or look through l-k archives for URL of recent variant...
Cheers,
Al

#7 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"AV" == Alexander Viro <[email protected]> writes:

AV> Double ugh. Why bother with ioctl() when you can just have a AV> second channel and do read()/write() on it?

Because you cannot rewrite -- or even re-compile -- every app this should support. OSS emulation by ALSA is a great example, given how many binary-only apps already exist which need OSS emulation on an ALSA box.

I'm sure sound is not the only real application for this.

-JimC

Signature · 2 lines
James H. Cloos, Jr.  <http://jhcloos.com/public_key>     1024D/ED7DAEA6 
<[email protected]>  E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
#8 lore raw
In reply to Alexander Viro
Quoted 11 lines · Alexander Viro
Quoted 4 lines · Jeremy Elson

Right now, my code looks something like this: (it might make more sense if you know that I've written a framework for writing user-space device drivers... I'm going to be releasing it soon, hopefully after I resolve this performance problem. Or maybe before, if it's hard.)

Ugh. Why not make that a named pipe and use zerocopy stuff for pipes? I.e. why bother with making it look like a character device rather than a FIFO?

What about ioctl? Device drivers sometimes need it ;-)

Signature · 10 lines
Abramo Bagnara                       mailto:[email protected]

Opera Unica                          Phone: +39.546.656023
Via Emilia Interna, 140
48014 Castel Bolognese (RA) - Italy

ALSA project is            http://www.alsa-project.org
sponsored by SuSE Linux    http://www.suse.com

It sounds good!
#9 lore raw
Quoted 9 lines · Abramo Bagnara
Quoted 8 lines · Alexander Viro
Quoted 4 lines · Jeremy Elson

Right now, my code looks something like this: (it might make more sense if you know that I've written a framework for writing user-space device drivers... I'm going to be releasing it soon, hopefully after I resolve this performance problem. Or maybe before, if it's hard.)

Quoted 5 lines
Quoted 3 lines

Ugh. Why not make that a named pipe and use zerocopy stuff for pipes? I.e. why bother with making it look like a character device rather than a FIFO?

What about ioctl? Device drivers sometimes need it ;-)

No, they don't. OOB data is equivalent to data on parallel channel.

#10 lore raw
Quoted 12 lines · Alexander Viro
Quoted 9 lines · Abramo Bagnara
Quoted 8 lines · Alexander Viro
Quoted 4 lines · Jeremy Elson

Right now, my code looks something like this: (it might make more sense if you know that I've written a framework for writing user-space device drivers... I'm going to be releasing it soon, hopefully after I resolve this performance problem. Or maybe before, if it's hard.)

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Ugh. Why not make that a named pipe and use zerocopy stuff for pipes? I.e. why bother with making it look like a character device rather than a FIFO?

What about ioctl? Device drivers sometimes need it ;-)

No, they don't. OOB data is equivalent to data on parallel channel.

Al, you're perfectly right in principle (although last time I've checked pipe and unix socket did not support OOB data. Is this changed recently?).

But you're forgetting that we need to cope with non collaborative applications (that *use* ioctl).

Signature · 10 lines
Abramo Bagnara                       mailto:[email protected]

Opera Unica                          Phone: +39.546.656023
Via Emilia Interna, 140
48014 Castel Bolognese (RA) - Italy

ALSA project is            http://www.alsa-project.org
sponsored by SuSE Linux    http://www.suse.com

It sounds good!
#11 lore raw
Quoted 5 lines · Abramo Bagnara
Quoted 1 line

No, they don't. OOB data is equivalent to data on parallel channel.

Al, you're perfectly right in principle (although last time I've checked pipe and unix socket did not support OOB data. Is this changed recently?).

ioctl() is OOB stuff. Replace with second pipe and you don't need it anymore.

Quoted 2 lines

But you're forgetting that we need to cope with non collaborative applications (that *use* ioctl).

Erm. If ioctls are device-specific - the program is already bound to specific driver. If they are for class of devices (and if I guessed right that's the case you are interested in - sound, isn't it?) we could let the stub driver in kernel open two pipes and redirect read()/write() on device to the first one turn ioctls into read()/write() on the second. That way you can get userland programs serving that stuff with no magic required.

I mean something along the lines of

foo_write(struct file *file, char *buf, size_t size, loff_t *ppos)

{
	return f1->f_ops->write(file, buf, size, ppos);
}
foo_ioctl(struct file *file, int cmd, long arg)
{
	loff_t dummy;
	switch (cmd) {
		case SNDCTL_DSP_RESET:
			f2->f_ops->write(file, "dsp_reset", 10, &dummy);
			return 0;
		....
	}
}

where f1 and f2 are named pipes opened at init time. Notice that it doesn't introduce extra copying - just splits the stream with OOB data into two normal byte streams that can be served by naive userland code. After that any client that uses current sound API is able to talk to userland drivers and said drivers don't have to pull any special tricks - they just open two pipes and do usual select()/read()/write() loop. Kernel side is also quite simple that way - plain redirect for read/write and trivial decoder for the sound ioctls.
Cheers,
Al

#12 lore raw
Quoted 29 lines · Alexander Viro
Quoted 4 lines

But you're forgetting that we need to cope with non collaborative applications (that *use* ioctl).

Erm. If ioctls are device-specific - the program is already bound to specific driver. If they are for class of devices (and if I guessed right that's the case you are interested in - sound, isn't it?) we could let the stub driver in kernel open two pipes and redirect read()/write() on device to the first one turn ioctls into read()/write() on the second. That way you can get userland programs serving that stuff with no magic required.

I mean something along the lines of

foo_write(struct file *file, char *buf, size_t size, loff_t *ppos)

{
        return f1->f_ops->write(file, buf, size, ppos);
}
foo_ioctl(struct file *file, int cmd, long arg)
{
        loff_t dummy;
        switch (cmd) {
                case SNDCTL_DSP_RESET:
                        f2->f_ops->write(file, "dsp_reset", 10, &dummy);
                        return 0;
                ....
        }
}
Quoted 8 lines

where f1 and f2 are named pipes opened at init time. Notice that it doesn't introduce extra copying - just splits the stream with OOB data into two normal byte streams that can be served by naive userland code. After that any client that uses current sound API is able to talk to userland drivers and said drivers don't have to pull any special tricks - they just open two pipes and do usual select()/read()/write() loop. Kernel side is also quite simple that way - plain redirect for read/write and trivial decoder for the sound ioctls.

I agree, and this is how I wanted to implement it (apart that I thought to packetize read/write/ioctl with an header and then to avoid to have two pipes).

This has two major drawbacks (that have discouraged me):
a) no mmap support
b) the kernel side need to know the details of ioctl contents

a) bring to have an incomplete solution
b) bring the kernel space to be OSS dependent then giving a not general solution for user space drivers.

The only thing I see that would cope with these drawbacks is to allow a privileged server application to mmap the address space of another client application. Then:

a) is solved trivially
b) is ok because server would read/write directly ioctl arguments from client address space.

Security is a non issue here as also for kernel space driver applications expose all their address space to driver.

Signature · 10 lines
Abramo Bagnara                       mailto:[email protected]

Opera Unica                          Phone: +39.546.656023
Via Emilia Interna, 140
48014 Castel Bolognese (RA) - Italy

ALSA project is            http://www.alsa-project.org
sponsored by SuSE Linux    http://www.suse.com

It sounds good!
#13 lore raw
In reply to Alexander Viro
Quoted 7 lines · Alexander Viro

Erm. If ioctls are device-specific - the program is already bound to specific driver. If they are for class of devices (and if I guessed right that's the case you are interested in - sound, isn't it?) we could let the stub driver in kernel open two pipes and redirect read()/write() on device to the first one turn ioctls into read()/write() on the second. That way you can get userland programs serving that stuff with no magic required.

One problem, in addition to the points Abramo made, is that there's no metadata that goes along with the data you're writing to the pipe. This is fine as long as there's only one process reading, but in the case of a single driver serving multiple instances of an open file, there has to be some way for the userspace driver to communicate to the kernel which open file needs to get the data.

A similar problem is that metadata needs to accompany the read request when the userspace driver gets it (i.e., current file position, file flags, size of the read that was requested, etc.) Although, that data might be transmitted on the OOB channel.

Also, even though my framework right now only does character devices, it's a pretty simple extension to support userspace block devices and network devices as well. In these cases I think using a pipe gets more and more complicated.

Quoted 1 line

From reading all the responses to the thread (thanks, everyone, for

your useful comments), it sounds like the best thing to do is use Manfred's zerocopy pipe code as a model and implement something similar in my framework. It's not really that much code anyway so it's not much of a waste reimplementing it. And, I think reimplementing those 30 lines will be much simpler than trying to manage 2 parallel channels. If the zerocopy patch becomes part of the kernel anyway, it gets even simpler, since I can probably just call pio_copy_to_user instead of copying it.

Regards,
Jer

#14 lore raw
In reply to Jeremy Elson
Quoted 6 lines · Jeremy Elson
Quoted 4 lines · Marcelo Tosatti
Quoted 2 lines · Alexander Viro

You are reinventing the wheel.
man ptrace (see PTRACE_{PEEK,POKE}{TEXT,DATA} and PTRACE_{ATTACH,CONT,DETACH})

Quoted 8 lines
Quoted 2 lines

With ptrace data will be copied twice. As far as I understood, Jeremy wants to avoid that.

Yes - I've been looking at the sys_ptrace code and it seems that it does two copies through the kernel (but, using access_process_vm instead of copy_from_user -- I'm not sure how they differ). I'm trying to reduce the number of copies by giving one process a pointer into another process's address space.

I've investigated this in past to do the same thing you're trying (in my case I was interested in a full user space implementation of OSS emulation layer for ALSA).

The other huge drawback for ptrace is the impossibility to use mmap.

With older versions of Linux it was feasible to mmap /proc/PID/mem and this was a very elegant way to do the job.

Unfortunately this feature has been dropped, but I don't know for which reasons.

Signature · 10 lines
Abramo Bagnara                       mailto:[email protected]

Opera Unica                          Phone: +39.546.656023
Via Emilia Interna, 140
48014 Castel Bolognese (RA) - Italy

ALSA project is            http://www.alsa-project.org
sponsored by SuSE Linux    http://www.suse.com

It sounds good!
#15 lore raw
In reply to Alexander Viro

Hi!

Quoted 5 lines
Quoted 2 lines

You are reinventing the wheel.
man ptrace (see PTRACE_{PEEK,POKE}{TEXT,DATA} and PTRACE_{ATTACH,CONT,DETACH})

With ptrace data will be copied twice. As far as I understood, Jeremy wants to avoid that.

mmap /proc/pid/memory?
Pavel

Signature · 2 lines
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.