Skip to contentSkip to conversation

userspace block driver?

DISCUSSION Jeff Garzik 14 participants 19 messages Nov 9, 2005 – Nov 16, 2005

lore
Share & tools
lore
#0 lore raw

Has anybody put any thought towards how a userspace block driver would work?

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

Anybody have any favorite methods? [similar to] mmap'd packet socket? ramfs?

TIA,

Jeff

#2 lore raw
Quoted 8 lines · Neil Brown
Quoted 2 lines · On Wednesday November 9, [email protected] wrote:

Has anybody put any thought towards how a userspace block driver would work?

Isn't this was enbd does?
http://www.it.uc3m.es/~ptb/nbd/

Is there something there relevant for modern kernels? I would sure hope I could come up with something more lightweight than that.

Jeff

#3 lore raw
Quoted 7 lines · Jeff Garzik
Quoted 6 lines · Neil Brown
Quoted 2 lines · On Wednesday November 9, [email protected] wrote:

Has anybody put any thought towards how a userspace block driver would work?

Quoted 5 lines
Quoted 2 lines

Isn't this was enbd does?
http://www.it.uc3m.es/~ptb/nbd/

Is there something there relevant for modern kernels? I would sure hope I could come up with something more lightweight than that.

I was going to say drbd, but then you did say more lightweight :-)

Is nbd completely screwed these days?

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Jeff Garzik
Quoted 7 lines · Neil Brown
Quoted 2 lines · On Wednesday November 9, [email protected] wrote:

Has anybody put any thought towards how a userspace block driver would work?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Isn't this was enbd does? http://www.it.uc3m.es/~ptb/nbd/

Is there something there relevant for modern kernels? I would sure hope I could come up with something more lightweight than that.

I was going to say drbd, but then you did say more lightweight :-)

Is nbd completely screwed these days?

nbd does more than I want.

_All_ that is needed is flipping requests <somehow> to/from userspace. nbd messes directly with sockets and such, which I don't want. It does way too much, hardcodes way too much.

loop is a closer model to a generic userspace block device than nbd, I think.

Though, answering your question directly, I do get the impression that in-kernel nbd has been left behind in favor of drbd and enbd, out in the few places where nbd-ish solutions are used.

Jeff

#5 lore raw
Quoted 8 lines · Jeff Garzik
Quoted 7 lines · Jens Axboe
Quoted 6 lines · Jeff Garzik
Quoted 5 lines · Neil Brown
Quoted 2 lines · On Wednesday November 9, [email protected] wrote:

Has anybody put any thought towards how a userspace block driver would work?

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Isn't this was enbd does? http://www.it.uc3m.es/~ptb/nbd/

Is there something there relevant for modern kernels? I would sure hope I could come up with something more lightweight than that.

I was going to say drbd, but then you did say more lightweight :-)

[...]

loop is a closer model to a generic userspace block device than nbd, I think.

That got me thinking... theoretically we should be able to do a FUSE server that served a single file that could be used by a loopback device, couldn't we?

IIRC, Miklos Szeredi tried hard to avoid the deadlock scenarios that nbd suffers from in FUSE, but I don't know if it would stand being called by the loopback device.

If it works, it should be extremely simple to do the server. Just check the FUSE hello world server example:

http://fuse.sourceforge.net/helloworld.html

I've CC'ed Miklos Szeredi to see if he can shed some light on the loopback <-> FUSE combination...

Signature · 5 lines
Paulo Marques - www.grupopie.com

The rule is perfect: in all matters of opinion our
adversaries are insane.
Mark Twain
#6 lore raw
Quoted 9 lines

That got me thinking... theoretically we should be able to do a FUSE server that served a single file that could be used by a loopback device, couldn't we?

IIRC, Miklos Szeredi tried hard to avoid the deadlock scenarios that nbd suffers from in FUSE, but I don't know if it would stand being called by the loopback device.

Quoted 7 lines

If it works, it should be extremely simple to do the server. Just check the FUSE hello world server example:

http://fuse.sourceforge.net/helloworld.html

I've CC'ed Miklos Szeredi to see if he can shed some light on the loopback <-> FUSE combination...

Loopback works fine on FUSE filesystems (unless 'direct_io' mount option is used).

Miklos

#7 lore raw
Quoted 8 lines

That got me thinking... theoretically we should be able to do a FUSE server that served a single file that could be used by a loopback device, couldn't we?

IIRC, Miklos Szeredi tried hard to avoid the deadlock scenarios that nbd suffers from in FUSE, but I don't know if it would stand being called by the loopback device.

N.B. though FUSE itself is free of deadlocks, as soon as you put something on top of it which has asyncronous page writeback it will not be safe anymore.

So the issues with NBD will still be there if you loopback mount a fuse file.

Miklos

#8 lore raw
Quoted 4 lines · Miklos Szeredi

N.B. though FUSE itself is free of deadlocks, as soon as you put something on top of it which has asyncronous page writeback it will not be safe anymore.

Why? What goes wrong?

#9 lore raw
Quoted 6 lines
Quoted 4 lines

N.B. though FUSE itself is free of deadlocks, as soon as you put something on top of it which has asyncronous page writeback it will not be safe anymore.

Why? What goes wrong?

Filesystem daemon can't use GFP_NOIO, and can't set PF_MEMALLOC. Even if it could, there's the problem with reply packets from network, which are not even handled in kernel yet (?).

FUSE sidesteps the issue, by doing writes synchronously and not allowing shared-writable mappings, hence never dirtying any pages.

The sync write is actually not so bad, the filesystem daemon can do it's own buffering safely (it's swapabble memory), or do async writes over the network (letting TCP handle the buffering).

Miklos

#10 lore raw
In reply to Jens Axboe
Quoted 1 line · Jens Axboe
Quoted 1 line
Quoted 1 line
Quoted 1 line

Has anybody put any thought towards how a userspace block driver

						^^^^^^^^^
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

would work?

I was going to say drbd, but then you did say more lightweight :-)

		     ^^^^

drbd is implemented all in-kernel, too.

The deadlock scenarios with running block IO through user-space are still somewhat unsolved, though. Not that block over network IO (in particular TCP) is much better even when implemented in-kernel...

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 4 lines
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"
#11 lore raw
In reply to Jeff Garzik
Quoted 3 lines · * Jeff Garzik ([email protected]) wrote:

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

Sounds a bit like stunnel + nbd.

thanks,
-chris

#12 lore raw
In reply to Jeff Garzik
Quoted 6 lines · Jeff Garzik

Has anybody put any thought towards how a userspace block driver would work?

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

I am afraid this is impossible without some heavy infrastructure work. You will almost inevitably deadlock. Yes, you can mlock() your driver, but that still will not tell the kernel that GFP_KERNEL must be replaced with GFP_NOIO if it is triggered by syscalls you are doing.

Regards
Oliver

#13 lore raw
Quoted 6 lines · Oliver Neukum
Quoted 2 lines · Jeff Garzik

Has anybody put any thought towards how a userspace block driver would work?

Quoted 11 lines
Quoted 5 lines

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

I am afraid this is impossible without some heavy infrastructure work.
You will almost inevitably deadlock. Yes, you can mlock() your driver, but that still will not tell the kernel that GFP_KERNEL must be replaced with GFP_NOIO if it is triggered by syscalls you are doing.

A simple patch can help here (in addition to mlockall()):

http://www.ussg.iu.edu/hypermail/linux/kernel/0407.3/0297.html

you might want to increase the free memory target as well.

#15 lore raw
In reply to Jeff Garzik
Quoted 8 lines · Jeff Garzik

Has anybody put any thought towards how a userspace block driver would work?

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

Anybody have any favorite methods? [similar to] mmap'd packet socket? ramfs?

Heh, thanks, Jeff, for bringing this subject up again, hasn't been that long ago

http://marc.theaimsgroup.com/?l=linux-kernel&m=113140332009208&w=2

, which was indeed asked with nbd in mind. To remind you and others in

http://marc.theaimsgroup.com/?t=111524157800004&r=1&w=2
http://marc.theaimsgroup.com/?t=111706463800001&r=1&w=2

I played a bit with extending nbd to map block devices to the client system more transparently, which means, as James Bottomley explained, basically supporting REQ_BLOCK_PC requests. He also suggested not to use ioctls on both sides, which is where I stopped. I can understand how to avoid implementing ioctl in the nbd driver and intercepting REQ_BLOCK_PC requests instead, but on the server side? Assume you get the request object on the client, send it to the server, and then? Even if there existed a generic interface to block devices, allowing to inject requests directly from user space into block queue, wouldn't the same problems with endianness, 32/64 bit stay? The advantage, perhaps, would be that the request structure is standard, so, the conversion would be universal?

So, my problem is - how to send a generic request to a device (disk / cdrom / loop / sg / st / ...) from the user space? Hence my recent question...

Thanks
Guennadi
---
Guennadi Liakhovetski

#16 lore raw
Quoted 10 lines · Jeff Garzik
Quoted 9 lines

Has anybody put any thought towards how a userspace block driver would work?

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

Anybody have any favorite methods? [similar to] mmap'd packet socket? ramfs?

Hm, how about a simple trick:

you write a "remote resource access" filesystem and do

mount -t remote_resourse -o access_control.conf 192.168.1.1 /mnt/server1

then you "just" write a library to overload open, close, read, write, ioctl,... do

export LD_PRELOAD=remote_libc.so

and then

mount /mnt/server1/hda1 /usr/local

In /mnt/server1/ you could have interesting things like

mouse0
dsp0

or even

network0
node0/cpu0
node0/ram0

Simple, sin't it?:-)

Thanks
Guennadi
---
Guennadi Liakhovetski

#17 lore raw
In reply to Jeff Garzik

Hi!

Quoted 8 lines

Has anybody put any thought towards how a userspace block driver would work?

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

Anybody have any favorite methods? [similar to] mmap'd packet socket? ramfs?

drivers/block/nbd?
Pavel

Signature · 1 line
Thanks, Sharp!
#18 lore raw
In reply to Jeff Garzik
Quoted 10 lines · Jeff Garzik

Has anybody put any thought towards how a userspace block driver would work?

Consider a block device implemented via an SSL network connection. I don't want to put SSL in the kernel, which means the only other alternative is to pass data to/from a userspace daemon.

Anybody have any favorite methods? [similar to] mmap'd packet socket? ramfs?

Here is a user block device i've been using for my own userspace block device drivers for some years now:

http://gort.metaparadigm.com/userblk/

This code is 2.6 based. 2.6 has made it much more reliable to do this due to the separation of writeback into the 'pdflush' thread lessening the likely hood of deadlocks. I've had very good results with this code under quite heavy memory pressure (you need a carefully written userspace which mlocks itself into memory and avoids doing certain things).

I wrote it because the nbd and enbd implementations didn't provide a nice and/or simple interface for a local userspace daemon.

enbd was closer to what I needed but when I looked at it I thought it's blocking on a ioctl was an ugly design - plus it was overcomplicated with enbd specific features.

I chose to use a kernel <-> user comms model based on Alan Cox's psdev with a char device using read and write and a mmap area for the block request data (potentially allowing me to implement zero copy in the future by mapping the bio into the user address space).

It is named 'ub' as it was written way before the USB driver although as I hadn't published my work no one was aware of this. I should come up with a new name. Perhaps 'bdu'?

I'm actually using the code in a production environment on a lab of 25 linux machines with a hybrid network block device / local disk caching implementation in userspace (to make netboot less disk intensive - similar to how Apple's netboot system work's). The userspace implementation also does local COW onto disk (to avoid the need for a stateless readonly type linux system which is hard to achieve). I have source for this available if anyone is interested. The link above only contains a simple userspace example implementation.

The 'metaboot' system which uses 'ub' does a lot of smart things such as 'commit' and 'rollback' of block deltas and has a cache coherency protocol to only invalidate changed blocks on clients local caches so I can upgrade a bunch of quasi netboot/cache machines with minimal network traffic (much more scalable then normal fatclient netbooting).

I can put up a webpage with links to CVS, etc it anyone is interested.

~mc