Skip to contentSkip to conversation

Use case for "loop: Issue O_DIRECT aio with pages"?

DISCUSSION Ted Ts'o 5 participants 7 messages Mar 1, 2012 – Mar 2, 2012 VFS

lore
Share & tools
lore

Ted Ts'o

author
#0 lore raw
Quoted 3 lines
Quoted 3 lines

My latest focus is on completing the work started on by Zach Brown to add a kernel interface to allow kernel code to submit aio, and having the loop device submit direct IO requests to the underlying file system.

So I've been taking a lot at your patch series, and one of the things which would help me understand the requirements behind what you are doing is *why* is it interesting to allow kernel code to submit aio requests, and loop devices to be able to direct I/O requests all the way to the underlying file systems. In other words, what's the use case that you're most interested in that these patches would enable?

Thanks, regards,

- Ted

#1 lore raw
Quoted 4 lines

*why* is it interesting to allow kernel code to submit aio requests, and loop devices to be able to direct I/O requests all the way to the underlying file systems.

As I understood it, lo those years ago, the use case was virtual guests provisioned on loopback devices on files in the host file system.

In this use case either having loop implement a second writeback cache or having it only capable of one sync IO in flight at a time is pretty bad when you're trying to offer consistent and performant file systems to the guest.

I believe one motivations for using loopback files is the ability to use ocfs2's reflinks to get cluster-wide cow snapshoting of guest file systems in files in ocfs2.

But I'll let Dave (or Chris?) share more about how this stuff is actually used.

Really, if we had our way, file systems would offer the bio submission and completion interface for file data. Then loopback dev IO <-> file system IO would be trivial. The aio+dio iocbs are an existing aproximation :/.

- z

#4 lore raw
In reply to Ted Ts'o

I don't care about loop in particular, but this is important for any in-kernel block device emulation using files, including the SCSI target and a possible vhost-{block,scsi} implementation.

It's also the right way to implement swapping, instead of the braindead swap over nfs proposals floating around currently.

author
#2 lore raw
In reply to Zach Brown
Quoted 5 lines · Zach Brown
Quoted 4 lines

*why* is it interesting to allow kernel code to submit aio requests, and loop devices to be able to direct I/O requests all the way to the underlying file systems.

Quoted 7 lines

As I understood it, lo those years ago, the use case was virtual guests provisioned on loopback devices on files in the host file system.

In this use case either having loop implement a second writeback cache or having it only capable of one sync IO in flight at a time is pretty bad when you're trying to offer consistent and performant file systems to the guest.

I'm confused though why the guests would be pointed at the loopback device, as opposed to just simply attaching to the file directly. Or is this some kind of Xen limitation? With qemu/kvm I don't see why it would make sense to indirect through the loop device....

- Ted

#5 lore raw
In reply to Ted Ts'o
Quoted 10 lines · Ted Ts'o
Quoted 3 lines
Quoted 3 lines

My latest focus is on completing the work started on by Zach Brown to add a kernel interface to allow kernel code to submit aio, and having the loop device submit direct IO requests to the underlying file system.

So I've been taking a lot at your patch series, and one of the things which would help me understand the requirements behind what you are doing is *why* is it interesting to allow kernel code to submit aio requests, and loop devices to be able to direct I/O requests all the way to the underlying file systems. In other words, what's the use case that you're most interested in that these patches would enable?

I'll let James talk about the parallels use case, which includes some nice building blocks on top of just exporting a file as a block device.

Oracle's main use for this is virtualized guests. We have files in dom0 that we want to turn into block devices for guests, and doing it from loop allows us to avoid passing the IO off to a userland process. Yes, kvm does this from qemu today, but the xen guys tell me the perf hit is pretty big on 64 bit xen handoffs for the ios.

Being able to use loop in a read/write workload where it is actually crash safe is an important use case. I always cringe when I hear people using loop for writable filesystem images.

-chris

#3 lore raw
In reply to Ted Ts'o
Quoted 4 lines

I'm confused though why the guests would be pointed at the loopback device, as opposed to just simply attaching to the file directly. Or is this some kind of Xen limitation?

I certainly haven't a clue. It sounds like it from Chris' message. Maybe some Xen folks could provide more detail.

- z

#6 lore raw
In reply to Chris Mason
Quoted 12 lines · Chris Mason
Quoted 10 lines · Ted Ts'o
Quoted 3 lines
Quoted 3 lines

My latest focus is on completing the work started on by Zach Brown to add a kernel interface to allow kernel code to submit aio, and having the loop device submit direct IO requests to the underlying file system.

So I've been taking a lot at your patch series, and one of the things which would help me understand the requirements behind what you are doing is *why* is it interesting to allow kernel code to submit aio requests, and loop devices to be able to direct I/O requests all the way to the underlying file systems. In other words, what's the use case that you're most interested in that these patches would enable?

Quoted 2 lines

I'll let James talk about the parallels use case, which includes some nice building blocks on top of just exporting a file as a block device.

Our use case is even more simple: containers are a simple slicing up of the existing operating system sharing the same kernel. There are performance impacts (mostly because the journal becomes a hot spot) from running a bunch of containers in a single filesystem as chroots (the current setup), so we want to transition to an image file based system like VMs use. To do that we need a device mapping to an image file per container and, since we don't have a hypervisor, the only way to do this today is via loop.

We basically want to redo the current loop device to make it far more performant (get rid of double caching using DIO etc.) and, of course, less crash prone.

James