extend e2fsprogs functionality to add EXT2_FLAG_DIRECT option

This patch extends functionality of e{2|4}fsprogs to add EXT2_FLAG_DIRECT flag to be passed to ext2fs_open2() function. This internally calls open() function with O_DIRECT and handles the memory alignment for both read and write operations. In some cases direct access to devices is necessary and that was the main reason for this patch to be done.

The main reason why this was done is that pygrub (used by xen virtualization user-space package, it's a python version of grub for paravirtualized guests) sometimes uses outdated version of grub.conf file. Modifications to xen package were *not* enough because e2fsprogs doesn't open the files directly. That's why I added EXT2_FLAG_DIRECT support to make read/write operations work directly when passed. It's been tested with pygrub like mentioned above for read operation and it's working fine.

Signed-off-by Michal Novotny <[email protected]>

Quoted 3 lines · Michal Novotny

This patch extends functionality of e{2|4}fsprogs to add EXT2_FLAG_DIRECT flag to be passed to ext2fs_open2() function. This internally calls open() function with O_DIRECT and handles the memory alignment for both read and write operations. In some cases direct access to devices is necessary and that was the main reason for this patch to be done.

Quoted 3 lines

The main reason why this was done is that pygrub (used by xen virtualization user-space package, it's a python version of grub for paravirtualized guests) sometimes uses outdated version of grub.conf file. Modifications to xen package were *not* enough because e2fsprogs doesn't open the files directly. That's why I added EXT2_FLAG_DIRECT support to make read/write operations work directly when passed. It's been tested with pygrub like mentioned above for read operation and it's working fine.

Signed-off-by Michal Novotny <[email protected]>

Can you add to this some kind of data flow overview? Seems like a really odd way to update this file system...

ric

Quoted 17 lines · Ric Wheeler

Quoted 16 lines · Michal Novotny

This patch extends functionality of e{2|4}fsprogs to add EXT2_FLAG_DIRECT flag to be passed to ext2fs_open2() function. This internally calls open() function with O_DIRECT and handles the memory alignment for both read and write operations.
In some cases direct access to devices is necessary and that was the main reason for this patch to be done.

The main reason why this was done is that pygrub (used by xen virtualization user-space package, it's a python version of grub for paravirtualized guests) sometimes uses outdated version of grub.conf file. Modifications to xen package were *not* enough because e2fsprogs doesn't open the files directly. That's why I added EXT2_FLAG_DIRECT support to make read/write operations work directly when passed. It's been tested with pygrub like mentioned above for read operation and it's working fine.

Quoted 7 lines

Quoted 1 line

Signed-off-by Michal Novotny <[email protected]>

Can you add to this some kind of data flow overview? Seems like a really odd way to update this file system...

ric

What do you mean by adding some kind of data flow overview? I am not working on file systems but I needed option open files directly to be added to e2fsprogs to make pygrub working right with not using cached/outdated data so this was the first patch I did for file system so I don't realy know what do you mean by data flow overview ...

Thanks,
Michal

Quoted 25 lines · Michal Novotny

Quoted 17 lines · Ric Wheeler

Quoted 16 lines · Michal Novotny

This patch extends functionality of e{2|4}fsprogs to add EXT2_FLAG_DIRECT flag to be passed to ext2fs_open2() function. This internally calls open() function with O_DIRECT and handles the memory alignment for both read and write operations.
In some cases direct access to devices is necessary and that was the main reason for this patch to be done.

The main reason why this was done is that pygrub (used by xen virtualization user-space package, it's a python version of grub for paravirtualized guests) sometimes uses outdated version of grub.conf file. Modifications to xen package were *not* enough because e2fsprogs doesn't open the files directly. That's why I added EXT2_FLAG_DIRECT support to make read/write operations work directly when passed. It's been tested with pygrub like mentioned above for read operation and it's working fine.

Quoted 7 lines

Quoted 1 line

Signed-off-by Michal Novotny <[email protected]>

Can you add to this some kind of data flow overview? Seems like a really odd way to update this file system...

ric

Quoted 8 lines

What do you mean by adding some kind of data flow overview? I am not working on file systems but I needed option open files directly to be added to e2fsprogs to make pygrub working right with not using cached/outdated data so this was the first patch I did for file system so I don't realy know what do you mean by data flow overview ...

Thanks,
Michal

Hi Michael,

What would be useful is where the data is written (guest/host) and the state of the file system (mounted/unmounted). Which app it is that writes the data that gets cached, who needs to read it and misses that cached data.

From the above, it is really not clear that you should not be able to use existing interfaces to flush any cached data to disk....

ric

I don't really know if I see your point but the thing here is that there was no way to open a file directly (ie. using O_DIRECT). The direct write support has been added only to make it possible to use both read and write directly. The main reason to create this patch was to add

direct read support and flush capability won't help me at all. I am working in Red Hat, Virtualization team on Xen so I am really not that much familiar with file systems but what I needed was an option to read the data directly (using O_DIRECT) in e2fsprogs. One bug was about pygrub (Python version of GRUB of Xen PV guests that is internally using e2fsprogs functionality to access data on ext2/3/4 partition to boot the PV guests) uses outdated/cached data so some modifications were necessary to open everything directly...

Quoted 4 lines · Michal Novotny

I don't really know if I see your point but the thing here is that there was no way to open a file directly (ie. using O_DIRECT). The direct write support has been added only to make it possible to use both read and write directly. The main reason to create this patch was to add

Quoted 9 lines

direct read support and flush capability won't help me at all. I am working in Red Hat, Virtualization team on Xen so I am really not that much familiar with file systems but what I needed was an option to read the data directly (using O_DIRECT) in e2fsprogs. One bug was about pygrub (Python version of GRUB of Xen PV guests that is internally using e2fsprogs functionality to access data on ext2/3/4 partition to boot the PV guests) uses outdated/cached data so some modifications were necessary to open everything directly...

So to get things staigt: you're using e2fsprogs to manipulate a life filesystem and thing using O_DIRECT saves your ass? I think you need to rething your model of operation fundamentally in that case.

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

Quoted 10 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the host, except for initial setup long time ago

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

- once the guest is life it uses the extN kernel driver to access the filesystem

nowhere in this cycle you should have any stale cached data. The kernel always makes sure to write back data on umount/reboot, as does e2fsprogs if actually used to write data (which you said is not the case anyway).

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block device implementation, but these can't cause any problems.

Quoted 10 lines · Christoph Hellwig

Quoted 9 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

Quoted 5 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 2 lines

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

Correct.

Quoted 2 lines

- once the guest is life it uses the extN kernel driver to access the filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 3 lines

nowhere in this cycle you should have any stale cached data. The kernel always makes sure to write back data on umount/reboot, as does e2fsprogs if actually used to write data (which you said is not the case anyway).

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 3 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block device implementation, but these can't cause any problems.

Michal

Quoted 41 lines · Michal Novotny

Quoted 10 lines · Christoph Hellwig

Quoted 9 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

Quoted 5 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 2 lines

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

Correct.

Quoted 2 lines

- once the guest is life it uses the extN kernel driver to access the filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 3 lines

nowhere in this cycle you should have any stale cached data. The kernel always makes sure to write back data on umount/reboot, as does e2fsprogs if actually used to write data (which you said is not the case anyway).

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 3 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block device implementation, but these can't cause any problems.

Quoted 1 line

Michal

If the guest is the only one (when running) that installs a new grub.conf file and kernel and it shuts down properly, you should be good. It if does not shut down cleanly, it could have a stale grub.conf file (or worse, a partially written one), but using O_DIRECT to bypass the file system cache should not help.

If we cannot reproduce this failure, sounds like we need to go back and get a better understanding of what the customer saw?

ric

Quoted 44 lines · Ric Wheeler

Quoted 42 lines · Michal Novotny

Quoted 10 lines · Christoph Hellwig

Quoted 9 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

Quoted 5 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 2 lines

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

Correct.

Quoted 2 lines

- once the guest is life it uses the extN kernel driver to access the filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 3 lines

nowhere in this cycle you should have any stale cached data. The kernel always makes sure to write back data on umount/reboot, as does e2fsprogs if actually used to write data (which you said is not the case anyway).

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 3 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block device implementation, but these can't cause any problems.

Michal

Quoted 6 lines

If the guest is the only one (when running) that installs a new grub.conf file and kernel and it shuts down properly, you should be good. It if does not shut down cleanly, it could have a stale grub.conf file (or worse, a partially written one), but using O_DIRECT to bypass the file system cache should not help.

If we cannot reproduce this failure, sounds like we need to go back and get a better understanding of what the customer saw?

ric

That's right. I am going write an e-mail regarding this information to the reproducer if this bug and tell him that I need more information about what's happening at the customer side.

Quoted 60 lines · Michal Novotny

Quoted 59 lines · Ric Wheeler

Quoted 43 lines · Michal Novotny

Quoted 15 lines · Christoph Hellwig

Quoted 6 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for
PV guest and after this is read and selected in pygrub then the guest is

Quoted 8 lines

booted using the kernel and initrd extracted from the image (after which
the file is closed). Once again, nothing uses write support and it was
added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the
only way to make it use non-cached data.

Quoted 5 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 2 lines

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

Correct.

Quoted 2 lines

- once the guest is life it uses the extN kernel driver to access the filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 6 lines

nowhere in this cycle you should have any stale cached data. The kernel
always makes sure to write back data on umount/reboot, as does e2fsprogs
if actually used to write data (which you said is not the case anyway).

Quoted 8 lines

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 4 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block
device implementation, but these can't cause any problems.

Michal

If the guest is the only one (when running) that installs a new grub.conf file and kernel and it shuts down properly, you should be good. It if does not shut down cleanly, it could have a stale grub.conf file (or worse, a partially written one), but using O_DIRECT to bypass the file system cache should not help.

Quoted 9 lines

Quoted 5 lines

If we cannot reproduce this failure, sounds like we need to go back and get a better understanding of what the customer saw?

ric

That's right. I am going write an e-mail regarding this information to the reproducer if this bug and tell him that I need more information about what's happening at the customer side.

One more thing to point out, let's have a look at:
https://bugzilla.redhat.com/show_bug.cgi?id=466681#c15 .This is about workaround to drop caches to be added to pygrub in the host machine using this command:

echo 1> /proc/sys/vm/drop_caches

So this really looks like the caching issue if it's working fine after dropping the caches. That may be the reason why this could be fine with this patch present in e2fsprogs.

Michal

Quoted 83 lines · Michal Novotny

Quoted 62 lines · Michal Novotny

Quoted 61 lines · Ric Wheeler

Quoted 45 lines · Michal Novotny

Quoted 15 lines · Christoph Hellwig

Quoted 6 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for
PV guest and after this is read and selected in pygrub then the guest is

Quoted 8 lines

booted using the kernel and initrd extracted from the image (after which
the file is closed). Once again, nothing uses write support and it was
added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the
only way to make it use non-cached data.

Quoted 6 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the
host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 2 lines

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

Correct.

Quoted 3 lines

- once the guest is life it uses the extN kernel driver to access the
filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 6 lines

nowhere in this cycle you should have any stale cached data. The kernel
always makes sure to write back data on umount/reboot, as does e2fsprogs
if actually used to write data (which you said is not the case anyway).

Quoted 8 lines

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 4 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block
device implementation, but these can't cause any problems.

Michal

If the guest is the only one (when running) that installs a new grub.conf file and kernel and it shuts down properly, you should be good. It if does not shut down cleanly, it could have a stale grub.conf file (or worse, a partially written one), but using O_DIRECT to bypass the file system cache should not help.

Quoted 9 lines

Quoted 5 lines

If we cannot reproduce this failure, sounds like we need to go back and get a better understanding of what the customer saw?

ric

That's right. I am going write an e-mail regarding this information to the reproducer if this bug and tell him that I need more information about what's happening at the customer side.

One more thing to point out, let's have a look at:
https://bugzilla.redhat.com/show_bug.cgi?id=466681#c15 .This is about workaround to drop caches to be added to pygrub in the host machine using this command:

echo 1> /proc/sys/vm/drop_caches

So this really looks like the caching issue if it's working fine after dropping the caches. That may be the reason why this could be fine with this patch present in e2fsprogs.

Quoted 1 line

Michal

That BZ has a pretty long and twisted history, but after a quick read, I still don't see why a cleanly shutdown guest would have issues with caching that using O_DIRECT on read would help.

We will need to dig into a bit more...

ric

Quoted 88 lines · Ric Wheeler

Quoted 77 lines · Michal Novotny

Quoted 62 lines · Michal Novotny

Quoted 61 lines · Ric Wheeler

Quoted 45 lines · Michal Novotny

Quoted 15 lines · Christoph Hellwig

Quoted 6 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for
PV guest and after this is read and selected in pygrub then the guest is

Quoted 8 lines

booted using the kernel and initrd extracted from the image (after which
the file is closed). Once again, nothing uses write support and it was
added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the
only way to make it use non-cached data.

Quoted 6 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the
host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 2 lines

- before booting a guest your "pygrub" tools needs to read files on it, and it's doing so using e2fsprogs

Correct.

Quoted 3 lines

- once the guest is life it uses the extN kernel driver to access the
filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 6 lines

nowhere in this cycle you should have any stale cached data. The kernel
always makes sure to write back data on umount/reboot, as does e2fsprogs
if actually used to write data (which you said is not the case anyway).

Quoted 8 lines

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 4 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block
device implementation, but these can't cause any problems.

Michal

If the guest is the only one (when running) that installs a new grub.conf file and kernel and it shuts down properly, you should be good. It if does not shut down cleanly, it could have a stale grub.conf file (or worse, a partially written one), but using O_DIRECT to bypass the file system cache should not help.

Quoted 9 lines

Quoted 5 lines

If we cannot reproduce this failure, sounds like we need to go back and get a better understanding of what the customer saw?

ric

That's right. I am going write an e-mail regarding this information to the reproducer if this bug and tell him that I need more information about what's happening at the customer side.

One more thing to point out, let's have a look at:
https://bugzilla.redhat.com/show_bug.cgi?id=466681#c15 .This is about workaround to drop caches to be added to pygrub in the host machine using this command:

Quoted 7 lines

echo 1> /proc/sys/vm/drop_caches

So this really looks like the caching issue if it's working fine after dropping the caches. That may be the reason why this could be fine with this patch present in e2fsprogs.

Michal

That BZ has a pretty long and twisted history, but after a quick read, I still don't see why a cleanly shutdown guest would have issues with caching that using O_DIRECT on read would help.

Quoted 4 lines

We will need to dig into a bit more...

ric

I am not saying we don't need to dig a little bit more, we surely do but unfortunately I am waiting for information from reporter. But I am also thinking that this O_DIRECT functionality support to bypass caches could be useful...

Thanks,
Michal

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 96 lines · Michal Novotny

Quoted 85 lines · Ric Wheeler

Quoted 84 lines · Michal Novotny

Quoted 63 lines · Michal Novotny

Quoted 62 lines · Ric Wheeler

Quoted 46 lines · Michal Novotny

Quoted 15 lines · Christoph Hellwig

Quoted 6 lines · Michal Novotny

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for
PV guest and after this is read and selected in pygrub then the guest is

Quoted 8 lines

booted using the kernel and initrd extracted from the image (after which
the file is closed). Once again, nothing uses write support and it was
added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the
only way to make it use non-cached data.

Quoted 6 lines

So what caches get in the way? From the above it seems the situation is the following:

- filesystem N is a guest filesystem. It's not usually mounted on the
host, except for initial setup long time ago

Yes, it is really a guest file system. This is not mounted in the host and the reason is to get actual version of grub.conf, initrd and kernel to be booted...

Quoted 3 lines

- before booting a guest your "pygrub" tools needs to read files on
it, and it's doing so using e2fsprogs

Correct.

Quoted 3 lines

- once the guest is life it uses the extN kernel driver to access the
filesystem

That's right. So this is no longer pygrub responsibility...

Quoted 6 lines

nowhere in this cycle you should have any stale cached data. The kernel
always makes sure to write back data on umount/reboot, as does e2fsprogs
if actually used to write data (which you said is not the case anyway).

Quoted 8 lines

In fact I was unable to run into those problems myself but reporter/customer did.

Quoted 4 lines

The only data that may be in the cache are unmodified data from reads on the block device from either e2fsprogs or a suboptimal virtual block
device implementation, but these can't cause any problems.

Michal

If the guest is the only one (when running) that installs a new grub.conf file and kernel and it shuts down properly, you should be good. It if does not shut down cleanly, it could have a stale grub.conf file (or worse, a partially written one), but using O_DIRECT to bypass the file system cache should not help.

Quoted 9 lines

Quoted 5 lines

If we cannot reproduce this failure, sounds like we need to go back and get a better understanding of what the customer saw?

ric

That's right. I am going write an e-mail regarding this information to the reproducer if this bug and tell him that I need more information about what's happening at the customer side.

One more thing to point out, let's have a look at:
https://bugzilla.redhat.com/show_bug.cgi?id=466681#c15 .This is about workaround to drop caches to be added to pygrub in the host machine using this command:

echo 1> /proc/sys/vm/drop_caches

So this really looks like the caching issue if it's working fine after dropping the caches. That may be the reason why this could be fine with this patch present in e2fsprogs.

Quoted 8 lines

Quoted 1 line

Michal

That BZ has a pretty long and twisted history, but after a quick read, I still don't see why a cleanly shutdown guest would have issues with caching that using O_DIRECT on read would help.

We will need to dig into a bit more...

ric

I am not saying we don't need to dig a little bit more, we surely do but unfortunately I am waiting for information from reporter. But I am also thinking that this O_DIRECT functionality support to bypass caches could be useful...

Quoted 7 lines

Thanks,
Michal

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

I can not see where the cache could cause this problem but is it possible that it is in the Host file system rather than than the guest where it is causing a problem;

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

Quoted 3 lines · Michal Novotny

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

Yes, the Xen blkback driver just submits I/O directly without using the right interfaces to force cache coherency. It might be relatively easy to hack a call in to flush all caches when it starts up, but given how it bypasses all abstractions it's almost impossible to give full coherency as if using the normal block device interfaces.

Quoted 9 lines · Christoph Hellwig

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

Quoted 3 lines

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

Quoted 10 lines · Michal Novotny

Quoted 9 lines · Christoph Hellwig

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

Quoted 5 lines

Quoted 3 lines

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

If XEN drivers by pass the normal IO and FS stack on the host, then I can understand why the hack to e2fsprogs works but it does not seem like a good fix.

Specifically, the data will continue to be cached (and if dirty, might be written back to the storage eventually).

If we need a work around, you need to drop VM caches for that device before you update the guest's files and possibly again afterwards (and make sure that nothing pulls the data into cache during the operation).

Basically, this sounds like the backend drivers are doing something really, really dangerous....

ric

Quoted 31 lines · Ric Wheeler

Quoted 10 lines · Michal Novotny

Quoted 9 lines · Christoph Hellwig

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

Quoted 5 lines

Quoted 3 lines

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

If XEN drivers by pass the normal IO and FS stack on the host, then I can understand why the hack to e2fsprogs works but it does not seem like a good fix.

Specifically, the data will continue to be cached (and if dirty, might be written back to the storage eventually).

If we need a work around, you need to drop VM caches for that device before you update the guest's files and possibly again afterwards (and make sure that nothing pulls the data into cache during the operation).

Basically, this sounds like the backend drivers are doing something really, really dangerous....

Quoted 2 lines

ric

Ok, so you think this is not good to do this patch for e2fsprogs for direct access support? The only things we could do now is to fix the backend drivers or create a workaround to drop caches? I need to discuss this further with guys in my team...

Thanks,
Michal

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 32 lines · Michal Novotny

Quoted 31 lines · Ric Wheeler

Quoted 10 lines · Michal Novotny

Quoted 9 lines · Christoph Hellwig

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

Quoted 5 lines

Quoted 3 lines

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

If XEN drivers by pass the normal IO and FS stack on the host, then I can understand why the hack to e2fsprogs works but it does not seem like a good fix.

Specifically, the data will continue to be cached (and if dirty, might be written back to the storage eventually).

If we need a work around, you need to drop VM caches for that device before you update the guest's files and possibly again afterwards (and make sure that nothing pulls the data into cache during the operation).

Basically, this sounds like the backend drivers are doing something really, really dangerous....

Quoted 6 lines

Quoted 2 lines

ric

Ok, so you think this is not good to do this patch for e2fsprogs for direct access support? The only things we could do now is to fix the backend drivers or create a workaround to drop caches? I need to discuss this further with guys in my team...

I do think that patching it up in e2fsprogs is unnecessarily invasive; it's fixing it at the wrong spot.

Any block dev IO from the host is dangerous; fixing it only in e2fsprogs for this one case doesn't seem like the right course of action.

-Eric

Quoted 46 lines · Eric Sandeen

Quoted 32 lines · Michal Novotny

Quoted 31 lines · Ric Wheeler

Quoted 10 lines · Michal Novotny

Quoted 9 lines · Christoph Hellwig

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

Quoted 5 lines

Quoted 3 lines

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

If XEN drivers by pass the normal IO and FS stack on the host, then I can understand why the hack to e2fsprogs works but it does not seem like a good fix.

Specifically, the data will continue to be cached (and if dirty, might be written back to the storage eventually).

If we need a work around, you need to drop VM caches for that device before you update the guest's files and possibly again afterwards (and make sure that nothing pulls the data into cache during the operation).

Basically, this sounds like the backend drivers are doing something really, really dangerous....

Quoted 6 lines

Quoted 2 lines

ric

Ok, so you think this is not good to do this patch for e2fsprogs for direct access support? The only things we could do now is to fix the backend drivers or create a workaround to drop caches? I need to discuss this further with guys in my team...

I do think that patching it up in e2fsprogs is unnecessarily invasive; it's fixing it at the wrong spot.

Any block dev IO from the host is dangerous; fixing it only in e2fsprogs for this one case doesn't seem like the right course of action.

Quoted 1 line

-Eric

It actually could produce some nastier issues where it would work a bit, the bad data gets flushed back to the backing store and then your O_DIRECT read would be broken.

Also, for normal users of e2fsprogs, they should never bypass the cache...

ric

Quoted 47 lines · Ric Wheeler

Quoted 46 lines · Eric Sandeen

Quoted 32 lines · Michal Novotny

Quoted 31 lines · Ric Wheeler

Quoted 10 lines · Michal Novotny

Quoted 9 lines · Christoph Hellwig

Ok, I looked at the issue. The problem is that the Xen backend drivers are (as expected) utterly braindead and submit bios directly from the virtualization backed without using proper abstractions and thus bypassing all the cache coherency features in the fileystems (the block device nodes are just another mini-filesystem in that respect). So when you first have buffered access in the host pages may stay in cache and get overwritten directly on disk by a Xen guest, and once the guest is down the host may still use the now stale cached data.

Quoted 5 lines

Quoted 3 lines

I would recommend to migrate your cutomers to KVM which uses the proper abtractions and thus doesn't have this problem. There's a reason after all why all the Xen dom0 mess never got merged to mainline.

So, do you think the problem is in the Xen backend drivers and to make it working right in Xen the driver fix is needed?

If XEN drivers by pass the normal IO and FS stack on the host, then I can understand why the hack to e2fsprogs works but it does not seem like a good fix.

Specifically, the data will continue to be cached (and if dirty, might be written back to the storage eventually).

If we need a work around, you need to drop VM caches for that device before you update the guest's files and possibly again afterwards (and make sure that nothing pulls the data into cache during the operation).

Basically, this sounds like the backend drivers are doing something really, really dangerous....

Quoted 6 lines

Quoted 2 lines

ric

Ok, so you think this is not good to do this patch for e2fsprogs for direct access support? The only things we could do now is to fix the backend drivers or create a workaround to drop caches? I need to discuss this further with guys in my team...

I do think that patching it up in e2fsprogs is unnecessarily invasive; it's fixing it at the wrong spot.

Any block dev IO from the host is dangerous; fixing it only in e2fsprogs for this one case doesn't seem like the right course of action.

Quoted 7 lines

Quoted 1 line

-Eric

It actually could produce some nastier issues where it would work a bit, the bad data gets flushed back to the backing store and then your O_DIRECT read would be broken.

Also, for normal users of e2fsprogs, they should never bypass the cache...

ric

Ok, good to know the normal users should never bypass the cache. That way this seems to be some kind of kernel bug...

Quoted 4 lines · Christoph Hellwig

So to get things staigt: you're using e2fsprogs to manipulate a life filesystem and thing using O_DIRECT saves your ass? I think you need to rething your model of operation fundamentally in that case.

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

Quoted 6 lines · Michal Novotny

Quoted 4 lines · Christoph Hellwig

So to get things staigt: you're using e2fsprogs to manipulate a life filesystem and thing using O_DIRECT saves your ass? I think you need to rething your model of operation fundamentally in that case.

Quoted 1 line

Not really, pygrub doesn't do any manipulation with file system and also, it's not working on a life file system. It's called before the guest boots up to read information about grub.conf/initrd and kernel for PV guest and after this is read and selected in pygrub then the guest is booted using the kernel and initrd extracted from the image (after which the file is closed). Once again, nothing uses write support and it was added just to make it use O_DIRECT for both read and write operations but only pygrub uses only read support and O_DIRECT passed here is the only way to make it use non-cached data.

Michal, I think the thing that is confusing everyone is that if you are not accessing a live filesystem, and you are not doing the writes yourself, then why is it bad to read cached data? How is it that the cached data becomes stale if pygrub isn't modifying it, and there is nothing else mounting the filesystem?

Quoted 4 lines · Michal Novotny

I don't really know if I see your point but the thing here is that there was no way to open a file directly (ie. using O_DIRECT). The direct write support has been added only to make it possible to use both read and write directly. The main reason to create this patch was to add

Quoted 8 lines

direct read support and flush capability won't help me at all. I am working in Red Hat, Virtualization team on Xen so I am really not that much familiar with file systems but what I needed was an option to read the data directly (using O_DIRECT) in e2fsprogs. One bug was about pygrub (Python version of GRUB of Xen PV guests that is internally using e2fsprogs functionality to access data on ext2/3/4 partition to boot the PV guests) uses outdated/cached data so some modifications were necessary to open everything directly...

So to get things staigt: you're using e2fsprogs to manipulate a life filesystem and thing using O_DIRECT saves your ass? I think you need to rething your model of operation fundamentally in that case.

Quoted 15 lines · Christoph Hellwig

Quoted 13 lines · Michal Novotny

I don't really know if I see your point but the thing here is that there was no way to open a file directly (ie. using O_DIRECT). The direct write support has been added only to make it possible to use both read and write directly. The main reason to create this patch was to add direct read support and flush capability won't help me at all. I am working in Red Hat, Virtualization team on Xen so I am really not that much familiar with file systems but what I needed was an option to read the data directly (using O_DIRECT) in e2fsprogs. One bug was about pygrub (Python version of GRUB of Xen PV guests that is internally using e2fsprogs functionality to access data on ext2/3/4 partition to boot the PV guests) uses outdated/cached data so some modifications were necessary to open everything directly...

Quoted 5 lines

So to get things staigt: you're using e2fsprogs to manipulate a life filesystem and thing using O_DIRECT saves your ass? I think you need to rething your model of operation fundamentally in that case.

Christoph -

It's my understanding that nobody is doing concurrent access.

If the host reads the block device via pygrub to boot the guest, the guest while running updates the same device when installing a new kernel (either through the fs, or by writing to the bdev; probably the former...), and then the guest shuts down - is there something which will sync the host's cache (cached from the prior read) with the updates from the guest?