getattr() results ignored when writeback cache is active
Hi,
I'm having another problem with FUSE's writeback cache in SSHFS.
As far as I can tell, the FUSE kernel module issues getattr() requests, but then silently discards the reported mtime and file size.
For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.
To me this looks like a bug.. am I missing something?
Steps to reproduce (with SSHFS from Git master at
https://github.com/libfuse/sshfs/ and the debugging patch from
https://github.com/libfuse/sshfs/issues/93):
mkdir /tmp/issue_93
touch /tmp/issue_93/file_1
mkdir /tmp/issue_93_mnt
dd if=/dev/urandom count=1 bs=2048 |base64 > /tmp/issue_93/file_1 2>/dev/null
./sshfs -f -o attr_timeout=0,entry_timeout=0,dir_cache=no,writeback_cache=yes \
localhost:/tmp/issue_93/ /tmp/issue_93_mnt/ &sleep 1
stat --format='Kernel: mtime %Y, size %s for %n' /tmp/issue_93_mnt/file_1 stat --format='Local: mtime %Y, size %s for %n' /tmp/issue_93/file_1
sleep 1
echo "Changing file.."
echo -e "\nrevision 2" >> /tmp/issue_93/file_1
stat --format='Kernel: mtime %Y, size %s for %n' /tmp/issue_93_mnt/file_1 stat --format='Local: mtime %Y, size %s for %n' /tmp/issue_93/file_1
fusermount -u /tmp/issue_93_mnt/
rm -r /tmp/issue_93 /tmp/issue_93_mnt
,----
| $ ../issue_93.sh | 1+0 records in | 1+0 records out | 2048 bytes (2.0 kB, 2.0 KiB) copied, 5.3011e-05 s, 38.6 MB/s | SSHFS: mtime 0, size 0 for /.Trash | SSHFS: mtime 0, size 0 for /.Trash-1000 | SSHFS: mtime 1505908107, size 2768 for /file_1 | SSHFS: mtime 1505908107, size 2768 for /file_1 | Kernel: mtime 1505908107, size 2768 for /tmp/issue_93_mnt/file_1 | Local: mtime 1505908107, size 2768 for /tmp/issue_93/file_1 | Changing file.. | SSHFS: mtime 1505908109, size 2780 for /file_1 | SSHFS: mtime 1505908109, size 2780 for /file_1 | Kernel: mtime 1505908107, size 2768 for /tmp/issue_93_mnt/file_1 | Local: mtime 1505908109, size 2780 for /tmp/issue_93/file_1
`----
Note how sshfs' getattr handler returns the correct mtime and ctime for the second stat() call, but the kernel returns the old values.
This is with kernel 4.9.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 4 lines · Nikolaus Rath
Hi,
I'm having another problem with FUSE's writeback cache in SSHFS.
Quoted 8 lines
As far as I can tell, the FUSE kernel module issues
getattr()requests, but then silently discards the reported mtime and file size.For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.
To me this looks like a bug.. am I missing something?
Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.
Apparently the above is not documented anywhere :(
Thanks,
Miklos
Quoted 12 lines · Miklos Szeredi
Quoted 11 lines · Nikolaus Rath
Hi,
I'm having another problem with FUSE's writeback cache in SSHFS.
As far as I can tell, the FUSE kernel module issues
getattr()requests, but then silently discards the reported mtime and file size.For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.
Quoted 6 lines
Quoted 1 line
To me this looks like a bug.. am I missing something?
Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.
Apparently the above is not documented anywhere :(
Ouch.
I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.
(This actually applies to large chunk of information that's currently only in the libfuse documentation).
Any chance of that happening? I understand that bringing Documentation/filesystems/fuse.txt up to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a new Documentation/filesystems/fuse/writeback.txt file? Together with the requirement that the filesystem has to support reading from files opened O_WRONLY?
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 31 lines · Nikolaus Rath
Quoted 12 lines · Miklos Szeredi
Quoted 11 lines · Nikolaus Rath
Hi,
I'm having another problem with FUSE's writeback cache in SSHFS.
As far as I can tell, the FUSE kernel module issues
getattr()requests, but then silently discards the reported mtime and file size.For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.
Quoted 6 lines
Quoted 1 line
To me this looks like a bug.. am I missing something?
Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.
Apparently the above is not documented anywhere :(
Ouch.
I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.
(This actually applies to large chunk of information that's currently only in the libfuse documentation).
Quoted 6 lines
Any chance of that happening? I understand that bringing
Documentation/filesystems/fuse.txtup to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a newDocumentation/filesystems/fuse/writeback.txtfile? Together with the requirement that the filesystem has to support reading from files openedO_WRONLY?
Something like this?
Fuse supports the following writeback modes:
- direct-io
- write-through
- writeback-cache
In direct-io mode (enabled by the FOPEN_DIRECT_IO flag) the page cache is completely bypassed for reads and writes.
In write-through mode (default) each write modifies the page cache as well as immediately being sent to userspace.
In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs. If a partial page is written, then the page needs to be first read from userspace. This means, thet even for files opened for O_WRONLY it is possible that read requests will be generated by the kernel.
Thanks,
Miklos
Quoted 16 lines · Miklos Szeredi
Quoted 16 lines
Quoted 4 lines
Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.
Apparently the above is not documented anywhere :(
Ouch.
I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.
(This actually applies to large chunk of information that's currently only in the libfuse documentation).
Quoted 8 lines
Quoted 6 lines
Any chance of that happening? I understand that bringing
Documentation/filesystems/fuse.txtup to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a newDocumentation/filesystems/fuse/writeback.txtfile? Together with the requirement that the filesystem has to support reading from files openedO_WRONLY?Something like this?
Anything is better than nothing. Well, almost anything, but that definitely includes your draft :-).
Quoted 1 line
Fuse supports the following writeback modes:
"write modes", or maybe "access modes", not "writeback modes", right?
Quoted 6 lines
- direct-io
- write-through
- writeback-cacheIn direct-io mode (enabled by the
FOPEN_DIRECT_IOflag) the page cache is completely bypassed for reads and writes.
I'd add "No read-ahead takes place." (if that's true)
Quoted 3 lines
In write-through mode (default) each write modifies the page cache as well as immediately being sent to userspace.
Reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the page-cache.
Quoted 2 lines
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to
Typo in "flaga"
Quoted 3 lines
the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests.
I think it would be great to be more specific than "later". Maybe "when the file is no longer opened by any process, when explicitly requested by an fsync() call, or when the dirty data is evicted from the cache by the kernel for other reasons" (if that's correct).
Quoted 5 lines
This
mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs. If a partial page is written, then the page needs to be first read from userspace. This means, thet even for files
"that", not "thet"
Quoted 2 lines
opened for
O_WRONLYit is possible that read requests will be generated by the kernel.
Nice!
Thanks!
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.
.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 24 lines · Nikolaus Rath
Quoted 23 lines · Miklos Szeredi
Quoted 22 lines
Quoted 4 lines
Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.
Apparently the above is not documented anywhere :(
Ouch.
I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.
(This actually applies to large chunk of information that's currently only in the libfuse documentation).
Any chance of that happening? I understand that bringing
Documentation/filesystems/fuse.txtup to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a newDocumentation/filesystems/fuse/writeback.txtfile? Together with the requirement that the filesystem has to support reading from files openedO_WRONLY?
Quoted 4 lines
Quoted 1 line
Something like this?
Anything is better than nothing. Well, almost anything, but that definitely includes your draft :-).
Thanks for your feedback. Fixed and expanded version:
Fuse supports the following I/O modes:
- direct-io
- cached
+ write-through
+ writeback-cache
The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the FUSE_OPEN reply.
In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.
In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.
The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the FUSE_INIT reply.
In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.
In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by fsync(2)). This mode assumes that the file is never changed outside the mounted filesystem (the size time attributes are kept uptodate by the kernel), so it's not suitable for any any network filesystem even if no write operations are actually carried out. If a partial page is written, then the page needs to be first read from userspace. This means, that even for files opened for O_WRONLY it is possible that read requests will be generated by the kernel.
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
Quoted 5 lines
Best,
-Nikolaus
Quoted 11 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Quoted 7 lines
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 29 lines · Miklos Szeredi
Thanks for your feedback. Fixed and expanded version:
Fuse supports the following I/O modes:
- direct-io
- cached
+ write-through
+ writeback-cacheThe direct-io mode can be selected with the
FOPEN_DIRECT_IOflag in theFUSE_OPENreply.In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.
In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.
The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the
FUSE_WRITEBACK_CACHEflag in theFUSE_INITreply.In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.
Quoted 4 lines
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by fsync(2)).
I think it would be useful to mention if the cache is flushed when the file is not opened anymore, or (if not) to explicitly say that that's not the case.
Quoted 2 lines
This mode assumes that the file is never changed outside the mounted filesystem
This sounds a little awkward. How about "assumes that all changes to the filesystem go through the FUSE kernel module"
Quoted 1 line
(the size time attributes are kept uptodate by the kernel),
"size and atime/ctime/mtime attributes", and I think "uptodate" should be up-to-date.
Quoted 1 line
so it's not suitable for any any network filesystem
duplicated "any"
I think this is great. I may work on similar documentation for some of the other stuff that currently lives only in include/fuse*.h
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Quoted 7 lines
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
Quoted 4 lines
In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).
In your example you modified file bypassing fuse mount:
Quoted 1 line
echo -e "\nrevision 2" >>
/tmp/issue_93/file_1
That's like mangling block device while ordinary local fs is running on top of it.
Quoted 4 lines
Best,
-Nikolaus
Quoted 25 lines · Maxim Patlasov
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Quoted 7 lines
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
Quoted 4 lines
In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).
Quoted 6 lines
In your example you modified file bypassing fuse mount:
Quoted 1 line
echo -e "\nrevision 2" >>
/tmp/issue_93/file_1That's like mangling block device while ordinary local fs is running on top of it.
Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 25 lines · Nikolaus Rath
Quoted 24 lines · Maxim Patlasov
Quoted 22 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Quoted 7 lines
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).
Quoted 8 lines
Quoted 5 lines
In your example you modified file bypassing fuse mount:
Quoted 1 line
echo -e "\nrevision 2" >>
/tmp/issue_93/file_1That's like mangling block device while ordinary local fs is running on top of it.
Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.
In your example the change didn't come through any FUSE layer. When another host writes to a network filesystem, the filesystem has a choice to process the request intelligently. In your example, you didn't give FUSE a chance because "/tmp/issue_93" was not fuse mount.
Quoted 4 lines
Best,
-Nikolaus
Quoted 29 lines · Maxim Patlasov
Quoted 27 lines · Nikolaus Rath
Quoted 26 lines · Maxim Patlasov
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Quoted 7 lines
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
Quoted 4 lines
In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).
In your example you modified file bypassing fuse mount:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
echo -e "\nrevision 2" >>
/tmp/issue_93/file_1That's like mangling block device while ordinary local fs is running on top of it.
Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.
In your example the change didn't come through any FUSE layer.
It did. The filesystem reported the updated mtime and size in its getattr() response. The kernel ignored it.
Quoted 3 lines
When another host writes to a network filesystem, the filesystem has a choice to process the request intelligently. In your example, you didn't give FUSE a chance because "
/tmp/issue_93" was not fuse mount.
Right. /tmp/issue_93 substitutes for the network. The filesystem gets notification through the network, aka by reading this mountpoint. It then attempts to pass this on to the FUSE layer, but is being ignored.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 40 lines · Nikolaus Rath
Quoted 29 lines · Maxim Patlasov
Quoted 27 lines · Nikolaus Rath
Quoted 26 lines · Maxim Patlasov
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs..."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".
Quoted 7 lines
Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.
Quoted 4 lines
In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).
In your example you modified file bypassing fuse mount:
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
echo -e "\nrevision 2" >>
/tmp/issue_93/file_1That's like mangling block device while ordinary local fs is running on top of it.
Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.
In your example the change didn't come through any FUSE layer.
It did. The filesystem reported the updated mtime and size in its
getattr()response. The kernel ignored it.
Quoted 6 lines
Quoted 3 lines
When another host writes to a network filesystem, the filesystem has a choice to process the request intelligently. In your example, you didn't give FUSE a chance because "
/tmp/issue_93" was not fuse mount.Right.
/tmp/issue_93substitutes for the network. The filesystem gets notification through the network, aka by reading this mountpoint. It then attempts to pass this on to the FUSE layer, but is being ignored.
Oh, I understand now. Your example actually demonstrates that FUSE_WRITEBACK_CACHE must not be used if it's possible to modify underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. That's correct. But initial statement:
Quoted 1 line
this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out
is not, because it's not impossible to protect underlying data against such an external change somehow. A network filesystem is not obliged to keep transparent correspondence like /tmp/issue_93_mnt/file_1 <--> /tmp/issue_93/file_1. It can keep all user data (and metadata) in internal structures making any external modifications impossible. Then, implementing exclusive write semantics would make using FUSE_WRITEBACK_CACHE safe. Agree?
Quoted 5 lines
Best,
-Nikolaus
Updated version.
Thanks,
Miklos
---
Fuse supports the following I/O modes:
- direct-io
- cached
+ write-through
+ writeback-cache
The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the FUSE_OPEN reply.
In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.
In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.
The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the FUSE_INIT reply.
In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.
In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and when the last ref to the file is being released on munmap(2)). This mode assumes that all changes to the filesystem go through the FUSE kernel module (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so it's generally not suitable for network filesystems. If a partial page is written, then the page needs to be first read from userspace. This means, that even for files opened for O_WRONLY it is possible that READ requests will be generated by the kernel.
Quoted 27 lines · Miklos Szeredi
Fuse supports the following I/O modes:
- direct-io
- cached
+ write-through
+ writeback-cacheThe direct-io mode can be selected with the
FOPEN_DIRECT_IOflag in theFUSE_OPENreply.In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.
In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.
The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the
FUSE_WRITEBACK_CACHEflag in theFUSE_INITreply.In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.
Quoted 11 lines
In writeback-cache mode (enabled by the
FUSE_WRITEBACK_CACHEflag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and when the last ref to the file is being released on munmap(2)). This mode assumes that all changes to the filesystem go through the FUSE kernel module (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so it's generally not suitable for network filesystems. If a partial page is written, then the page needs to be first read from userspace. This means, that even for files opened forO_WRONLYit is possible that READ requests will be generated by the kernel.
Looks great to me.
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 8 lines · Maxim Patlasov
Oh, I understand now. Your example actually demonstrates that
FUSE_WRITEBACK_CACHEmust not be used if it's possible to modify underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. That's correct. But initial statement:Quoted 2 lines
this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out
Quoted 7 lines
is not, because it's not impossible to protect underlying data against such an external change somehow. A network filesystem is not obliged to keep transparent correspondence like
/tmp/issue_93_mnt/file_1<-->/tmp/issue_93/file_1. It can keep all user data (and metadata) in internal structures making any external modifications impossible. Then, implementing exclusive write semantics would make usingFUSE_WRITEBACK_CACHEsafe. Agree?
I agree with what you're saying, but I don't agree that this should go into the documentation for which the sentence in dispute was intended :-). I think in 93% of all cases, the simplified statement will be more helpful to a potential FUSE user than the correct one.
If this bothers you a lot, can we compromise and just be a little more vague overall? I.e.,
| This mode of operation is therefore generally not suitable for | filesystems where modifications may not come through the (local) FUSE | layer (i.e, most network filesystems).
Best,
-Nikolaus
Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«Quoted 20 lines · Nikolaus Rath
Quoted 7 lines · Maxim Patlasov
Oh, I understand now. Your example actually demonstrates that
FUSE_WRITEBACK_CACHEmust not be used if it's possible to modify underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. That's correct. But initial statement:Quoted 2 lines
this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out
Quoted 7 lines
is not, because it's not impossible to protect underlying data against such an external change somehow. A network filesystem is not obliged to keep transparent correspondence like
/tmp/issue_93_mnt/file_1<-->/tmp/issue_93/file_1. It can keep all user data (and metadata) in internal structures making any external modifications impossible. Then, implementing exclusive write semantics would make usingFUSE_WRITEBACK_CACHEsafe. Agree?I agree with what you're saying, but I don't agree that this should go into the documentation for which the sentence in dispute was intended :-). I think in 93% of all cases, the simplified statement will be more helpful to a potential FUSE user than the correct one.
Quoted 6 lines
If this bothers you a lot, can we compromise and just be a little more vague overall? I.e.,
| This mode of operation is therefore generally not suitable for | filesystems where modifications may not come through the (local) FUSE | layer (i.e, most network filesystems).
The way how Miklos conveyed it:
Quoted 4 lines
This mode
assumes that all changes to the filesystem go through the FUSE kernel module (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so it's generally not suitable for network filesystems.
looks perfect. Thank you!
Maxim