Skip to contentSkip to conversation

getattr() results ignored when writeback cache is active

fuse Nikolaus Rath 4 participants 19 messages Sep 20, 2017 – Sep 26, 2017 VFS

lore
Share & tools
lore
#0 lore raw

Hi,

I'm having another problem with FUSE's writeback cache in SSHFS.

As far as I can tell, the FUSE kernel module issues getattr() requests, but then silently discards the reported mtime and file size.

For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.

To me this looks like a bug.. am I missing something?

Steps to reproduce (with SSHFS from Git master at
https://github.com/libfuse/sshfs/ and the debugging patch from
https://github.com/libfuse/sshfs/issues/93):

mkdir /tmp/issue_93
touch /tmp/issue_93/file_1
mkdir /tmp/issue_93_mnt

dd if=/dev/urandom count=1 bs=2048 |base64 > /tmp/issue_93/file_1 2>/dev/null

./sshfs -f -o attr_timeout=0,entry_timeout=0,dir_cache=no,writeback_cache=yes \
        localhost:/tmp/issue_93/ /tmp/issue_93_mnt/ &

sleep 1
stat --format='Kernel: mtime %Y, size %s for %n' /tmp/issue_93_mnt/file_1 stat --format='Local: mtime %Y, size %s for %n' /tmp/issue_93/file_1

sleep 1
echo "Changing file.."
echo -e "\nrevision 2" >> /tmp/issue_93/file_1

stat --format='Kernel: mtime %Y, size %s for %n' /tmp/issue_93_mnt/file_1 stat --format='Local: mtime %Y, size %s for %n' /tmp/issue_93/file_1

fusermount -u /tmp/issue_93_mnt/

rm -r /tmp/issue_93 /tmp/issue_93_mnt

,----

| $ ../issue_93.sh 
| 1+0 records in
| 1+0 records out
| 2048 bytes (2.0 kB, 2.0 KiB) copied, 5.3011e-05 s, 38.6 MB/s
| SSHFS:  mtime 0, size 0 for /.Trash
| SSHFS:  mtime 0, size 0 for /.Trash-1000
| SSHFS:  mtime 1505908107, size 2768 for /file_1
| SSHFS:  mtime 1505908107, size 2768 for /file_1
| Kernel: mtime 1505908107, size 2768 for /tmp/issue_93_mnt/file_1
| Local:  mtime 1505908107, size 2768 for /tmp/issue_93/file_1
| Changing file..
| SSHFS:  mtime 1505908109, size 2780 for /file_1
| SSHFS:  mtime 1505908109, size 2780 for /file_1
| Kernel: mtime 1505908107, size 2768 for /tmp/issue_93_mnt/file_1
| Local:  mtime 1505908109, size 2780 for /tmp/issue_93/file_1

`----

Note how sshfs' getattr handler returns the correct mtime and ctime for the second stat() call, but the kernel returns the old values.

This is with kernel 4.9.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#1 lore raw
Quoted 4 lines · Nikolaus Rath

Hi,

I'm having another problem with FUSE's writeback cache in SSHFS.

Quoted 8 lines

As far as I can tell, the FUSE kernel module issues getattr() requests, but then silently discards the reported mtime and file size.

For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.

To me this looks like a bug.. am I missing something?

Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.

Apparently the above is not documented anywhere :(

Thanks,
Miklos

#2 lore raw
Quoted 12 lines · Miklos Szeredi
Quoted 11 lines · Nikolaus Rath

Hi,

I'm having another problem with FUSE's writeback cache in SSHFS.

As far as I can tell, the FUSE kernel module issues getattr() requests, but then silently discards the reported mtime and file size.

For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.

Quoted 6 lines
Quoted 1 line

To me this looks like a bug.. am I missing something?

Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.

Apparently the above is not documented anywhere :(

Ouch.

I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.

(This actually applies to large chunk of information that's currently only in the libfuse documentation).

Any chance of that happening? I understand that bringing Documentation/filesystems/fuse.txt up to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a new Documentation/filesystems/fuse/writeback.txt file? Together with the requirement that the filesystem has to support reading from files opened O_WRONLY?

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#3 lore raw
Quoted 31 lines · Nikolaus Rath
Quoted 12 lines · Miklos Szeredi
Quoted 11 lines · Nikolaus Rath

Hi,

I'm having another problem with FUSE's writeback cache in SSHFS.

As far as I can tell, the FUSE kernel module issues getattr() requests, but then silently discards the reported mtime and file size.

For SSHFS, this means that if a file has been accessed, and is then changed on the server, the changed attributes don't make it to the client and the file appears truncated or \0-filled.

Quoted 6 lines
Quoted 1 line

To me this looks like a bug.. am I missing something?

Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.

Apparently the above is not documented anywhere :(

Ouch.

I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.

(This actually applies to large chunk of information that's currently only in the libfuse documentation).

Quoted 6 lines

Any chance of that happening? I understand that bringing Documentation/filesystems/fuse.txt up to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a new Documentation/filesystems/fuse/writeback.txt file? Together with the requirement that the filesystem has to support reading from files opened O_WRONLY?

Something like this?

Fuse supports the following writeback modes:

- direct-io
- write-through
- writeback-cache

In direct-io mode (enabled by the FOPEN_DIRECT_IO flag) the page cache is completely bypassed for reads and writes.

In write-through mode (default) each write modifies the page cache as well as immediately being sent to userspace.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs. If a partial page is written, then the page needs to be first read from userspace. This means, thet even for files opened for O_WRONLY it is possible that read requests will be generated by the kernel.

Thanks,
Miklos

#4 lore raw
Quoted 16 lines · Miklos Szeredi
Quoted 16 lines
Quoted 4 lines

Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.

Apparently the above is not documented anywhere :(

Ouch.

I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.

(This actually applies to large chunk of information that's currently only in the libfuse documentation).

Quoted 8 lines
Quoted 6 lines

Any chance of that happening? I understand that bringing Documentation/filesystems/fuse.txt up to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a new Documentation/filesystems/fuse/writeback.txt file? Together with the requirement that the filesystem has to support reading from files opened O_WRONLY?

Something like this?

Anything is better than nothing. Well, almost anything, but that definitely includes your draft :-).

Quoted 1 line

Fuse supports the following writeback modes:

"write modes", or maybe "access modes", not "writeback modes", right?

Quoted 6 lines

- direct-io
- write-through
- writeback-cache

In direct-io mode (enabled by the FOPEN_DIRECT_IO flag) the page cache is completely bypassed for reads and writes.

I'd add "No read-ahead takes place." (if that's true)

Quoted 3 lines

In write-through mode (default) each write modifies the page cache as well as immediately being sent to userspace.

Reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the page-cache.

Quoted 2 lines

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to

Typo in "flaga"

Quoted 3 lines

the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests.

I think it would be great to be more specific than "later". Maybe "when the file is no longer opened by any process, when explicitly requested by an fsync() call, or when the dirty data is evicted from the cache by the kernel for other reasons" (if that's correct).

Quoted 5 lines

This
mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs. If a partial page is written, then the page needs to be first read from userspace. This means, thet even for files

"that", not "thet"

Quoted 2 lines

opened for O_WRONLY it is possible that read requests will be generated by the kernel.

Nice!

Thanks!
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#5 lore raw
Quoted 24 lines · Nikolaus Rath
Quoted 23 lines · Miklos Szeredi
Quoted 22 lines
Quoted 4 lines

Writeback cache assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs currently.

Apparently the above is not documented anywhere :(

Ouch.

I will of course put this into the libfuse documentation, but it would be much nicer if things like that could be documented somewhere in the kernel. After all, these are not properties of libfuse but the kernel module - and some filesystems use the fuse interface without using libfuse.

(This actually applies to large chunk of information that's currently only in the libfuse documentation).

Any chance of that happening? I understand that bringing Documentation/filesystems/fuse.txt up to date would be a major endeavour, but maybe one could just start by putting at least this information into e.g. a new Documentation/filesystems/fuse/writeback.txt file? Together with the requirement that the filesystem has to support reading from files opened O_WRONLY?

Quoted 4 lines
Quoted 1 line

Something like this?

Anything is better than nothing. Well, almost anything, but that definitely includes your draft :-).

Thanks for your feedback. Fixed and expanded version:

Fuse supports the following I/O modes:

- direct-io
- cached
+ write-through
+ writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by fsync(2)). This mode assumes that the file is never changed outside the mounted filesystem (the size time attributes are kept uptodate by the kernel), so it's not suitable for any any network filesystem even if no write operations are actually carried out. If a partial page is written, then the page needs to be first read from userspace. This means, that even for files opened for O_WRONLY it is possible that read requests will be generated by the kernel.

#6 lore raw
Quoted 29 lines · Miklos Szeredi

Thanks for your feedback. Fixed and expanded version:

Fuse supports the following I/O modes:

- direct-io
- cached
+ write-through
+ writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.

Quoted 4 lines

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by fsync(2)).

I think it would be useful to mention if the cache is flushed when the file is not opened anymore, or (if not) to explicitly say that that's not the case.

Quoted 2 lines

This mode assumes that the file is never changed outside the mounted filesystem

This sounds a little awkward. How about "assumes that all changes to the filesystem go through the FUSE kernel module"

Quoted 1 line

(the size time attributes are kept uptodate by the kernel),

"size and atime/ctime/mtime attributes", and I think "uptodate" should be up-to-date.

Quoted 1 line

so it's not suitable for any any network filesystem

duplicated "any"

I think this is great. I may work on similar documentation for some of the other stuff that currently lives only in include/fuse*.h

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#7 lore raw

Updated version.

Thanks,
Miklos
---

Fuse supports the following I/O modes:

- direct-io
- cached
+ write-through
+ writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and when the last ref to the file is being released on munmap(2)). This mode assumes that all changes to the filesystem go through the FUSE kernel module (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so it's generally not suitable for network filesystems. If a partial page is written, then the page needs to be first read from userspace. This means, that even for files opened for O_WRONLY it is possible that READ requests will be generated by the kernel.

#8 lore raw
Quoted 27 lines · Miklos Szeredi

Fuse supports the following I/O modes:

- direct-io
- cached
+ write-through
+ writeback-cache

The direct-io mode can be selected with the FOPEN_DIRECT_IO flag in the FUSE_OPEN reply.

In direct-io mode the page cache is completely bypassed for reads and writes. No read-ahead takes place. Shared mmap is disabled.

In cached mode reads may be satisfied from the page cache, and data may be read-ahead by the kernel to fill the cache. The cache is always kept consistent after any writes to the file. All mmap modes are supported.

The cached mode has two sub modes controlling how writes are handled. The write-through mode is the default and is supported on all kernels. The writeback-cache mode may be selected by the FUSE_WRITEBACK_CACHE flag in the FUSE_INIT reply.

In write-through mode each write is immediately sent to userspace as one or more WRITE requests, as well as updating any cached pages (and caching previously uncached, but fully written pages). No READ requests are ever sent for writes, so when an uncached page is partially written, the page is discarded.

Quoted 11 lines

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flag) writes go to the cache only, which means that the write(2) syscall can often complete very fast. Dirty pages are written back implicitly (background writeback or page reclaim on memory pressure) or explicitly (invoked by close(2), fsync(2) and when the last ref to the file is being released on munmap(2)). This mode assumes that all changes to the filesystem go through the FUSE kernel module (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so it's generally not suitable for network filesystems. If a partial page is written, then the page needs to be first read from userspace. This means, that even for files opened for O_WRONLY it is possible that READ requests will be generated by the kernel.

Looks great to me.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#9 lore raw
In reply to Miklos Szeredi
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#10 lore raw
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

Quoted 5 lines

Best,
-Nikolaus

#11 lore raw
Quoted 11 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Quoted 7 lines

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#12 lore raw
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Quoted 7 lines

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

Quoted 4 lines

In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).

In your example you modified file bypassing fuse mount:

Quoted 1 line

echo -e "\nrevision 2" >> /tmp/issue_93/file_1

That's like mangling block device while ordinary local fs is running on top of it.

Quoted 4 lines

Best,
-Nikolaus

#13 lore raw
Quoted 25 lines · Maxim Patlasov
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Quoted 7 lines

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

Quoted 4 lines

In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).

Quoted 6 lines

In your example you modified file bypassing fuse mount:

Quoted 1 line

echo -e "\nrevision 2" >> /tmp/issue_93/file_1

That's like mangling block device while ordinary local fs is running on top of it.

Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#14 lore raw
Quoted 25 lines · Nikolaus Rath
Quoted 24 lines · Maxim Patlasov
Quoted 22 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Quoted 7 lines

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).

Quoted 8 lines
Quoted 5 lines

In your example you modified file bypassing fuse mount:

Quoted 1 line

echo -e "\nrevision 2" >> /tmp/issue_93/file_1

That's like mangling block device while ordinary local fs is running on top of it.

Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.

In your example the change didn't come through any FUSE layer. When another host writes to a network filesystem, the filesystem has a choice to process the request intelligently. In your example, you didn't give FUSE a chance because "/tmp/issue_93" was not fuse mount.

Quoted 4 lines

Best,
-Nikolaus

#15 lore raw
Quoted 29 lines · Maxim Patlasov
Quoted 27 lines · Nikolaus Rath
Quoted 26 lines · Maxim Patlasov
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Quoted 7 lines

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

Quoted 4 lines

In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).

In your example you modified file bypassing fuse mount:

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

echo -e "\nrevision 2" >> /tmp/issue_93/file_1

That's like mangling block device while ordinary local fs is running on top of it.

Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.

In your example the change didn't come through any FUSE layer.

It did. The filesystem reported the updated mtime and size in its getattr() response. The kernel ignored it.

Quoted 3 lines

When another host writes to a network filesystem, the filesystem has a choice to process the request intelligently. In your example, you didn't give FUSE a chance because "/tmp/issue_93" was not fuse mount.

Right. /tmp/issue_93 substitutes for the network. The filesystem gets notification through the network, aka by reading this mountpoint. It then attempts to pass this on to the FUSE layer, but is being ignored.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#16 lore raw
Quoted 40 lines · Nikolaus Rath
Quoted 29 lines · Maxim Patlasov
Quoted 27 lines · Nikolaus Rath
Quoted 26 lines · Maxim Patlasov
Quoted 18 lines · Nikolaus Rath
Quoted 10 lines · Maxim Patlasov
Quoted 8 lines · Nikolaus Rath
Quoted 5 lines · Miklos Szeredi

In writeback-cache mode (enabled by the FUSE_WRITEBACK_CACHE flaga) writes go to the cache only, which means that the write(2) syscall can often complete very fast. The dirty pages are later sent to userspace using write requests. This mode assumes that the file is never changed outside the mounted filesystem, so it's not suitable for any network fs.

.."this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out".

Quoted 7 lines

Not true. A network filesystem can guarantee that the file is never changed outside by implementing exclusive write lease semantics: when someone opens file for writing first time the metadata server grants exclusive rights for that mount, then declines all subsequent open requests from other mounts; and similarly while a file is being kept opened for reading, the metadata server declines all open-for-writing requests from other mounts.

Quoted 4 lines

In practice that doesn't seem to work, see the example in my first message. The file is only ever accessed on one mount at a time, yet the changes do not propagate (and would result in data corruption if another mount would attempt to read or modify the file afterwards).

In your example you modified file bypassing fuse mount:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

echo -e "\nrevision 2" >> /tmp/issue_93/file_1

That's like mangling block device while ordinary local fs is running on top of it.

Huh? That's exactly what a network filesystem would see when another host writes to the filesystem. The change doesn't come through the local FUSE layer.

In your example the change didn't come through any FUSE layer.

It did. The filesystem reported the updated mtime and size in its getattr() response. The kernel ignored it.

Quoted 6 lines
Quoted 3 lines

When another host writes to a network filesystem, the filesystem has a choice to process the request intelligently. In your example, you didn't give FUSE a chance because "/tmp/issue_93" was not fuse mount.

Right. /tmp/issue_93 substitutes for the network. The filesystem gets notification through the network, aka by reading this mountpoint. It then attempts to pass this on to the FUSE layer, but is being ignored.

Oh, I understand now. Your example actually demonstrates that FUSE_WRITEBACK_CACHE must not be used if it's possible to modify underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. That's correct. But initial statement:

Quoted 1 line

this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out

is not, because it's not impossible to protect underlying data against such an external change somehow. A network filesystem is not obliged to keep transparent correspondence like /tmp/issue_93_mnt/file_1 <--> /tmp/issue_93/file_1. It can keep all user data (and metadata) in internal structures making any external modifications impossible. Then, implementing exclusive write semantics would make using FUSE_WRITEBACK_CACHE safe. Agree?

Quoted 5 lines

Best,
-Nikolaus

#17 lore raw
Quoted 8 lines · Maxim Patlasov

Oh, I understand now. Your example actually demonstrates that FUSE_WRITEBACK_CACHE must not be used if it's possible to modify underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. That's correct. But initial statement:

Quoted 2 lines

this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out

Quoted 7 lines

is not, because it's not impossible to protect underlying data against such an external change somehow. A network filesystem is not obliged to keep transparent correspondence like /tmp/issue_93_mnt/file_1 <--> /tmp/issue_93/file_1. It can keep all user data (and metadata) in internal structures making any external modifications impossible. Then, implementing exclusive write semantics would make using FUSE_WRITEBACK_CACHE safe. Agree?

I agree with what you're saying, but I don't agree that this should go into the documentation for which the sentence in dispute was intended :-). I think in 93% of all cases, the simplified statement will be more helpful to a potential FUSE user than the correct one.

If this bothers you a lot, can we compromise and just be a little more vague overall? I.e.,

| This mode of operation is therefore generally not suitable for
| filesystems where modifications may not come through the (local) FUSE
| layer (i.e, most network filesystems).

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#18 lore raw
Quoted 20 lines · Nikolaus Rath
Quoted 7 lines · Maxim Patlasov

Oh, I understand now. Your example actually demonstrates that FUSE_WRITEBACK_CACHE must not be used if it's possible to modify underlying data (/tmp/issue_93/file_1) externally w.r.t fuse filesystem. That's correct. But initial statement:

Quoted 2 lines

this mode of operation is not suitable for any network filesystem even if no write operations are actually carried out

Quoted 7 lines

is not, because it's not impossible to protect underlying data against such an external change somehow. A network filesystem is not obliged to keep transparent correspondence like /tmp/issue_93_mnt/file_1 <--> /tmp/issue_93/file_1. It can keep all user data (and metadata) in internal structures making any external modifications impossible. Then, implementing exclusive write semantics would make using FUSE_WRITEBACK_CACHE safe. Agree?

I agree with what you're saying, but I don't agree that this should go into the documentation for which the sentence in dispute was intended :-). I think in 93% of all cases, the simplified statement will be more helpful to a potential FUSE user than the correct one.

Quoted 6 lines

If this bothers you a lot, can we compromise and just be a little more vague overall? I.e.,

| This mode of operation is therefore generally not suitable for
| filesystems where modifications may not come through the (local) FUSE
| layer (i.e, most network filesystems).

The way how Miklos conveyed it:

Quoted 4 lines

This mode
assumes that all changes to the filesystem go through the FUSE kernel module (size and atime/ctime/mtime attributes are kept up-to-date by the kernel), so it's generally not suitable for network filesystems.

looks perfect. Thank you!

Maxim