Skip to contentSkip to conversation

interaction between O_APPEND and writeback cache

fuse Nikolaus Rath 3 participants 5 messages Aug 4, 2017 – Aug 7, 2017 VFS

lore
Share & tools
lore
#0 lore raw

Hello,

I am confused about how O_APPEND is supposed to interact with the writeback cache.

As far as I can tell, the O_APPEND flag is currently passed to the filesystem process, so my expectation is that the filesystem process is responsible for ignoring any offset in write requests and instead write at the current end of the file[1].

However, with writeback cache enabled the filesystem process cannot tell which data is "new" and came from userspace, should be appended, and which data is old and just made a round-trip to the kernel. So it seems to me that the filesystem process should probably leave the handling of O_APPEND to the kernel. But then, shouldn't the kernel filter out this flag when sending the open request?

On the other hand, when the kernel handles O_APPEND, then it is no longer atomic (think of a network fuse filesystem).

It seems to me something is not right here. Either the kernel should enforce O_APPEND when writeback caching is enabled and filter out the flag (sacrificing atomicity), or the kernel should pass O_APPEND to the filesystem but either disable writeback caching or somehow keep track of what data is actually new.

Am I missing something?

(At first I thought that maybe the filesystem's write is supposed to ignore data that comes before the current end of the file if writeback caching is active. But that wouldn't work for a network filesystem either).

Best,
-Nikolaus

[1]: Actually, if the offset is bigger than the size of the file, should it be ignored or should the file be extended?

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#1 lore raw
Quoted 10 lines · Nikolaus Rath

Hello,

I am confused about how O_APPEND is supposed to interact with the writeback cache.

As far as I can tell, the O_APPEND flag is currently passed to the filesystem process, so my expectation is that the filesystem process is responsible for ignoring any offset in write requests and instead write at the current end of the file[1].

Quoted 6 lines

However, with writeback cache enabled the filesystem process cannot tell which data is "new" and came from userspace, should be appended, and which data is old and just made a round-trip to the kernel. So it seems to me that the filesystem process should probably leave the handling of O_APPEND to the kernel. But then, shouldn't the kernel filter out this flag when sending the open request?

Indeed, when writing back the cache the kernel should definitely not set O_APPEND.

Quoted 3 lines

On the other hand, when the kernel handles O_APPEND, then it is no longer atomic (think of a network fuse filesystem).

Yes, network filesystem generally needs to handle consistency of caches across nodes and O_APPEND in no exception (i.e. you cannot have two nodes writing O_APPEND to cache at the same time, because that will not work).

Thanks,
Miklos

#2 lore raw
Quoted 18 lines · Miklos Szeredi
Quoted 10 lines · Nikolaus Rath

Hello,

I am confused about how O_APPEND is supposed to interact with the writeback cache.

As far as I can tell, the O_APPEND flag is currently passed to the filesystem process, so my expectation is that the filesystem process is responsible for ignoring any offset in write requests and instead write at the current end of the file[1].

Quoted 6 lines

However, with writeback cache enabled the filesystem process cannot tell which data is "new" and came from userspace, should be appended, and which data is old and just made a round-trip to the kernel. So it seems to me that the filesystem process should probably leave the handling of O_APPEND to the kernel. But then, shouldn't the kernel filter out this flag when sending the open request?

Quoted 2 lines

Indeed, when writing back the cache the kernel should definitely not set O_APPEND.

Well, 4.9 certainly does it though. Should I try to make a patch, or are you or Maxim going to do that shortly anyway?

Do you think it makes sense to filter out O_APPEND in libfuse as well (to work around the issue for present day kernels)?

Quoted 7 lines
Quoted 2 lines

On the other hand, when the kernel handles O_APPEND, then it is no longer atomic (think of a network fuse filesystem).

Yes, network filesystem generally needs to handle consistency of caches across nodes and O_APPEND in no exception (i.e. you cannot have two nodes writing O_APPEND to cache at the same time, because that will not work).

This poses a bit of a problem though. So a network filesystem either cannot use writeback caching or O_APPEND will (silently) not work.

With the current behavior (O_APPEND being passed to open() when writeback is enabled) the filesystem would at least have a chance to return an error, i.e. instead of a silent failure there would be a noisy error. With that in mind, maybe the current behavior isn't so bad? We'd just have to document that if writeback cache is enabled and O_APPEND is received, the filesystem has to decide if it is fine with the kernel handling O_APPEND (and in that case ignore the flag for subsequent writes) or return an error.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#3 lore raw
In reply to Miklos Szeredi
Quoted 18 lines · Miklos Szeredi
Quoted 10 lines · Nikolaus Rath

Hello,

I am confused about how O_APPEND is supposed to interact with the writeback cache.

As far as I can tell, the O_APPEND flag is currently passed to the filesystem process, so my expectation is that the filesystem process is responsible for ignoring any offset in write requests and instead write at the current end of the file[1].

Quoted 6 lines

However, with writeback cache enabled the filesystem process cannot tell which data is "new" and came from userspace, should be appended, and which data is old and just made a round-trip to the kernel. So it seems to me that the filesystem process should probably leave the handling of O_APPEND to the kernel. But then, shouldn't the kernel filter out this flag when sending the open request?

Quoted 2 lines

Indeed, when writing back the cache the kernel should definitely not set O_APPEND.

Well, 4.9 certainly does it though. Should I try to make a patch, or are you or Maxim going to do that shortly anyway?

Do you think it makes sense to filter out O_APPEND in libfuse as well (to work around the issue for present day kernels)?

Quoted 7 lines
Quoted 2 lines

On the other hand, when the kernel handles O_APPEND, then it is no longer atomic (think of a network fuse filesystem).

Yes, network filesystem generally needs to handle consistency of caches across nodes and O_APPEND in no exception (i.e. you cannot have two nodes writing O_APPEND to cache at the same time, because that will not work).

This poses a bit of a problem though. So a network filesystem either cannot use writeback caching or O_APPEND will (silently) not work.

With the current behavior (O_APPEND being passed to open() when writeback is enabled) the filesystem would at least have a chance to return an error, i.e. instead of a silent failure there would be a noisy error. With that in mind, maybe the current behavior isn't so bad? We'd just have to document that if writeback cache is enabled and O_APPEND is received, the filesystem has to decide if it is fine with the kernel handling O_APPEND (and in that case ignore the flag for subsequent writes) or return an error.

Best,
-Nikolaus

Signature · 3 lines
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«
#4 lore raw

Hi Nikolaus,

Quoted 18 lines · Nikolaus Rath
Quoted 17 lines · Miklos Szeredi
Quoted 16 lines · Nikolaus Rath

Hello,

I am confused about how O_APPEND is supposed to interact with the writeback cache.

As far as I can tell, the O_APPEND flag is currently passed to the filesystem process, so my expectation is that the filesystem process is responsible for ignoring any offset in write requests and instead write at the current end of the file[1].

However, with writeback cache enabled the filesystem process cannot tell which data is "new" and came from userspace, should be appended, and which data is old and just made a round-trip to the kernel. So it seems to me that the filesystem process should probably leave the handling of O_APPEND to the kernel. But then, shouldn't the kernel filter out this flag when sending the open request?

Quoted 7 lines
Quoted 2 lines

Indeed, when writing back the cache the kernel should definitely not set O_APPEND.

Well, 4.9 certainly does it though. Should I try to make a patch, or are you or Maxim going to do that shortly anyway?

Do you think it makes sense to filter out O_APPEND in libfuse as well (to work around the issue for present day kernels)?

I think it's up to filesystem how to handle O_APPEND. The kernel shouldn't filter it out.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

On the other hand, when the kernel handles O_APPEND, then it is no longer atomic (think of a network fuse filesystem).

Yes, network filesystem generally needs to handle consistency of caches across nodes and O_APPEND in no exception (i.e. you cannot have two nodes writing O_APPEND to cache at the same time, because that will not work).

This poses a bit of a problem though. So a network filesystem either cannot use writeback caching or O_APPEND will (silently) not work.

Quoted 8 lines

With the current behavior (O_APPEND being passed to open() when writeback is enabled) the filesystem would at least have a chance to return an error, i.e. instead of a silent failure there would be a noisy error. With that in mind, maybe the current behavior isn't so bad? We'd just have to document that if writeback cache is enabled and O_APPEND is received, the filesystem has to decide if it is fine with the kernel handling O_APPEND (and in that case ignore the flag for subsequent writes) or return an error.

Yes, I agree. For some filesystems O_APPEND is problematic, for others not. Let them decide.

Thanks,
Maxim