flock.2: add CIFS details

From: Aurelien Aptel <[email protected]>

Similarly to NFS, CIFS flock() locks behave differently than the standard. Document those differences.

Signed-off-by Aurelien Aptel <[email protected]>

---
  man2/flock.2 | 22 ++++++++++++++++++++++
  1 file changed, 22 insertions(+)

man2/flock.2 +0 -0

index 61d4b5396..9271b8fef 100644

man2/flock.2 +4 -0

+.PP
+Since Linux 5.5,
+.BR flock ()
+are emulated with SMB byte-range locks on the

man2/flock.2 +0 -0

  .SH SEE ALSO
  .BR flock (1),
  .BR close (2),

On 3/2/2021 10:48 AM, Aurélien Aptel wrote:
I'd suggest removing this sentence. It doesn't really add anything to the definition.

This is discussing the scenario where a process on the server performs an flock(), right? That's perhaps confusingly special. How about

"In Linux kernels up to 5.4, flock() is not propagated over SMB. A file with such locks will not appear locked for remote clients."

"protocol, which provides mandatory locking semantics."

From: Aurelien Aptel <[email protected]>

Similarly to NFS, CIFS flock() locks behave differently than the standard. Document those differences.

Here is the rendered text:

CIFS details
In Linux kernels up to 5.4, flock() is not propagated over SMB. A file with such locks will not appear locked for other SMB clients.

Since Linux 5.5, flock() are emulated with SMB byte-range locks on the entire file. Similarly to NFS, this means that fcntl(2) and flock() locks interact with one another over SMB. Another important side-effect is that the locks are not advisory anymore: a write on a locked file will always fail with EACCESS. This difference originates from the de- sign of locks in the SMB protocol, which provides mandatory locking se- mantics. The nobrl mount option can be used switch back to pre-5.5 ker- nel behavior.

Signed-off-by Aurelien Aptel <[email protected]>

---
  man2/flock.2 | 21 +++++++++++++++++++++
  1 file changed, 21 insertions(+)

man2/flock.2 +0 -0

index 61d4b5396..7c4e7e8c9 100644

man2/flock.2 +0 -0

  .SH SEE ALSO
  .BR flock (1),
  .BR close (2),

From: Aurelien Aptel <[email protected]>

Similarly to NFS, CIFS flock() locks behave differently than the standard. Document those differences.

Here is the rendered text:

CIFS details
In Linux kernels up to 5.4, flock() is not propagated over SMB. A file with such locks will not appear locked for remote clients.

Since Linux 5.5, flock() locks are emulated with SMB byte-range locks on the entire file. Similarly to NFS, this means that fcntl(2) and flock() locks interact with one another. Another important side-effect is that the locks are not advisory anymore: a write on a locked file will always fail with EACCESS. This difference originates from the de- sign of locks in the SMB protocol, which provides mandatory locking se- mantics. The nobrl mount option (see mount.cifs(8)) turns off fnctl(2) and flock() lock propagation to remote clients and makes flock() locks advisory again.

Signed-off-by Aurelien Aptel <[email protected]>

---
  man2/flock.2 | 29 +++++++++++++++++++++++++++++
  1 file changed, 29 insertions(+)

man2/flock.2 +7 -0

index 61d4b5396..4b6e5cc24 100644
@@ -239,6 +239,35 @@ see the discussion of the
  .I "local_lock"
  option in
  .BR nfs (5).
+.SS CIFS details
+In Linux kernels up to 5.4,
+.BR flock ()
+is not propagated over SMB. A file with such locks will not appear
+locked for remote clients.
+.PP
+Since Linux 5.5,

man2/flock.2 +12 -0

+This difference originates from the design of locks in the SMB
+protocol, which provides mandatory locking semantics. The
+.I nobrl
+mount option (see
+.BR mount.cifs (8))
+turns off
+.BR fnctl (2)
+and
+.BR flock ()
+lock propagation to remote clients and makes
+.BR flock ()
+locks advisory again.
  .SH SEE ALSO
  .BR flock (1),
  .BR close (2),

It looks great, and sorry to be a pest, but I just noticed - it's EACCES (not EACCESS).

From: Aurelien Aptel <[email protected]>

Similarly to NFS, CIFS flock() locks behave differently than the standard. Document those differences.

Here is the rendered text:

CIFS details
In Linux kernels up to 5.4, flock() is not propagated over SMB. A file with such locks will not appear locked for remote clients.

Since Linux 5.5, flock() locks are emulated with SMB byte-range locks on the entire file. Similarly to NFS, this means that fcntl(2) and flock() locks interact with one another. Another important side-effect is that the locks are not advisory anymore: a write on a locked file
will always fail with EACCES. This difference originates from the de-
sign of locks in the SMB protocol, which provides mandatory locking se- mantics. The nobrl mount option (see mount.cifs(8)) turns off fnctl(2) and flock() lock propagation to remote clients and makes flock() locks advisory again.

Signed-off-by Aurelien Aptel <[email protected]>

Reviewed-by Tom Talpey <[email protected]>

---

man2/flock.2 +9 -0

+mount option (see
+.BR mount.cifs (8))
+turns off
+.BR fnctl (2)
+and
+.BR flock ()
+lock propagation to remote clients and makes
+.BR flock ()
+locks advisory again.
  .SH SEE ALSO
  .BR flock (1),
  .BR close (2),

I agree with Tom. It's much easier to read if you just say that 'nobrl' torns off the non-locale behaviour, and acts as 5.4 and earlier kernels.
Unless there's any subtlety that makes it different. Is there any?

BTW, you should use "semantic newlines":

Quoted 3 lines · Alejandro Colomar (man-pages)

I agree with Tom. It's much easier to read if you just say that 'nobrl' torns off the non-locale behaviour, and acts as 5.4 and earlier kernels.
Unless there's any subtlety that makes it different. Is there any?

nobrl also makes fnctl() locks local.
In 5.4 and earlier kernel, flock() is local but fnctl() isn't.

Quoted 1 line

BTW, you should use "semantic newlines":

Ok, I'll redo once we agree on the text.

Quoted 11 lines · Aurélien Aptel

Quoted 3 lines · Alejandro Colomar (man-pages)

I agree with Tom. It's much easier to read if you just say that 'nobrl' torns off the non-locale behaviour, and acts as 5.4 and earlier kernels.
Unless there's any subtlety that makes it different. Is there any?

nobrl also makes fnctl() locks local.
In 5.4 and earlier kernel, flock() is local but fnctl() isn't.

Quoted 1 line

BTW, you should use "semantic newlines":

Ok, I'll redo once we agree on the text.

I wonder if it's best to leave the nobrl details to the mount.cifs manpage, and just make a reference from here.

Another advantage of putting this in a cifs.ko-specific manpage is that it would be significantly easier to maintain. The details of a 5.4-to-5.5 transition are going to fade over time, and the APIs in fcntl(2)/flock(2) really aren't driving that.

If not, it's going to be messy... Aurélien is this correct?

cifs.ko flock()
- local in <= 5.4
- remote by default in >= 5.5
- local if nobrl in >= 5.5

cifs.ko fcntl()
- remote by default in X.Y
- local if nobrl in X.Y

Not sure what the value(s) of X.Y actually might be.

It seems odd that "nobrl" means "handle locking locally, and never send to server". I mean, there is always byte-range locking, right?

Are there any other options or configurations that alter this?

Quoted 13 lines · Tom Talpey

Quoted 11 lines · Aurélien Aptel

Quoted 3 lines · Alejandro Colomar (man-pages)

I agree with Tom. It's much easier to read if you just say that 'nobrl' torns off the non-locale behaviour, and acts as 5.4 and earlier kernels.
Unless there's any subtlety that makes it different. Is there any?

nobrl also makes fnctl() locks local.
In 5.4 and earlier kernel, flock() is local but fnctl() isn't.

Quoted 1 line

BTW, you should use "semantic newlines":

Ok, I'll redo once we agree on the text.

Quoted 7 lines

I wonder if it's best to leave the nobrl details to the mount.cifs manpage, and just make a reference from here.

Another advantage of putting this in a cifs.ko-specific manpage is that it would be significantly easier to maintain. The details of a 5.4-to-5.5 transition are going to fade over time, and the APIs in fcntl(2)/flock(2) really aren't driving that.

I was trying to write in the same style as the NFS details just above (see existing man page). Give basic overview of the issues.

Quoted 10 lines

If not, it's going to be messy... Aurélien is this correct?

cifs.ko flock()
- local in <= 5.4
- remote by default in >= 5.5
- local if nobrl in >= 5.5

cifs.ko fcntl()
- remote by default in X.Y
- local if nobrl in X.Y

Correct.

Quoted 1 line

Not sure what the value(s) of X.Y actually might be.

AFAIK fcntl() was always remote by default.
And nobrl was added in 2.6.15 (15 years ago). I wouldn't bother mentionning X.Y, it's already complex enough as it is.

Quoted 2 lines

It seems odd that "nobrl" means "handle locking locally, and never send to server". I mean, there is always byte-range locking, right?

Yes the option name can be confusing. Byte-range locking is always possible, but with "nobrl" it's local-only.

Quoted 1 line

Are there any other options or configurations that alter this?

I've taken another long look at the cifs.ko and samba code. There are many knobs that would make an accurate matrix table pretty big.

* If the mount point is done on an SMB1+UNIX Extensions connection, locking becomes advisory. Unless forcemandatorylock option is passed. This will eventually be implemented for SMB3 posix extensions as well (I've started a thread on the samba-technical mailing list).

* If cifs.ko can get guarantees (via oplocks or leases) that it is the only user of a file, it caches read/writes but also locking locally. If the lease is broke then it will send the locks. The levels of caching cifs.ko can do can be changed with the cache mount option.

The nobrl mount option (see mount.cifs(8)) turns off fnctl(2)
and flock() lock propagation to remote clients and makes flock() locks advisory again.

and simply state (perhaps)

"Remote and mandatory locking semantics may vary with SMB protocol,
mount options and server type. See mount.cifs(8) for additional information."

Quoted 5 lines · Tom Talpey

and simply state (perhaps)

"Remote and mandatory locking semantics may vary with SMB protocol,
mount options and server type. See mount.cifs(8) for additional information."

This would be the complete addition to the man page? I feel like we

should at least say it is *likely* that:
- locks will be mandatory
- flock() is emulated via fnctl() and so they interact with each other

Which are the 2 aspects that really diverges from the expected behaviour of flock() and likely to hit people in the wild. Mentionning this will send people trying to debug their app in the right direction.

Ok, then I agree with your last paragraph. Here's the current version, with semantic newlines:

In Linux kernels up to 5.4, flock() is not propagated over SMB. A file with such locks will not appear locked for remote clients.

Since Linux 5.5, flock() locks are emulated with SMB byte-range locks on the entire file. Similarly to NFS, this means that fcntl(2) and flock() locks interact with one another. Another important side-effect is that the locks are not advisory anymore:
a write on a locked file will always fail with EACCES.

It is not only about writing to a locked file. It is also about any IO against a locked file if such a file is locked through another file handle. Right?

Quoted 3 lines · Pavel Shilovsky

It is not only about writing to a locked file. It is also about any IO against a locked file if such a file is locked through another file handle. Right?

Yes that was implied, the write was a simple example to illustrate. I'll update to make it more generic:

Another important side-effect is that the locks are not advisory anymore:
any IO on a locked file will always fail with EACCES,
even when done from a separate file descriptor.

If you have comments please provide direct text suggestion to save time.

Cheers,
--
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Sure. Thanks!

I would put more details from
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex
:

"""
Another important side-effect is that the locks are not advisory anymore:
any IO on an exclusively locked file will always fail with EACCES when done from a separate file descriptor; write calls on
a file locked for shared access will fail with EACCES when done from any file descriptor including the one used to lock the file.
"""

Thoughts?

--
Best regards,
Pavel Shilovsky

вт, 16 мар. 2021 г. в 03:42, Aurélien Aptel <[email protected]>:

Quoted 22 lines

Quoted 3 lines · Pavel Shilovsky

It is not only about writing to a locked file. It is also about any IO against a locked file if such a file is locked through another file handle. Right?

Yes that was implied, the write was a simple example to illustrate. I'll update to make it more generic:

Another important side-effect is that the locks are not advisory anymore:
any IO on a locked file will always fail with EACCES,
even when done from a separate file descriptor.

If you have comments please provide direct text suggestion to save time.

Cheers,
--
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Quoted 14 lines

Sure. Thanks!

I would put more details from
https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex
:

"""
Another important side-effect is that the locks are not advisory anymore:
any IO on an exclusively locked file will always fail with EACCES when done from a separate file descriptor; write calls on
a file locked for shared access will fail with EACCES when done from any file descriptor including the one used to lock the file.
"""

Quoted 1 line

Thoughts?

I think it'll be important to define what "exclusive" and "shared" mean from a Linux/POSIX API perspective, and that will get into dragon territory. I don't think it's a good idea to attempt that in this manpage. It is best to leave Windows semantics, and interop with Windows clients, out of it.

IOW, I personally prefer Aurélien's simple version for now.

Tom.

Quoted 29 lines

--
Best regards,
Pavel Shilovsky

вт, 16 мар. 2021 г. в 03:42, Aurélien Aptel <[email protected]>:

Quoted 22 lines

Quoted 3 lines · Pavel Shilovsky

It is not only about writing to a locked file. It is also about any IO against a locked file if such a file is locked through another file handle. Right?

Yes that was implied, the write was a simple example to illustrate. I'll update to make it more generic:

Another important side-effect is that the locks are not advisory anymore:
any IO on a locked file will always fail with EACCES,
even when done from a separate file descriptor.

If you have comments please provide direct text suggestion to save time.

Cheers,
--
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

Friendly ping to the man page maintainers

Cheers,

mantics. The nobrl mount option (see mount.cifs(8)) turns off fnctl(2)

Hi Tom,

Thanks for taking a look.

Quoted 3 lines · Tom Talpey

On 3/2/2021 10:48 AM, Aurélien Aptel wrote:
I'd suggest removing this sentence. It doesn't really add anything to the definition.

OK.

Quoted 2 lines

This is discussing the scenario where a process on the server performs an flock(), right? That's perhaps confusingly special. How about

This is about clients. Let's say the same app is running on 2 different Linux system that have the same Windows Server share mounted.

The scenario is those 2 app instances use the same file on the share and are trying to synchronize access using flock().

Pre-5.5, CIFS flock() is using the generic flock() implementation from the Linux VFS layer which only knows about syscall made by local apps and isn't aware that the file can be accessed under its feet from the network.

In 5.5 and above, CIFS flock() is implemented using SMB locks, which have different semantics than what POSIX defines, i.e. you cannot ignore the locks and write, write() will fail with EPERM. So this version can be used for file sync with other clients but works slightly differently. It is a best-effort attempt.

Does this clarification changes anything to your suggestions?

Quoted 5 lines

"In Linux kernels up to 5.4, flock() is not propagated over SMB. A file with such locks will not appear locked for remote clients."

"protocol, which provides mandatory locking semantics."

OK. As it turns out, there is actually a 'nobrl' mount option to get back pre-5.5 behavior. I'll mention it and use your suggestions in v2.

Cheers,

I don't fully understand your response. What does "knows about syscall from local apps" mean, from a practical perspective? That it never forwards any flock() call to the server? Or that it somehow caches the flock() results, and never checks if the server has a conflict from another client?

Quoted 5 lines · Tom Talpey

I don't fully understand your response. What does "knows about syscall from local apps" mean, from a practical perspective? That it never forwards any flock() call to the server? Or that it somehow caches the flock() results, and never checks if the server has a conflict from another client?

Yes that's what I'm trying to say. Locking never goes on the wire. Server is not aware of locking, and thus neither are any other clients.