APFS, mmap, and git diff reporting no changes when there are changes?

Hi All - I used to maintain Cygwin/X (the X Server for Cygwin on Windows) 20 years ago... it's been a while since I've used a mailing list.

I'm working on my own implementation of copy on write cloning for git on a couple Linux fs options and APFS on Mac. While working on this I created tests around mmap'd files and how to handle sharing them when creating worktrees with clones.

I discovered that mmap'd files on Mac can cause existing git to misreport no changes on various commands, while tools that read the bytes of the files will see the changed bytes. I could not find any discussion of this being an intentional tradeoff of speed over correctness for git on APFS. I don't believe this is a security problem. However, any tool that mmaps a file, changes it, does not call msync(), and holds that mmap open will cause this misreport to continue until munmap() is called, at which point the ctime of the file changes (mtime does not change on this path).

One could argue that not calling msync() (which updates both ctime and mtime) is bad form and that git not reporting the changes in this case is an acceptable tradeoff for performance reasons. However, on Linux, git correctly detects the changes via the mmap even without calling msync().

One could also argue that this is Apple's bug or deficiency. But that could take months / years to resolve?

I'm writing to ask a few questions before either extracting and submitting just the tests or carefully vetting the fixes and preparing my own PRs:
1. Is this already known or is this nothing?
2. How far do we want to go for correctness?
3. Does anyone else have experience with this on APFS and have some advice?

I have an initial LLM-written PR for this on my fork (I am NOT asking for review on this... if you want to look, look, but if you don't want
to: please don't): https://github.com/huntharo/git/pull/14

The impact outside of APFS file systems is zero. The trade off on APFS depends on how many files are mmap'd and how large they are. If no files are ever mmap'd by any tools, ever, then the cost can be zero. The technique I'm using looks at generation number being 0 as the indication that we have to examine the file contents as the generation number drops to 0 while mapped.

There is a related follow-on problem: changing a file via mmap/munmap (or any other mechanism that updates ctime but not mtime) in the same second as the last commit will cause git to continue to report no changes even with the file unmapped.

Thanks for any input you can provide here. If these aren't really considered to be a problem then I'll leave them unsolved in my CoW clone patches when I'm ready to submit.

Harold

Git doesn't use mmap on files that change in place, only on content-addressed files like packs, indexes, and (for interoperability code) object maps, all of which have hashes in their names, or files which are replaced by rename(2), like the index.

In general, it's not safe to use mmap on files that can change in place because if you access beyond the end of the file, you can get SIGBUS, but you don't know if the file may have been modified at any point, so you effectively have to deal with crashing anywhere in your program.

This behaviour is allowed by POSIX:

The last data modification and last file status change timestamps of a file that is mapped with MAP_SHARED and PROT_WRITE shall be marked for update at some point in the interval between a write reference to the mapped region and the next call to msync() with MS_ASYNC or MS_SYNC for that portion of the file by any process.

However, Git relies on the operating system to reflect an accurate modification time. If you write files to disk with mmap, then you're responsible for calling msync() in a timely manner. However, as I said, that's extremely improvident for things like files in the working tree, which is why text editors don't use mmap.

Please read the entirety of SubmittingPatches, including
https://git-scm.com/docs/SubmittingPatches#ai, before sending that in.

brian m. carlson (they/them)
Toronto, Ontario, CA
Speaking for myself only

Am 09.11.26 um 23:38 schrieb brian m. carlson:

Not to mention that our mmap emulation for Windows cannot reproduce the Linux behavior (timely update of timestamps) and is more like the macOS behavior (delayed update of timestamps). We cannot have an accurate (atomic) msync emulation, either, because not even FlushViewOfFile[1] updates the timestamp.

[1]
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-flushviewoffile

-- Hannes