safer git?
Hi, due to a hardware or firmware issue, my machine is currently crashing rather a lot. If that happens around the time I'm performing any git activity, it seems quite likely to cause substantial corruption of the git state.
I've responded both by running sync(1) every 60 seconds and by pushing work to github more frequently, but the latter isn't appropriate for everything; twice recently I've lost a fair bit of work as a result.
Q: Is there a mode in which I can run git that would make it a bit more robust against crashes, at the cost of being a bit slower?
The primary symptom is that files modified shortly before a crash show up existing but zero-length after the crash. For source files I mostly know what to do in that situation, but git fsck shows a lot of files under '.git/objects' that are empty, which seems to make things hard to recover:
% git fsck
error: object file .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
error: unable to mmap .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file or directory
error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 [... a dozen similar entries ...]
error: object file .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
error: unable to mmap .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or directory
error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 Checking object directories: 100% (256/256), done.
Checking objects: 100% (1577/1577), done.
error: refs/stash: invalid sha1 pointer 0000000000000000000000000000000000000000
error: bad ref for .git/logs/refs/stash
dangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107
%
Last time I checked out the previous state from github in a new directory and was able to find and copy over most of my work before continuing. On this occasion I did a git stash save shortly before the crash, and I'm not sure how to get that back. I see René Scharfe's suggestion of:
git fsck --unreachable |
grep commit | cut -d\ -f3 |
xargs git log --merges --no-walk --grep=WIP
from a recent message, but that is only showing me an older stash item.
Thanks in advance for any suggestions,
Hugo van der Sanden
Sorry, I should have added: I'm using Ubuntu 18.04.5 (bionic) with git version 2.17.1.
Hugo
Quoted 2 lines · On October 24, 2020 4:19 PM, [email protected] wrote:
Hi, due to a hardware or firmware issue, my machine is currently crashing rather a lot. If that happens around the time I'm performing any git
activity,
Quoted 3 lines
it seems quite likely to cause substantial corruption of the git state.
I've responded both by running sync(1) every 60 seconds and by pushing
work
Quoted 1 line
to github more frequently, but the latter isn't appropriate for
everything;
Quoted 3 lines
twice recently I've lost a fair bit of work as a result.
Q: Is there a mode in which I can run git that would make it a bit more
robust
Quoted 5 lines
against crashes, at the cost of being a bit slower?
The primary symptom is that files modified shortly before a crash show up existing but zero-length after the crash. For source files I mostly know
what
Quoted 1 line
to do in that situation, but
git fsckshows a lot of files under
'.git/objects'
Quoted 26 lines
that are empty, which seems to make things hard to recover:
% git fsck
error: object file .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
error: unable to mmap .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file or directory
error:0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or
missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 [... a dozen similar entries ...]
error: object file .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
error: unable to mmap .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or directory
error:f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or
missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 Checking object directories: 100% (256/256), done.
Checking objects: 100% (1577/1577), done.
error: refs/stash: invalid sha1 pointer 0000000000000000000000000000000000000000
error: bad ref for .git/logs/refs/stash
dangling commit1c0ea4e6159952501957012d2b9db7d68b52d107
%
Quoted 1 line
Last time I checked out the previous state from github in a new directory
and
Quoted 2 lines
was able to find and copy over most of my work before continuing. On this occasion I did a
git stash saveshortly before the crash, and I'm not
sure
Quoted 5 lines
how to get that back. I see René Scharfe's suggestion of:
git fsck --unreachable |
grep commit | cut -d\ -f3 |
xargs git log --merges --no-walk --grep=WIP from a recent message, but
that is only showing me an older stash item.
I would suggest turning off write-through buffering on your disk. Let writes complete immediately instead of being deferred to sync. Also, this does feel like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.
Good luck,
Randall
-- Brief whoami:
NonStop developer since approximately 211288444200000000 UNIX developer since approximately 421664400
-- In my real life, I talk too much.
[I somehow didn't get the original message, so replying inline below.]
Quoted 27 lines · Randall S. Becker
Quoted 26 lines · On October 24, 2020 4:19 PM, [email protected] wrote:
Q: Is there a mode in which I can run git that would make it a bit more robust
against crashes, at the cost of being a bit slower?The primary symptom is that files modified shortly before a crash show up existing but zero-length after the crash. For source files I mostly know what to do in that situation, but
git fsckshows a lot of files under '.git/objects' that are empty, which seems to make things hard to recover:% git fsck
error: object file .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
error: unable to mmap .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file or directory
error:0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or
missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 [... a dozen similar entries ...]
error: object file .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
error: unable to mmap .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or directory
error:f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or
Quoted 8 lines
Quoted 8 lines
missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 Checking object directories: 100% (256/256), done. Checking objects: 100% (1577/1577), done.
error: refs/stash: invalid sha1 pointer 0000000000000000000000000000000000000000
error: bad ref for .git/logs/refs/stash
dangling commit1c0ea4e6159952501957012d2b9db7d68b52d107%
You can try setting core.fsyncObjectFiles to true. That's the only knob that Git has for that at the moment, although there was some discussion about adding a new feature for other files[0].
I suspect a lot of the zero-byte files and any files that end up as all-zeros are due to your file system. The default file system on Ubuntu is ext4, IIRC, and if that's what you're using, you can set data=journal instead of data=ordered as a mount option. For the root file system, you'll need to pass rootflags=data=journal as a kernel boot option.
That may be significantly slower, but until you get your hardware problem sorted out, it may very well be worth it for you. I'd try this option before the one below because it'll have less of an impact on performance and may solve most or all of your problems.
Quoted 11 lines
Quoted 1 line
Last time I checked out the previous state from github in a new directory
and
Quoted 2 lines
was able to find and copy over most of my work before continuing. On this occasion I did a
git stash saveshortly before the crash, and I'm notsure
Quoted 5 lines
how to get that back. I see René Scharfe's suggestion of:
git fsck --unreachable |
grep commit | cut -d\ -f3 |
xargs git log --merges --no-walk --grep=WIP from a recent message, but
that is only showing me an older stash item.
Quoted 3 lines
I would suggest turning off write-through buffering on your disk. Let writes complete immediately instead of being deferred to sync. Also, this does feel like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.
Turning off buffering and caching for your disk drive may make things _really_ slow, but it will definitely improve data integrity.
I know hardware problems are always a hassle, so I hope you get things figured out and fixed soon.
[0] I admit I am not running the very latest version and the new feature may have already landed; if so, I apologize for the out-of-date information and for not keeping up with the list.
Signature · 2 lines
brian m. carlson (he/him or they/them) Houston, Texas, US
"brian m. carlson" wrote:
:You can try setting core.fsyncObjectFiles to true.
Super, that's exactly the sort of thing I was hoping to find.
:I suspect a lot of the zero-byte files and any files that end up as :all-zeros are due to your file system. The default file system on :Ubuntu is ext4, IIRC, and if that's what you're using, you can set :data=journal instead of data=ordered as a mount option.
It is indeed ext4. I'll consider this option; for now I've turned off write caching as suggested by Randall, which feels like a lighter-weight approach that should give almost all of the benefit.
:"Randall S. Becker" wrote:
:> I would suggest turning off write-through buffering on your disk. Let writes :> complete immediately instead of being deferred to sync. Also, this does feel :> like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.
fsck doesn't seem to be complaining, but I've set it to run every 20 mounts. What I do see is a handful of "orphaned inodes" being reclaimed on boot after every crash.
:Turning off buffering and caching for your disk drive may make things :_really_ slow, but it will definitely improve data integrity.
I haven't noticed a big slowdown so far; I'm rarely doing a _lot_ of writes.
Thanks greatly to both of you for the suggestions.
Hugo
Quoted 76 lines · brian m. carlson
[I somehow didn't get the original message, so replying inline below.]
Quoted 11 lines · Randall S. Becker
Quoted 10 lines · On October 24, 2020 4:19 PM, [email protected] wrote:
Q: Is there a mode in which I can run git that would make it a bit more robust against crashes, at the cost of being a bit slower?
The primary symptom is that files modified shortly before a crash show up existing but zero-length after the crash. For source files I mostly know what to do in that situation, but
git fsckshows a lot of files under '.git/objects' that are empty, which seems to make things hard to recover:Quoted 5 lines
Quoted 5 lines
% git fsck
error: object file .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
error: unable to mmap .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No suchfile
Quoted 8 lines
Quoted 8 lines
or directory
error:0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or
missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 [... a dozen similar entries ...]
error: object file .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
error: unable to mmap .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No suchfile
Quoted 10 lines
Quoted 10 lines
or directory
error:f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or
missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 Checking object directories: 100% (256/256), done. Checking objects: 100% (1577/1577), done.
error: refs/stash: invalid sha1 pointer 0000000000000000000000000000000000000000
error: bad ref for .git/logs/refs/stash dangling commit1c0ea4e6159952501957012d2b9db7d68b52d107
%You can try setting core.fsyncObjectFiles to true. That's the only knob that Git has for that at the moment, although there was some discussion about adding a new feature for other files[0].
I suspect a lot of the zero-byte files and any files that end up as all-zeros are due to your file system. The default file system on Ubuntu is ext4, IIRC, and if that's what you're using, you can set data=journal instead of data=ordered as a mount option. For the root file system, you'll need to pass rootflags=data=journal as a kernel boot option.
That may be significantly slower, but until you get your hardware problem sorted out, it may very well be worth it for you. I'd try this option before the one below because it'll have less of an impact on performance and may solve most or all of your problems.
Quoted 13 lines
Quoted 2 lines
Last time I checked out the previous state from github in a new directory
and
Quoted 3 lines
was able to find and copy over most of my work before continuing. On this occasion I did a
git stash saveshortly before the crash, and I'm notsure
Quoted 5 lines
how to get that back. I see René Scharfe's suggestion of:
git fsck --unreachable |
grep commit | cut -d\ -f3 |
xargs git log --merges --no-walk --grep=WIP from a recent message,
but that is only showing me an older stash item.Quoted 3 lines
I would suggest turning off write-through buffering on your disk. Let writes complete immediately instead of being deferred to sync. Also, this does feel like a disk issue, so fsck or chkdsk /f (or whatever) on your
disk urgently.
Turning off buffering and caching for your disk drive may make things _really_ slow, but it will definitely improve data integrity.
Quoted 6 lines
I know hardware problems are always a hassle, so I hope you get things figured out and fixed soon.
[0] I admit I am not running the very latest version and the new feature may have already landed; if so, I apologize for the out-of-date information and for not keeping up with the list.
Thanks for pointing out the git knob. I didn't know about that one and it seems helpful in general.
Regards,
Randall