Leveldb in google-chrome incompatible with btrfs?
I don't know who else uses google-chrome here, but for me, for as long as I've used btrfs (3+ years now), I've had no end of troubles recovering from a linux crash, and google-chrome has had problems recovering my tabs and usually cmoplains about plenty of problems, some are corruption looking.
The way I recover, every time, is to simply rsync over the previous snapshot's data for ~/.config/google-chrome-dir back over the current corrupted one, and move on.
I had a bit more look at this with a chrome engineer, and there seems to be at least a clear problem with leveldb where its file containing the index name, does not match the files on disk.
Details are in https://bugs.chromium.org/p/chromium/issues/detail?id=738961
but I'll summarize here.
All the files are in a non corrupted state, but I end up with this:
saruman:~/.config/google-chrome-dir/google-chrome-beta/Default/IndexedDB/https_docs.google.com_0.indexeddb.leveldb$ l (...) -rw------- 1 merlin merlin 16 Oct 4 2016 CURRENT -rw------- 1 merlin merlin 0 Apr 9 2016 LOCK -rw------- 1 merlin merlin 0 Jul 5 22:31 LOG -rw------- 1 merlin merlin 0 Jul 5 22:31 LOG.old -rw------- 1 merlin merlin 23 Jul 3 08:31 MANIFEST-000001 -rw------- 1 merlin merlin 8639 May 12 17:36 MANIFEST-022745 saruman:~/.config/google-chrome-dir/google-chrome-beta/Default/IndexedDB/https_docs.google.com_0.indexeddb.leveldb$ cat CURRENT MANIFEST-010814
In other words, I think things go like this:
- MANIFEST-010814 is replaced by MANIFEST-022745,
- MANIFEST-022745 is written into the file CURRENT
- ideally both MANIFEST-010814 and MANIFEST-022745 should be present on disk and MANIFEST-010814 deleted only after CURRENT has been written to with the new pointer, but I'm not sure if or how that is done.
- my laptop crashes
- after reboot, btrfs has not been able to commit the update to CURRENT to disk in a consistent state, therefore discards the COW data to CURRENT and leaves it in its prior state, which now holds the old data
- somehow MANIFEST-010814 has however been deleted, so now CURRENT points to a non existent file.
I don't know how leveldb works or where its (f)sync statements are. but I'm guessing it works reliably on ext4 since it's used by many users and I'm the first one reporting this problem with leveldb.
Does anyone know if it's leveldb relying on non POSIX semantics that just happen to work out on ext4, or if btrfs COW and atomicity doesn't quite handle multi file updates in a way that is expected by a spec, or by application developers?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 4 lines · Marc MERLIN
I don't know who else uses google-chrome here, but for me, for as long as I've used btrfs (3+ years now), I've had no end of troubles recovering from a linux crash, and google-chrome has had problems recovering my tabs and usually cmoplains about plenty of problems, some are corruption looking.
I've also had issues with chrome and Btrfs, not just you.
[snip]
Quoted 4 lines
Does anyone know if it's leveldb relying on non POSIX semantics that just happen to work out on ext4, or if btrfs COW and atomicity doesn't quite handle multi file updates in a way that is expected by a spec, or by application developers?
A quick google search turned this up: https://github.com/google/leveldb/issues/195.
Unless anything has changed since that issue was last updated, it does sound like LevelDB is making some unsafe assumptions. I'll take a look.
Quoted 10 lines · Omar Sandoval
Quoted 4 lines · Marc MERLIN
I don't know who else uses google-chrome here, but for me, for as long as I've used btrfs (3+ years now), I've had no end of troubles recovering from a linux crash, and google-chrome has had problems recovering my tabs and usually cmoplains about plenty of problems, some are corruption looking.
I've also had issues with chrome and Btrfs, not just you.
[snip]
Quoted 8 lines
Quoted 4 lines
Does anyone know if it's leveldb relying on non POSIX semantics that just happen to work out on ext4, or if btrfs COW and atomicity doesn't quite handle multi file updates in a way that is expected by a spec, or by application developers?
A quick google search turned this up: https://github.com/google/leveldb/issues/195.
Unless anything has changed since that issue was last updated, it does sound like LevelDB is making some unsafe assumptions. I'll take a look.
Thanks Omar, this very much looks related indeed.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 16 lines · Marc MERLIN
Quoted 15 lines · Omar Sandoval
Quoted 4 lines · Marc MERLIN
I don't know who else uses google-chrome here, but for me, for as long as I've used btrfs (3+ years now), I've had no end of troubles recovering from a linux crash, and google-chrome has had problems recovering my tabs and usually cmoplains about plenty of problems, some are corruption looking.
I've also had issues with chrome and Btrfs, not just you.
[snip]
Quoted 4 lines
Does anyone know if it's leveldb relying on non POSIX semantics that just happen to work out on ext4, or if btrfs COW and atomicity doesn't quite handle multi file updates in a way that is expected by a spec, or by application developers?
Quoted 7 lines
Quoted 3 lines
A quick google search turned this up: https://github.com/google/leveldb/issues/195.
Unless anything has changed since that issue was last updated, it does sound like LevelDB is making some unsafe assumptions. I'll take a look.Thanks Omar, this very much looks related indeed.
Marc
Hm, tracing a simple program using LevelDB I put together, it looks like the relevant sequence of events on open is pretty much
create new MANIFEST-nnnnnn
write out new MANIFEST-nnnnnnfsync() parent directoryfdatasync() MANIFEST-nnnnnn
create temporary CURRENT
write out temporary CURRENT to point to new MANIFEST-nnnnnn name fdatasync() temporary CURRENT
rename temporary CURRENT to CURRENT
unlink old MANIFEST-nnnnnn
There's no fsync of the parent directory between renaming CURRENT and unlinking the old MANIFEST, but that shouldn't be an issue on Btrfs. Either both of those operations will be committed in the same filesystem transaction, or the rename will be committed before the unlink. I can't think of any way that the unlink would end up reordered before the rename.
What doesn't add up about your bug report is that your CURRENT points to a MANIFEST-010814 way behind all of the other files in that directory, which are numbered 022745+. If there were a bug here, I'd expect the stale MANIFEST file would be one older than the new one. The filenames seem to be allocated sequentially, so that old MANIFEST file CURRENT refers to must be really old, which doesn't make sense. I don't see how Btrfs would screw that up :) I'd be interested to see if you can make the same condition trigger again.
Quoted 9 lines · Omar Sandoval
What doesn't add up about your bug report is that your CURRENT points to a MANIFEST-010814 way behind all of the other files in that directory, which are numbered 022745+. If there were a bug here, I'd expect the stale MANIFEST file would be one older than the new one. The filenames seem to be allocated sequentially, so that old MANIFEST file CURRENT refers to must be really old, which doesn't make sense. I don't see how Btrfs would screw that up :) I'd be interested to see if you can make the same condition trigger again.
First, thanks for looking at it.
Second, you are right on the numbers being so far apart that something was wrong. I checked my snapshots, and I've been carrying that MANIFEST-010814 for a long time.
In other words, it's a old stale manifest that never got deleted.
The new real old one apparently got deleted, the new one was created but didn't make it to disk, but the pointer in CURRENT did get repointed to the new one that never made it to actual disk.
So I think what happened is something like this:
MANIFEST-new got created
echo MANIFEST-new > CURRENT
MANIFEST-old got deleted
system crashed
MANIFEST-old was indeed deleted, and MANIFEST-new never made it to disk.
Does that sound more plausible?
As for redoing this at will, apparently I may have been hit by the skylake hyperthreading CPU bug that I just installed a microcode update for, which was causing random crashes, which hopefully are now solved.
I can't say if those in turn messed with btrfs writing data, but I'd rather not recreate this since it's my real filesystem I care about and don't want to corrupt on purpose :)
That said, the google-chrome on my previous haswell CPU also had routine problems when restarting chrome, although at this point I don't know if they were due to leveldb or sqlite or something else.
I'm just mentioning this to say that I'm pretty sure that the haswell HT bug isn't the sole culprit of this problem, likely just the trigger of some of my crashes.
Hope this helps
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 28 lines · Marc MERLIN
Quoted 9 lines · Omar Sandoval
What doesn't add up about your bug report is that your CURRENT points to a MANIFEST-010814 way behind all of the other files in that directory, which are numbered 022745+. If there were a bug here, I'd expect the stale MANIFEST file would be one older than the new one. The filenames seem to be allocated sequentially, so that old MANIFEST file CURRENT refers to must be really old, which doesn't make sense. I don't see how Btrfs would screw that up :) I'd be interested to see if you can make the same condition trigger again.
First, thanks for looking at it.
Second, you are right on the numbers being so far apart that something was wrong. I checked my snapshots, and I've been carrying that MANIFEST-010814 for a long time.
In other words, it's a old stale manifest that never got deleted.The new real old one apparently got deleted, the new one was created but didn't make it to disk, but the pointer in CURRENT did get repointed to the new one that never made it to actual disk.
So I think what happened is something like this:
MANIFEST-new got created
echo MANIFEST-new > CURRENT
MANIFEST-old got deleted
system crashed
Quoted 3 lines
MANIFEST-old was indeed deleted, and MANIFEST-new never made it to disk.
Does that sound more plausible?
In the bug report, you commented that CURRENT contained MANIFEST-010814, is that indeed the case or was it actually something newer? If it was the newer one, then it's still tricky how we'd end up that way but not as outlandish.
Quoted 6 lines
As for redoing this at will, apparently I may have been hit by the skylake hyperthreading CPU bug that I just installed a microcode update for, which was causing random crashes, which hopefully are now solved.
I can't say if those in turn messed with btrfs writing data, but I'd rather not recreate this since it's my real filesystem I care about and don't want to corrupt on purpose :)
Understandable :)
Quoted 14 lines
That said, the google-chrome on my previous haswell CPU also had routine problems when restarting chrome, although at this point I don't know if they were due to leveldb or sqlite or something else.
I'm just mentioning this to say that I'm pretty sure that the haswell HT bug isn't the sole culprit of this problem, likely just the trigger of some of my crashes.Hope this helps
MarcSignature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/
Quoted 4 lines · Omar Sandoval
In the bug report, you commented that CURRENT contained MANIFEST-010814, is that indeed the case or was it actually something newer? If it was the newer one, then it's still tricky how we'd end up that way but not as outlandish.
You are correct, my bad.
At this point, I'm going to have to assume that something bad happened with rsync when I rsync'ed an old profile over the one that caused chrome to fail to restart.
Especially because CURRENT was dated Oct 4th, which does not make sense.
Now that I know what to look for, I'll have a much closer look next time this happens, with the understanding that it would be a while if I've successfully fixed the reason why my laptop was crashing too often.
But you said you've also seen issues with google-chrome profile and btrfs. What did you experience?
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 11 lines · Marc MERLIN
Quoted 4 lines · Omar Sandoval
In the bug report, you commented that CURRENT contained MANIFEST-010814, is that indeed the case or was it actually something newer? If it was the newer one, then it's still tricky how we'd end up that way but not as outlandish.
You are correct, my bad.
At this point, I'm going to have to assume that something bad happened with rsync when I rsync'ed an old profile over the one that caused chrome to fail to restart.
Especially because CURRENT was dated Oct 4th, which does not make sense.
Okay, this makes sense.
Quoted 3 lines
Now that I know what to look for, I'll have a much closer look next time this happens, with the understanding that it would be a while if I've successfully fixed the reason why my laptop was crashing too often.
Sounds good.
Quoted 2 lines
But you said you've also seen issues with google-chrome profile and btrfs. What did you experience?
I never debugged it, but I had to blow away the profile a couple of times. Then there's this weird one which looks like a Btrfs bug:
┌[osandov@vader ~/.config] └$ ls -al google-chrome-busted/**
ls: cannot access 'google-chrome-busted/Local State': No such file or directory google-chrome-busted/Default:
ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory total 0
drwx------ 1 osandov users 12 Feb 7 16:50 .
drwx------ 1 osandov users 14 Feb 7 16:50 ..
-????????? ? ? ? ? ? .com.google.Chrome.VfAUNx -????????? ? ? ? ? ? Preferences
I was also struggling with this issue for quite some time. Today my 2 months old disk is crashed (which really surprises me). It doesn't even being shown by fdisk -l.
After buying this this disk, I installed a Debian on a BTRFS partition on an LVM partition on a LUKS partition. From that day till today, while doing my work, there was always a video running on Youtube in the background. Sometimes my editor (atom, it also shares a lot with chromium) was slowing down which was making me end up restarting the Chromium (and it was restarting barely with killall -9 chromium. Another mentionable event is, 3 days ago my laptop suddenly restarted.
I suspect Chromium usage on BTRFS and disk the crash are somehow related.
2017-07-07 8:46 GMT+03:00 Omar Sandoval <[email protected]>:
Quoted 42 lines · Marc MERLIN
Quoted 11 lines · Omar Sandoval
Quoted 4 lines
In the bug report, you commented that CURRENT contained MANIFEST-010814, is that indeed the case or was it actually something newer? If it was the newer one, then it's still tricky how we'd end up that way but not as outlandish.
You are correct, my bad.
At this point, I'm going to have to assume that something bad happened with rsync when I rsync'ed an old profile over the one that caused chrome to fail to restart.
Especially because CURRENT was dated Oct 4th, which does not make sense.Okay, this makes sense.
Quoted 3 lines
Now that I know what to look for, I'll have a much closer look next time this happens, with the understanding that it would be a while if I've successfully fixed the reason why my laptop was crashing too often.
Sounds good.
Quoted 2 lines
But you said you've also seen issues with google-chrome profile and btrfs. What did you experience?
I never debugged it, but I had to blow away the profile a couple of times. Then there's this weird one which looks like a Btrfs bug:
┌[osandov@vader ~/.config] └$ ls -al google-chrome-busted/**ls: cannot access 'google-chrome-busted/Local State': No such file or directory google-chrome-busted/Default:
ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory total 0
drwx------ 1 osandov users 12 Feb 7 16:50 .
drwx------ 1 osandov users 14 Feb 7 16:50 ..-????????? ? ? ? ? ? .com.google.Chrome.VfAUNx -????????? ? ? ? ? ? Preferences--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
(removing pwnall at chromium.org to cut spam)
Quoted 11 lines · Omar Sandoval
┌[osandov@vader ~/.config] └$ ls -al google-chrome-busted/**ls: cannot access 'google-chrome-busted/Local State': No such file or directory google-chrome-busted/Default:
ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory total 0
drwx------ 1 osandov users 12 Feb 7 16:50 .
drwx------ 1 osandov users 14 Feb 7 16:50 ..-????????? ? ? ? ? ? .com.google.Chrome.VfAUNx -????????? ? ? ? ? ? Preferences
Yeah, that's definitely not chrome's fault :)
Quoted 11 lines · Cerem Cem ASLAN
I was also struggling with this issue for quite some time. Today my 2 months old disk is crashed (which really surprises me). It doesn't even being shown by
fdisk -l.After buying this this disk, I installed a Debian on a BTRFS partition on an LVM partition on a LUKS partition. From that day till today, while doing my work, there was always a video running on Youtube in the background. Sometimes my editor (atom, it also shares a lot with chromium) was slowing down which was making me end up restarting the Chromium (and it was restarting barely with
killall -9 chromium. Another mentionable event is, 3 days ago my laptop suddenly restarted.
Sorry, I'm not quite making sense out of this.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ The day before I re-checked to see if I could recover any of my files and it seems that disk is all alive. Nothing missed, nothing deleted. I run btrfs scrub, there are only a few uncorrectable errors (they were there before the event). I'm using the disk as before for two days and there is nothing unusual. Quite interesting.
2017-07-07 19:50 GMT+03:00 Marc MERLIN <[email protected]>:
Quoted 38 lines
(removing pwnall at chromium.org to cut spam)
Quoted 11 lines · Omar Sandoval
┌[osandov@vader ~/.config] └$ ls -al google-chrome-busted/**ls: cannot access 'google-chrome-busted/Local State': No such file or directory google-chrome-busted/Default:
ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or directory
ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': No such file or directory total 0
drwx------ 1 osandov users 12 Feb 7 16:50 .
drwx------ 1 osandov users 14 Feb 7 16:50 ..-????????? ? ? ? ? ? .com.google.Chrome.VfAUNx -????????? ? ? ? ? ? PreferencesYeah, that's definitely not chrome's fault :)
Quoted 11 lines · Cerem Cem ASLAN
I was also struggling with this issue for quite some time. Today my 2 months old disk is crashed (which really surprises me). It doesn't even being shown by
fdisk -l.After buying this this disk, I installed a Debian on a BTRFS partition on an LVM partition on a LUKS partition. From that day till today, while doing my work, there was always a video running on Youtube in the background. Sometimes my editor (atom, it also shares a lot with chromium) was slowing down which was making me end up restarting the Chromium (and it was restarting barely with
killall -9 chromium. Another mentionable event is, 3 days ago my laptop suddenly restarted.Sorry, I'm not quite making sense out of this.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/