nfs home directory and google chrome.
Subsystems (2)
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT
[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)
and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set
storage.nfs_filesystem: true
This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.
Quoted 32 lines · Kenneth Johansson
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
autofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
sync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 2 lines
Quoted 2 lines
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
Quoted 6 lines · Patrick Goetz
We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set
storage.nfs_filesystem: true
This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.
Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.
I did a strace and the first -EIO I get look like this
fdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)
then the same thing happens for other files like
fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)
fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)
Quoted 33 lines · Kenneth Johansson
Quoted 32 lines
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
autofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
sync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 2 lines
Quoted 2 lines
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
Quoted 19 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz
We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set
storage.nfs_filesystem: true
This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.
Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.
I did a strace and the first
-EIOI get look like thisfdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)
Quoted 7 lines
then the same thing happens for other files like
fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)
fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)
Are you using soft mounts?
(What are your mount options?)
--b.
Quoted 61 lines
Quoted 60 lines · Kenneth Johansson
Quoted 55 lines
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTOsync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 2 lines
Quoted 2 lines
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz
We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set
storage.nfs_filesystem: true
This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.
Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.
I did a strace and the first
-EIOI get look like thisfdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)
then the same thing happens for other files like
Quoted 8 lines
Quoted 5 lines
fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)
fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)
Are you using soft mounts?
(What are your mount options?)
auto.home /home autofs rw,relatime,fd=18,pgrp=2682,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=67621 0 0
/home/kenjo nfs4 rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.16,fsc,local_lock=none,addr=172.16.2.6 0 0
what I actualy set manually in auto.home is
-tcp,fsc,noatime,ac,actimeo=120
Quoted 61 lines
--b.
Quoted 59 lines
Quoted 59 lines · Kenneth Johansson
Quoted 55 lines
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTOsync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 2 lines
Quoted 2 lines
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
Quoted 35 lines · Kenneth Johansson
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz
We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set
storage.nfs_filesystem: true
This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.
Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.
I did a strace and the first
-EIOI get look like thisfdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)
then the same thing happens for other files like
Quoted 8 lines
Quoted 5 lines
fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)
fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)
Are you using soft mounts?
(What are your mount options?)
auto.home /home autofs rw,relatime,fd=18,pgrp=2682,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=67621 0 0
Quoted 6 lines
/home/kenjonfs4 rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.16,fsc,local_lock=none,addr=172.16.2.6 0 0what I actualy set manually in auto.home is
-tcp,fsc,noatime,ac,actimeo=120
OK, that looks fine.
Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.
Maybe there's some way to turn off that locking as a workaround.
The simplest thing we can do to help might be implementing "courteous server" behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
--b.
Quoted 64 lines
Quoted 63 lines
--b.
Quoted 59 lines
Quoted 59 lines · Kenneth Johansson
Quoted 55 lines
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTOsync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 2 lines
Quoted 2 lines
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
Quoted 9 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.Maybe there's some way to turn off that locking as a workaround.
Quoted 8 lines
The simplest thing we can do to help might be implementing "courteous server"
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
I've had courtesy locks on my back burner for Ganesha though I hadn't thought about that there might actually be an important practical issue. Does any other server implement them? If we suggest this as a solution to the Chrome suspend issue, it might be good to assure that the major server vendors implement this.
There is a problem with the courtesy locks for this solution though... The clientid is still going to be expired, and the locks are associated with the clientid, so unless we allow courtesy re-instatement of expired clientids, courtesy locks don't actually solve the problem...
Option - use NFSv3 instead :-) The lack of lock expiry due to AWOL client would work in a suspended client's favor... Note also that a suspended client could be a VM, for example, VirtualBox allows saving and suspending a VM in running state.
Interesting problem...
Frank
Quoted 25 lines · Frank Filz
Quoted 10 lines
-----Original Message----- From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.
Maybe there's some way to turn off that locking as a workaround.
Quoted 8 lines
The simplest thing we can do to help might be implementing "courteous server" behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
I've had courtesy locks on my back burner for Ganesha though I hadn't thought about that there might actually be an important practical issue. Does any other server implement them? If we suggest this as a solution to the Chrome suspend issue, it might be good to assure that the major server vendors implement this.
Quoted 5 lines
There is a problem with the courtesy locks for this solution though... The clientid is still going to be expired, and the locks are associated with the clientid, so unless we allow courtesy re-instatement of expired clientids, courtesy locks don't actually solve the problem...
The server's not required to expire the clientid when the lease expires. A server that chooses to be "courteous" can let it hang around.
As a first implementation our server would probably wait until there's a lock conflict, then destroy all the client's state. But we could also choose to revoke only those locks we have to. The client uses TEST_STATEID, I think, to sort out what's happened in that case.
I believe the Linux client implements all of this. I'm not sure about the status of other servers.
--b.
Quoted 18 lines · Frank Filz
Quoted 15 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.Maybe there's some way to turn off that locking as a workaround.
The simplest thing we can do to help might be implementing "courteous server"
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
Quoted 5 lines
Quoted 3 lines
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
I've had courtesy locks on my back burner for Ganesha though I hadn't thought about that there might actually be an important practical issue.
We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.
Extending the lease period impacts the length an NFS server is in grace after a reboot, so it's not always a good solution.
Quoted 1 line
Does any other server implement them? If we suggest this as a solution to the Chrome suspend issue, it might be good to assure that the major server vendors implement this.
We think OnTAP does, at least.
Quoted 1 line
There is a problem with the courtesy locks for this solution though... The clientid is still going to be expired, and the locks are associated with the clientid, so unless we allow courtesy re-instatement of expired clientids, courtesy locks don't actually solve the problem...
An NFSv4 server is not required to expire a lease after the lease period expires.
A courteous server would simply allow a conflicting lock request to take an expired lock after a client's lease expired. If no conflicting lock operations occur, then the missing client could come back and find its lease state intact (unless of course the server has restarted or purged the lease for other reasons).
Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.
Quoted 6 lines
Option - use NFSv3 instead :-) The lack of lock expiry due to AWOL client would work in a suspended client's favor... Note also that a suspended client could be a VM, for example, VirtualBox allows saving and suspending a VM in running state.
Interesting problem...
Frank
--
Chuck Lever
Quoted 7 lines
-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Wednesday, October 7, 2020 8:40 AM
To: Frank Filz <[email protected]>
Cc: Bruce Fields <[email protected]>; Kenneth Johansson
Quoted 15 lines
Patrick Goetz <[email protected]>; Linux NFS Mailing List <linux- [email protected]>
Subject: Re: nfs home directory and google chrome.Quoted 9 lines · Frank Filz
Quoted 8 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default
90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further
IO to that
Quoted 15 lines
file descriptor.
Quoted 4 lines
Quoted 4 lines
Maybe there's some way to turn off that locking as a workaround.
The simplest thing we can do to help might be implementing "courteous
server"
Quoted 9 lines
Quoted 7 lines
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
I've had courtesy locks on my back burner for Ganesha though I hadn't
thought
Quoted 7 lines
about that there might actually be an important practical issue.
We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.
Extending the lease period impacts the length an NFS server is in grace
after a
Quoted 4 lines
reboot, so it's not always a good solution.
Quoted 1 line
Does any other server implement them? If we suggest this as a solution
to the
Quoted 1 line
Chrome suspend issue, it might be good to assure that the major server
vendors
Quoted 6 lines
implement this.
We think OnTAP does, at least.
Quoted 1 line
There is a problem with the courtesy locks for this solution though...
The
Quoted 1 line
clientid is still going to be expired, and the locks are associated with
the clientid,
Quoted 1 line
so unless we allow courtesy re-instatement of expired clientids, courtesy
locks
Quoted 3 lines
don't actually solve the problem...
An NFSv4 server is not required to expire a lease after the lease period
expires.
Quoted 2 lines
A courteous server would simply allow a conflicting lock request to take
an
Quoted 1 line
expired lock after a client's lease expired. If no conflicting lock
operations occur,
Quoted 1 line
then the missing client could come back and find its lease state intact
(unless of
Quoted 5 lines
course the server has restarted or purged the lease for other reasons).
Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.
Ah that document would be helpful. Does the document discuss conditions where a server might abandon a courtesy hold on a client id and expire it out anyway? For example, to conserve resources.
Thanks
Frank
Quoted 83 lines · Frank Filz
Quoted 8 lines
-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Wednesday, October 7, 2020 8:40 AM
To: Frank Filz <[email protected]>
Cc: Bruce Fields <[email protected]>; Kenneth JohanssonQuoted 15 lines
Patrick Goetz <[email protected]>; Linux NFS Mailing List <linux- [email protected]>
Subject: Re: nfs home directory and google chrome.Quoted 9 lines · Frank Filz
Quoted 8 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default
90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any furtherIO to that
Quoted 15 lines
file descriptor.
Quoted 4 lines
Quoted 4 lines
Maybe there's some way to turn off that locking as a workaround.
The simplest thing we can do to help might be implementing "courteous
server"
Quoted 9 lines
Quoted 7 lines
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
I've had courtesy locks on my back burner for Ganesha though I hadn't
thought
Quoted 7 lines
about that there might actually be an important practical issue.
We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.
Extending the lease period impacts the length an NFS server is in grace
after a
Quoted 4 lines
reboot, so it's not always a good solution.
Quoted 1 line
Does any other server implement them? If we suggest this as a solution
to the
Quoted 1 line
Chrome suspend issue, it might be good to assure that the major server
vendors
Quoted 6 lines
implement this.
We think OnTAP does, at least.
Quoted 1 line
There is a problem with the courtesy locks for this solution though...
The
Quoted 1 line
clientid is still going to be expired, and the locks are associated with
the clientid,
Quoted 1 line
so unless we allow courtesy re-instatement of expired clientids, courtesy
locks
Quoted 3 lines
don't actually solve the problem...
An NFSv4 server is not required to expire a lease after the lease period
expires.
Quoted 2 lines
A courteous server would simply allow a conflicting lock request to take
an
Quoted 1 line
expired lock after a client's lease expired. If no conflicting lock
operations occur,
Quoted 1 line
then the missing client could come back and find its lease state intact
(unless of
Quoted 2 lines
course the server has restarted or purged the lease for other reasons).
Quoted 7 lines
Quoted 3 lines
Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.
Ah that document would be helpful. Does the document discuss conditions where a server might abandon a courtesy hold on a client id and expire it out anyway? For example, to conserve resources.
Yes. It covers appropriate server responses to a client to report that it has done this.
Bill will post the document soon in a separate thread.
--
Chuck Lever
Quoted 5 lines · Frank Filz
Quoted 5 lines
-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Wednesday, October 7, 2020 8:40 AM
To: Frank Filz <[email protected]>
Cc: Bruce Fields <[email protected]>; Kenneth Johansson
Quoted 8 lines
Quoted 8 lines
Patrick Goetz <[email protected]>; Linux NFS Mailing List <linux- [email protected]>
Subject: Re: nfs home directory and google chrome.Quoted 2 lines · Frank Filz
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default
90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further
IO to that
Quoted 10 lines
Quoted 10 lines
file descriptor.
Quoted 4 lines
Quoted 4 lines
Maybe there's some way to turn off that locking as a workaround.
The simplest thing we can do to help might be implementing "courteous
server"
Quoted 4 lines
Quoted 4 lines
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
I've had courtesy locks on my back burner for Ganesha though I hadn't
thought
Quoted 7 lines
Quoted 7 lines
about that there might actually be an important practical issue.
We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.
Extending the lease period impacts the length an NFS server is in grace
after a
Quoted 4 lines
Quoted 4 lines
reboot, so it's not always a good solution.
Quoted 1 line
Does any other server implement them? If we suggest this as a solution
to the
Quoted 1 line
Quoted 1 line
Chrome suspend issue, it might be good to assure that the major server
vendors
Quoted 6 lines
Quoted 6 lines
implement this.
We think OnTAP does, at least.
Quoted 1 line
There is a problem with the courtesy locks for this solution though...
The
Quoted 1 line
Quoted 1 line
clientid is still going to be expired, and the locks are associated with
the clientid,
Quoted 1 line
Quoted 1 line
so unless we allow courtesy re-instatement of expired clientids, courtesy
locks
Quoted 3 lines
Quoted 3 lines
don't actually solve the problem...
An NFSv4 server is not required to expire a lease after the lease period
expires.
The way the FreeBSD server is implemented is that does not expire a clientID (and all associated open/byte range lock state) when a lease expires. However, a conflicting Open/Lock request results in the ClientID and all associated opens/byte range locks being expired at that point in time.
I was never sure if a NFSv4 client would expect NFS4ERR_EXPIRED to be returned for some lock, but not all state related operations. so I chose to expire ClientID + all Opens and Locks at the same time.
--> However, this is only triggered by a conflicting Open/Lock request or
server resource depletion and not simply a client failing to renew a lease.
rick
ps: It has been like this for almost 20years and I have not heard of it
causing problems.
Quoted 2 lines
A courteous server would simply allow a conflicting lock request to take
an
Quoted 1 line
expired lock after a client's lease expired. If no conflicting lock
operations occur,
Quoted 1 line
then the missing client could come back and find its lease state intact
(unless of
Quoted 5 lines
course the server has restarted or purged the lease for other reasons).
Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.
Ah that document would be helpful. Does the document discuss conditions where a server might abandon a courtesy hold on a client id and expire it out anyway? For example, to conserve resources.
Thanks
Frank
Quoted 40 lines · J. Bruce Fields
Quoted 39 lines · Kenneth Johansson
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz
We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set
storage.nfs_filesystem: true
This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.
Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.
I did a strace and the first
-EIOI get look like thisfdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)
then the same thing happens for other files like
Quoted 8 lines
Quoted 5 lines
fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)
fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)
Are you using soft mounts?
(What are your mount options?)
auto.home /home autofs rw,relatime,fd=18,pgrp=2682,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=67621 0 0
/home/kenjonfs4 rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.16,fsc,local_lock=none,addr=172.16.2.6 0 0what I actualy set manually in auto.home is
Quoted 8 lines
Quoted 1 line
-tcp,fsc,noatime,ac,actimeo=120
OK, that looks fine.
Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.
So I tested by just pulling the network cable for a few minutes and the effect is the same. this time the password database survived but the history went bye bye.
History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT
The real problem here is that chrome destroys the database file and do not communicate that there is a problem to the end user unless started from a terminal and it can't recover ever. If it had just removed the file and recreated a new one from the server it would not be a big problem. What the user notice is that password handling and form filling do work very poorly and the only solution is really "rm -rf ~/.config/google-chrome/" to force it to recreate the files.
Quoted 74 lines
Maybe there's some way to turn off that locking as a workaround.
The simplest thing we can do to help might be implementing "courteous server" behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.
There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.
--b.
Quoted 61 lines
Quoted 61 lines
--b.
Quoted 58 lines
Quoted 58 lines · Kenneth Johansson
Quoted 55 lines
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTOsync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 2 lines
Quoted 2 lines
This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<
So this is just an update to how to avoid this problem.
I switched to nfs v3 and no more issues. Since the switch chrome have not stopped syncing with the google server even once. suspend resume causes no issues and everything looks ok. So it's clear that google-chrome currently does not like nfs v4 and I need chrome to work more than I need to run nfs v4.
Quoted 30 lines · Kenneth Johansson
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
autofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
sync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.
Quoted 3 lines · Kenneth Johansson
So this is just an update to how to avoid this problem.
I switched to nfs v3 and no more issues.
Yes, that's also consistent with the explanation that the problem is client lease expiry.
NFSv4 locks are lease-based--the client loses all its locks if it doesn't contact the server regularly (by default, about every 90 seconds). So, if you suspend or lose contact with the server for too long, then you lose your locks.
NFSv3 (NLM) locks are not. The client keeps them until it unlocks them or explicitly tells the server to remove them all (such as if it comes back up after crashing). So, there's no risk of losing locks when you suspend, but there's more of a risk of stuck locks that get in other client's way when one client dies.
Once we implement "courteous server", locks will only be removed once the client loses contact for more than 90 seconds *and* either another client requests a conflicting lock, or the server just runs out of memory for client state. I think that'll be a better compromise.
--b.
Quoted 57 lines
Since the switch chrome
have not stopped syncing with the google server even once. suspend resume causes no issues and everything looks ok. So it's clear that google-chrome currently does not like nfs v4 and I need chrome to work more than I need to run nfs v4.Quoted 49 lines · Kenneth Johansson
So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.
it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.
How to get the error.
1. start google from a terminal with "google-chrome"
2. suspend the computer
3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning
4. look for printout that looks like something like this
[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
autofill_model_type_state(model_type, value)VALUES(?,?)[16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO
sync_model_metadata(id,model_metadata)VALUES(1, ?)[16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTOautofill_model_type_state(model_type, value)VALUES(?,?)and so on. if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.