Skip to contentSkip to conversation

nfs home directory and google chrome.

DISCUSSION Kenneth Johansson 6 participants 15 messages Oct 4, 2020 – Oct 29, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

#1 lore raw

We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set

storage.nfs_filesystem: true

This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.

Quoted 32 lines · Kenneth Johansson

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

Quoted 2 lines
Quoted 2 lines

This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf.                        <<

#2 lore raw
Quoted 6 lines · Patrick Goetz

We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set

   storage.nfs_filesystem: true

This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.

Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.

I did a strace and the first -EIO I get look like this

fdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)

then the same thing happens for other files like

fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)

fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)

Quoted 33 lines · Kenneth Johansson
Quoted 32 lines

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

Quoted 2 lines
Quoted 2 lines

This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<

#3 lore raw
Quoted 19 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz

We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set

   storage.nfs_filesystem: true

This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.

Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.

I did a strace and the first -EIO I get look like this

fdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)

Quoted 7 lines

then the same thing happens for other files like

fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)

fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)

Are you using soft mounts?

(What are your mount options?)

--b.

Quoted 61 lines
Quoted 60 lines · Kenneth Johansson
Quoted 55 lines

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

Quoted 2 lines
Quoted 2 lines

This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<

#4 lore raw
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz

We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set

   storage.nfs_filesystem: true

This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.

Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.

I did a strace and the first -EIO I get look like this

fdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)

then the same thing happens for other files like

Quoted 8 lines
Quoted 5 lines

fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)

fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)

Are you using soft mounts?

(What are your mount options?)

auto.home /home autofs rw,relatime,fd=18,pgrp=2682,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=67621 0 0

/home/kenjo nfs4 rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.16,fsc,local_lock=none,addr=172.16.2.6 0 0

what I actualy set manually in auto.home is

-tcp,fsc,noatime,ac,actimeo=120

Quoted 61 lines

--b.

Quoted 59 lines
Quoted 59 lines · Kenneth Johansson
Quoted 55 lines

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

Quoted 2 lines
Quoted 2 lines

This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<

#5 lore raw
Quoted 35 lines · Kenneth Johansson
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz

We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set

   storage.nfs_filesystem: true

This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.

Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.

I did a strace and the first -EIO I get look like this

fdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)

then the same thing happens for other files like

Quoted 8 lines
Quoted 5 lines

fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)

fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)

Are you using soft mounts?

(What are your mount options?)

auto.home /home autofs rw,relatime,fd=18,pgrp=2682,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=67621 0 0

Quoted 6 lines

/home/kenjo nfs4 rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.16,fsc,local_lock=none,addr=172.16.2.6 0 0

what I actualy set manually in auto.home is

-tcp,fsc,noatime,ac,actimeo=120

OK, that looks fine.

Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.

Maybe there's some way to turn off that locking as a workaround.

The simplest thing we can do to help might be implementing "courteous server" behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

--b.

Quoted 64 lines
Quoted 63 lines

--b.

Quoted 59 lines
Quoted 59 lines · Kenneth Johansson
Quoted 55 lines

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

Quoted 2 lines
Quoted 2 lines

This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<

#6 lore raw
Quoted 9 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.

Maybe there's some way to turn off that locking as a workaround.

Quoted 8 lines

The simplest thing we can do to help might be implementing "courteous server"
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

I've had courtesy locks on my back burner for Ganesha though I hadn't thought about that there might actually be an important practical issue. Does any other server implement them? If we suggest this as a solution to the Chrome suspend issue, it might be good to assure that the major server vendors implement this.

There is a problem with the courtesy locks for this solution though... The clientid is still going to be expired, and the locks are associated with the clientid, so unless we allow courtesy re-instatement of expired clientids, courtesy locks don't actually solve the problem...

Option - use NFSv3 instead :-) The lack of lock expiry due to AWOL client would work in a suspended client's favor... Note also that a suspended client could be a VM, for example, VirtualBox allows saving and suspending a VM in running state.

Interesting problem...

Frank

#7 lore raw
Quoted 25 lines · Frank Filz
Quoted 10 lines

-----Original Message----- From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.

Maybe there's some way to turn off that locking as a workaround.

Quoted 8 lines

The simplest thing we can do to help might be implementing "courteous server" behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

I've had courtesy locks on my back burner for Ganesha though I hadn't thought about that there might actually be an important practical issue. Does any other server implement them? If we suggest this as a solution to the Chrome suspend issue, it might be good to assure that the major server vendors implement this.

Quoted 5 lines

There is a problem with the courtesy locks for this solution though... The clientid is still going to be expired, and the locks are associated with the clientid, so unless we allow courtesy re-instatement of expired clientids, courtesy locks don't actually solve the problem...

The server's not required to expire the clientid when the lease expires. A server that chooses to be "courteous" can let it hang around.

As a first implementation our server would probably wait until there's a lock conflict, then destroy all the client's state. But we could also choose to revoke only those locks we have to. The client uses TEST_STATEID, I think, to sort out what's happened in that case.

I believe the Linux client implements all of this. I'm not sure about the status of other servers.

--b.

#8 lore raw
In reply to Frank Filz
Quoted 18 lines · Frank Filz
Quoted 15 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.

Maybe there's some way to turn off that locking as a workaround.

The simplest thing we can do to help might be implementing "courteous server"
behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

Quoted 5 lines
Quoted 3 lines

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

I've had courtesy locks on my back burner for Ganesha though I hadn't thought about that there might actually be an important practical issue.

We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.

Extending the lease period impacts the length an NFS server is in grace after a reboot, so it's not always a good solution.

Quoted 1 line

Does any other server implement them? If we suggest this as a solution to the Chrome suspend issue, it might be good to assure that the major server vendors implement this.

We think OnTAP does, at least.

Quoted 1 line

There is a problem with the courtesy locks for this solution though... The clientid is still going to be expired, and the locks are associated with the clientid, so unless we allow courtesy re-instatement of expired clientids, courtesy locks don't actually solve the problem...

An NFSv4 server is not required to expire a lease after the lease period expires.

A courteous server would simply allow a conflicting lock request to take an expired lock after a client's lease expired. If no conflicting lock operations occur, then the missing client could come back and find its lease state intact (unless of course the server has restarted or purged the lease for other reasons).

Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.

Quoted 6 lines

Option - use NFSv3 instead :-) The lack of lock expiry due to AWOL client would work in a suspended client's favor... Note also that a suspended client could be a VM, for example, VirtualBox allows saving and suspending a VM in running state.

Interesting problem...

Frank

--
Chuck Lever

#9 lore raw
Quoted 7 lines

-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Wednesday, October 7, 2020 8:40 AM
To: Frank Filz <[email protected]>
Cc: Bruce Fields <[email protected]>; Kenneth Johansson

<[email protected]>;

Quoted 15 lines

Patrick Goetz <[email protected]>; Linux NFS Mailing List <linux- [email protected]>
Subject: Re: nfs home directory and google chrome.

Quoted 9 lines · Frank Filz
Quoted 8 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default
90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further

IO to that

Quoted 15 lines

file descriptor.

Quoted 4 lines
Quoted 4 lines

Maybe there's some way to turn off that locking as a workaround.

The simplest thing we can do to help might be implementing "courteous

server"

Quoted 9 lines
Quoted 7 lines

behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

I've had courtesy locks on my back burner for Ganesha though I hadn't

thought

Quoted 7 lines

about that there might actually be an important practical issue.

We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.

Extending the lease period impacts the length an NFS server is in grace

after a

Quoted 4 lines

reboot, so it's not always a good solution.

Quoted 1 line

Does any other server implement them? If we suggest this as a solution

to the

Quoted 1 line

Chrome suspend issue, it might be good to assure that the major server

vendors

Quoted 6 lines

implement this.

We think OnTAP does, at least.

Quoted 1 line

There is a problem with the courtesy locks for this solution though...

The

Quoted 1 line

clientid is still going to be expired, and the locks are associated with

the clientid,

Quoted 1 line

so unless we allow courtesy re-instatement of expired clientids, courtesy

locks

Quoted 3 lines

don't actually solve the problem...

An NFSv4 server is not required to expire a lease after the lease period

expires.

Quoted 2 lines

A courteous server would simply allow a conflicting lock request to take

an

Quoted 1 line

expired lock after a client's lease expired. If no conflicting lock

operations occur,

Quoted 1 line

then the missing client could come back and find its lease state intact

(unless of

Quoted 5 lines

course the server has restarted or purged the lease for other reasons).

Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.

Ah that document would be helpful. Does the document discuss conditions where a server might abandon a courtesy hold on a client id and expire it out anyway? For example, to conserve resources.

Thanks

Frank

#10 lore raw
Quoted 83 lines · Frank Filz
Quoted 8 lines

-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Wednesday, October 7, 2020 8:40 AM
To: Frank Filz <[email protected]>
Cc: Bruce Fields <[email protected]>; Kenneth Johansson

<[email protected]>;

Quoted 15 lines

Patrick Goetz <[email protected]>; Linux NFS Mailing List <linux- [email protected]>
Subject: Re: nfs home directory and google chrome.

Quoted 9 lines · Frank Filz
Quoted 8 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default
90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further

IO to that

Quoted 15 lines

file descriptor.

Quoted 4 lines
Quoted 4 lines

Maybe there's some way to turn off that locking as a workaround.

The simplest thing we can do to help might be implementing "courteous

server"

Quoted 9 lines
Quoted 7 lines

behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

I've had courtesy locks on my back burner for Ganesha though I hadn't

thought

Quoted 7 lines

about that there might actually be an important practical issue.

We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.

Extending the lease period impacts the length an NFS server is in grace

after a

Quoted 4 lines

reboot, so it's not always a good solution.

Quoted 1 line

Does any other server implement them? If we suggest this as a solution

to the

Quoted 1 line

Chrome suspend issue, it might be good to assure that the major server

vendors

Quoted 6 lines

implement this.

We think OnTAP does, at least.

Quoted 1 line

There is a problem with the courtesy locks for this solution though...

The

Quoted 1 line

clientid is still going to be expired, and the locks are associated with

the clientid,

Quoted 1 line

so unless we allow courtesy re-instatement of expired clientids, courtesy

locks

Quoted 3 lines

don't actually solve the problem...

An NFSv4 server is not required to expire a lease after the lease period

expires.

Quoted 2 lines

A courteous server would simply allow a conflicting lock request to take

an

Quoted 1 line

expired lock after a client's lease expired. If no conflicting lock

operations occur,

Quoted 1 line

then the missing client could come back and find its lease state intact

(unless of

Quoted 2 lines

course the server has restarted or purged the lease for other reasons).

Quoted 7 lines
Quoted 3 lines

Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.

Ah that document would be helpful. Does the document discuss conditions where a server might abandon a courtesy hold on a client id and expire it out anyway? For example, to conserve resources.

Yes. It covers appropriate server responses to a client to report that it has done this.

Bill will post the document soon in a separate thread.

--
Chuck Lever

#11 lore raw
In reply to Frank Filz
Quoted 5 lines · Frank Filz
Quoted 5 lines

-----Original Message-----
From: Chuck Lever [mailto:[email protected]]
Sent: Wednesday, October 7, 2020 8:40 AM
To: Frank Filz <[email protected]>
Cc: Bruce Fields <[email protected]>; Kenneth Johansson

<[email protected]>;

Quoted 8 lines
Quoted 8 lines

Patrick Goetz <[email protected]>; Linux NFS Mailing List <linux- [email protected]>
Subject: Re: nfs home directory and google chrome.

Quoted 2 lines · Frank Filz
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]] Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default
90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further

IO to that

Quoted 10 lines
Quoted 10 lines

file descriptor.

Quoted 4 lines
Quoted 4 lines

Maybe there's some way to turn off that locking as a workaround.

The simplest thing we can do to help might be implementing "courteous

server"

Quoted 4 lines
Quoted 4 lines

behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

I've had courtesy locks on my back burner for Ganesha though I hadn't

thought

Quoted 7 lines
Quoted 7 lines

about that there might actually be an important practical issue.

We've found that instantly bringing the hammer down on NFSv4 leases has negative operational consequences in environments where minutes-long network partitions are part of life.

Extending the lease period impacts the length an NFS server is in grace

after a

Quoted 4 lines
Quoted 4 lines

reboot, so it's not always a good solution.

Quoted 1 line

Does any other server implement them? If we suggest this as a solution

to the

Quoted 1 line
Quoted 1 line

Chrome suspend issue, it might be good to assure that the major server

vendors

Quoted 6 lines
Quoted 6 lines

implement this.

We think OnTAP does, at least.

Quoted 1 line

There is a problem with the courtesy locks for this solution though...

The

Quoted 1 line
Quoted 1 line

clientid is still going to be expired, and the locks are associated with

the clientid,

Quoted 1 line
Quoted 1 line

so unless we allow courtesy re-instatement of expired clientids, courtesy

locks

Quoted 3 lines
Quoted 3 lines

don't actually solve the problem...

An NFSv4 server is not required to expire a lease after the lease period

expires.
The way the FreeBSD server is implemented is that does not expire a clientID (and all associated open/byte range lock state) when a lease expires. However, a conflicting Open/Lock request results in the ClientID and all associated opens/byte range locks being expired at that point in time.

I was never sure if a NFSv4 client would expect NFS4ERR_EXPIRED to be returned for some lock, but not all state related operations. so I chose to expire ClientID + all Opens and Locks at the same time.
--> However, this is only triggered by a conflicting Open/Lock request or
server resource depletion and not simply a client failing to renew a lease.

rick
ps: It has been like this for almost 20years and I have not heard of it
causing problems.

Quoted 2 lines

A courteous server would simply allow a conflicting lock request to take

an

Quoted 1 line

expired lock after a client's lease expired. If no conflicting lock

operations occur,

Quoted 1 line

then the missing client could come back and find its lease state intact

(unless of

Quoted 5 lines

course the server has restarted or purged the lease for other reasons).

Oracle has an open design document that can be posted here for more comment and review. We agree that this is much better server behavior and would like more server implementations to adopt it.

Ah that document would be helpful. Does the document discuss conditions where a server might abandon a courtesy hold on a client id and expire it out anyway? For example, to conserve resources.

Thanks

Frank

#12 lore raw
In reply to J. Bruce Fields
Quoted 40 lines · J. Bruce Fields
Quoted 39 lines · Kenneth Johansson
Quoted 22 lines · J. Bruce Fields
Quoted 21 lines · Kenneth Johansson
Quoted 10 lines · Patrick Goetz

We had a similar problem with Firefox, most notably with Mac OSX users who have NFS-mounted home directories. There's an about:config solution for Firefox; namely set

   storage.nfs_filesystem: true

This forces a specific network file locking mechanism which makes sqlite behave better. I'm guessing google chrome has something similar.

Since I have used chrome for years without any problems my guess it that its something that changed with nfs in my setup.

I did a strace and the first -EIO I get look like this

fdatasync(94</home/kenjo/.config/google-chrome/Default/Login Data>) = -1 EIO (Input/output error)

then the same thing happens for other files like

Quoted 8 lines
Quoted 5 lines

fdatasync(83</home/kenjo/.config/google-chrome/Default/Web Data>) = -1 EIO (Input/output error)

fdatasync(74</home/kenjo/.config/google-chrome/Default/History>) = -1 EIO (Input/output error)

Are you using soft mounts?

(What are your mount options?)

auto.home /home autofs rw,relatime,fd=18,pgrp=2682,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=67621 0 0

/home/kenjo nfs4 rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.16.2.16,fsc,local_lock=none,addr=172.16.2.6 0 0

what I actualy set manually in auto.home is

Quoted 8 lines
Quoted 1 line

-tcp,fsc,noatime,ac,actimeo=120

OK, that looks fine.

Maybe I overlooked the obvious: if Chrome holds a lock on that file when you suspend, and if you stay in suspend for longer than the NFSv4 lease time (default 90 seconds), then the client will lose its lease, hence any file locks. I think these days the client then returns EIO on any further IO to that file descriptor.

So I tested by just pulling the network cable for a few minutes and the effect is the same. this time the password  database survived but the history went bye bye.

History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT

The real problem here is that chrome destroys the database file and do not communicate that there is a problem to the end user unless started from a terminal and it can't recover ever. If it had just removed the file and recreated a new one from the server it would not be a big problem. What the user notice is that password handling and form filling do work very poorly and the only solution is really  "rm -rf  ~/.config/google-chrome/" to force it to recreate the files.

Quoted 74 lines

Maybe there's some way to turn off that locking as a workaround.

The simplest thing we can do to help might be implementing "courteous server" behavior: instead of automatically removing locks after a client's lease expires, it can wait until there's an actual lock conflict. That might be enough for your case.

There's been a little planning done and it's not a big project, but I don't think it's actually at the top of anyone's todo list right now, so I'm not sure when that will get done.

--b.

Quoted 61 lines
Quoted 61 lines

--b.

Quoted 58 lines
Quoted 58 lines · Kenneth Johansson
Quoted 55 lines

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
COMMIT
[16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql:
INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

Quoted 2 lines
Quoted 2 lines

This message is from an external sender. Learn more about why this << matters at https://links.utexas.edu/rtyclf. <<

#13 lore raw
In reply to Kenneth Johansson

So this is just an update to how to avoid this problem.

I switched to nfs v3 and no more issues. Since the switch chrome have not stopped syncing with the google server even once. suspend resume causes no issues and everything looks ok.  So it's clear that google-chrome currently does not like nfs v4 and I need chrome to work more than I need to run nfs v4.

Quoted 30 lines · Kenneth Johansson

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.

#14 lore raw
Quoted 3 lines · Kenneth Johansson

So this is just an update to how to avoid this problem.

I switched to nfs v3 and no more issues.

Yes, that's also consistent with the explanation that the problem is client lease expiry.

NFSv4 locks are lease-based--the client loses all its locks if it doesn't contact the server regularly (by default, about every 90 seconds). So, if you suspend or lose contact with the server for too long, then you lose your locks.

NFSv3 (NLM) locks are not. The client keeps them until it unlocks them or explicitly tells the server to remove them all (such as if it comes back up after crashing). So, there's no risk of losing locks when you suspend, but there's more of a risk of stuck locks that get in other client's way when one client dies.

Once we implement "courteous server", locks will only be removed once the client loses contact for more than 90 seconds *and* either another client requests a conflicting lock, or the server just runs out of memory for client state. I think that'll be a better compromise.

--b.

Quoted 57 lines

Since the switch chrome
have not stopped syncing with the google server even once. suspend resume causes no issues and everything looks ok.  So it's clear that google-chrome currently does not like nfs v4 and I need chrome to work more than I need to run nfs v4.

Quoted 49 lines · Kenneth Johansson

So I have had for a long time problems with google chrome and suspend resume causing it to mangle its sqlite database.

it looks to only happen if I use nfs mounted home directory. I'm not sure exactly what is happening but lets first see if this happens to anybody else.

How to get the error.

1. start google from a terminal with "google-chrome"

2. suspend the computer

3. wait a while. There is some type of minimum time here I do not know what its is but I basically get the error every time of I suspend in evening and resume in morning

4. look for printout that looks like something like this

[16789:18181:1004/125852.529750:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.529744:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:16829:1004/125852.530261:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?) [16789:16789:1004/125852.563571:ERROR:sync_metadata_store_change_list.cc(34)] Autofill datatype error was encountered: Failed to update ModelTypeState. [16789:19002:1004/125902.534103:ERROR:database.cc(1692)] History sqlite error 1034, errno 5: disk I/O error, sql: COMMIT [16789:19002:1004/125902.536903:ERROR:database.cc(1692)] Thumbnail sqlite error 778, errno 5: disk I/O error, sql: COMMIT

[16789:19002:1004/130044.120379:ERROR:database.cc(1692)] Passwords sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO sync_model_metadata (id, model_metadata) VALUES(1, ?) [16789:16829:1004/130044.120388:ERROR:database.cc(1692)] Web sqlite error 1034, errno 5: disk I/O error, sql: INSERT OR REPLACE INTO autofill_model_type_state (model_type, value) VALUES(?,?)

and so on.  if you use google sync you can also check "chrome://sync-internals" to see if something is wrong with the database.