Regression, bisected: sqlite locking failure on nfs
Subsystems (3)
After installing 2.6.37-rc1, attempting to use sqlite in any capacity on NFS gives a locking error:
% echo 'select * from blah;' | sqlite3 blah.sqlite
Error: near line 1: database is locked
% echo 'create table blargh(INT);' | sqlite3 blargh.sqlite
Error: near line 1: database is locked
The result is that a lot of high-profile applications which make use of sqlite fail mysteriously. Bisection reveals the following, and reverting the implicated commit solves the issue:
9247685088398cf21bcb513bd2832b4cd42516c4 is the first bad commit commit 9247685088398cf21bcb513bd2832b4cd42516c4
Author: Chuck Lever <[email protected]> Date: Wed Oct 20 11:53:01 2010 -0400
SUNRPC: Properly initialize sock_xprt.srcaddr in all cases
The source address field in the transport's sock_xprt is initialized ONLY IF the RPC application passed a pointer to a source address during the call to rpc_create(). However, xs_bind() subsequently uses the value of this field without regard to whether the source address was initialized during transport creation or not.
So far we've been lucky: the uninitialized value of this field is zeroes. xs_bind(), until recently, used only the sin[6]_addr field in this sockaddr, and all zeroes is a valid value for this: it means ANYADDR. This is a happy coincidence.
However, xs_bind() now wants to use the sa_family field as well, and expects it to be initialized to something other than zero.
Therefore, the source address sockaddr field should be fully initialized at transport create time in _every_ case, not just when the RPC application wants to use a specific bind address.
Bruce added a workaround for this missing initialization by adjusting commit 6bc9638a, but the "right" way to do this is to ensure that the source address sockaddr is always correctly initialized from the get-go.
This patch doesn't introduce a behavior change. It's simply a clean-up of Bruce's fix, to prevent future problems of this kind. It may look like overkill, but
a) it clearly documents the default initial value of this field,
b) it doesn't assume that the sockaddr_storage memory is first initialized to any particular value, and
c) it will fail verbosely if some unknown address family is passed in
Originally introduced by commit d3bc9a1d.
:040000 040000 843e2423b5a45f7bc13b9e30cc8e5bcb072dbb8f 7148b5256ce78fd5c98a01f1bdcac46faa80f773 M net
git bisect start
# bad: [c8ddb2713c624f432fa5fe3c7ecffcdda46ea0d4] Linux 2.6.37-rc1
git bisect bad c8ddb2713c624f432fa5fe3c7ecffcdda46ea0d4
# good: [f6f94e2ab1b33f0082ac22d71f66385a60d8157f] Linux 2.6.36
git bisect good f6f94e2ab1b33f0082ac22d71f66385a60d8157f
# good: [33081adf8b89d5a716d7e1c60171768d39795b39] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound-2.6
git bisect good 33081adf8b89d5a716d7e1c60171768d39795b39
# bad: [00ebb6382b8d9c7c15b5f8ad230670d8161d38dd] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/cjb/mmc
git bisect bad 00ebb6382b8d9c7c15b5f8ad230670d8161d38dd
# bad: [520045db940a381d2bee1c1b2179f7921b40fb10] Merge branches 'upstream/xenfs' and 'upstream/core' of git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen
git bisect bad 520045db940a381d2bee1c1b2179f7921b40fb10
# bad: [4390110fef9e5c64e10c6ca19d586932242c9a8a] Merge branch 'for-2.6.37' of git://linux-nfs.org/~bfields/linux
git bisect bad 4390110fef9e5c64e10c6ca19d586932242c9a8a
# good: [7b6181e06841f5ad15c4ff708b967b4db65a64de] Merge branch 'omap-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap-2.6
git bisect good 7b6181e06841f5ad15c4ff708b967b4db65a64de
# good: [e0e170bd7ded2ec16e2813d63c0faff43193fde8] Merge branch 'next' of git://git.monstr.eu/linux-2.6-microblaze
git bisect good e0e170bd7ded2ec16e2813d63c0faff43193fde8
# good: [22f793268de3b4dff8abfcd873ba7afc1f34224f] sunrpc: Factor out v4 sockets creation
git bisect good 22f793268de3b4dff8abfcd873ba7afc1f34224f
# good: [a4dd8dce14014665862ce7911b38cb2c69e366dd] Merge branch 'nfs-for-2.6.37' of git://git.linux-nfs.org/projects/trondmy/nfs-2.6
git bisect good a4dd8dce14014665862ce7911b38cb2c69e366dd
# bad: [cd5b814458e5554457c6e62f17aed122145b065e] nfsd4: don't cache seq_misordered replies
git bisect bad cd5b814458e5554457c6e62f17aed122145b065e
# good: [8c14ff2aaf26d58aa2258a59bd419c906d105938] sunrpc: Remove UDP worker wrappers
git bisect good 8c14ff2aaf26d58aa2258a59bd419c906d105938
# good: [8f3a6de313391b6910aa7db185eb9f3e930a51cf] sunrpc: Turn list_for_each-s into the ..._entry-s
git bisect good 8f3a6de313391b6910aa7db185eb9f3e930a51cf
# good: [4232e8634ad82c5a53389e4016de15a8b15c09c3] SUNRPC: Use conventional switch statement when reclassifying sockets
git bisect good 4232e8634ad82c5a53389e4016de15a8b15c09c3
# bad: [9247685088398cf21bcb513bd2832b4cd42516c4] SUNRPC: Properly initialize sock_xprt.srcaddr in all cases
git bisect bad 9247685088398cf21bcb513bd2832b4cd42516c4
Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Quoted 6 lines · Nick Bowler
After installing 2.6.37-rc1, attempting to use sqlite in any capacity on NFS gives a locking error:
% echo 'select * from blah;' | sqlite3 blah.sqlite
Error: near line 1: database is locked
Quoted 6 lines
% echo 'create table blargh(INT);' | sqlite3 blargh.sqlite
Error: near line 1: database is lockedThe result is that a lot of high-profile applications which make use of sqlite fail mysteriously. Bisection reveals the following, and reverting the implicated commit solves the issue:
Nick, thanks for the report. Is 2.6.37-rc1 running on your clients or on your server? Does anything interesting appear in the kernel log when your test case fails?
Quoted 83 lines
9247685088398cf21bcb513bd2832b4cd42516c4is the first bad commit commit9247685088398cf21bcb513bd2832b4cd42516c4Author: Chuck Lever <[email protected]> Date: Wed Oct 20 11:53:01 2010 -0400SUNRPC: Properly initialize sock_xprt.srcaddr in all cases
The source address field in the transport's
sock_xprtis initialized ONLY IF the RPC application passed a pointer to a source address during the call torpc_create(). However,xs_bind()subsequently uses the value of this field without regard to whether the source address was initialized during transport creation or not.So far we've been lucky: the uninitialized value of this field is zeroes.
xs_bind(), until recently, used only the sin[6]_addr field in this sockaddr, and all zeroes is a valid value for this: it means ANYADDR. This is a happy coincidence.However,
xs_bind()now wants to use thesa_familyfield as well, and expects it to be initialized to something other than zero.Therefore, the source address sockaddr field should be fully initialized at transport create time in _every_ case, not just when the RPC application wants to use a specific bind address.
Bruce added a workaround for this missing initialization by adjusting commit
6bc9638a, but the "right" way to do this is to ensure that the source address sockaddr is always correctly initialized from the get-go.This patch doesn't introduce a behavior change. It's simply a clean-up of Bruce's fix, to prevent future problems of this kind. It may look like overkill, but
a) it clearly documents the default initial value of this field,
b) it doesn't assume that the
sockaddr_storagememory is first initialized to any particular value, andc) it will fail verbosely if some unknown address family is passed in
Originally introduced by commit
d3bc9a1d.Signed-off-by Chuck Lever <[email protected]>Signed-off-by J. Bruce Fields <[email protected]>:040000 040000
843e2423b5a45f7bc13b9e30cc8e5bcb072dbb8f7148b5256ce78fd5c98a01f1bdcac46faa80f773M netgit bisect start
# bad: [c8ddb2713c624f432fa5fe3c7ecffcdda46ea0d4] Linux 2.6.37-rc1
git bisect badc8ddb2713c624f432fa5fe3c7ecffcdda46ea0d4
# good: [f6f94e2ab1b33f0082ac22d71f66385a60d8157f] Linux 2.6.36
git bisect goodf6f94e2ab1b33f0082ac22d71f66385a60d8157f
# good: [33081adf8b89d5a716d7e1c60171768d39795b39] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound-2.6
git bisect good33081adf8b89d5a716d7e1c60171768d39795b39
# bad: [00ebb6382b8d9c7c15b5f8ad230670d8161d38dd] Merge branch 'for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/cjb/mmc
git bisect bad00ebb6382b8d9c7c15b5f8ad230670d8161d38dd
# bad: [520045db940a381d2bee1c1b2179f7921b40fb10] Merge branches 'upstream/xenfs' and 'upstream/core' ofgit://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen
git bisect bad520045db940a381d2bee1c1b2179f7921b40fb10
# bad: [4390110fef9e5c64e10c6ca19d586932242c9a8a] Merge branch 'for-2.6.37' ofgit://linux-nfs.org/~bfields/linux
git bisect bad4390110fef9e5c64e10c6ca19d586932242c9a8a
# good: [7b6181e06841f5ad15c4ff708b967b4db65a64de] Merge branch 'omap-for-linus' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap-2.6
git bisect good7b6181e06841f5ad15c4ff708b967b4db65a64de
# good: [e0e170bd7ded2ec16e2813d63c0faff43193fde8] Merge branch 'next' ofgit://git.monstr.eu/linux-2.6-microblaze
git bisect goode0e170bd7ded2ec16e2813d63c0faff43193fde8
# good: [22f793268de3b4dff8abfcd873ba7afc1f34224f] sunrpc: Factor out v4 sockets creation
git bisect good22f793268de3b4dff8abfcd873ba7afc1f34224f
# good: [a4dd8dce14014665862ce7911b38cb2c69e366dd] Merge branch 'nfs-for-2.6.37' ofgit://git.linux-nfs.org/projects/trondmy/nfs-2.6
git bisect gooda4dd8dce14014665862ce7911b38cb2c69e366dd
# bad: [cd5b814458e5554457c6e62f17aed122145b065e] nfsd4: don't cacheseq_misorderedreplies
git bisect badcd5b814458e5554457c6e62f17aed122145b065e
# good: [8c14ff2aaf26d58aa2258a59bd419c906d105938] sunrpc: Remove UDP worker wrappers
git bisect good8c14ff2aaf26d58aa2258a59bd419c906d105938
# good: [8f3a6de313391b6910aa7db185eb9f3e930a51cf] sunrpc: Turn list_for_each-s into the ..._entry-s
git bisect good8f3a6de313391b6910aa7db185eb9f3e930a51cf
# good: [4232e8634ad82c5a53389e4016de15a8b15c09c3] SUNRPC: Use conventional switch statement when reclassifying sockets
git bisect good4232e8634ad82c5a53389e4016de15a8b15c09c3
# bad: [9247685088398cf21bcb513bd2832b4cd42516c4] SUNRPC: Properly initialize sock_xprt.srcaddr in all cases
git bisect bad9247685088398cf21bcb513bd2832b4cd42516c4Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 10 lines · Chuck Lever
Quoted 9 lines · Nick Bowler
After installing 2.6.37-rc1, attempting to use sqlite in any capacity on NFS gives a locking error:
% echo 'select * from blah;' | sqlite3 blah.sqlite
Error: near line 1: database is locked% echo 'create table blargh(INT);' | sqlite3 blargh.sqlite
Error: near line 1: database is locked
Quoted 6 lines
Quoted 3 lines
The result is that a lot of high-profile applications which make use of sqlite fail mysteriously. Bisection reveals the following, and reverting the implicated commit solves the issue:
Nick, thanks for the report. Is 2.6.37-rc1 running on your clients or on your server?
Sorry for not being clear: the client is running 2.6.37-rc1. The
server is running RHEL 5.5.
Quoted 2 lines
Does anything interesting appear in the kernel log when your test case fails?
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
(192.168.8.199 is the address of the failing client). I can only assume that these are a result of my recent issues, since I don't have access to the system log (with timestamps) on that machine.
Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Quoted 33 lines · Nick Bowler
Quoted 10 lines · Chuck Lever
Quoted 9 lines · Nick Bowler
After installing 2.6.37-rc1, attempting to use sqlite in any capacity on NFS gives a locking error:
% echo 'select * from blah;' | sqlite3 blah.sqlite
Error: near line 1: database is locked% echo 'create table blargh(INT);' | sqlite3 blargh.sqlite
Error: near line 1: database is lockedQuoted 6 lines
Quoted 3 lines
The result is that a lot of high-profile applications which make use of sqlite fail mysteriously. Bisection reveals the following, and reverting the implicated commit solves the issue:
Nick, thanks for the report. Is 2.6.37-rc1 running on your clients or on your server?
Sorry for not being clear: the client is running 2.6.37-rc1. The
server is running RHEL 5.5.Quoted 2 lines
Does anything interesting appear in the kernel log when your test case fails?
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
Quoted 3 lines
(192.168.8.199 is the address of the failing client). I can only assume that these are a result of my recent issues, since I don't have access to the system log (with timestamps) on that machine.
That's the problem this patch is supposed to prevent. I'll investigate further.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 35 lines · Chuck Lever
Quoted 33 lines · Nick Bowler
Quoted 10 lines · Chuck Lever
Quoted 9 lines · Nick Bowler
After installing 2.6.37-rc1, attempting to use sqlite in any capacity on NFS gives a locking error:
% echo 'select * from blah;' | sqlite3 blah.sqlite
Error: near line 1: database is locked% echo 'create table blargh(INT);' | sqlite3 blargh.sqlite
Error: near line 1: database is lockedQuoted 6 lines
Quoted 3 lines
The result is that a lot of high-profile applications which make use of sqlite fail mysteriously. Bisection reveals the following, and reverting the implicated commit solves the issue:
Nick, thanks for the report. Is 2.6.37-rc1 running on your clients or on your server?
Sorry for not being clear: the client is running 2.6.37-rc1. The
server is running RHEL 5.5.Quoted 2 lines
Does anything interesting appear in the kernel log when your test case fails?
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
Quoted 6 lines
Quoted 3 lines
(192.168.8.199 is the address of the failing client). I can only assume that these are a result of my recent issues, since I don't have access to the system log (with timestamps) on that machine.
That's the problem this patch is supposed to prevent. I'll investigate further.
I suspect nlmclnt_lookup_host() is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it to
const struct sockaddr source = {
.sa_family = AF_UNSPEC,
};
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;
from xs_create_sock().
Trond
Quoted 8 lines · Trond Myklebust
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};
Quoted 7 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().Trond
Does this fix the regression?
Trond
----------------------------------------------------------------------------------------------
NLM: Fix a regression in lockd
From: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
Bisected to commit 9247685088398cf21bcb513bd2832b4cd42516c4 (SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)
Apparently, removing the 'transport->srcaddr.ss_family = family' from xs_create_sock() triggers this due to nlmclnt_lookup_host() incorrectly initialising the srcaddr family to AF_UNSPEC.
---
fs/lockd/host.c | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-)
fs/lockd/host.c +0 -5
@@ -238,9 +238,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,
@@ -249,8 +246,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source, - .src_len = sizeof(source), .noresvport = noresvport, };
Quoted 67 lines · Trond Myklebust
Quoted 8 lines · Trond Myklebust
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};Quoted 7 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().Trond
Does this fix the regression?
Trond
----------------------------------------------------------------------------------------------
NLM: Fix a regression in lockdFrom: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!Bisected to commit
9247685088398cf21bcb513bd2832b4cd42516c4(SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)Apparently, removing the '
transport->srcaddr.ss_family= family' fromxs_create_sock()triggers this due tonlmclnt_lookup_host()incorrectly initialising the srcaddr family toAF_UNSPEC.Reported-by Nick Bowler <[email protected]>Signed-off-by Trond Myklebust <[email protected]>---
fs/lockd/host.c| 5 -----1 files changed, 0 insertions(+), 5 deletions(-)fs/lockd/host.c +0
-3@@ -238,9 +238,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,
fs/lockd/host.c struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, · line 251
Quoted 8 lines
fs/lockd/host.c +0
-2@@ -249,8 +246,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source, - .src_len = sizeof(source), .noresvport = noresvport, };
What about that memcpy() in nlm_lookup_host()? With this patch, wouldn't it be copying garbage into the host's srcaddr field?
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 72 lines · Chuck Lever
Quoted 70 lines · Trond Myklebust
Quoted 8 lines · Trond Myklebust
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};Quoted 7 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().Trond
Does this fix the regression?
Trond
----------------------------------------------------------------------------------------------
NLM: Fix a regression in lockdFrom: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!Bisected to commit
9247685088398cf21bcb513bd2832b4cd42516c4(SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)Apparently, removing the '
transport->srcaddr.ss_family= family' fromxs_create_sock()triggers this due tonlmclnt_lookup_host()incorrectly initialising the srcaddr family toAF_UNSPEC.Reported-by Nick Bowler <[email protected]>Signed-off-by Trond Myklebust <[email protected]>---
fs/lockd/host.c| 5 -----1 files changed, 0 insertions(+), 5 deletions(-)fs/lockd/host.c +0
-4@@ -238,9 +238,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,@@ -249,8 +246,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source,
Quoted 9 lines
fs/lockd/host.c struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, · line 251
Quoted 5 lines
fs/lockd/host.c +0
-1- .src_len = sizeof(source), .noresvport = noresvport, };What about that
memcpy()innlm_lookup_host()? With this patch, wouldn't it be copying garbage into the host's srcaddr field?
It shouldn't. ni->src_len is now zero.
Trond
Quoted 83 lines · Trond Myklebust
Quoted 72 lines · Chuck Lever
Quoted 70 lines · Trond Myklebust
Quoted 8 lines · Trond Myklebust
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};Quoted 7 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().Trond
Does this fix the regression?
Trond
----------------------------------------------------------------------------------------------
NLM: Fix a regression in lockdFrom: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!Bisected to commit
9247685088398cf21bcb513bd2832b4cd42516c4(SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)Apparently, removing the '
transport->srcaddr.ss_family= family' fromxs_create_sock()triggers this due tonlmclnt_lookup_host()incorrectly initialising the srcaddr family toAF_UNSPEC.Reported-by Nick Bowler <[email protected]>Signed-off-by Trond Myklebust <[email protected]>---
fs/lockd/host.c| 5 -----1 files changed, 0 insertions(+), 5 deletions(-)fs/lockd/host.c +0
-4@@ -238,9 +238,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,@@ -249,8 +246,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source,Quoted 9 lines
fs/lockd/host.c struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, · line 251
Quoted 5 lines
fs/lockd/host.c +0
-1- .src_len = sizeof(source), .noresvport = noresvport, };What about that
memcpy()innlm_lookup_host()? With this patch, wouldn't it be copying garbage into the host's srcaddr field?
Quoted 1 line
It shouldn't.
ni->src_lenis now zero.
Yech. All this still assumes that ANYADDR is all zeroes, and that the memory this is going into is already initialized to zeroes. It's asking for trouble if we re-arrange all this someday.
I've got an untested one line patch that should fix this for any upper layer caller. Posting now.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 85 lines · Chuck Lever
Quoted 83 lines · Trond Myklebust
Quoted 81 lines · Chuck Lever
Quoted 67 lines · Trond Myklebust
Quoted 8 lines · Trond Myklebust
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};Quoted 7 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().Trond
Does this fix the regression?
Trond
----------------------------------------------------------------------------------------------
NLM: Fix a regression in lockdFrom: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!Bisected to commit
9247685088398cf21bcb513bd2832b4cd42516c4(SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)Apparently, removing the '
transport->srcaddr.ss_family= family' fromxs_create_sock()triggers this due tonlmclnt_lookup_host()incorrectly initialising the srcaddr family toAF_UNSPEC.Reported-by Nick Bowler <[email protected]>Signed-off-by Trond Myklebust <[email protected]>---
fs/lockd/host.c| 5 -----1 files changed, 0 insertions(+), 5 deletions(-)fs/lockd/host.c +0
-3@@ -238,9 +238,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,fs/lockd/host.c struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, · line 251
Quoted 8 lines
fs/lockd/host.c +0
-2@@ -249,8 +246,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source, - .src_len = sizeof(source), .noresvport = noresvport, };What about that
memcpy()innlm_lookup_host()? With this patch, wouldn't it be copying garbage into the host's srcaddr field?
Quoted 6 lines
Quoted 1 line
It shouldn't.
ni->src_lenis now zero.Yech. All this still assumes that ANYADDR is all zeroes, and that the memory this is going into is already initialized to zeroes. It's asking for trouble if we re-arrange all this someday.
I've got an untested one line patch that should fix this for any upper layer caller. Posting now.
No! Upper layer callers should simply not be setting .saddress. Pretty much the only thing that _should_ be setting .saddress is the lockd callback, and possibly the nfsv4 server callback.
Trond
Quoted 88 lines · Trond Myklebust
Quoted 87 lines · Chuck Lever
Quoted 84 lines · Trond Myklebust
Quoted 72 lines · Chuck Lever
Quoted 70 lines · Trond Myklebust
Quoted 8 lines · Trond Myklebust
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};Quoted 7 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().Trond
Does this fix the regression?
Trond
----------------------------------------------------------------------------------------------
NLM: Fix a regression in lockdFrom: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!Bisected to commit
9247685088398cf21bcb513bd2832b4cd42516c4(SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)Apparently, removing the '
transport->srcaddr.ss_family= family' fromxs_create_sock()triggers this due tonlmclnt_lookup_host()incorrectly initialising the srcaddr family toAF_UNSPEC.Reported-by Nick Bowler <[email protected]>Signed-off-by Trond Myklebust <[email protected]>---
fs/lockd/host.c| 5 -----1 files changed, 0 insertions(+), 5 deletions(-)fs/lockd/host.c +0
-4@@ -238,9 +238,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,@@ -249,8 +246,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source,Quoted 9 lines
fs/lockd/host.c struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, · line 251
Quoted 5 lines
fs/lockd/host.c +0
-1- .src_len = sizeof(source), .noresvport = noresvport, };What about that
memcpy()innlm_lookup_host()? With this patch, wouldn't it be copying garbage into the host's srcaddr field?It shouldn't.
ni->src_lenis now zero.
Quoted 8 lines
Quoted 4 lines
Yech. All this still assumes that ANYADDR is all zeroes, and that the memory this is going into is already initialized to zeroes. It's asking for trouble if we re-arrange all this someday.
I've got an untested one line patch that should fix this for any upper layer caller. Posting now.
No! Upper layer callers should simply not be setting .saddress. Pretty much the only thing that _should_ be setting .saddress is the lockd callback, and possibly the nfsv4 server callback.
Then the take 2 patch description should be updated to reflect your actual intent. I don't think it's especially clear from your patch description why "nlmclnt_lookup_host() ... initialising the srcaddr family to AF_UNSPEC" is incorrect from an architectural standpoint. The specifics of "fixing bad behavior" are obvious already, but the architectural problem needs to be documented.
Otherwise I'm OK with the general idea of take 2, since it passes in a NULL pointer when no special bind address is desired, as other callers of rpc_create() do. Of course, take 2 has to pass Nick's test case too.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 6 lines · Trond Myklebust
Quoted 2 lines · Chuck Lever
What about that
memcpy()innlm_lookup_host()? With this patch, wouldn't it be copying garbage into the host's srcaddr field?It shouldn't.
ni->src_lenis now zero.
Urgh, but nlmclnt_bind_host() will still set .saddress.
OK. Here is take 2 of the patch...
Trond
---------------------------------------------------------------------------------------------------------------
NLM: Fix a regression in lockd
From: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
Bisected to commit 9247685088398cf21bcb513bd2832b4cd42516c4 (SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)
Apparently, removing the 'transport->srcaddr.ss_family = family' from xs_create_sock() triggers this due to nlmclnt_lookup_host() incorrectly initialising the srcaddr family to AF_UNSPEC.
---
fs/lockd/host.c | 11 ++++------- include/linux/lockd/lockd.h | 1 + 2 files changed, 5 insertions(+), 7 deletions(-)
fs/lockd/host.c +4 -7
@@ -124,7 +124,7 @@ static struct nlm_host *nlm_lookup_host(struct nlm_lookup_host_info *ni)continue ;if (host->h_server != ni->server)continue ; -if (ni->server && +if (ni->server && host->h_srcaddrlen !=0 && !rpc_cmp_addr (nlm_srcaddr (host), ni->src_sap))continue ;
@@ -167,6 +167,7 @@ static struct nlm_host *nlm_lookup_host(struct nlm_lookup_host_info *ni) host->h_addrlen = ni->salen;rpc_set_port (nlm_addr (host),0 );memcpy (nlm_srcaddr (host), ni->src_sap, ni->src_len); + host->h_srcaddrlen = ni->src_len; host->h_version = ni->version; host->h_proto = ni->protocol; host->h_rpcclnt =NULL ;
@@ -238,9 +239,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,
@@ -249,8 +247,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source, - .src_len = sizeof(source), .noresvport = noresvport, };
@@ -357,7 +353,6 @@ nlm_bind_host(struct nlm_host *host) .protocol = host->h_proto, .address = nlm_addr(host), .addrsize = host->h_addrlen, - .saddress = nlm_srcaddr(host), .timeout = &timeparms, .servername = host->h_name, .program = &nlm_program,
@@ -376,6 +371,8 @@ nlm_bind_host(struct nlm_host *host) args.flags |=RPC_CLNT_CREATE_HARDRTRY ;if (host->h_noresvport) args.flags |=RPC_CLNT_CREATE_NONPRIVPORT ; +if (host->h_srcaddrlen) + args.saddress =nlm_srcaddr (host), clnt =rpc_create (&args);if (!IS_ERR (clnt))
include/linux/lockd/lockd.h +1 -0
@@ -43,6 +43,7 @@ struct nlm_host {struct sockaddr_storage h_addr;/* peer address */ size_t h_addrlen;struct sockaddr_storage h_srcaddr;/* our address (optional) */ +size_t h_srcaddrlen;struct rpc_clnt *h_rpcclnt;/* RPC client to talk to peer */ char *h_name;/* remote hostname */ u32 h_version;/* interface version */
Quoted 3 lines · Trond Myklebust
Urgh, but
nlmclnt_bind_host()will still set .saddress.OK. Here is take 2 of the patch...
[...]
Quoted 1 line
NLM: Fix a regression in lockd
Problem occurs with this one (on top of 2.6.37-rc1), too :(
Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Quoted 9 lines · Nick Bowler
Quoted 3 lines · Trond Myklebust
Urgh, but
nlmclnt_bind_host()will still set .saddress.OK. Here is take 2 of the patch...
[...]
Quoted 1 line
NLM: Fix a regression in lockd
Problem occurs with this one (on top of 2.6.37-rc1), too :(
OK. Can you please try the following? It only differs from the last one by a small bugfix, and a WARN_ON() in order to try to catch other occurrences of the same bug.
Cheers
Trond
------------------------------------------------------------------------------------------------
NLM: Fix a regression in lockd
From: Trond Myklebust <[email protected]>
Nick Bowler reports:
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
Bisected to commit 9247685088398cf21bcb513bd2832b4cd42516c4 (SUNRPC:
Properly initialize sock_xprt.srcaddr in all cases)
Apparently, removing the 'transport->srcaddr.ss_family = family' from xs_create_sock() triggers this due to nlmclnt_lookup_host() incorrectly initialising the srcaddr family to AF_UNSPEC.
---
fs/lockd/host.c | 11 ++++------- include/linux/lockd/lockd.h | 1 + net/sunrpc/clnt.c | 2 ++ 3 files changed, 7 insertions(+), 7 deletions(-)
fs/lockd/host.c +4 -7
@@ -124,7 +124,7 @@ static struct nlm_host *nlm_lookup_host(struct nlm_lookup_host_info *ni)continue ;if (host->h_server != ni->server)continue ; -if (ni->server && +if (ni->server && ni->src_len !=0 && !rpc_cmp_addr (nlm_srcaddr (host), ni->src_sap))continue ;
@@ -167,6 +167,7 @@ static struct nlm_host *nlm_lookup_host(struct nlm_lookup_host_info *ni) host->h_addrlen = ni->salen;rpc_set_port (nlm_addr (host),0 );memcpy (nlm_srcaddr (host), ni->src_sap, ni->src_len); + host->h_srcaddrlen = ni->src_len; host->h_version = ni->version; host->h_proto = ni->protocol; host->h_rpcclnt =NULL ;
@@ -238,9 +239,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, const char *hostname, int noresvport) { -const struct sockaddr source = { - .sa_family =AF_UNSPEC , - };struct nlm_lookup_host_info ni = { .server =0 , .sap = sap,
@@ -249,8 +247,6 @@ struct nlm_host *nlmclnt_lookup_host(const struct sockaddr *sap, .version = version, .hostname = hostname, .hostname_len = strlen(hostname), - .src_sap = &source, - .src_len = sizeof(source), .noresvport = noresvport, };
@@ -357,7 +353,6 @@ nlm_bind_host(struct nlm_host *host) .protocol = host->h_proto, .address = nlm_addr(host), .addrsize = host->h_addrlen, - .saddress = nlm_srcaddr(host), .timeout = &timeparms, .servername = host->h_name, .program = &nlm_program,
@@ -376,6 +371,8 @@ nlm_bind_host(struct nlm_host *host) args.flags |=RPC_CLNT_CREATE_HARDRTRY ;if (host->h_noresvport) args.flags |=RPC_CLNT_CREATE_NONPRIVPORT ; +if (host->h_srcaddrlen) + args.saddress =nlm_srcaddr (host); clnt =rpc_create (&args);if (!IS_ERR (clnt))
include/linux/lockd/lockd.h +1 -0
@@ -43,6 +43,7 @@ struct nlm_host {struct sockaddr_storage h_addr;/* peer address */ size_t h_addrlen;struct sockaddr_storage h_srcaddr;/* our address (optional) */ +size_t h_srcaddrlen;struct rpc_clnt *h_rpcclnt;/* RPC client to talk to peer */ char *h_name;/* remote hostname */ u32 h_version;/* interface version */
net/sunrpc/clnt.c +2 -0
@@ -322,6 +322,8 @@ struct rpc_clnt *rpc_create(struct rpc_create_args *args) args->servername = servername; } +WARN_ON (args->saddress && args->saddress->sa_family != args->address->sa_family); + xprt =xprt_create_transport (&xprtargs);if (IS_ERR (xprt))return (struct rpc_clnt *)xprt;
Nick Bowler
Tested-byQuoted 3 lines · Trond Myklebust
OK. Can you please try the following? It only differs from the last one by a small bugfix, and a
WARN_ON()in order to try to catch other occurrences of the same bug.
[...]
Quoted 1 line
NLM: Fix a regression in lockd
Things seem to be working fine now, thanks.
Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Quoted 2 lines · Trond Myklebust
On Mon, 2010-11-01 at 15:22 -0400, Trond Myklebust wrote:
Does this fix the regression?
[...]
Quoted 1 line
NLM: Fix a regression in lockd
Unfortunately, the problem persists with this patch applied on top of 2.6.37-rc1.
Signature · 1 line
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
Quoted 51 lines · Trond Myklebust
Quoted 35 lines · Chuck Lever
Quoted 33 lines · Nick Bowler
Quoted 10 lines · Chuck Lever
Quoted 9 lines · Nick Bowler
After installing 2.6.37-rc1, attempting to use sqlite in any capacity on NFS gives a locking error:
% echo 'select * from blah;' | sqlite3 blah.sqlite
Error: near line 1: database is locked% echo 'create table blargh(INT);' | sqlite3 blargh.sqlite
Error: near line 1: database is lockedQuoted 6 lines
Quoted 3 lines
The result is that a lot of high-profile applications which make use of sqlite fail mysteriously. Bisection reveals the following, and reverting the implicated commit solves the issue:
Nick, thanks for the report. Is 2.6.37-rc1 running on your clients or on your server?
Sorry for not being clear: the client is running 2.6.37-rc1. The
server is running RHEL 5.5.Quoted 2 lines
Does anything interesting appear in the kernel log when your test case fails?
There are no unusual messages on the client... but I just logged into the server and I see lots of messages of the following form:
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!
nfsd: request from insecure port (192.168.8.199:35766)!Quoted 6 lines
Quoted 3 lines
(192.168.8.199 is the address of the failing client). I can only assume that these are a result of my recent issues, since I don't have access to the system log (with timestamps) on that machine.
That's the problem this patch is supposed to prevent. I'll investigate further.
I suspect
nlmclnt_lookup_host()is to blame. It appears to be the _only_ thing in the kernel that actually sets this 'srcaddr' field, and it sets it toconst struct sockaddr source = {
.sa_family =AF_UNSPEC,
};
Quoted 5 lines
You triggered the bug by removing the line
transport->srcaddr.ss_family = family;from
xs_create_sock().
Thanks. Actually that line was added by Bruce very recently because Pavel's patches changed xs_bind() so it can't tolerate an AF_UNSPEC address. My patch attempts to replace the workaround with something more permanent... but looks like I didn't find all the places that needed to be fixed.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com