nfs_update_inode: inode X mode changed, Y to Z
Subsystems (2)
Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS
server, and I am receiving the above log message on a semi-regular basis
in the NFS client system-logs. When this occurs, one of the users
receives an error (although there seems to be no way to easily collate
the users experience with the system log, short of using the timestamp).
Sometimes it's "read/write error", sometimes it's more specific
about something that was a file now being a directory (SVN tends to be
rather more verbose). Most of the time this is just a nuisance, but at times (such as last week or so) it occurs so often that it blocks any work getting done.
The NFS Server is running OpenSuSE 10.2 on a Dell PowerEdge 2900
with the PERC5/i controller. The NFS Clients are all SLES10 with the standard mountoptions, and running in TCP mode (something about UDP + NFS + GbE leads to subtle data corruption).
I have been able to find some references to this problem in Google,
but no solutions, and no discussion about what the problem stems from, nor if any fixes have been attempted.
Quoted 11 lines · Adam Schrotenboer
Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS server, and I am receiving the above log message on a semi-regular basis in the NFS client system-logs. When this occurs, one of the users receives an error (although there seems to be no way to easily collate the users experience with the system log, short of using the timestamp).
Sometimes it's "read/write error", sometimes it's more specific
about something that was a file now being a directory (SVN tends to be rather more verbose). Most of the time this is just a nuisance, but at times (such as last week or so) it occurs so often that it blocks any work getting done.
Quoted 4 lines
The NFS Server is running OpenSuSE 10.2 on a Dell PowerEdge 2900
with the PERC5/i controller. The NFS Clients are all SLES10 with the standard mountoptions, and running in TCP mode (something about UDP + NFS + GbE leads to subtle data corruption).
Could you be a little bit more specific about the nature of the NFS
server: are you using knfsd or is this the legacy userland server? Please also specify what kind of filesystem you are exporting, and the export options you are using.
As for the client side, when do these errors occur? Have you seen them occur after a prolonged period of use, or are they more frequent immediately after a reboot or mount operation?
Cheers
Trond
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · Adam Schrotenboer
Running SLES 10 on multiple compute nodes, with an OpenSuSE 10.2 NFS
server, and I am receiving the above log message on a semi-regular basis
in the NFS client system-logs. When this occurs, one of the users
receives an error (although there seems to be no way to easily collate
the users experience with the system log, short of using the timestamp).
Sometimes it's "read/write error", sometimes it's more specific
about something that was a file now being a directory (SVN tends to be
rather more verbose). Most of the time this is just a nuisance, but at times (such as last week or so) it occurs so often that it blocks any work getting done.
Quoted 9 lines
Quoted 3 lines
The NFS Server is running OpenSuSE 10.2 on a Dell PowerEdge 2900
with the PERC5/i controller. The NFS Clients are all SLES10 with the standard mountoptions, and running in TCP mode (something about UDP + NFS + GbE leads to subtle data corruption).Could you be a little bit more specific about the nature of the NFS
server: are you using knfsd or is this the legacy userland server? Please also specify what kind of filesystem you are exporting, and the export options you are using.
We're using the kernel NFSd. The exported FS is XFS./mnt/storage0/users 192.168.1.6(rw,sync,insecure,no_subtree_check,no_root_squash) 192.168.1.0/24(rw,sync,insecure,no_subtree_check)
Quoted 4 lines
As for the client side, when do these errors occur? Have you seen them occur after a prolonged period of use, or are they more frequent immediately after a reboot or mount operation?
They occur mostly randomly, we have not noticed any particular pattern to their occurrence, we have not noticed any correlation with mount-time.
Here is an excerpt from one of our servers. Note that all servers were shutdown on Feb 29th and restarted on March 1st due to a planned power outage. All systems are backed up by UPS.
Quoted 3 lines
Cheers
Trond
Quoted 3 lines · On Wednesday March 5, [email protected] wrote:
They occur mostly randomly, we have not noticed any particular pattern to their occurrence, we have not noticed any correlation with mount-time. Here is an excerpt from one of our servers. Note that all servers were
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrown
Quoted 7 lines
shutdown on Feb 29th and restarted on March 1st due to a planned power outage. All systems are backed up by UPS.
Quoted 3 lines
Cheers
Trond
Quoted 9 lines · Neil Brown
On Wednesday March 5, [email protected] wrote:
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrown
sorry, must have missed the paste
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode changed, 0040755 to 0100644 Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode changed, 0040755 to 0100755 Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode changed, 0040755 to 0100644 Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode changed, 0120777 to 0100644 Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode changed, 0040755 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 mode changed, 0120777 to 0100644
Quoted 29 lines · On Wednesday March 5, [email protected] wrote:
Quoted 9 lines · Neil Brown
On Wednesday March 5, [email protected] wrote:
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrownsorry, must have missed the paste
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode changed, 0040755 to 0100644 Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode changed, 0040755 to 0100755 Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode changed, 0040755 to 0100644 Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode changed, 0120777 to 0100644 Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode changed, 0040755 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 mode
Quoted 3 lines
changed, 0120777 to 0100644
Hmmm...
Directories and symlinks "changing" to regular files.
32bit inode numbers with the high bit set.
Random times, but a burst at 3pm one afternoon.
The fact that it happens so rarely make is very hard to collect more data....
What happens when a user experiences an error? What sort of file is the error reported against? Does the file remain inaccessible or does it start working again soon? Is the same file accessible from another client?
NeilBrown
Quoted 33 lines · Neil Brown
Quoted 14 lines · On Wednesday March 5, [email protected] wrote:
Quoted 10 lines · Neil Brown
On Wednesday March 5, [email protected] wrote:
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrownsorry, must have missed the paste
Quoted 12 lines
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode changed, 0040755 to 0100644 Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode changed, 0040755 to 0100755 Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode changed, 0040755 to 0100644 Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode changed, 0120777 to 0100644 Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode changed, 0040755 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 mode changed, 0120777 to 0100644Hmmm...
Directories and symlinks "changing" to regular files.32bit inode numbers with the high bit set.
Quoted 7 lines
Random times, but a burst at 3pm one afternoon.
The fact that it happens so rarely make is very hard to collect more data....
What happens when a user experiences an error?
well, sometimes it shows up as a read/write error (that is the error code returned by the application). SVN tend to be more verbose (although I have no examples)
Quoted 2 lines
What sort of file is
the error reported against?
It does not to my knowledge only happen to certain classes of files, although it does seem to happen most often when lots of files are being created.
Quoted 2 lines
Does the file remain inaccessible or does
it start working again soon?
No data, maybe Thomas knows something.
Quoted 3 lines
Is the same file accessible from another
client?
No data, esp as it never happens to me personally.
Quoted 2 lines
NeilBrown
Another excerpt from last night.
Mar 12 00:53:16 wanda kernel: nfs: server 192.168.1.27 not responding, still trying Mar 12 00:53:18 wanda kernel: nfs: server 192.168.1.27 OK Mar 12 00:54:29 wanda kernel: nfs_update_inode: inode 1075049148 mode changed, 0040755 to 0100644 Mar 12 00:54:49 wanda kernel: nfs_update_inode: inode 1611920610 mode changed, 0100644 to 0040755 Mar 12 00:54:49 wanda kernel: nfs_update_inode: inode 1611920612 mode changed, 0040755 to 0100644 Mar 12 00:54:50 wanda kernel: nfs_update_inode: inode 2685631887 mode changed, 0100644 to 0040755 Mar 12 00:54:50 wanda kernel: nfs_update_inode: inode 3222506021 mode changed, 0100644 to 0040755 Mar 12 00:54:55 wanda kernel: nfs_update_inode: inode 1372131 mode changed, 0100644 to 0040755 Mar 12 00:55:25 wanda kernel: nfs_update_inode: inode 3759413120 mode changed, 0040755 to 0100644 Mar 12 00:56:05 wanda kernel: nfs_update_inode: inode 1075051607 mode changed, 0100644 to 0040755 Mar 12 00:56:05 wanda kernel: nfs_update_inode: inode 1075051608 mode changed, 0040755 to 0100644 Mar 12 00:56:05 wanda kernel: nfs_update_inode: inode 1075051612 mode changed, 0100644 to 0040755 Mar 12 00:56:05 wanda kernel: nfs_update_inode: inode 1075051613 mode changed, 0040755 to 0100644 Mar 12 00:56:06 wanda kernel: nfs_update_inode: inode 1075051616 mode changed, 0100644 to 0040755 Mar 12 00:56:06 wanda kernel: nfs_update_inode: inode 1075051617 mode changed, 0040755 to 0100644 Mar 12 00:56:09 wanda kernel: nfs_update_inode: inode 1075051624 mode changed, 0100644 to 0040755 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926434 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926438 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926442 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926446 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926450 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926454 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926458 mode changed, 0040755 to 0100644 Mar 12 01:00:33 wanda kernel: nfs_update_inode: inode 1611926462 mode changed, 0040755 to 0100644 Mar 12 01:00:34 wanda kernel: nfs_update_inode: inode 1611926463 mode changed, 0100644 to 0040755 Mar 12 09:36:46 wanda kernel: nfs_update_inode: inode 538186966 mode changed, 0100644 to 0040755 Mar 12 09:36:46 wanda kernel: nfs_update_inode: inode 538186967 mode changed, 0040755 to 0100644 Mar 12 09:36:46 wanda kernel: nfs_update_inode: inode 538186970 mode changed, 0100644 to 0040755 Mar 12 09:36:46 wanda kernel: nfs_update_inode: inode 538186972 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186974 mode changed, 0100644 to 0040755 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186977 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186979 mode changed, 0100644 to 0040755 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186982 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186984 mode changed, 0100644 to 0040755 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186987 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186989 mode changed, 0100644 to 0040755 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186992 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186994 mode changed, 0100644 to 0040755 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186997 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538186999 mode changed, 0100644 to 0040755 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538187002 mode changed, 0040755 to 0100644 Mar 12 09:36:47 wanda kernel: nfs_update_inode: inode 538187004 mode changed, 0100644 to 0040755
It keeps going...
Another machine, different times.
Mar 12 03:35:19 minnow kernel: nfs_update_inode: inode 2685628653 mode changed, 0100644 to 0040755 Mar 12 03:38:01 minnow kernel: nfs_update_inode: inode 2685628658 mode changed, 0100644 to 0040755 Mar 12 03:38:04 minnow kernel: nfs_update_inode: inode 2685628707 mode changed, 0100644 to 0040755 Mar 12 03:38:04 minnow kernel: nfs_update_inode: inode 2685628715 mode changed, 0100644 to 0040755 Mar 12 03:38:04 minnow kernel: nfs_update_inode: inode 2685628719 mode changed, 0100644 to 0040755 Mar 12 03:38:04 minnow kernel: nfs_update_inode: inode 2685628731 mode changed, 0100644 to 0040755 Mar 12 03:38:05 minnow kernel: nfs_update_inode: inode 3222503513 mode changed, 0100644 to 0040755 Mar 12 03:38:05 minnow kernel: nfs_update_inode: inode 2685628818 mode changed, 0100644 to 0040755 Mar 12 03:38:06 minnow kernel: nfs_update_inode: inode 2685628842 mode changed, 0100644 to 0040755 Mar 12 03:38:06 minnow kernel: nfs_update_inode: inode 2685628846 mode changed, 0100644 to 0040755 Mar 12 03:38:06 minnow kernel: nfs_update_inode: inode 2685628851 mode changed, 0100644 to 0040755 Mar 12 03:38:06 minnow kernel: nfs_update_inode: inode 2685628856 mode changed, 0100644 to 0040755 Mar 12 03:38:07 minnow kernel: nfs_update_inode: inode 3222503585 mode changed, 0100644 to 0040755 Mar 12 03:38:09 minnow kernel: nfs_update_inode: inode 2685628923 mode changed, 0100644 to 0040755 Mar 12 03:38:09 minnow kernel: nfs_update_inode: inode 2685628927 mode changed, 0100644 to 0040755 Mar 12 04:00:48 minnow kernel: nfs_update_inode: inode 2149158555 mode changed, 0100644 to 0040755 Mar 12 04:00:51 minnow kernel: nfs_update_inode: inode 1611926234 mode changed, 0040755 to 0100644 Mar 12 04:01:04 minnow kernel: nfs_update_inode: inode 1611926235 mode changed, 0100644 to 0040755 Mar 12 04:01:04 minnow kernel: nfs_update_inode: inode 1611926238 mode changed, 0040755 to 0100644 Mar 12 04:01:04 minnow kernel: nfs_update_inode: inode 1611926240 mode changed, 0100644 to 0040755 Mar 12 04:01:04 minnow kernel: nfs_update_inode: inode 1611926243 mode changed, 0040755 to 0100644
I have other machines it has occurred to as well. The only usage pattern I have is that it happened mostly while running Mentor's Precision, but I am doubting that that is relevant.
This is the line from /proc/mounts on Minnow:
192.168.1.27:/mnt/storage0/users /home/users nfs rw,v3,rsize=32768,wsize=32768,hard,lock,proto=tcp,addr=192.168.1.27 0 0
192.168.1.27 is our machine dolphin, which is running OpenSuSE 10.2 x86_64 on Intel(R) Xeon(R) CPU E5310 @ 1.60GHz with 4G of RAM, and is an XFS file system running on a Dell PERC5/i
adam@dolphin:~$ df -h /mnt/storage0/ Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.8T 2.0T 807G 72% /mnt/storage0 from /etc/fstab /dev/disk/by-label/storage0 /mnt/storage0 xfs rw,usrquota,grpquota 0 0 It is not running with inode64, but probably should be. Hasn't become a problem yet. Otoh, I understand that NFS v3 doesn't support 64bit inode numbers. FYI almost all of our NFS clients are x86_64 Linux(except 2)
I have CC:d Fred Revenu, who was the person who brought it to my attention this morning that it had happened again last night.
Quoted 7 lines · Adam Schrotenboer
Another excerpt from last night.
Mar 12 00:53:16 wanda kernel: nfs: server 192.168.1.27 not responding, still trying Mar 12 00:53:18 wanda kernel: nfs: server 192.168.1.27 OK Mar 12 00:54:29 wanda kernel: nfs_update_inode: inode 1075049148 modechanged, 0040755 to 0100644
Hang on. That does not look like an XID collision problem...
That code path basically means that the fileids/inode numbers match, which would not be the case if we were talking about an XID collision causing the reply cache to replay an old request...
That message rather symptomatic of a filehandle reuse problem. In other words a NFS filehandle appears to be reused to label a regular file after it has been used for a directory with the same fileid/inode number. That is a definite server bug.
Are you sure that you are using the regular kernel nfs server?
Signature · 6 lines
Trond Myklebust NFS client maintainer NetApp [email protected] www.netapp.com
Quoted 15 lines · Trond Myklebust
Quoted 6 lines · Adam Schrotenboer
Another excerpt from last night.
Mar 12 00:53:16 wanda kernel: nfs: server 192.168.1.27 not responding, still trying Mar 12 00:53:18 wanda kernel: nfs: server 192.168.1.27 OK Mar 12 00:54:29 wanda kernel: nfs_update_inode: inode 1075049148 mode changed, 0040755 to 0100644Hang on. That does not look like an XID collision problem...
That code path basically means that the fileids/inode numbers match, which would not be the case if we were talking about an XID collision causing the reply cache to replay an old request...
Quoted 8 lines
That message rather symptomatic of a filehandle reuse problem. In other words a NFS filehandle appears to be reused to label a regular file after it has been used for a directory with the same fileid/inode number. That is a definite server bug.
Are you sure that you are using the regular kernel nfs server?
Quite. OpenSuSE 10.2 stock unmodified kernel.
dolphin:~ # uname -a
Linux dolphin 2.6.18.2-34-default #1 SMP Mon Nov 27 11:46:27 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux
dolphin:~ # top -bn1 |grep nfsd
PID PPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3916 1 root 15 0 0 0 0 S 0 0.0 3:04.42 nfsd 3942 1 root 15 0 0 0 0 S 0 0.0 3:11.88 nfsd
(there's quite a bit more of these threads than this (32), but they all look the same in top or ps)
Quoted 37 lines · Neil Brown
Quoted 29 lines · On Wednesday March 5, [email protected] wrote:
Quoted 9 lines · Neil Brown
On Wednesday March 5, [email protected] wrote:
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrownsorry, must have missed the paste
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode changed, 0040755 to 0100644 Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode changed, 0040755 to 0100755 Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode changed, 0040755 to 0100644 Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode changed, 0120777 to 0100644 Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode changed, 0040755 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 modeQuoted 3 lines
changed, 0120777 to 0100644
Hmmm...
Directories and symlinks "changing" to regular files.
Quoted 7 lines
32bit inode numbers with the high bit set.
Random times, but a burst at 3pm one afternoon.
The fact that it happens so rarely make is very hard to collect more data....
I've reported this problem in the past for 2.6.13.4, so it's not a new problem. Happens very rarely indeed.
See http://lkml.org/lkml/2007/2/26/102 and
http://lkml.org/lkml/2007/3/1/53 for details.
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 42 lines · Jesper Juhl
Quoted 41 lines · Neil Brown
Quoted 29 lines · On Wednesday March 5, [email protected] wrote:
Quoted 9 lines · Neil Brown
On Wednesday March 5, [email protected] wrote:
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrownsorry, must have missed the paste
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode changed, 0040755 to 0100644 Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode changed, 0040755 to 0100755 Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode changed, 0040755 to 0100644 Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode changed, 0120777 to 0100644 Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode changed, 0040755 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 modeQuoted 3 lines
changed, 0120777 to 0100644
Hmmm...
Directories and symlinks "changing" to regular files.32bit inode numbers with the high bit set.
Random times, but a burst at 3pm one afternoon.
Quoted 8 lines
Quoted 3 lines
The fact that it happens so rarely make is very hard to collect more data....
I've reported this problem in the past for 2.6.13.4, so it's not a new problem. Happens very rarely indeed.
See http://lkml.org/lkml/2007/2/26/102 and
http://lkml.org/lkml/2007/3/1/53 for details.
What was the exported filesystem?
--b.
Quoted 47 lines · J. Bruce Fields
Quoted 46 lines · Jesper Juhl
Quoted 44 lines · Neil Brown
Quoted 29 lines · On Wednesday March 5, [email protected] wrote:
Quoted 9 lines · Neil Brown
On Wednesday March 5, [email protected] wrote:
Where :-)
A excerpt from the logs to show actual error messages could be helpful.
Thanks,
NeilBrownsorry, must have missed the paste
Feb 28 17:27:58 koi kernel: nfs_update_inode: inode 3221350147 mode changed, 0040755 to 0100644 Feb 28 21:57:22 koi kernel: nfs_update_inode: inode 2149056279 mode changed, 0040755 to 0100755 Mar 5 01:43:21 koi kernel: nfs_update_inode: inode 3222473680 mode changed, 0040755 to 0100644 Mar 5 08:52:19 koi kernel: nfs_update_inode: inode 3222473568 mode changed, 0120777 to 0100644 Mar 5 15:00:01 koi kernel: nfs_update_inode: inode 3222473569 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473674 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473675 mode changed, 0120777 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473672 mode changed, 0040755 to 0100644 Mar 5 15:00:31 koi kernel: nfs_update_inode: inode 3222473673 modeQuoted 3 lines
changed, 0120777 to 0100644
Hmmm...
Directories and symlinks "changing" to regular files.32bit inode numbers with the high bit set.
Random times, but a burst at 3pm one afternoon.
The fact that it happens so rarely make is very hard to collect more data....
Quoted 8 lines
Quoted 4 lines
I've reported this problem in the past for 2.6.13.4, so it's not a new problem. Happens very rarely indeed.
See http://lkml.org/lkml/2007/2/26/102 and
http://lkml.org/lkml/2007/3/1/53 for details.What was the exported filesystem?
XFS
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 6 lines · On Wednesday March 12, [email protected] wrote:
Quoted 3 lines · J. Bruce Fields
What was the exported filesystem?
XFS
It's a bit of along shot, but could you try mounting the XFS file system with
-o ikeep
and see if it makes a difference.
When you have "ikeep", I can find the code that increments the generation number between different uses of the one inode number.
When you have "noikeep" (which I think is the default) it doesn't keep the inode of disk when deleted and so (presumably) needs generate a random generation number for each use. But I cannot find the code that does that. I'm probably not looking in the right place, but I don't think it can hurt to try "-o ikeep".
NeilBrown
Quoted 16 lines · Neil Brown
Quoted 7 lines · On Wednesday March 12, [email protected] wrote:
Quoted 4 lines · J. Bruce Fields
What was the exported filesystem?
XFS
It's a bit of along shot, but could you try mounting the XFS file
system with
-o ikeepand see if it makes a difference.
Quoted 9 lines
When you have "ikeep", I can find the code that increments the generation number between different uses of the one inode number.
When you have "noikeep" (which I think is the default) it doesn't keep the inode of disk when deleted and so (presumably) needs generate a random generation number for each use. But I cannot find the code that does that. I'm probably not looking in the right place, but I don't think it can hurt to try "-o ikeep".
It's about a year since I posted my bug report and I'm no longer responsible for the servers that showed the problem. I don't even know what kernel they are running these days and if they still show this problem. I'll try and talk to one of the guys who is currently managing them on monday and see what information I can gather.
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 20 lines · Neil Brown
Quoted 8 lines · On Wednesday March 12, [email protected] wrote:
Quoted 3 lines · J. Bruce Fields
What was the exported filesystem?
XFS
It's a bit of along shot, but could you try mounting the XFS file system with
-o ikeepand see if it makes a difference.
When you have "ikeep", I can find the code that increments the generation number between different uses of the one inode number.
Quoted 8 lines
When you have "noikeep" (which I think is the default) it doesn't keep the inode of disk when deleted and so (presumably) needs generate a random generation number for each use. But I cannot find the code that does that. I'm probably not looking in the right place, but I don't think it can hurt to try "-o ikeep".
NeilBrown
Ok, I've unmounted and remounted with that option enabled (/proc/mounts confirms it's enabled). We'll see what happens.
Quoted 21 lines · Adam Schrotenboer
Quoted 20 lines · Neil Brown
Quoted 3 lines · On Wednesday March 12, [email protected] wrote:
Quoted 1 line · J. Bruce Fields
What was the exported filesystem?
XFS
It's a bit of a long shot, but could you try mounting the XFS file system with
-o ikeepand see if it makes a difference.
When you have "ikeep", I can find the code that increments the generation number between different uses of the one inode number.
When you have "noikeep" (which I think is the default) it doesn't keep the inode of disk when deleted and so (presumably) needs generate a random generation number for each use. But I cannot find the code that does that. I'm probably not looking in the right place, but I don't think it can hurt to try "-o ikeep".
Quoted 3 lines
Quoted 2 lines
NeilBrown
Ok, I've unmounted and remounted with that option enabled (
/proc/mountsconfirms it's enabled). We'll see what happens.
Well, it's been almost 2 weeks (11 days anyhow) and I am not seeing
the nfs_update_inode message in the syslogs of any of our compute servers. I need to talk to the various people who work with them to verify, but it looks like this problem has been resolved.
Thank you very much for all of your help.
Quoted 24 lines · Adam Schrotenboer
Quoted 23 lines · Adam Schrotenboer
Quoted 22 lines · Neil Brown
Quoted 3 lines · On Wednesday March 12, [email protected] wrote:
Quoted 1 line · J. Bruce Fields
What was the exported filesystem?
XFS
It's a bit of a long shot, but could you try mounting the XFS file system with
-o ikeepand see if it makes a difference.
When you have "ikeep", I can find the code that increments the generation number between different uses of the one inode number.
When you have "noikeep" (which I think is the default) it doesn't keep the inode of disk when deleted and so (presumably) needs generate a random generation number for each use. But I cannot find the code that does that. I'm probably not looking in the right place, but I don't think it can hurt to try "-o ikeep".
NeilBrown
Quoted 7 lines
Quoted 2 lines
Ok, I've unmounted and remounted with that option enabled (
/proc/mountsconfirms it's enabled). We'll see what happens.Well, it's been almost 2 weeks (11 days anyhow) and I am not seeing
thenfs_update_inodemessage in the syslogs of any of our compute servers. I need to talk to the various people who work with them to verify, but it looks like this problem has been resolved.
That's a workaround, at least, but it's unfortunate if a special mount option is required to get correct behavior for nfs exports. Is there anything we can do?
--b.
Quoted 28 lines · J. Bruce Fields
Quoted 27 lines · Adam Schrotenboer
Quoted 25 lines · Adam Schrotenboer
Quoted 14 lines · Neil Brown
Quoted 3 lines · On Wednesday March 12, [email protected] wrote:
Quoted 1 line · J. Bruce Fields
What was the exported filesystem?
XFS
It's a bit of a long shot, but could you try mounting the XFS file system with
-o ikeepand see if it makes a difference.
When you have "ikeep", I can find the code that increments the generation number between different uses of the one inode number.
Quoted 8 lines
When you have "noikeep" (which I think is the default) it doesn't keep the inode of disk when deleted and so (presumably) needs generate a random generation number for each use. But I cannot find the code that does that. I'm probably not looking in the right place, but I don't think it can hurt to try "-o ikeep".
NeilBrown
Ok, I've unmounted and remounted with that option enabled (
/proc/mountsconfirms it's enabled). We'll see what happens.
Quoted 9 lines
Quoted 4 lines
Well, it's been almost 2 weeks (11 days anyhow) and I am not seeing
thenfs_update_inodemessage in the syslogs of any of our compute servers. I need to talk to the various people who work with them to verify, but it looks like this problem has been resolved.That's a workaround, at least, but it's unfortunate if a special mount option is required to get correct behavior for nfs exports. Is there anything we can do?
I suggest taking it up with the XFS developers...
Dear XFS developers.
Adam (and Jesper, though that was some time ago) was having problems
with an XFS filesystem that was exported via NFS. The client would occasionally report the message given in the subject line. Examining the NFS code suggested that the most likely explanation was that the generation number used in the file handle was the same every time that the inode number was re-used.
Examining the XFS code suggested that when the 'ikeep' mount option was used, the generation number be explicitly incremented for each re-use, while without 'ikeep', no evidence of setting the generation number could be found. Maybe it defaults to zero.
Experimental evidence suggests that setting 'ikeep' removes the symptom.
Question: Is is possible that without 'ikeep', XFS does not even try to provide unique generation numbers? If this is the case, could it please be fixed. If it is not the case, please help me find the code responsible.
Thanks,
NeilBrown
Quoted 15 lines · NeilBrown
I suggest taking it up with the XFS developers...
Dear XFS developers.
Adam (and Jesper, though that was some time ago) was having problems
with an XFS filesystem that was exported via NFS. The client would occasionally report the message given in the subject line. Examining the NFS code suggested that the most likely explanation was that the generation number used in the file handle was the same every time that the inode number was re-used.Examining the XFS code suggested that when the 'ikeep' mount option was used, the generation number be explicitly incremented for each re-use, while without 'ikeep', no evidence of setting the generation number could be found. Maybe it defaults to zero.
Quoted 6 lines
Experimental evidence suggests that setting 'ikeep' removes the symptom.
Question: Is is possible that without 'ikeep', XFS does not even try to provide unique generation numbers? If this is the case, could it please be fixed. If it is not the case, please help me find the code responsible.
Unless you specify the "ikeep" mount option, XFS will remove unused inode clusters. The newly freed blocks can be then used to store data or possibly a new inode cluster. If the blocks get reused for inodes, you'll end up with inodes whose generation numbers regressed. (inode number = f(block number))
Using the "ikeep" mount option causes to _never_ free empty inode clusters. This means that if you create many files and then unlink them, you'll end up with many unused inodes that are still allocated (and taking up disk space) but free to be used by the next creat(2)/mkdir(2)/etc..
This "problem" is inherent to any file system which dynamically allocates inodes.
Josef 'Jeff' Sipek.
Signature · 3 lines
Linux, n.: Generous programmers from around the world all join forces to help you shoot yourself in the foot for free.
Quoted 15 lines · Josef 'Jeff' Sipek
Unless you specify the "ikeep" mount option, XFS will remove unused inode clusters. The newly freed blocks can be then used to store data or possibly
a new inode cluster. If the blocks get reused for inodes, you'll end up with inodes whose generation numbers regressed. (inode number = f(block number))Using the "ikeep" mount option causes to _never_ free empty inode clusters.
This means that if you create many files and then unlink them, you'll end up
with many unused inodes that are still allocated (and taking up disk space)
but free to be used by the next creat(2)/mkdir(2)/etc..
Quoted 2 lines
This "problem" is inherent to any file system which dynamically allocates inodes.
Yes, I understand all that.
However you still need to do something about the generation number. It must be set to something.
When you allocate an inode that doesn't currently exist on the device, you obviously cannot increment the old value and use that. However you can do a lot better than always using 0.
The simplest would be to generate a 'random' number (get_random_bytes). Slightly better would be to generate a random number at boot time and use that, incrementing it each time it is used to set the generation number for an inode.
Even better would be store store that 'next generation number' in the superblock so there would be even less risk of the 'random' generation producing repeats.
This is what ext3 does. It doesn't dynamically allocate inodes, but it doesn't want to pay the cost of reading an old inode from storage just to see what the generation number is. So it has a number in the superblock which is incremented on each inode allocation and is used as the generation number.
Certainly anything would be better than always using the same number.
NeilBrown
On Wed, Mar 26, 2008 at 08:38:22AM +1100, NeilBrown wrote:
...
Quoted 2 lines
However you still need to do something about the generation number. It must be set to something.
Right.
Quoted 2 lines
When you allocate an inode that doesn't currently exist on the device, you obviously cannot increment the old value and use that.
Makes sense.
Quoted 1 line
However you can do a lot better than always using 0.
I looked at the code (xfs_ialloc.c:xfs_ialloc_ag_alloc)
290 /*
291 * Set initial values for the inodes in this buffer.
292 */
293 xfs_biozero(fbuf, 0, ninodes << args.mp->m_sb.sb_inodelog);
294 for (i = 0; i < ninodes; i++) {
295 free = XFS_MAKE_IPTR(args.mp, fbuf, i); 296 free->di_core.di_magic = cpu_to_be16(XFS_DINODE_MAGIC); 297 free->di_core.di_version = version; 298 free->di_next_unlinked = cpu_to_be32(NULLAGINO); 299 xfs_ialloc_log_di(tp, fbuf, i,
300 XFS_DI_CORE_BITS | XFS_DI_NEXT_UNLINKED);
301 }
xfs_biozero(...) turns into a memset(buf, 0, len), and since the loop that follows doesn't change the generation number, it'll stay 0.
Quoted 4 lines
The simplest would be to generate a 'random' number (
get_random_bytes). Slightly better would be to generate a random number at boot time and use that, incrementing it each time it is used to set the generation number for an inode.
I'm not familiar enough with NFS, do you want something that's monotonically increasing or do you just test for inequality? If it is inequality, why not just use something like the jiffies - that should be unique enough.
Quoted 8 lines
Even better would be store store that 'next generation number' in the superblock so there would be even less risk of the 'random' generation producing repeats.
This is what ext3 does. It doesn't dynamically allocate inodes, but it doesn't want to pay the cost of reading an old inode from storage just to see what the generation number is. So it has a number in the superblock which is incremented on each inode allocation and is used as the generation number.
Something tells me that the SGI folks might not be all too happy with the in-sb number...XFS tries to be as parallel as possible, and this would cause the counter variable to bounce around their NUMA systems. Perhaps a per-ag variable would be better, but I remember reading that parallelizing updates to some inode count variable (I forget which) in the superblock \cite{dchinner-ols2006} led to a rather big improvement. It's almost morning down under, so I guess we'll get their comments on this soon.
Josef 'Jeff' Sipek.
Signature · 4 lines
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. - Brian W. Kernighan
Quoted 41 lines · Josef 'Jeff' Sipek
On Wed, Mar 26, 2008 at 08:38:22AM +1100, NeilBrown wrote:
...Quoted 2 lines
However you still need to do something about the generation number. It must be set to something.
Right.
Quoted 2 lines
When you allocate an inode that doesn't currently exist on the device, you obviously cannot increment the old value and use that.
Makes sense.
Quoted 1 line
However you can do a lot better than always using 0.
I looked at the code (xfs_ialloc.c:xfs_ialloc_ag_alloc)
290 /*
291 * Set initial values for the inodes in this buffer.
292 */
293 xfs_biozero(fbuf, 0, ninodes <<args.mp->m_sb.sb_inodelog);
294 for (i = 0; i < ninodes; i++) {
295 free =XFS_MAKE_IPTR(args.mp, fbuf, i);
296free->di_core.di_magic=cpu_to_be16(XFS_DINODE_MAGIC);
297free->di_core.di_version= version;
298free->di_next_unlinked=cpu_to_be32(NULLAGINO);
299 xfs_ialloc_log_di(tp, fbuf, i,
300XFS_DI_CORE_BITS|XFS_DI_NEXT_UNLINKED);
301 }
xfs_biozero(...)turns into amemset(buf, 0, len), and since the loop that follows doesn't change the generation number, it'll stay 0.Quoted 4 lines
The simplest would be to generate a 'random' number (
get_random_bytes). Slightly better would be to generate a random number at boot time and use that, incrementing it each time it is used to set the generation number for an inode.
Quoted 6 lines
I'm not familiar enough with NFS, do you want something that's monotonically
increasing or do you just test for inequality? If it is inequality, why not
just use something like the jiffies - that should be unique enough.
What we need is for the "filehandle" to be stable and unique. By 'stable' I mean that every time I get the filehandle for a particular file, I get the same string of bytes.
By 'uniqie' I mean that if I get two filehandles for two different files, they must differ in at least one bit.
If a file is deleted and the inode is re-used for a new file, then the old and new files are different and must have different file handles.
The filehandle is traditionally generated from the inode number and a generation number, but the filesystem can actually do whatever it likes. xfs does it with xfs_fs_encode_fh().
Certainly you could initialise the i_generation to jiffies in xfs_ialloc_ag_alloc. That would be a suitable fix. get_random_bytes might be better, but the difference probably wouldn't be noticeable.
NeilBrown
Hi Neil,
Quoted 20 lines · NeilBrown
Quoted 15 lines · Josef 'Jeff' Sipek
Unless you specify the "ikeep" mount option, XFS will remove unused inode clusters. The newly freed blocks can be then used to store data or possibly
a new inode cluster. If the blocks get reused for inodes, you'll end up with inodes whose generation numbers regressed. (inode number = f(block number))Using the "ikeep" mount option causes to _never_ free empty inode clusters.
This means that if you create many files and then unlink them, you'll end up
with many unused inodes that are still allocated (and taking up disk space)
but free to be used by the next creat(2)/mkdir(2)/etc..Quoted 2 lines
This "problem" is inherent to any file system which dynamically allocates inodes.
Quoted 9 lines
Yes, I understand all that.
However you still need to do something about the generation number. It must be set to something.
When you allocate an inode that doesn't currently exist on the device, you obviously cannot increment the old value and use that. However you can do a lot better than always using 0.
Yes, this is a known problem.
We came across it in about August last year I believe in the context of
DMF as it wants to keep persistent file handles with gen#s in them:
SGI bug:
969192: Default mount option "noikeep" makes the inode generation number non-persistent
I vaguely remember at the time that a number of different schemes were tossed around but in the end we just turned off the ikeep
for DMAPI mounted filesystems.
I thought we had a bug open to do a real fix but can't see
it at the moment. Will look into it and discuss with our group.
Cheers,
--Tim
Quoted 4 lines · Josef 'Jeff' Sipek
On Wed, Mar 26, 2008 at 08:38:22AM +1100, NeilBrown wrote:
...Quoted 2 lines
However you still need to do something about the generation number. It must be set to something.
.....
Quoted 11 lines
Quoted 8 lines
Even better would be store store that 'next generation number' in the superblock so there would be even less risk of the 'random' generation producing repeats.
This is what ext3 does. It doesn't dynamically allocate inodes, but it doesn't want to pay the cost of reading an old inode from storage just to see what the generation number is. So it has a number in the superblock which is incremented on each inode allocation and is used as the generation number.Something tells me that the SGI folks might not be all too happy with the in-sb number...
.....
Quoted 1 line
Perhaps a per-ag variable would be better,
/me goes back to the bug from last year about stable inode/gen numbers for a HSM.
dgc> Right, except the last thing we want is yet more global state needing to dgc> be updated in inode allocation. The best way to do this is a max generation dgc> number per AG (held in the AGI) so that it can be updated at the same time dgc> inodes are freed and not cause additional serialisation.
Which was soundly rejected by the HSM folk because it wraps at 4 billion inode create/unlink cycles in an AG rather than per inode. The only thing they were happy with was the old behaviour and so they now mount their filesystems with ikeep. At that point the issue was dropped on the floor; the NFS side of things apparently weren't causing any problems so we didn't consider it urgent to fix....
Given this state of affairs (i.e. HSM using ikeep), I guess we can do anything we want for the noikeep case. I'll cook up a patch that does something similar to ext3 generation numbers for the initial seeding....
Quoted 3 lines
but I remember reading that parallelizing updates
to some inode count variable (I forget which) in the superblock \cite{dchinner-ols2006} led to a rather big improvement.
That was for in memory counters not on disk, and the problem really was free block counts rather than free inode counts. Yes, I converted the inode counters at the same time, but that wasn't the limiting factor. Updates to the on disk superblock, OTOH, are a limiting factor and that was the lazy superblock counter modifications solve....
Cheers,
Dave.
Signature · 3 lines
Dave Chinner Principal Engineer SGI Australian Software Group
Quoted 3 lines · David Chinner
Given this state of affairs (i.e. HSM using ikeep), I guess we can do anything we want for the noikeep case. I'll cook up a patch that does something similar to ext3 generation numbers for the initial seeding....
Patch below for comments. It passes xfsqa, but there's no userspace support for it yet. 2.6.26 is the likely target for this change.
Cheers,
Dave.
Signature · 145 lines
Dave Chinner Principal Engineer SGI Australian Software Group --- Don't initialise new inode generation numbers to zero When we allocation new inode chunks, we initialise the generation numbers to zero. This works fine until we delete a chunk and then reallocate it, resulting in the same inode numbers but with a reset generation count. This can result in inode/generation pairs of different inodes occurring relatively close together. Given that the inode/gen pair makes up the "unique" portion of an NFS filehandle on XFS, this can result in file handles cached on clients being seen on the wire from the server but refer to a different file. This causes .... issues for NFS clients. Hence we need a unique generation number initialisation for each inode to prevent reuse of a small portion of the generation number space. Make this initialiser per-allocation group so that it is not a single point of contention in the filesystem, and increment it on every allocation within an AG to reduce the chance that a generation number is reused for a given inode number if the inode chunk is deleted and reallocated immediately afterwards. It is safe to add the agi_newinogen field to the AGI without using a feature bit. If an older kernel is used, it simply will not update the field on allocation. If the kernel is updated and the field has garbage in it, then it's like having a random seed to the generation number.... Signed-off-by: Dave Chinner <[email protected]> --- fs/xfs/xfs_ag.h | 4 +++- fs/xfs/xfs_ialloc.c | 30 ++++++++++++++++++++++-------- 2 files changed, 25 insertions(+), 9 deletions(-) Index: 2.6.x-xfs-new/fs/xfs/xfs_ag.h =================================================================== --- 2.6.x-xfs-new.orig/fs/xfs/xfs_ag.h 2008-01-18 18:30:06.000000000 +1100 +++ 2.6.x-xfs-new/fs/xfs/xfs_ag.h 2008-03-26 13:03:41.122918236 +1100 @@ -121,6 +121,7 @@ typedef struct xfs_agi { * still being referenced. */ __be32 agi_unlinked[XFS_AGI_UNLINKED_BUCKETS]; + __be32 agi_newinogen; /* inode cluster generation */ } xfs_agi_t; #define XFS_AGI_MAGICNUM 0x00000001 @@ -134,7 +135,8 @@ typedef struct xfs_agi { #define XFS_AGI_NEWINO 0x00000100 #define XFS_AGI_DIRINO 0x00000200 #define XFS_AGI_UNLINKED 0x00000400 -#define XFS_AGI_NUM_BITS 11 +#define XFS_AGI_NEWINOGEN 0x00000800 +#define XFS_AGI_NUM_BITS 12 #define XFS_AGI_ALL_BITS ((1 << XFS_AGI_NUM_BITS) - 1) /* disk block (xfs_daddr_t) in the AG */ Index: 2.6.x-xfs-new/fs/xfs/xfs_ialloc.c =================================================================== --- 2.6.x-xfs-new.orig/fs/xfs/xfs_ialloc.c 2008-03-25 15:41:27.000000000 +1100 +++ 2.6.x-xfs-new/fs/xfs/xfs_ialloc.c 2008-03-26 14:29:47.998554368 +1100 @@ -309,6 +309,8 @@ xfs_ialloc_ag_alloc( free = XFS_MAKE_IPTR(args.mp, fbuf, i); free->di_core.di_magic = cpu_to_be16(XFS_DINODE_MAGIC); free->di_core.di_version = version; + free->di_core.di_gen = agi->agi_newinogen; + be32_add_cpu(&agi->agi_newinogen, 1); free->di_next_unlinked = cpu_to_be32(NULLAGINO); xfs_ialloc_log_di(tp, fbuf, i, XFS_DI_CORE_BITS | XFS_DI_NEXT_UNLINKED); @@ -347,7 +349,8 @@ xfs_ialloc_ag_alloc( * Log allocation group header fields */ xfs_ialloc_log_agi(tp, agbp, - XFS_AGI_COUNT | XFS_AGI_FREECOUNT | XFS_AGI_NEWINO); + XFS_AGI_COUNT | XFS_AGI_FREECOUNT | + XFS_AGI_NEWINO | XFS_AGI_NEWINOGEN); /* * Modify/log superblock values for inode count and inode free count. */ @@ -896,11 +899,12 @@ nextag: ino = XFS_AGINO_TO_INO(mp, agno, rec.ir_startino + offset); XFS_INOBT_CLR_FREE(&rec, offset); rec.ir_freecount--; + be32_add_cpu(&agi->agi_newinogen, 1); if ((error = xfs_inobt_update(cur, rec.ir_startino, rec.ir_freecount, rec.ir_free))) goto error0; be32_add(&agi->agi_freecount, -1); - xfs_ialloc_log_agi(tp, agbp, XFS_AGI_FREECOUNT); + xfs_ialloc_log_agi(tp, agbp, XFS_AGI_FREECOUNT | XFS_AGI_NEWINOGEN); down_read(&mp->m_peraglock); mp->m_perag[tagno].pagi_freecount--; up_read(&mp->m_peraglock); @@ -1320,6 +1324,11 @@ xfs_ialloc_compute_maxlevels( /* * Log specified fields for the ag hdr (inode section) + * + * We don't log the unlinked inode fields through here; they + * get logged directly to the buffer. Hence we have a discontinuity + * in the fields we are logging and we need two calls to map all + * the dirtied parts of the agi.... */ void xfs_ialloc_log_agi( @@ -1342,22 +1351,27 @@ xfs_ialloc_log_agi( offsetof(xfs_agi_t, agi_newino), offsetof(xfs_agi_t, agi_dirino), offsetof(xfs_agi_t, agi_unlinked), + offsetof(xfs_agi_t, agi_newinogen), sizeof(xfs_agi_t) }; + int log_newino = fields & XFS_AGI_NEWINOGEN; + #ifdef DEBUG xfs_agi_t *agi; /* allocation group header */ agi = XFS_BUF_TO_AGI(bp); ASSERT(be32_to_cpu(agi->agi_magicnum) == XFS_AGI_MAGIC); #endif - /* - * Compute byte offsets for the first and last fields. - */ + fields &= ~XFS_AGI_NEWINOGEN; + + /* Compute byte offsets for the first and last fields. */ xfs_btree_offsets(fields, offsets, XFS_AGI_NUM_BITS, &first, &last); - /* - * Log the allocation group inode header buffer. - */ xfs_trans_log_buf(tp, bp, first, last); + if (log_newino) { + xfs_btree_offsets(XFS_AGI_NEWINOGEN, offsets, XFS_AGI_NUM_BITS, + &first, &last); + xfs_trans_log_buf(tp, bp, first, last); + } } /*
Quoted 10 lines · David Chinner
Quoted 4 lines · David Chinner
Given this state of affairs (i.e. HSM using ikeep), I guess we can do anything we want for the noikeep case. I'll cook up a patch that does something similar to ext3 generation numbers for the initial seeding....
Patch below for comments. It passes xfsqa, but there's no userspace support for it yet. 2.6.26 is the likely target for this change.
2.6.26 merge window begins now. Has this been pushed yet? Is it in linux-next tree ?