Skip to contentSkip to conversation

2.6.24.3 nfs server on xfs keeps producing nfsd: non-standard errno: -117

DISCUSSION Stuart Rowan 3 participants 6 messages Mar 17, 2008 – Mar 20, 2008

lore
Share & tools
lore

Stuart Rowan

author
#0 lore raw

Hi,

Firstly thanks for the great filesystem and apologies if this ends up being NFS rather than XFS being weird! I'm not subscribed so please do keep me CC'd.

I have *millions* of lines of (>200k per minute according to syslog):
nfsd: non-standard errno: -117
being sent out of dmesg

Now errno 117 is
#define EUCLEAN 117 /* Structure needs cleaning */
which seems to be only used from a quick grep by XFS and JFFS and smbfs.

My nfs server export two locations
/home
/home/archive
both of these are XFS partitions, hence my suspicion that the -117 is coming from XFS.

xfs_repair -n says the filesystems are clean
xfs_repair has been run multiple times to completion on the filesystems, all is fine.

The XFS partitions are lvm volumes as follows
data/home 900G
data/archive 400G
The volume group, data, is sda3
sda3 is a 6 drive 3ware 9550SXU-8LP RAID10 array

The NFS server is currently in use (indeed the message only starts once clients connect) and works absolutely fine.

How do I find out what (if anything) is wrong with my filesystem / appropriately silence this message?

Many thanks,
Stu.

#1 lore raw

Hi Stuart,

Quoted 4 lines · Stuart Rowan

Hi,

Firstly thanks for the great filesystem and apologies if this ends up being NFS rather than XFS being weird! I'm not subscribed so please do keep me CC'd.

Quoted 9 lines

I have *millions* of lines of (>200k per minute according to syslog):
nfsd: non-standard errno: -117
being sent out of dmesg

Now errno 117 is
#define EUCLEAN 117 /* Structure needs cleaning */
which seems to be only used from a quick grep by XFS and JFFS and smbfs.

In XFS we mapped EFSCORRUPTED to EUCLEAN as EFSCORRUPTED didn't exist on Linux.
However, normally if this error is encountered in XFS then we output an appropriate msg to the syslog.
Our default error level is 3 and most reports are rated at 1 so should show up I would have thought.

--Tim

Quoted 21 lines

My nfs server export two locations
/home
/home/archive
both of these are XFS partitions, hence my suspicion that the -117 is coming from XFS.

xfs_repair -n says the filesystems are clean
xfs_repair has been run multiple times to completion on the filesystems, all is fine.

The XFS partitions are lvm volumes as follows
data/home 900G
data/archive 400G
The volume group, data, is sda3
sda3 is a 6 drive 3ware 9550SXU-8LP RAID10 array

The NFS server is currently in use (indeed the message only starts once clients connect) and works absolutely fine.

How do I find out what (if anything) is wrong with my filesystem / appropriately silence this message?

Many thanks,
Stu.

author
#2 lore raw

Hi Tim,

Quoted 16 lines · Timothy Shimmin

Hi Stuart,

Quoted 4 lines · Stuart Rowan

Hi,

Firstly thanks for the great filesystem and apologies if this ends up being NFS rather than XFS being weird! I'm not subscribed so please do keep me CC'd.

Quoted 9 lines

I have *millions* of lines of (>200k per minute according to syslog):
nfsd: non-standard errno: -117
being sent out of dmesg

Now errno 117 is
#define EUCLEAN 117 /* Structure needs cleaning */
which seems to be only used from a quick grep by XFS and JFFS and smbfs.

Quoted 9 lines

In XFS we mapped EFSCORRUPTED to EUCLEAN as EFSCORRUPTED didn't exist on Linux.
However, normally if this error is encountered in XFS then we output an appropriate msg to the syslog.
Our default error level is 3 and most reports are rated at 1 so should show up I would have thought.

--Tim

Thanks for the swift reply -- reading previous mailing list posts, that was my expectation too!
I've attached the relevant section of /var/log/kern.log inline below -- there is no error message! Is there another way of inspecting what (if anything) a given XFS file-system thinks is wrong with it.

Thanks,
Stu.

Mar 17 23:01:50 evenlode kernel: SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
Mar 17 23:01:50 evenlode kernel: SGI XFS Quota Management subsystem
Mar 17 23:01:50 evenlode kernel: Filesystem "dm-0": Disabling barriers, not supported by the underlying device
Mar 17 23:01:50 evenlode kernel: XFS mounting filesystem dm-0
Mar 17 23:01:50 evenlode kernel: Ending clean XFS mount for filesystem: dm-0
Mar 17 23:01:50 evenlode kernel: Filesystem "dm-1": Disabling barriers, not supported by the underlying device
Mar 17 23:01:50 evenlode kernel: XFS mounting filesystem dm-1
Mar 17 23:01:50 evenlode kernel: Ending clean XFS mount for filesystem: dm-1
Mar 17 23:01:50 evenlode kernel: e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Mar 17 23:01:50 evenlode kernel: RPC: Registered udp transport module.
Mar 17 23:01:50 evenlode kernel: RPC: Registered tcp transport module.
Mar 17 23:01:50 evenlode kernel: NET: Registered protocol family 10
Mar 17 23:01:50 evenlode kernel: lo: Disabled Privacy Extensions
Mar 17 23:01:50 evenlode kernel: IA-32 Microcode Update Driver: v1.14a <[email protected]>
Mar 17 23:01:51 evenlode kernel: Installing knfsd (copyright (C) 1996 [email protected]).
Mar 17 23:01:51 evenlode kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Mar 17 23:01:51 evenlode kernel: NFSD: starting 90-second grace period
Mar 17 23:01:57 evenlode kernel: eth0: no IPv6 routers present
Mar 17 23:03:04 evenlode kernel: 3w-9xxx: scsi0: AEN: INFO (0x04:0x000C): Initialize started:unit=0, subunit=0.
Mar 17 23:03:04 evenlode kernel: 3w-9xxx: scsi0: AEN: INFO (0x04:0x000C): Initialize started:unit=0, subunit=2.
Mar 17 23:04:40 evenlode kernel: nfsd: non-standard errno: -117
Mar 17 23:05:11 evenlode last message repeated 93970 times
Mar 17 23:06:12 evenlode last message repeated 188363 times

mount:
/dev/mapper/data-home on /home type xfs (rw,logbufs=8,usrquota) /dev/mapper/data-archive on /home/archive type xfs (rw,logbufs=8)

evenlode:~# xfs_info /home
meta-data=/dev/data/home         isize=256    agcount=32, agsize=7372784 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=235929088, imaxpct=25
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=0
realtime =none                   extsz=65536  blocks=0, rtextents=0
evenlode:~# xfs_info /home/archive
meta-data=/dev/data/archive      isize=256    agcount=16, agsize=6553600 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=104857600, imaxpct=25
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=0
realtime =none                   extsz=65536  blocks=0, rtextents=0
author
#3 lore raw
In reply to Timothy Shimmin

Timothy Shimmin wrote, on 18/03/08 00:28:

Quoted 16 lines

Hi Stuart,

Quoted 4 lines · Stuart Rowan

Hi,

Firstly thanks for the great filesystem and apologies if this ends up being NFS rather than XFS being weird! I'm not subscribed so please do keep me CC'd.

Quoted 9 lines

I have *millions* of lines of (>200k per minute according to syslog):
nfsd: non-standard errno: -117
being sent out of dmesg

Now errno 117 is
#define EUCLEAN 117 /* Structure needs cleaning */
which seems to be only used from a quick grep by XFS and JFFS and smbfs.

Quoted 32 lines

In XFS we mapped EFSCORRUPTED to EUCLEAN as EFSCORRUPTED didn't exist on Linux.
However, normally if this error is encountered in XFS then we output an appropriate msg to the syslog.
Our default error level is 3 and most reports are rated at 1 so should show up I would have thought.

--Tim

Quoted 21 lines

My nfs server export two locations
/home
/home/archive
both of these are XFS partitions, hence my suspicion that the -117 is coming from XFS.

xfs_repair -n says the filesystems are clean
xfs_repair has been run multiple times to completion on the filesystems, all is fine.

The XFS partitions are lvm volumes as follows
data/home 900G
data/archive 400G
The volume group, data, is sda3
sda3 is a 6 drive 3ware 9550SXU-8LP RAID10 array

The NFS server is currently in use (indeed the message only starts once clients connect) and works absolutely fine.

How do I find out what (if anything) is wrong with my filesystem / appropriately silence this message?

Many thanks,
Stu.

I briefly changed the sysctl fs.xfs.error_level to 6 and then back to 3

It gives the following message and backtrace

Quoted 24 lines
Mar 18 13:35:15 evenlode kernel: nfsd: non-standard errno: -117
Mar 18 13:35:15 evenlode kernel: 0x0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Mar 18 13:35:15 evenlode kernel: Filesystem "dm-0": XFS internal error xfs_itobp at line 360 of file fs/xfs/xfs_inode.c.  Caller 0xffffffff8821224d
Mar 18 13:35:15 evenlode kernel: Pid: 2791, comm: nfsd Not tainted 2.6.24.3-generic #1
Mar 18 13:35:15 evenlode kernel: Mar 18 13:35:15 evenlode kernel: Call Trace:
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820f784>] :xfs:xfs_itobp+0x141/0x17b
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820d7c9>] :xfs:xfs_iget_core+0x352/0x63a
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8029095f>] alloc_inode+0x152/0x1c2
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820db4c>] :xfs:xfs_iget+0x9b/0x13f
Mar 18 13:35:15 evenlode kernel:  [<ffffffff882243d1>] :xfs:xfs_vget+0x4d/0xbb
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8822f0d2>] :xfs:xfs_nfs_get_inode+0x2e/0x42
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8822f1f9>] :xfs:xfs_fs_fh_to_dentry+0x64/0x97
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88332554>] :exportfs:exportfs_decode_fh+0x30/0x1dc
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88347c16>] :nfsd:nfsd_acceptable+0x0/0xca
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8023b11a>] set_current_groups+0x148/0x153
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8834df5f>] :nfsd:nfsd_setuser+0x11c/0x171
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88347762>] :nfsd:nfsd_setuser_and_check_port+0x52/0x57
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88347edb>] :nfsd:fh_verify+0x1fb/0x4a4
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88261024>] :sunrpc:svc_tcp_recvfrom+0x7ab/0x843
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88349141>] :nfsd:nfsd_open+0x1f/0x170
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88349538>] :nfsd:nfsd_read+0x7f/0xc4
Mar 18 13:35:15 evenlode kernel:  [<ffffffff88350321>] :nfsd:nfsd3_proc_read+0x117/0x15a
Quoted 8 lines
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8834524e>] :nfsd:nfsd_dispatch+0xde/0x1c2
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8825e192>] :sunrpc:svc_process+0x3f7/0x6e9
Mar 18 13:35:15 evenlode kernel:  [<ffffffff803fba8d>] __down_read+0x12/0x9a
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8834582f>] :nfsd:nfsd+0x191/0x2ae
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8020cbe8>] child_rip+0xa/0x12
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8834569e>] :nfsd:nfsd+0x0/0x2ae
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8020cbde>] child_rip+0x0/0x12
Mar 18 13:35:15 evenlode kernel:

Does that help?

Thanks,
Stu.

#4 lore raw
Quoted 25 lines · Stuart Rowan

Timothy Shimmin wrote, on 18/03/08 00:28:

Quoted 24 lines

Hi Stuart,

Quoted 8 lines · Stuart Rowan

I have *millions* of lines of (>200k per minute according to syslog):
nfsd: non-standard errno: -117
being sent out of dmesg

Now errno 117 is
#define EUCLEAN 117 /* Structure needs cleaning */

In XFS we mapped EFSCORRUPTED to EUCLEAN as EFSCORRUPTED didn't exist on Linux.
However, normally if this error is encountered in XFS then we output an appropriate msg to the syslog.
Our default error level is 3 and most reports are rated at 1 so should show up I would have thought.

--Tim

Quoted 4 lines

xfs_repair -n says the filesystems are clean
xfs_repair has been run multiple times to completion on the filesystems, all is fine.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

The NFS server is currently in use (indeed the message only starts once clients connect) and works absolutely fine.

How do I find out what (if anything) is wrong with my filesystem / appropriately silence this message?

I briefly changed the sysctl fs.xfs.error_level to 6 and then back to 3

Good idea (I was thinking about that :-).

Somehow, your subject line referring to 2.6.24 didn't stick in my brain (that's pretty old).
So I was looking at recent code which I can't see has this error case from xfs_itobp() (it is now in xfs_imap_to_bp()).

Looking at old code, I see 2 EFSCORRUPTED paths with the following one triggering at XFS_ERRLEVEL_HIGH (and presumably why you didn't see it until now) ...

montep    |1.198|                            |  /*
montep    |1.198|                            |   * Validate the magic number and version of every inode in the buffer
montep    |1.198|                            |   * (if DEBUG kernel) or the first inode in the buffer, otherwise.
montep    |1.198|                            |   */
nathans   |1.303|2.4.x-xfs:slinx:74929a      |#ifdef DEBUG
montep    |1.198|                            |  ni = BBTOB(imap.im_len) >> mp->m_sb.sb_inodelog;
montep    |1.198|                            |#else
montep    |1.198|                            |  ni = 1;
montep    |1.198|                            |#endif
montep    |1.198|                            |  for (i = 0; i < ni; i++) {
doucette  |1.245|irix6.5f:irix:09146b        |          int             di_ok;
doucette  |1.245|irix6.5f:irix:09146b        |          xfs_dinode_t    *dip;
doucette  |1.245|irix6.5f:irix:09146b        |
lord      |1.292|2.4.0-test1-xfs:slinx:65571a|          dip = (xfs_dinode_t *)xfs_buf_offset(bp,
montep    |1.198|                            |                                  (i << mp->m_sb.sb_inodelog));
dxm       |1.285|2.4.0-test1-xfs:slinx:62350a|          di_ok = INT_GET(dip->di_core.di_magic, ARCH_CONVERT) == XFS_DINODE_MAGIC &&
dxm       |1.285|2.4.0-test1-xfs:slinx:62350a|                      XFS_DINODE_GOOD_VERSION(INT_GET(dip->di_core.di_version, ARCH_CONVERT));
overby    |1.362|2.4.x-xfs:slinx:136445a     |          if (unlikely(XFS_TEST_ERROR(!di_ok, mp, XFS_ERRTAG_ITOBP_INOTOBP,
overby    |1.362|2.4.x-xfs:slinx:136445a     |                           XFS_RANDOM_ITOBP_INOTOBP))) {
montep    |1.198|                            |#ifdef DEBUG
nathans   |1.337|2.4.x-xfs:slinx:119399a     |                  prdev("bad inode magic/vsn daddr 0x%llx #%d (magic=%x)",
nathans   |1.337|2.4.x-xfs:slinx:119399a     |                          mp->m_dev, (unsigned long long)imap.im_blkno, i,
nathans   |1.303|2.4.x-xfs:slinx:74929a      |                          INT_GET(dip->di_core.di_magic, ARCH_CONVERT));
montep    |1.198|                            |#endif
lord      |1.376|2.4.x-xfs:slinx:150747a     |                  XFS_CORRUPTION_ERROR("xfs_itobp", XFS_ERRLEVEL_HIGH,
overby    |1.362|2.4.x-xfs:slinx:136445a     |                                       mp, dip);
montep    |1.198|                            |                  xfs_trans_brelse(tp, bp);
sup       |1.216|                            |                  return XFS_ERROR(EFSCORRUPTED);
montep    |1.198|                            |          }
ajs       |1.143|                            |  }

So the first inode in the buffer has the wrong magic# or version#. I'm surprised that this wasn't picked up by repair or check.

--Tim

Quoted 20 lines

It gives the following message and backtrace

Quoted 4 lines
Mar 18 13:35:15 evenlode kernel: nfsd: non-standard errno: -117
Mar 18 13:35:15 evenlode kernel: 0x0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Mar 18 13:35:15 evenlode kernel: Filesystem "dm-0": XFS internal error xfs_itobp at line 360 of file fs/xfs/xfs_inode.c.  Caller 0xffffffff8821224d
Mar 18 13:35:15 evenlode kernel: Pid: 2791, comm: nfsd Not tainted 2.6.24.3-generic #1
Mar 18 13:35:15 evenlode kernel: Mar 18 13:35:15 evenlode kernel: Call Trace:
Quoted 8 lines
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820f784>] :xfs:xfs_itobp+0x141/0x17b
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820d7c9>] :xfs:xfs_iget_core+0x352/0x63a
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8029095f>] alloc_inode+0x152/0x1c2
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820db4c>] :xfs:xfs_iget+0x9b/0x13f
Mar 18 13:35:15 evenlode kernel:  [<ffffffff882243d1>] :xfs:xfs_vget+0x4d/0xbb

Does that help?

Thanks,
Stu.

#5 lore raw

XFS internal error xfs_itobp at line 360 of file fs/xfs/xfs_inode.c. (was Re: 2.6.24.3 nfs server on xfs keeps producing nfsd: non-standard errno: -117)

Quoted 31 lines · Timothy Shimmin
Quoted 30 lines · Stuart Rowan

Timothy Shimmin wrote, on 18/03/08 00:28:

Quoted 29 lines

Hi Stuart,

Quoted 8 lines · Stuart Rowan

I have *millions* of lines of (>200k per minute according to syslog):
nfsd: non-standard errno: -117
being sent out of dmesg

Now errno 117 is
#define EUCLEAN 117 /* Structure needs cleaning */

In XFS we mapped EFSCORRUPTED to EUCLEAN as EFSCORRUPTED didn't exist on Linux.
However, normally if this error is encountered in XFS then we output an appropriate msg to the syslog.
Our default error level is 3 and most reports are rated at 1 so should show up I would have thought.

--Tim

Quoted 8 lines

xfs_repair -n says the filesystems are clean
xfs_repair has been run multiple times to completion on the filesystems, all is fine.

The NFS server is currently in use (indeed the message only starts once clients connect) and works absolutely fine.

How do I find out what (if anything) is wrong with my filesystem / appropriately silence this message?

Quoted 9 lines
Quoted 2 lines

I briefly changed the sysctl fs.xfs.error_level to 6 and then back to 3

Good idea (I was thinking about that :-).

Somehow, your subject line referring to 2.6.24 didn't stick in my brain (that's pretty old).
So I was looking at recent code which I can't see has this error case from xfs_itobp() (it is now in xfs_imap_to_bp()).

Pretty old for you, latest released Linux kernel to me :-P

Quoted 35 lines

Looking at old code, I see 2 EFSCORRUPTED paths with the following one triggering at XFS_ERRLEVEL_HIGH (and presumably why you didn't see it until now) ...

montep    |1.198|                            |  /*
montep    |1.198|                            |   * Validate the magic number and version of every inode in the buffer
montep    |1.198|                            |   * (if DEBUG kernel) or the first inode in the buffer, otherwise.
montep    |1.198|                            |   */
nathans   |1.303|2.4.x-xfs:slinx:74929a      |#ifdef DEBUG
montep    |1.198|                            |  ni = BBTOB(imap.im_len) >> mp->m_sb.sb_inodelog;
montep    |1.198|                            |#else
montep    |1.198|                            |  ni = 1;
montep    |1.198|                            |#endif
montep    |1.198|                            |  for (i = 0; i < ni; i++) {
doucette  |1.245|irix6.5f:irix:09146b        |          int di_ok;
doucette  |1.245|irix6.5f:irix:09146b        |          xfs_dinode_t    *dip;
doucette  |1.245|irix6.5f:irix:09146b        |
lord      |1.292|2.4.0-test1-xfs:slinx:65571a|          dip = (xfs_dinode_t *)xfs_buf_offset(bp,
montep    |1.198|                            | (i << mp->m_sb.sb_inodelog));
dxm       |1.285|2.4.0-test1-xfs:slinx:62350a|          di_ok = INT_GET(dip->di_core.di_magic, ARCH_CONVERT) == XFS_DINODE_MAGIC &&
dxm       |1.285|2.4.0-test1-xfs:slinx:62350a| XFS_DINODE_GOOD_VERSION(INT_GET(dip->di_core.di_version, ARCH_CONVERT));
overby    |1.362|2.4.x-xfs:slinx:136445a     |          if (unlikely(XFS_TEST_ERROR(!di_ok, mp, XFS_ERRTAG_ITOBP_INOTOBP,
overby    |1.362|2.4.x-xfs:slinx:136445a     | XFS_RANDOM_ITOBP_INOTOBP))) {
montep    |1.198|                            |#ifdef DEBUG
nathans   |1.337|2.4.x-xfs:slinx:119399a     |                  prdev("bad inode magic/vsn daddr 0x%llx #%d (magic=%x)",
nathans   |1.337|2.4.x-xfs:slinx:119399a     | mp->m_dev, (unsigned long long)imap.im_blkno, i,
nathans   |1.303|2.4.x-xfs:slinx:74929a      | INT_GET(dip->di_core.di_magic, ARCH_CONVERT));
montep    |1.198|                            |#endif
lord      |1.376|2.4.x-xfs:slinx:150747a     | XFS_CORRUPTION_ERROR("xfs_itobp", XFS_ERRLEVEL_HIGH,
overby    |1.362|2.4.x-xfs:slinx:136445a     | mp, dip);
montep    |1.198|                            | xfs_trans_brelse(tp, bp);
sup       |1.216|                            |                  return XFS_ERROR(EFSCORRUPTED);
montep    |1.198|                            |          }
ajs       |1.143|                            |  }
Quoted 5 lines

So the first inode in the buffer has the wrong magic# or version#. I'm surprised that this wasn't picked up by repair or check.

--Tim

I have some more information! The server, evenlode, was previously serving NFS exports of ext3 filesystems. Last week we rsycned the data to the new server running XFS.

Eventually I spotted the high error rate was linked to the volume of NFS read calls (200k / minute). A quick tcpdump gave me a couple of likely looking hosts. I logged into one (bonny) and found gnome-panel using 100% CPU. I killed that and these messages have now reduced to a handful an hour. That gnome-panel will have had the NFS server and underlying NFS backing filesystem (ext3-> XFS) changed underneath it.

So my questions ...

Is it possible that the errors are related to duff request data being sent by the NFS clients because they are still referencing e.g. inodes as they were when the NFS server was ext3 backed?

Is it also possible that things like the rather high request rate (200k/sec) although that's reduced now, made a race in e.g. the XFS code triggerable?

As you say, it's rather suprising that this sort of issue is not being caught by xfs_repair (-n) and that's what leads me to suspect something else at play ...

Cheers,
Stu.

Quoted 23 lines
Quoted 20 lines

It gives the following message and backtrace

Quoted 4 lines
Mar 18 13:35:15 evenlode kernel: nfsd: non-standard errno: -117
Mar 18 13:35:15 evenlode kernel: 0x0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Mar 18 13:35:15 evenlode kernel: Filesystem "dm-0": XFS internal error xfs_itobp at line 360 of file fs/xfs/xfs_inode.c.  Caller 0xffffffff8821224d
Mar 18 13:35:15 evenlode kernel: Pid: 2791, comm: nfsd Not tainted 2.6.24.3-generic #1
Mar 18 13:35:15 evenlode kernel: Mar 18 13:35:15 evenlode kernel: Call Trace:
Quoted 8 lines
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820f784>] :xfs:xfs_itobp+0x141/0x17b
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8821224d>] :xfs:xfs_iread+0x71/0x1e8
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820d7c9>] :xfs:xfs_iget_core+0x352/0x63a
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8029095f>] alloc_inode+0x152/0x1c2
Mar 18 13:35:15 evenlode kernel:  [<ffffffff8820db4c>] :xfs:xfs_iget+0x9b/0x13f
Mar 18 13:35:15 evenlode kernel:  [<ffffffff882243d1>] :xfs:xfs_vget+0x4d/0xbb

Does that help?

Thanks,
Stu.