Skip to contentSkip to conversation

Is it possible the check an frozen XFS filesytem to avoid downtime

DISCUSSION Martin Steigerwald 3 participants 10 messages Jul 14, 2008 – Oct 28, 2008

lore
Share & tools
lore

Martin Steigerwald

author
#0 lore raw

Hi!

We seen in-memory corruption on two XFS filesystem on a server heartbeat cluster of one of our customers:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8824eb5d

Call Trace:
 [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
 [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
 [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
 [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
 [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
 [<ffffffff8022a06d>] notify_change+0x156/0x2ef
 [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
 [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
 [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
 [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
 [<ffffffff8025b20b>] __down_read+0x12/0x9a
 [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
 [<ffffffff80255f38>] child_rip+0xa/0x12
 [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
 [<ffffffff80255f2e>] child_rip+0x0/0x12

xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673
Filesystem "dm-1": Corruption of in-memory data detected. Shutting down
filesystem: dm-1
Please umount the filesystem, and rectify the problem(s)

on

Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) ([email protected]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007

We plan to do a takeover so that the server which appears to have memory errors can be memtested.

After the takeover we would like to make sure that the XFS filesystems are intact. Is it possible to do so without taking the filesystem completely offline?

I thought about mounting read only and it might be the best choice available, but then it will *fail* write accesses. I would prefer if these are just stalled.

I tried xfs_freeze -f on my laptop home directory, but then did not machine to get it check via xfs_check or xfs_repair -nd... is it possible at all?

Ciao,

Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
#1 lore raw

Hi Martin,

Quoted 48 lines · Martin Steigerwald

Hi!

We seen in-memory corruption on two XFS filesystem on a server heartbeat cluster of one of our customers:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8824eb5d

Call Trace:
 [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
 [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
 [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
 [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
 [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
 [<ffffffff8022a06d>] notify_change+0x156/0x2ef
 [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
 [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
 [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
 [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
 [<ffffffff8025b20b>] __down_read+0x12/0x9a
 [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
 [<ffffffff80255f38>] child_rip+0xa/0x12
 [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
 [<ffffffff80255f2e>] child_rip+0x0/0x12

xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673
Filesystem "dm-1": Corruption of in-memory data detected. Shutting down
filesystem: dm-1
Please umount the filesystem, and rectify the problem(s)

on

Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) ([email protected]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007

We plan to do a takeover so that the server which appears to have memory errors can be memtested.

After the takeover we would like to make sure that the XFS filesystems are intact. Is it possible to do so without taking the filesystem completely offline?

Quoted 8 lines

I thought about mounting read only and it might be the best choice available, but then it will *fail* write accesses. I would prefer if these are just stalled.

I tried xfs_freeze -f on my laptop home directory, but then did not machine to get it check via xfs_check or xfs_repair -nd... is it possible at all?

Ciao,

When I last tried (and I don't think Barry has done anything to it to change things) it wouldn't work.
However, I think it could/should be changed to make it work.

My notes from the SGI bug:

958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem

Quoted 60 lines

We've been asked a few times about the possibility of running xfs_check or xfs_repair -n on a frozen filesystem.
And a while back I looked into what some of the hinderances were. And now I've forgotten ;-))

I think there are hinderances for libxfs_init (check_open()) and for having a dirty log.

For libxfs_init, I found that I couldn't run the tools without error'ing out. I think I found out that I needed the INACTIVE flag,
without READONLY/DANGEROUSLY, like xfs_logprint does.

----------------------------------------
Date: Thu, 19 Oct 2006 11:24:06 +1000
From: Timothy Shimmin <[email protected]>
To: [email protected]
cc: [email protected]
Subject: Re: init.c patch
------------------------------------------------------
Ok, my understanding of the READONLY/DANGEROUSLY flags were wrong.
I thought they were just overriding flags when you were guaranteeing you were only reading and it would be more permissive,
but they are for doing stuff on readonly (ro) mounts.
They are rather confusing to me. When you go with defaults for repair and db then it doesn't set the INACTIVE flag.
It means if I do _not_ want to be fatal then I need to set INACTIVE but not set READONLY or DANGEROUSLY - which is what logprint does.

I would have thought they'd be an option which for commands which don't modify anything, that they can read from a non-ro mounted filesystem (at the users risk) - which is what logprint does. i.e an option which just sets INACTIVE and only produces a warning.

The other alternative is to be able to test for a frozen fs as you suggested.
----------------------------------------------------------

Lachlan suggested using a check_isfrozen() routine instead of overriding check_isactive().

And as far as the dirty log is concerned...
It will be dirty when it is frozen, but in a special way. It will have an unmount record followed by a dummy record - solely used so that when mounted again it can do
the unlinked list processing.
So we could add code to test if the log just had an unmount record followed by a dummy record and continue anyway knowing that the metadata was consistent.
e.g. in xfs_repair/phase2.c:zero_log() it calls xlog_find_tail() and tests if (head_blk != tail_blk) to know if the log is dirty. I think libxfs should provide a routine: libxfs_dirty_log or in the libxlog code with a suitable name,
which could say how dirty the log is ;-)
Is it dirty such that we have real transactions to replay or does it just have to do the unlinked processing as in the case of a frozen filesystem.
It would be nice anyway to have an abstraction here because it is finding out the head and tail blocks solely for this purpose and doesn't really care what they are.

Quoted 1 line

--Tim

--Tim

author
#2 lore raw

Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin:

Quoted 1 line

Hi Martin,

Hi Tim,

Quoted 57 lines
Quoted 56 lines · Martin Steigerwald

Hi!

We seen in-memory corruption on two XFS filesystem on a server heartbeat cluster of one of our customers:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8824eb5d

Call Trace:
 [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
 [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
 [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
 [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
 [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
 [<ffffffff8022a06d>] notify_change+0x156/0x2ef
 [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
 [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
 [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
 [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
 [<ffffffff8025b20b>] __down_read+0x12/0x9a
 [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
 [<ffffffff80255f38>] child_rip+0xa/0x12
 [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
 [<ffffffff80255f2e>] child_rip+0x0/0x12

xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673 Filesystem "dm-1": Corruption of in-memory data detected. Shutting down
filesystem: dm-1
Please umount the filesystem, and rectify the problem(s)

on

Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) ([email protected]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007

We plan to do a takeover so that the server which appears to have memory errors can be memtested.

After the takeover we would like to make sure that the XFS filesystems are intact. Is it possible to do so without taking the filesystem completely offline?

I thought about mounting read only and it might be the best choice available, but then it will *fail* write accesses. I would prefer if these are just stalled.

I tried xfs_freeze -f on my laptop home directory, but then did not machine to get it check via xfs_check or xfs_repair -nd... is it possible at all?

Quoted 5 lines
Quoted 1 line

Ciao,

When I last tried (and I don't think Barry has done anything to it to change things) it wouldn't work.
However, I think it could/should be changed to make it work.

Okay... we recommended the customer to do it the safe way unmounting the filesystem completely. He did and the filesystem appear to be intact *phew*. XFS appeared to detect the in memory corruption early enough.

Its a bit strange however, cause we now know that the server sports ECC RAM. Well we will see what memtest86+ has to say about it.

Quoted 33 lines

My notes from the SGI bug:

958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem

Quoted 29 lines

We've been asked a few times about the possibility of running xfs_check or xfs_repair -n on a frozen filesystem.
And a while back I looked into what some of the hinderances were. And now I've forgotten ;-))

I think there are hinderances for libxfs_init (check_open()) and for having a dirty log.

For libxfs_init, I found that I couldn't run the tools without error'ing out. I think I found out that I needed the INACTIVE flag, without READONLY/DANGEROUSLY, like xfs_logprint does.

----------------------------------------
Date: Thu, 19 Oct 2006 11:24:06 +1000
From: Timothy Shimmin <[email protected]>
To: [email protected]
cc: [email protected]
Subject: Re: init.c patch
------------------------------------------------------
Ok, my understanding of the READONLY/DANGEROUSLY flags were wrong. I thought they were just overriding flags when you were guaranteeing
you were only reading and it would be more permissive,
but they are for doing stuff on readonly (ro) mounts.

They are rather confusing to me. When you go with defaults for repair
and db then it doesn't set the INACTIVE flag.
It means if I do _not_ want to be fatal then I need to set INACTIVE but
not set READONLY or DANGEROUSLY - which is what logprint does.

I think that there should be different options for readonly / frozen fs checking and dangerous repair... since I think readonly checks are a different thing than repairing a mounted filesystem and hoping that the running XFS will not choke upon the filesystem that xfs_repair changes under its hood.

I expected a "-r" for read only in xfs_check and xfs_repair, well but for xfs_repair this option is already taken for specifying the realtime volume.

Ciao,

Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
#3 lore raw
Quoted 6 lines · Martin Steigerwald

Okay... we recommended the customer to do it the safe way unmounting the filesystem completely. He did and the filesystem appear to be intact *phew*. XFS appeared to detect the in memory corruption early enough.

Its a bit strange however, cause we now know that the server sports ECC RAM. Well we will see what memtest86+ has to say about it.

in-memory corruption could mean, but certainly does not absolutely mean, problematic memory. It could be, and usually is, a plain ol' bug (in xfs or elsewhere).

-Eric

author
#4 lore raw

Am Dienstag, 15. Juli 2008 17:27:39 schrieb Eric Sandeen:

Quoted 11 lines
Quoted 6 lines · Martin Steigerwald

Okay... we recommended the customer to do it the safe way unmounting the filesystem completely. He did and the filesystem appear to be intact *phew*. XFS appeared to detect the in memory corruption early enough.

Its a bit strange however, cause we now know that the server sports ECC RAM. Well we will see what memtest86+ has to say about it.

in-memory corruption could mean, but certainly does not absolutely mean, problematic memory. It could be, and usually is, a plain ol' bug (in xfs or elsewhere).

Thanks.

Yes, I thought about this, too. But then the machine ran over one year without any visible issues. And it happened only on one server, not on the other. It happened on the server that does NFS tough... could be an NFS related issue then. The other one does MySQL with the database stored on a XFS volume, too. But there haven't been any visible issues.

Well we will see whether it happens again on the server that has taken over and is now doing both MySQL and NFS. If it does, I think we will update to one of the lastest Debian Etch backport kernels (2.6.24 or even 2.6.25) on one of the servers and see whether that helps.

Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
author
#5 lore raw
In reply to Eric Sandeen

Am Dienstag, 15. Juli 2008 schrieb Eric Sandeen:

Quoted 11 lines
Quoted 6 lines · Martin Steigerwald

Okay... we recommended the customer to do it the safe way unmounting the filesystem completely. He did and the filesystem appear to be intact *phew*. XFS appeared to detect the in memory corruption early enough.

Its a bit strange however, cause we now know that the server sports ECC RAM. Well we will see what memtest86+ has to say about it.

in-memory corruption could mean, but certainly does not absolutely mean, problematic memory. It could be, and usually is, a plain ol' bug (in xfs or elsewhere).

Ok, just as a follow up:

Now we got similar XFS errors on the second backend server, this time on a local hardware RAID1 while on the first backend server it was on logical volumes on a soft RAID spread over two dislocated external hardware RAID boxes.

So this appears to be an XFS bug to me. Maybe when running for long time it corrupts its in-memory structures. Fortunately we did not see errors in on-disk structures.

A colleague did a kernel update on the inactive backend 1 server from 2.6.21 to 2.6.26 kernel from backports.org, tommorow backend 2 will follow. Let's see whether that solves the issue.

Anyway it seems to be a hard to trigger bug and before bugging you with something in kernel 2.6.21, we at least update to the latest backports.org kernel.

Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
#6 lore raw
Quoted 7 lines · Martin Steigerwald

A colleague did a kernel update on the inactive backend 1 server from 2.6.21 to 2.6.26 kernel from backports.org, tommorow backend 2 will follow. Let's see whether that solves the issue.

Anyway it seems to be a hard to trigger bug and before bugging you with something in kernel 2.6.21, we at least update to the latest backports.org kernel.

Honestly, I'd try a 2.6.27 kernel if you can, a few more problems were fixed there.

-Eric

author
#7 lore raw

Am Monday 27 October 2008 18:15:19 schrieb Eric Sandeen:

Quoted 11 lines
Quoted 7 lines · Martin Steigerwald

A colleague did a kernel update on the inactive backend 1 server from 2.6.21 to 2.6.26 kernel from backports.org, tommorow backend 2 will follow. Let's see whether that solves the issue.

Anyway it seems to be a hard to trigger bug and before bugging you with something in kernel 2.6.21, we at least update to the latest backports.org kernel.

Honestly, I'd try a 2.6.27 kernel if you can, a few more problems were fixed there.

Thanks for the hint. backports.org doesn't contain a 2.6.26 and AFAIK lenny will ship with 2.6.26 as well, thus I'd like to try that one before doing a backport of 2.6.27 from sid myself. Thus I hope the issue that is triggered on those servers is fixed with 2.6.26 or some prior kernel since 2.6.21 already. If not, we will consider backporting 2.6.27.

At least on my notebook I didn't have any issues so far with any kernel since 2.6.17.7. But even with TuxOnIce it doesn't have uptimes of 100 days or more, since I compile a new kernel for it once in a while.

Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
#8 lore raw
In reply to Martin Steigerwald
Quoted 105 lines · Martin Steigerwald

Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin:

Quoted 1 line

Hi Martin,

Hi Tim,

Quoted 53 lines
Quoted 52 lines · Martin Steigerwald

Hi!

We seen in-memory corruption on two XFS filesystem on a server heartbeat cluster of one of our customers:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8824eb5d

Call Trace:
 [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
 [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
 [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
 [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
 [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
 [<ffffffff8022a06d>] notify_change+0x156/0x2ef
 [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
 [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
 [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
 [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
 [<ffffffff8025b20b>] __down_read+0x12/0x9a
 [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
 [<ffffffff80255f38>] child_rip+0xa/0x12
 [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
 [<ffffffff80255f2e>] child_rip+0x0/0x12

xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673 Filesystem "dm-1": Corruption of in-memory data detected. Shutting down
filesystem: dm-1
Please umount the filesystem, and rectify the problem(s)

on

Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) ([email protected]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007

We plan to do a takeover so that the server which appears to have memory errors can be memtested.

After the takeover we would like to make sure that the XFS filesystems are intact. Is it possible to do so without taking the filesystem completely offline?

I thought about mounting read only and it might be the best choice available, but then it will *fail* write accesses. I would prefer if these are just stalled.

Quoted 8 lines
Quoted 5 lines

I tried xfs_freeze -f on my laptop home directory, but then did not machine to get it check via xfs_check or xfs_repair -nd... is it possible at all?

Ciao,

When I last tried (and I don't think Barry has done anything to it to change things) it wouldn't work.
However, I think it could/should be changed to make it work.

Okay... we recommended the customer to do it the safe way unmounting the filesystem completely. He did and the filesystem appear to be intact *phew*. XFS appeared to detect the in memory corruption early enough.

Its a bit strange however, cause we now know that the server sports ECC RAM. Well we will see what memtest86+ has to say about it.

Quoted 24 lines

My notes from the SGI bug:

958642: running xfs_check and "xfs_repair -n" on a frozen xfs filesystem

Quoted 20 lines

We've been asked a few times about the possibility of running xfs_check or xfs_repair -n on a frozen filesystem.
And a while back I looked into what some of the hinderances were. And now I've forgotten ;-))

I think there are hinderances for libxfs_init (check_open()) and for having a dirty log.

For libxfs_init, I found that I couldn't run the tools without error'ing out. I think I found out that I needed the INACTIVE flag, without READONLY/DANGEROUSLY, like xfs_logprint does.

----------------------------------------
Date: Thu, 19 Oct 2006 11:24:06 +1000
From: Timothy Shimmin <[email protected]>
To: [email protected]
cc: [email protected]
It means if I do _not_ want to be fatal then I need to set INACTIVE but
not set READONLY or DANGEROUSLY - which is what logprint does.

I think that there should be different options for readonly / frozen fs checking and dangerous repair... since I think readonly checks are a different thing than repairing a mounted filesystem and hoping that the running XFS will not choke upon the filesystem that xfs_repair changes under its hood.

Quoted 4 lines

I expected a "-r" for read only in xfs_check and xfs_repair, well but for xfs_repair this option is already taken for specifying the realtime volume.

Ciao,

My comments mentioned above were for the c flags used in
libxfs and the commands.
I was curious to know why logprint was able to continue printing and doing its stuff but repair and check couldn't and would exit.
What actual _command_ options/flags we use is a different matter and I probably confused the issue by using the term flags.

Looking at the code:

#define LIBXFS_ISREADONLY       0x0002  /* disallow all mounted filesystems */
#define LIBXFS_ISINACTIVE       0x0004  /* allow mounted only if mounted ro */
#define LIBXFS_DANGEROUSLY      0x0008  /* repairing a device mounted ro    */

-n No modify mode. Specifies that xfs_repair should not modify the
filesystem but should only scan the filesystem and indicate what repairs would have been made.

-d Repair dangerously. Allow xfs_repair to repair an XFS filesystem
mounted read only. This is typically done on a root fileystem from single user mode, immediately followed by a reboot.

if (no_modify) /* -n */
args->isreadonly = (LIBXFS_ISREADONLY | LIBXFS_ISINACTIVE);
else if (dangerously) /* -d */
args->isreadonly = (LIBXFS_DANGEROUSLY | LIBXFS_ISINACTIVE);

The comment for LIBXFS_ISINACTIVE seems a bit misleading as I believe it will allow things to continue on a filesystem which is not marked as readonly as xfs_logprint does.

Looking at:

init.c:

        if (!readonly && !inactive && platform_check_ismounted(path, *blockfile, NULL, 1))
                return 0;
        
        if (inactive && check_isactive(path, *blockfile, ((readonly|dangerously)?1:0)))
                return 0;

static int
check_isactive(char *name, char *block, int fatal) {
....

        return platform_check_iswritable(name, block, &st, fatal);
}

Okay, so this is why it worked fine for just ISINACTIVE but not when either ISREADONLY or DANGEROUSLY are set.

Yes, it would be nice for the commands to detect that the filesystem is frozen and then allow things to proceed in a non modifying mode.

--Tim

author
#9 lore raw
In reply to Timothy Shimmin

Am Dienstag, 15. Juli 2008 05:38:23 schrieb Timothy Shimmin:

Quoted 59 lines

Hi Martin,

Quoted 56 lines · Martin Steigerwald

Hi!

We seen in-memory corruption on two XFS filesystem on a server heartbeat cluster of one of our customers:

XFS internal error XFS_WANT_CORRUPTED_GOTO at line 1563 of file fs/xfs/xfs_alloc.c. Caller 0xffffffff8824eb5d

Call Trace:
 [<ffffffff8824cff3>] :xfs:xfs_free_ag_extent+0x1a6/0x6b5
 [<ffffffff8824eb5d>] :xfs:xfs_free_extent+0xa9/0xc9
 [<ffffffff88258636>] :xfs:xfs_bmap_finish+0xf0/0x169
 [<ffffffff88278b4c>] :xfs:xfs_itruncate_finish+0x180/0x2c1
 [<ffffffff8829071a>] :xfs:xfs_setattr+0x841/0xe59
 [<ffffffff8022e868>] sock_common_recvmsg+0x30/0x45
 [<ffffffff8829adc8>] :xfs:xfs_vn_setattr+0x121/0x144
 [<ffffffff8022a06d>] notify_change+0x156/0x2ef
 [<ffffffff883bf9c6>] :nfsd:nfsd_setattr+0x334/0x4b1
 [<ffffffff883c61d6>] :nfsd:nfsd3_proc_setattr+0xa2/0xae
 [<ffffffff883bb24d>] :nfsd:nfsd_dispatch+0xdd/0x19e
 [<ffffffff8833a10e>] :sunrpc:svc_process+0x3cb/0x6d9
 [<ffffffff8025b20b>] __down_read+0x12/0x9a
 [<ffffffff883bb816>] :nfsd:nfsd+0x192/0x2b0
 [<ffffffff80255f38>] child_rip+0xa/0x12
 [<ffffffff883bb684>] :nfsd:nfsd+0x0/0x2b0
 [<ffffffff80255f2e>] child_rip+0x0/0x12

xfs_force_shutdown(dm-1,0x8) called from line 4261 of file fs/xfs/xfs_bmap.c. Return address = 0xffffffff88258673 Filesystem "dm-1": Corruption of in-memory data detected. Shutting down
filesystem: dm-1
Please umount the filesystem, and rectify the problem(s)

on

Linux version 2.6.21-1-amd64 (Debian 2.6.21-4~bpo.1) ([email protected]) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Tue Jun 5 07:43:32 UTC 2007

We plan to do a takeover so that the server which appears to have memory errors can be memtested.

After the takeover we would like to make sure that the XFS filesystems are intact. Is it possible to do so without taking the filesystem completely offline?

I thought about mounting read only and it might be the best choice available, but then it will *fail* write accesses. I would prefer if these are just stalled.

I tried xfs_freeze -f on my laptop home directory, but then did not machine to get it check via xfs_check or xfs_repair -nd... is it possible at all?

Quoted 5 lines
Quoted 1 line

Ciao,

When I last tried (and I don't think Barry has done anything to it to change things) it wouldn't work.
However, I think it could/should be changed to make it work.

I am wondering whether it would need to set an option at all.

Shouldn't checking a filesystem that is not being written too be safe? So xfs_check and xfs_repair -n could just check whether fs is readonly or frozen and if so continue without requiring a special option? They can print the fact aka

Checking a frozen filesystem

or

Checking a read only filesystem

in the beginning tough.

Only thing is the log itself... when that is not cleared upon a freeze or readonly mount, it might be a problem for xfs_check and xfs_repair -n.

Ciao,

Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90