XFS corruption with failover
Folks:
We're deploying XFS in a configuration where the file system is being exported with NFS. XFS is being mounted on Linux, with default options; an iSCSI volume is the formatted media. We're working out a failover solution for this deployment utilizing Linux HA. Things appear to work correctly in the general case, but in continuous testing we're getting XFS superblock corruption on a very reproducible basis.
The sequence of events in our test scenario:
1. NFS server #1 online
2. Run IO to NFS server #1 from NFS client
3. NFS server #1 offline, (via passing 'b' to /proc/sysrq-trigger)
4. NFS server #2 online
5. XFS mounted as part of failover mechanism, mount fails
The mount fails with the following:
<snip>
kernel: XFS mounting filesystem sde
kernel: Starting XFS recovery on filesystem: sde (logdev: internal)
kernel: XFS: xlog_recover_process_data: bad clientid
kernel: XFS: log mount/recovery failed: error 5
kernel: XFS: log mount failed
</snip>
When running xfs_repair:
<snip>
[root@machine ~]# xfs_repair /dev/sde xfs_repair: warning - cannot set blocksize on block device /dev/sde: Invalid argument Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs ...
</snip>
Any advice or insight into what we're doing wrong would be very much appreciated. My apologies in advance for the somewhat off-topic question.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Le Thu, 13 Aug 2009 15:17:22 -0500 vous écriviez:
Quoted 3 lines
Any advice or insight into what we're doing wrong would be very much appreciated. My apologies in advance for the somewhat off-topic question.
By killing abruptly the primary server while doing IO, you're probably pushing the envelope... You may have a somewhat better luck with a cluster fs, OCFS2 works very well for me usually (GFS is a complete PITA to setup).
The better option would be to disallow completely write caching on the client side (because this is probably where it's going wrong) however I don't know how. You can get it to flush extremely often by playing with /proc/sys/vm/dirty_expire_centiseconds and /proc/sys/vm/dirty_writeback_centisecs, though. Safer settings generally imply terrible performance, though, you've been warned.
Ah another thing may be some cache option in the iSCSI target. what target are you using?
Signature · 8 lines
-------------------------------------------------- Emmanuel Florac www.intellique.com -------------------------------------------------- _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · Emmanuel Florac
Le Thu, 13 Aug 2009 15:17:22 -0500 vous écriviez:
Quoted 3 lines
Any advice or insight into what we're doing wrong would be very much appreciated. My apologies in advance for the somewhat off-topic question.
By killing abruptly the primary server while doing IO, you're probably pushing the envelope...
I don't think it's pushing too much. XFS was designed to survive such events.
Quoted 8 lines
You may have a somewhat better luck with a
cluster fs, OCFS2 works very well for me usually (GFS is a complete PITA to setup).The better option would be to disallow completely write caching on the client side (because this is probably where it's going wrong) however I don't know how.
Client's caching can't affect the metadata and the log in particular operations on the server. Client may indeed loose some data, but that's completely different issue.
Felix
Quoted 5 lines
You can get it to flush extremely
often by playing with/proc/sys/vm/dirty_expire_centisecondsand/proc/sys/vm/dirty_writeback_centisecs, though. Safer settings generally imply terrible performance, though, you've been warned.
Quoted 12 lines
Ah another thing may be some cache option in the iSCSI target. what target are you using?
Signature · 8 lines
-------------------------------------------------- Emmanuel Florac www.intellique.com -------------------------------------------------- _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Felix Blyakher
I don't think it's pushing too much. XFS was designed to survive such events.
And that was my understanding, based on all I've read about the design and intended usage. XFS has been remarkably resilient in the face of various poor operating conditions, and this is the only environment under which failure has been observed. It's for this reason that I assumed it's something we're doing wrong, and not an inherent issue with the file system.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Emmanuel Florac
By killing abruptly the primary server while doing IO, you're probably pushing the envelope... You may have a somewhat better luck with a cluster fs, OCFS2 works very well for me usually (GFS is a complete PITA to setup).
Acknowledged; we've looked at GFS, and I've been meaning to read up on OCFS2. For various reasons, particularly performance, ease of deployment and flexible growth, XFS has been the clear winner in our particular case (and our case is fairly unique, as our volume is backed by a distributed storage device).
Quoted 4 lines
You can get it to flush extremely
often by playing with/proc/sys/vm/dirty_expire_centisecondsand/proc/sys/vm/dirty_writeback_centisecs, though. Safer settings generally imply terrible performance, though, you've been warned.
Okay, interesting, I wasn't aware of these and will look into it.
Quoted 2 lines
Ah another thing may be some cache option in the iSCSI target. what target are you using?
No caching target side - I can speak definitely on that because I wrote it (integrates with our data dispersal stack [1]). Also, we're utilizing the same target when failing over; it's just the ISCSI initiator (aka, the NFS server) that is changing.
Thank you kindly for the quick response.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · John Quigley
Folks:
We're deploying XFS in a configuration where the file system is being exported with NFS. XFS is being mounted on Linux, with default options; an iSCSI volume is the formatted media. We're working out a failover solution for this deployment utilizing Linux HA. Things appear to work correctly in the general case, but in continuous testing we're getting XFS superblock corruption on a very reproducible basis. The sequence of events in our test scenario:
1. NFS server #1 online
2. Run IO to NFS server #1 from NFS client
3. NFS server #1 offline, (via passing 'b' to/proc/sysrq-trigger)
4. NFS server #2 online
5. XFS mounted as part of failover mechanism, mount fails
Quoted 7 lines
The mount fails with the following:
<snip>
kernel: XFS mounting filesystem sde
kernel: Starting XFS recovery on filesystem: sde (logdev: internal)
kernel: XFS:xlog_recover_process_data: bad clientid
kernel: XFS: log mount/recovery failed: error 5
This is an IO error. Is the block device (/dev/sde) accessible from the server #2 OK? Can you dd from that device?
Quoted 5 lines
kernel: XFS: log mount failed
</snip>When running
xfs_repair:
That's not a good time to run xfs_repair. There were no indication that the filesystem is corrupted.
Let's take for a sec "NFS server #2" out of the picture. Can you mount the filesystem from the original server after it reboots?
Felix
Quoted 10 lines
<snip>
[root@machine ~]#xfs_repair/dev/sdexfs_repair: warning - cannot set blocksize on block device/dev/sde: Invalid argument Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs ...
</snip>
Quoted 8 lines
Any advice or insight into what we're doing wrong would be very much appreciated. My apologies in advance for the somewhat off-topic question.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 21 lines · Felix Blyakher
Quoted 19 lines · John Quigley
Folks:
We're deploying XFS in a configuration where the file system is being exported with NFS. XFS is being mounted on Linux, with default options; an iSCSI volume is the formatted media. We're working out a failover solution for this deployment utilizing Linux HA. Things appear to work correctly in the general case, but in continuous testing we're getting XFS superblock corruption on a very reproducible basis.
The sequence of events in our test scenario:1. NFS server #1 online
2. Run IO to NFS server #1 from NFS client
3. NFS server #1 offline, (via passing 'b' to/proc/sysrq-trigger)
4. NFS server #2 online
5. XFS mounted as part of failover mechanism, mount failsThe mount fails with the following:
Quoted 8 lines
Quoted 5 lines
<snip>
kernel: XFS mounting filesystem sde
kernel: Starting XFS recovery on filesystem: sde (logdev: internal)
kernel: XFS:xlog_recover_process_data: bad clientid
kernel: XFS: log mount/recovery failed: error 5This is an IO error. Is the block device (
/dev/sde) accessible from the server #2 OK? Can you dd from that device?
Are you sure?
if (ohead->oh_clientid != XFS_TRANSACTION &&
ohead->oh_clientid != XFS_LOG) {
xlog_warn(
"XFS: xlog_recover_process_data: bad clientid");
ASSERT(0);
return (XFS_ERROR(EIO));
}so it does say EIO but that seems to me to be the wrong error; loks more like a bad log to me.
It does make me wonder if there's any sort of per-initiator caching on the iscsi target or something. </handwave>
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 44 lines · ----- "Eric Sandeen
Quoted 11 lines · Felix Blyakher
Quoted 9 lines · John Quigley
Folks:
We're deploying XFS in a configuration where the file system is being exported with NFS. XFS is being mounted on Linux, with default options; an iSCSI volume is the formatted media. We're working out a failover solution for this deployment utilizing Linux
HA. Things appear to work correctly in the general case, but in continuous testing we're getting XFS superblock corruption on a
very
Quoted 9 lines
Quoted 9 lines
reproducible basis.
The sequence of events in our test scenario:1. NFS server #1 online
2. Run IO to NFS server #1 from NFS client
3. NFS server #1 offline, (via passing 'b' to/proc/sysrq-trigger)
4. NFS server #2 online
5. XFS mounted as part of failover mechanism, mount failsQuoted 5 lines
Quoted 5 lines
The mount fails with the following:
<snip>
kernel: XFS mounting filesystem sde
kernel: Starting XFS recovery on filesystem: sde (logdev:internal)
Quoted 5 lines
Quoted 2 lines
kernel: XFS:
xlog_recover_process_data: bad clientid
kernel: XFS: log mount/recovery failed: error 5This is an IO error. Is the block device (
/dev/sde) accessible from the server #2 OK? Can you dd from that device?Are you sure?
if (ohead->oh_clientid != XFS_TRANSACTION && ohead->oh_clientid != XFS_LOG) { xlog_warn( "XFS: xlog_recover_process_data: bad clientid"); ASSERT(0); return (XFS_ERROR(EIO)); }
Quoted 7 lines
so it does say EIO but that seems to me to be the wrong error; loks more
like a bad log to me.It does make me wonder if there's any sort of per-initiator caching on
the iscsi target or something. </handwave>
Should barriers be enabled in XFS then?
Quoted 7 lines
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 44 lines · Lachlan McIlroy
Quoted 42 lines · ----- "Eric Sandeen
Quoted 11 lines · Felix Blyakher
Quoted 9 lines · John Quigley
Folks:
We're deploying XFS in a configuration where the file system is being exported with NFS. XFS is being mounted on Linux, with default options; an iSCSI volume is the formatted media. We're working out a failover solution for this deployment utilizing Linux
HA. Things appear to work correctly in the general case, but in continuous testing we're getting XFS superblock corruption on a
very
Quoted 9 lines
Quoted 9 lines
reproducible basis.
The sequence of events in our test scenario:1. NFS server #1 online
2. Run IO to NFS server #1 from NFS client
3. NFS server #1 offline, (via passing 'b' to/proc/sysrq-trigger)
4. NFS server #2 online
5. XFS mounted as part of failover mechanism, mount failsQuoted 5 lines
Quoted 5 lines
The mount fails with the following:
<snip>
kernel: XFS mounting filesystem sde
kernel: Starting XFS recovery on filesystem: sde (logdev:internal)
Quoted 4 lines
Quoted 2 lines
kernel: XFS:
xlog_recover_process_data: bad clientid
kernel: XFS: log mount/recovery failed: error 5This is an IO error. Is the block device (
/dev/sde) accessible from the server #2 OK? Can you dd from that device?Are you sure?
if (ohead->oh_clientid != XFS_TRANSACTION && ohead->oh_clientid != XFS_LOG) { xlog_warn( "XFS: xlog_recover_process_data: bad clientid"); ASSERT(0); return (XFS_ERROR(EIO)); }
Quoted 8 lines
Quoted 7 lines
so it does say EIO but that seems to me to be the wrong error; loks more
like a bad log to me.It does make me wonder if there's any sort of per-initiator caching on
the iscsi target or something. </handwave>Should barriers be enabled in XFS then?
Could try it but I bet the iscsi target doesn't claim to support them...
-eric
Quoted 7 lines
Quoted 6 lines
-Eric
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · Eric Sandeen
Quoted 1 line
Should barriers be enabled in XFS then?
Could try it but I bet the iscsi target doesn't claim to support them...
The target implementation, being new, is fairly naive and does not support this (or have any caching facilities, for that matter) at this time.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 10 lines · Eric Sandeen
Are you sure?
if (ohead->oh_clientid != XFS_TRANSACTION && ohead->oh_clientid != XFS_LOG) { xlog_warn( "XFS: xlog_recover_process_data: bad clientid"); ASSERT(0); return (XFS_ERROR(EIO)); }
Quoted 2 lines
so it does say EIO but that seems to me to be the wrong error; loks more like a bad log to me.
Hey Eric:
That would certainly be consistent with our experience, as the only way we're able to bring the file system back online is by zeroing the log.
Quoted 2 lines
It does make me wonder if there's any sort of per-initiator caching on the iscsi target or something. </handwave>
There isn't, as mentioned above, though we have several intermediate layers between the file system and iSCSI initiator, including multipath and LVM, both of which I was initially suspicious of. In testing with a similar scenario but in a more isolate fashion without those two intermediates, the behavior was still present. Also, just to clarify the topology:
/-----[Failover Secondary]------\
/ \
NFS Client ----/ \-----[ISCSI Target]----[Distributed Storage]
\ /
\ /
\-----[Failover Primary]--------/Those two failover machines, Primary and Secondary, act as the NFS server, the XFS mountpoint and ISCSI initiator. Only one failover machine is logged into the ISCSI target/has XFS mounted.
Thanks very much for your cycles on this guys.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Eric Sandeen
Quoted 2 lines
This is an IO error. Is the block device (
/dev/sde) accessible from the server #2 OK? Can you dd from that device?Are you sure?
No, I'm not. Replied first without looking at the code ^)
Quoted 10 lines
if (ohead->oh_clientid != XFS_TRANSACTION &&
ohead->oh_clientid != XFS_LOG) {
xlog_warn(
"XFS: xlog_recover_process_data: bad clientid");
ASSERT(0);
return (XFS_ERROR(EIO));
}Quoted 2 lines
so it does say EIO but that seems to me to be the wrong error; loks more like a bad log to me.
Agree. It does look like corrupted (incomplete) log.
Quoted 4 lines
It does make me wonder if there's any sort of per-initiator caching on the iscsi target or something. </handwave>
Yep, somewhere piece of log left out in a cache, and wasn't flushed to disk.
Felix
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 2 lines · Felix Blyakher
This is an IO error. Is the block device (
/dev/sde) accessible from the server #2 OK? Can you dd from that device?
Interesting suggestion; I don't recall having seen any indication of IO errors, but I'm testing again this evening, and will report back on what I find with dd.
Quoted 2 lines
That's not a good time to run
xfs_repair. There were no indication that the filesystem is corrupted.
Interesting again. I ran the tool merely because I had no other recourse to remedy this.
Quoted 3 lines
Let's take for a sec "NFS server #2" out of the picture. Can you mount the filesystem from the original server after it reboots?
This has been tested and fails, so it does appear to be a corruption issue on the stable media. Based on your first comment, I'm re-running tests to verify that the ISCSI volumes are fully online and ready to accept IO before mounting the file system.
Thank you very much.
- John Quigley
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs