xfsdump: more projid32bit fixes

extend fs_info and fs_getid to return fs flags (features) from GEOM call as well. Not used yet in this patch. Will be used in subsequent patch to detect 32-bit project ID filesystems.

Signed-off-by Eric Sandeen <[email protected]>

---

Because 32-bit project ID is not default and semi-rare, default to version 3 dumps unless dumping a filesystem with the 32-bit project ID feature flag set.

XFS_FSOP_GEOM_FLAGS_PROJID32 is a newish flag so I've redefined it here just in caes as well.

Signed-off-by Eric Sandeen <[email protected]>

---

common/fs.h +4 -0

index 878385b..7e63f8e 100644
@@ -26,6 +26,10 @@
  #define FS_MAXNAMELEN_DEFAULT	256
  #define FS_MAXPATHLEN_DEFAULT	1024
 
+#ifndef XFS_FSOP_GEOM_FLAGS_PROJID32
+#define XFS_FSOP_GEOM_FLAGS_PROJID32 0x0800
+#endif
+
  /* fs_info - decides if a source name describes a file system, and if
   * so returns useful information about that file system.
   *

common/global.h +0 -0

index 5138ed8..a847c5d 100644

Quoted 30 lines · Eric Sandeen

Because 32-bit project ID is not default and semi-rare, default to version 3 dumps unless dumping a filesystem with the 32-bit project ID feature flag set.

XFS_FSOP_GEOM_FLAGS_PROJID32 is a newish flag so I've redefined it here just in caes as well.

Signed-off-by Eric Sandeen <[email protected]>

---

common/fs.h +4 -0

index 878385b..7e63f8e 100644
@@ -26,6 +26,10 @@
  #define FS_MAXNAMELEN_DEFAULT    256
  #define FS_MAXPATHLEN_DEFAULT    1024
 
+#ifndef XFS_FSOP_GEOM_FLAGS_PROJID32
+#define XFS_FSOP_GEOM_FLAGS_PROJID32 0x0800
+#endif
+
  /* fs_info - decides if a source name describes a file system, and if
   * so returns useful information about that file system.
   *

common/global.h +0 -0

index 5138ed8..a847c5d 100644

common/global.h line 31

Quoted 3 lines

common/global.h +0 -0

@@ -29,14 +29,14 @@
  #define GLOBAL_HDR_VERSION_2    2
  #define GLOBAL_HDR_VERSION_3    3
  #define GLOBAL_HDR_VERSION_4    4

Hmm looks a a patch from your previous version.

common/global.h line 42

Quoted 13 lines

common/global.h +2 -2

-    /* version 4 adds 32-bit projid (projid_hi)
+    /* version 4 adds 32-bit projid (projid_hi) only used if projid32 in place.
       * version 3 uses the full 32-bit inode generation number in direnthdr_t.
       * version 2 adds encoding of holes and a change to on-tape inventory format.
       * version 1 adds extended file attribute dumping.
       * version 0 xfsrestore can't handle media produced
       * by version 1 xfsdump.
       */
-#define GLOBAL_HDR_VERSION    GLOBAL_HDR_VERSION_4
+#define GLOBAL_HDR_VERSION    GLOBAL_HDR_VERSION_3
 
  #define GLOBAL_HDR_STRING_SZ    0x100
  #define GLOBAL_HDR_TIME_SZ    4

I used the following for testing purposes.

Index: b/common/global.h
===================================================================

common/global.h +3 -1

@@ -28,7 +28,9 @@
 #define GLOBAL_HDR_VERSION_1    1
 #define GLOBAL_HDR_VERSION_2    2
 #define GLOBAL_HDR_VERSION_3    3
-    /* version 3 uses the full 32-bit inode generation number in direnthdr_t.
+#define GLOBAL_HDR_VERSION_4    4
+    /* version 4 adds 32-bit projid (projid_hi) only used if projid32 in place.
+     * version 3 uses the full 32-bit inode generation number in direnthdr_t.
      * version 2 adds encoding of holes and a change to on-tape inventory format.
      * version 1 adds extended file attribute dumping.
      * version 0 xfsrestore can't handle media produced

Rest of the patch looks good. Correct the common/global.h section and you can add a:

Reviewed-by Rich Johnston <[email protected]>

16 bit project ids than by the feature flag, but I don't see

I recently sent a patch for 32-bit project IDs for xfsdump, to properly restore the top 16 bits, which otherwise get lost. This forced a new dump format version 4 (we were currently at 3).

One thing missing is that we should not restore a dump with 32-bit project IDs onto a filesystem w/o that format; the restore will fail to restore the top 16 bits (but otherwise it returns success; attribute setting failures are not fatal (!?))

Also, 32-bit project ID is a bit uncommon; bumping the format (and making older restore incompatible) is a bit draconian.

3 patches here:

1/3: extend fs info call to get fs flags as well
2/3: default back to V3 and go to V4 only if the projid32 flag is set 3/3: fail restore if the target XFS fs doesn't have projid32 set

I have to say, I'm not super happy with this. I have nagging fear of feature-flag-itis, and I'm not sure how extensible this is as newer versions may appear. But anyway, here's a place to start.

(p.s. anybody have wkendall's new email?) ;)

Quoted 10 lines

If we restore old dumps w/ new xfsdump, nothing special is needed; 0 gets restored for the top 16 bits (vs. garbage, which WOULD be bad).

So bumping the version really only prevents old restore from restoring newer dumps.

If I *didn't* bump the version, then old restore would work, and would simply not restore the top 16 bits - just like an old dump+restore option did.

Had a look at xfsdump, and I agree, there's no need to bump the format version. Nice of someone to leave some zeroed pad bytes next to the project id.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 22 lines · Eric Sandeen

I recently sent a patch for 32-bit project IDs for xfsdump, to properly restore the top 16 bits, which otherwise get lost. This forced a new dump format version 4 (we were currently at 3).

One thing missing is that we should not restore a dump with 32-bit project IDs onto a filesystem w/o that format; the restore will fail to restore the top 16 bits (but otherwise it returns success; attribute setting failures are not fatal (!?))

Also, 32-bit project ID is a bit uncommon; bumping the format (and making older restore incompatible) is a bit draconian.

3 patches here:

1/3: extend fs info call to get fs flags as well
2/3: default back to V3 and go to V4 only if the projid32 flag is set 3/3: fail restore if the target XFS fs doesn't have projid32 set

I have to say, I'm not super happy with this. I have nagging fear of feature-flag-itis, and I'm not sure how extensible this is as newer versions may appear. But anyway, here's a place to start.

Quoted 1 line

(p.s. anybody have wkendall's new email?) ;)

I spoke with Bill, and he actually didn't feel that a new version was needed for the projid32 fix. I'd like to get some discussion here, and reach an agreement. *NOT* bumping the version simplifies a whole lot of things.

Here's what I'd said to Bill:

Quoted 10 lines

Quoted 10 lines

If we restore old dumps w/ new xfsdump, nothing special is needed; 0 gets restored for the top 16 bits (vs. garbage, which WOULD be bad).

So bumping the version really only prevents old restore from restoring newer dumps.

If I *didn't* bump the version, then old restore would work, and would simply not restore the top 16 bits - just like an old dump+restore option did.

And Bill replied:

Quoted 3 lines

Had a look at xfsdump, and I agree, there's no need to bump the format version. Nice of someone to leave some zeroed pad bytes next to the project id.

so what are people's thoughts? Moving to a new version has complexity & compatibility consequences...

Hey Eric,

Quoted 23 lines · Eric Sandeen

Quoted 22 lines · Eric Sandeen

I recently sent a patch for 32-bit project IDs for xfsdump, to properly restore the top 16 bits, which otherwise get lost. This forced a new dump format version 4 (we were currently at 3).

One thing missing is that we should not restore a dump with 32-bit project IDs onto a filesystem w/o that format; the restore will fail to restore the top 16 bits (but otherwise it returns success; attribute setting failures are not fatal (!?))

Also, 32-bit project ID is a bit uncommon; bumping the format (and making older restore incompatible) is a bit draconian.

3 patches here:

1/3: extend fs info call to get fs flags as well
2/3: default back to V3 and go to V4 only if the projid32 flag is set 3/3: fail restore if the target XFS fs doesn't have projid32 set

I have to say, I'm not super happy with this. I have nagging fear of feature-flag-itis, and I'm not sure how extensible this is as newer versions may appear. But anyway, here's a place to start.

Quoted 6 lines

Quoted 1 line

(p.s. anybody have wkendall's new email?) ;)

I spoke with Bill, and he actually didn't feel that a new version was needed for the projid32 fix. I'd like to get some discussion here, and reach an agreement. *NOT* bumping the version simplifies a whole lot of things.

Can you post a list of things that it simplifies? That would help with the discussion.

Quoted 13 lines

Here's what I'd said to Bill:

Quoted 10 lines

Quoted 10 lines

If we restore old dumps w/ new xfsdump, nothing special is needed; 0 gets restored for the top 16 bits (vs. garbage, which WOULD be bad).

So bumping the version really only prevents old restore from restoring newer dumps.

If I *didn't* bump the version, then old restore would work, and would simply not restore the top 16 bits - just like an old dump+restore option did.

Quoted 8 lines

And Bill replied:

Quoted 3 lines

Had a look at xfsdump, and I agree, there's no need to bump the format version. Nice of someone to leave some zeroed pad bytes next to the project id.

so what are people's thoughts? Moving to a new version has complexity & compatibility consequences...

Initially I think that moving to the new version was the right thing to do. It's good for users to have some warning when there are consequences of upgrading to a new release of xfsdump.

A new dump format version will make old xfs_restore fail with an error if a v4 dump is encountered, rather than do the restore and chop off the top 16 bits of the project id silently. I think that for a user it is reasonable to expect to be notified if project ids are going to be lost. Looks like there is no way for an older xfsrestore to force an override, so if there is to be some level of compatability between versions, newer xfsdump would have to still be able to create v3 dumps.

What other complexity/compat issues are there?

Regards,
Ben

Quoted 68 lines · Ben Myers

Quoted 23 lines · Eric Sandeen

Quoted 22 lines · Eric Sandeen

I recently sent a patch for 32-bit project IDs for xfsdump, to properly restore the top 16 bits, which otherwise get lost. This forced a new dump format version 4 (we were currently at 3).

One thing missing is that we should not restore a dump with 32-bit project IDs onto a filesystem w/o that format; the restore will fail to restore the top 16 bits (but otherwise it returns success; attribute setting failures are not fatal (!?))

Also, 32-bit project ID is a bit uncommon; bumping the format (and making older restore incompatible) is a bit draconian.

3 patches here:

1/3: extend fs info call to get fs flags as well
2/3: default back to V3 and go to V4 only if the projid32 flag is set 3/3: fail restore if the target XFS fs doesn't have projid32 set

I have to say, I'm not super happy with this. I have nagging fear of feature-flag-itis, and I'm not sure how extensible this is as newer versions may appear. But anyway, here's a place to start.

Quoted 6 lines

Quoted 1 line

(p.s. anybody have wkendall's new email?) ;)

I spoke with Bill, and he actually didn't feel that a new version was needed for the projid32 fix. I'd like to get some discussion here, and reach an agreement. *NOT* bumping the version simplifies a whole lot of things.

Can you post a list of things that it simplifies? That would help with the discussion.

Quoted 13 lines

Here's what I'd said to Bill:

Quoted 10 lines

Quoted 10 lines

If we restore old dumps w/ new xfsdump, nothing special is needed; 0 gets restored for the top 16 bits (vs. garbage, which WOULD be bad).

So bumping the version really only prevents old restore from restoring newer dumps.

If I *didn't* bump the version, then old restore would work, and would simply not restore the top 16 bits - just like an old dump+restore option did.

Quoted 8 lines

And Bill replied:

Quoted 3 lines

Had a look at xfsdump, and I agree, there's no need to bump the format version. Nice of someone to leave some zeroed pad bytes next to the project id.

so what are people's thoughts? Moving to a new version has complexity & compatibility consequences...

Initially I think that moving to the new version was the right thing to do. It's good for users to have some warning when there are consequences of upgrading to a new release of xfsdump.

A new dump format version will make old xfs_restore fail with an error if a v4 dump is encountered, rather than do the restore and chop off the top 16 bits of the project id silently. I think that for a user it is reasonable to expect to be notified if project ids are going to be lost. Looks like there is no way for an older xfsrestore to force an override, so if there is to be some level of compatability between versions, newer xfsdump would have to still be able to create v3 dumps.

Quoted 1 line

What other complexity/compat issues are there?

Well, I started putting in some feature flag detection. So in a super-perfect world we could:

* Dump to to V4 only if the PROJID32 flag is set on the fs being dumped, else V3

* Refuse / warn if restoring a < V4 dump onto a fs w/ the PROJID32 set

* Refuse / warn if restoring a >= V4 dump onto a non-PROJID32 fs

* Extend the -K option to allow specifying/forcing arbitrary dump levels

But:

* No released kernel yet reports the feature flag, so we don't have guarantees about feature presence
- So going to V4 only if flag found will silently fail prior to kernel 3.7, since no feature flag is available there.
- so then we should default to V4 regardless of feature flag test ~ so then we can't make restore of V4 require the feature flag on the fs
* Older dump/restore already has the silent failure problem
* By the time you are restoring a <V4 dump onto a PROJID32 fs, it's unlikely that you have further options anyway (you're possibly in recovery, can't go back & make V4 dumps to restore from)

* Warning on >= V4 dump to non-PROJID32 fs only makes sense if V4 is contingent on feature set/recognized at dump time but we cant' reliably detect that.
* projid32bit is a non-default option today, so it's an infrequent case...

Hey Eric,

Quoted 71 lines · Eric Sandeen

Quoted 69 lines · Ben Myers

Quoted 23 lines · Eric Sandeen

Quoted 22 lines · Eric Sandeen

I recently sent a patch for 32-bit project IDs for xfsdump, to properly restore the top 16 bits, which otherwise get lost. This forced a new dump format version 4 (we were currently at 3).

One thing missing is that we should not restore a dump with 32-bit project IDs onto a filesystem w/o that format; the restore will fail to restore the top 16 bits (but otherwise it returns success; attribute setting failures are not fatal (!?))

Also, 32-bit project ID is a bit uncommon; bumping the format (and making older restore incompatible) is a bit draconian.

3 patches here:

1/3: extend fs info call to get fs flags as well
2/3: default back to V3 and go to V4 only if the projid32 flag is set 3/3: fail restore if the target XFS fs doesn't have projid32 set

I have to say, I'm not super happy with this. I have nagging fear of feature-flag-itis, and I'm not sure how extensible this is as newer versions may appear. But anyway, here's a place to start.

Quoted 6 lines

Quoted 1 line

(p.s. anybody have wkendall's new email?) ;)

I spoke with Bill, and he actually didn't feel that a new version was needed for the projid32 fix. I'd like to get some discussion here, and reach an agreement. *NOT* bumping the version simplifies a whole lot of things.

Can you post a list of things that it simplifies? That would help with the discussion.

Quoted 13 lines

Here's what I'd said to Bill:

Quoted 10 lines

Quoted 10 lines

If we restore old dumps w/ new xfsdump, nothing special is needed; 0 gets restored for the top 16 bits (vs. garbage, which WOULD be bad).

So bumping the version really only prevents old restore from restoring newer dumps.

If I *didn't* bump the version, then old restore would work, and would simply not restore the top 16 bits - just like an old dump+restore option did.

Quoted 8 lines

And Bill replied:

Quoted 3 lines

Had a look at xfsdump, and I agree, there's no need to bump the format version. Nice of someone to leave some zeroed pad bytes next to the project id.

so what are people's thoughts? Moving to a new version has complexity & compatibility consequences...

Initially I think that moving to the new version was the right thing to do. It's good for users to have some warning when there are consequences of upgrading to a new release of xfsdump.

A new dump format version will make old xfs_restore fail with an error if a v4 dump is encountered, rather than do the restore and chop off the top 16 bits of the project id silently. I think that for a user it is reasonable to expect to be notified if project ids are going to be lost. Looks like there is no way for an older xfsrestore to force an override, so if there is to be some level of compatability between versions, newer xfsdump would have to still be able to create v3 dumps.

What other complexity/compat issues are there?

Quoted 4 lines

Well, I started putting in some feature flag detection. So in a super-perfect world we could:

* Dump to to V4 only if the PROJID32 flag is set on the fs being dumped, else V3

It might be better to just do v4 always (except if -K is specified). I guess I don't understand what the additional level of complexity buys us.

Quoted 1 line

* Refuse / warn if restoring a < V4 dump onto a fs w/ the PROJID32 set

Lets see if I understand...

In this case we're warning the user because we are concerned that they may have been running with projid32 and dumped the filesystem with an older xfsdump which didn't capture the top 16 bits of the projid. The restore of the filesystem would result in the loss of those top 16 bits.

Quoted 1 line

* Refuse / warn if restoring a >= V4 dump onto a non-PROJID32 fs

In this case we're warning the user because... the filesystem would drop the top 16 bits of the valid project ids in the dump?

Quoted 1 line

* Extend the -K option to allow specifying/forcing arbitrary dump levels

This would allow the newer xfsdump to create a v3 dump level.

Quoted 12 lines

But:

* No released kernel yet reports the feature flag, so we don't have guarantees about feature presence
- So going to V4 only if flag found will silently fail prior to kernel 3.7, since no feature flag is available there.
- so then we should default to V4 regardless of feature flag test ~ so then we can't make restore of V4 require the feature flag on the fs
* Older dump/restore already has the silent failure problem
* By the time you are restoring a <V4 dump onto a PROJID32 fs, it's unlikely that you have further options anyway (you're possibly in recovery, can't go back & make V4 dumps to restore from)

xfsdump/restore aren't just for those recovery situations: some people might dump on one machine and pipe it to another host during an upgrade.

Quoted 3 lines

* Warning on >= V4 dump to non-PROJID32 fs only makes sense if V4 is contingent on feature set/recognized at dump time but we cant' reliably detect that.
* projid32bit is a non-default option today, so it's an infrequent case...

It sounds like the super-perfect world needs to have kernel support for reporting whether a filesystem has the projid32 feature flag set. Are there other options for getting that data? xfs_db?

In this somewhat-less-than-perfect world, I think it might be ok to extend the -K option to have some backward compatability in the near term, and then work toward improved warnings once your fsgeom series hits upstream. Maybe I'm missing something.

Regards,
Ben

I recently sent a patch for 32-bit project IDs for xfsdump, to properly restore the top 16 bits, which otherwise get lost. This forced a new dump format version 4 (we were currently at 3).

One thing missing is that we should not restore a dump with 32-bit project IDs onto a filesystem w/o that format; the restore will fail to restore the top 16 bits (but otherwise it returns success; attribute setting failures are not fatal (!?))

Also, 32-bit project ID is a bit uncommon; bumping the format (and making older restore incompatible) is a bit draconian.

3 patches here:

1/3: extend fs info call to get fs flags as well
2/3: default back to V3 and go to V4 only if the projid32 flag is set 3/3: fail restore if the target XFS fs doesn't have projid32 set

I have to say, I'm not super happy with this. I have nagging fear of feature-flag-itis, and I'm not sure how extensible this is as newer versions may appear. But anyway, here's a place to start.

(p.s. anybody have wkendall's new email?) ;)

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs