Skip to contentSkip to conversation

Deprecating V4 on-disk format

XFS SUMMIT Dave Chinner 6 participants 10 messages May 13, 2020 – May 26, 2020

lore
Share & tools
lore
#0 lore raw

Topic: Deprecating V4 On-disk Format

Scope:
Long term life cycle planning Supporting old filesystems with new kernels. Unfixable integrity issues in v4 format. Reducing feature matrix for testing

Proposal:

The CRC-enabled V5 format has been the upstream default format now since commit 566ebd5ae5fa ("mkfs: default to CRC enabled filesystems") dated May 11 2015 (5 years ago!) and released in xfsprogs v3.2.3. It is the default in all major distros, and has been for some time.

We know that the v4 format has unfixable integrity issues apart from the obvious lack of CRCs and self-describing metadata structures; it has race conditions in log recovery realted to inode creation and other such issues that could only be solved with an on-disk format change of some kind. We are not adding new features to v4 formats, so anyone wanting to use new XFS features must use v5 format filesystems.

We also know that the number of v4 filesysetms in production is slowly decreasing as systems are replaced as part of the natural life cycle of production systems.

All this adds up to the realisation that existing v4 filesystems are effectively in the "Maintenance Mode" era of the software life cycle. The next stage in the life cycle is "Phasing Out" before we drop support for it altogether, also know around here as "deprecated" which is a sign that support will "soon" cease.

I'd like to move the v4 format to the "deprecated" state as a signal to users that it should really not be considered viable for new systems. New systems running modern kernels and userspace should all be using the v5 format, so this mostly only affects existing filesystems.

Note: I am not proposing that we drop support for the v4 format any time soon. What I am proposing is an "end of lifecycle" tag similar to the way we use EXPERIMENTAL to indicate that the functionality is available but we don't recommend it for production systems yet.

Hence what I am proposing is that we introduce a DEPRECATED alert at mount time to inform users that the functionality exists, but it will not be maintained indefinitely into the future. For distros with a ten year support life, this means that a near-future release will pick up the DEPRECATED tag but still support the filesystem for the support life of that release. A "future +1" release may not support the v4 format at all.

Discussion points:

- How practical is this?
- What should we have mkfs do when directed to create a v4 format filesystem?
- How long before we decide to remove v4 support from the upstream kernel and tools? 5 years after deprecation? 10 years?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#1 lore raw
Quoted 46 lines · Dave Chinner

Topic: Deprecating V4 On-disk Format

Scope:
Long term life cycle planning Supporting old filesystems with new kernels. Unfixable integrity issues in v4 format. Reducing feature matrix for testing

Proposal:

The CRC-enabled V5 format has been the upstream default format now since commit 566ebd5ae5fa ("mkfs: default to CRC enabled filesystems") dated May 11 2015 (5 years ago!) and released in xfsprogs v3.2.3. It is the default in all major distros, and has been for some time.

We know that the v4 format has unfixable integrity issues apart from the obvious lack of CRCs and self-describing metadata structures; it has race conditions in log recovery realted to inode creation and other such issues that could only be solved with an on-disk format change of some kind. We are not adding new features to v4 formats, so anyone wanting to use new XFS features must use v5 format filesystems.

We also know that the number of v4 filesysetms in production is slowly decreasing as systems are replaced as part of the natural life cycle of production systems.

All this adds up to the realisation that existing v4 filesystems are effectively in the "Maintenance Mode" era of the software life cycle. The next stage in the life cycle is "Phasing Out" before we drop support for it altogether, also know around here as "deprecated" which is a sign that support will "soon" cease.

I'd like to move the v4 format to the "deprecated" state as a signal to users that it should really not be considered viable for new systems. New systems running modern kernels and userspace should all be using the v5 format, so this mostly only affects existing filesystems.

Note: I am not proposing that we drop support for the v4 format any time soon. What I am proposing is an "end of lifecycle" tag similar to the way we use EXPERIMENTAL to indicate that the functionality is available but we don't recommend it for production systems yet.

Quoted 7 lines

Hence what I am proposing is that we introduce a DEPRECATED alert at mount time to inform users that the functionality exists, but it will not be maintained indefinitely into the future. For distros with a ten year support life, this means that a near-future release will pick up the DEPRECATED tag but still support the filesystem for the support life of that release. A "future +1" release may not support the v4 format at all.

/me regrets that he is frequently failing to clear enough space out of his schedule to respond to all of these adequately. But here goes, random thoughts at 23:23. :/

Quoted 3 lines

Discussion points:

- How practical is this?

Well, we've killed off old features before... v1 inodes, v2 directories, etc. So clearly this can be done, given enough preparation time.

And we probably ought to do it, before we start to resemble the ext4 quota nightmare.

Quoted 2 lines

- What should we have mkfs do when directed to create a v4 format filesystem?

It probably ought to print a warning...

That said, way back when we were arguing with the syzbot people, one of us suggested that we hide V4 behind a CONFIG_XFS_DEPRECATED=y option, so that people who want to harden their kernel against the unfixable structural problems in the V4 format could effectively lose V4 support early. Maybe we should add that for a few years?

Quoted 2 lines

- How long before we decide to remove v4 support from the upstream kernel and tools? 5 years after deprecation? 10 years?

That probably depends a lot on how much our respective employers want to keep those old XFSes going. Some of our customers are about ready to certify that they can support their distro defaults changing from ext3 to XFS v4, but those folks have support contracts that won't terminate until whenever the so^Hun goes out.

--D

Quoted 7 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 61 lines · Darrick J. Wong
Quoted 46 lines · Dave Chinner

Topic: Deprecating V4 On-disk Format

Scope:
Long term life cycle planning Supporting old filesystems with new kernels. Unfixable integrity issues in v4 format. Reducing feature matrix for testing

Proposal:

The CRC-enabled V5 format has been the upstream default format now since commit 566ebd5ae5fa ("mkfs: default to CRC enabled filesystems") dated May 11 2015 (5 years ago!) and released in xfsprogs v3.2.3. It is the default in all major distros, and has been for some time.

We know that the v4 format has unfixable integrity issues apart from the obvious lack of CRCs and self-describing metadata structures; it has race conditions in log recovery realted to inode creation and other such issues that could only be solved with an on-disk format change of some kind. We are not adding new features to v4 formats, so anyone wanting to use new XFS features must use v5 format filesystems.

We also know that the number of v4 filesysetms in production is slowly decreasing as systems are replaced as part of the natural life cycle of production systems.

All this adds up to the realisation that existing v4 filesystems are effectively in the "Maintenance Mode" era of the software life cycle. The next stage in the life cycle is "Phasing Out" before we drop support for it altogether, also know around here as "deprecated" which is a sign that support will "soon" cease.

I'd like to move the v4 format to the "deprecated" state as a signal to users that it should really not be considered viable for new systems. New systems running modern kernels and userspace should all be using the v5 format, so this mostly only affects existing filesystems.

Note: I am not proposing that we drop support for the v4 format any time soon. What I am proposing is an "end of lifecycle" tag similar to the way we use EXPERIMENTAL to indicate that the functionality is available but we don't recommend it for production systems yet.

Quoted 7 lines

Hence what I am proposing is that we introduce a DEPRECATED alert at mount time to inform users that the functionality exists, but it will not be maintained indefinitely into the future. For distros with a ten year support life, this means that a near-future release will pick up the DEPRECATED tag but still support the filesystem for the support life of that release. A "future +1" release may not support the v4 format at all.

/me regrets that he is frequently failing to clear enough space out of his schedule to respond to all of these adequately. But here goes, random thoughts at 23:23. :/

Quoted 2 lines

Discussion points:

Quoted 7 lines
Quoted 1 line

- How practical is this?

Well, we've killed off old features before... v1 inodes, v2 directories, etc. So clearly this can be done, given enough preparation time.

And we probably ought to do it, before we start to resemble the ext4 quota nightmare.

*nod*

Quoted 11 lines
Quoted 3 lines

- What should we have mkfs do when directed to create a v4 format filesystem?

It probably ought to print a warning...

That said, way back when we were arguing with the syzbot people, one of us suggested that we hide V4 behind a CONFIG_XFS_DEPRECATED=y option, so that people who want to harden their kernel against the unfixable structural problems in the V4 format could effectively lose V4 support early. Maybe we should add that for a few years?

That sounds like a good idea - being able to config it out provides a mechanism to isolate v4 only code paths over time....

Quoted 8 lines
Quoted 2 lines

- How long before we decide to remove v4 support from the upstream kernel and tools? 5 years after deprecation? 10 years?

That probably depends a lot on how much our respective employers want to keep those old XFSes going. Some of our customers are about ready to certify that they can support their distro defaults changing from ext3 to XFS v4, but those folks have support contracts that won't terminate until whenever the so^Hun goes out.

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

However, with the way typical enterprise feature policies work, a feature needs to be deprecated for a major release before it can be removed.

So, realistically, the deprecation decision is not for the near term
- it's for dropping support in the "current + 2" major release that nobody is even thinking about yet. i.e. if we don't deprecate it in the next couple of years, then we'll still need to be maintaining the v4 filesystem format in upstream for the next 15 years. Which, IMO, is about 10 years too long...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but defaulted to v4. That means that unless you were extremely cautious (like I am :) 99% of RH/COs v7 will be running v4 volumes for the coming years. How many years, would you ask?

As for the lifecycle of a filesystem, I just ended support on a 40 TB archival server I set up back in 2007. I still have a number of supported systems from the years 2008-2010, and about a hundred from 2010-2013. That's how reliable XFS is, unfortunately :)

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
#4 lore raw
Quoted 10 lines · Emmanuel Florac

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

Quoted 4 lines

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but defaulted to v4. That means that unless you were extremely cautious (like I am :) 99% of RH/COs v7 will be running v4 volumes for the coming years. How many years, would you ask?

[Trying again hopefully without HTML]

So initial RHEL/CentOS 7 releases did create XFS v4 file systems. However from RHEL/CentOS 7.3 [1] (circa Nov 2016) they are creating XFS v5 file systems by default.

[1] RHEL 7.3 Release Notes > Chapter 9. File Systems
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.3_release_notes/new_features_file_systems

# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
# mkfs.xfs -V
mkfs.xfs version 4.5.0
# mkfs.xfs /dev/sdb13
...
# xfs_db -c version -r /dev/sdb13
versionnum [0xb4a5+0x18a] =
V5,NLINK,DIRV2,ALIGN,LOGV2,EXTFLG,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE
#5 lore raw
Quoted 16 lines · Mike Fleetwood
Quoted 10 lines · Emmanuel Florac

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

Quoted 4 lines

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but defaulted to v4. That means that unless you were extremely cautious (like I am :) 99% of RH/COs v7 will be running v4 volumes for the coming years. How many years, would you ask?

Quoted 5 lines

[Trying again hopefully without HTML]

So initial RHEL/CentOS 7 releases did create XFS v4 file systems. However from RHEL/CentOS 7.3 [1] (circa Nov 2016) they are creating XFS v5 file systems by default.

That's correct. Still a little shocked that we switched midstream but it worked out fine. :)

But as Dave said downthread, distro defaults and support is unrelated to upstream; distros already signed up to support their shipped features for the lifetime of the OS (or whatever their support policy says)

In fact, deprecating old stuff upstream will help future distro releases manage their support by getting rid of antiquated features and behaviors...

-Eric

Quoted 15 lines

[1] RHEL 7.3 Release Notes > Chapter 9. File Systems
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.3_release_notes/new_features_file_systems

# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)
# mkfs.xfs -V
mkfs.xfs version 4.5.0
# mkfs.xfs /dev/sdb13
...
# xfs_db -c version -r /dev/sdb13
versionnum [0xb4a5+0x18a] =
V5,NLINK,DIRV2,ALIGN,LOGV2,EXTFLG,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE
#6 lore raw
In reply to Emmanuel Florac
Quoted 9 lines · Emmanuel Florac

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

Quoted 4 lines

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but defaulted to v4. That means that unless you were extremely cautious (like I am :) 99% of RH/COs v7 will be running v4 volumes for the coming years. How many years, would you ask?

Largely irrelevant to the question at hand, as support is dependent on the distro lifecycle here. Essentially whatever is in RHEL7 is supported by RH until the end of it's life.

In RHEL8, we default to v5 filesystems, but fully support v4. That will be the case for the rest of it's life. Unless the user specifically asks for it, no new v4 filesystems are being created on current RHEL releases.

If we were to deprecate v4 now, then it will be marked as deprecated in the next major RHEL release. That means it's still fully supported in that release for it's entire life, but it will be removed in the next major release after that. So we are still talking about at least 15+ years of enterprise distro support for the format, even if upstream drops it sooner...

Quoted 4 lines

As for the lifecycle of a filesystem, I just ended support on a 40 TB archival server I set up back in 2007. I still have a number of supported systems from the years 2008-2010, and about a hundred from 2010-2013. That's how reliable XFS is, unfortunately :)

Yup, 10-15 years is pretty much the expected max life of storage systems before the hardware really needs to be retired. We made v5 the default 5 years ago, so give it another 10 years (the sort of timeframe we are talking about here) and just about everything will be running v5 and that's when v4 can likely be dropped.

The other thing to consider is that we need to drop v4 before we get to y2038 support issues as the format will never support dates beyond that. Essentially, we need to have the deprecation discussion and take action in the near future so that people have stopped using it before y2038 comes along and v4 filesystems break everything.

Not enough people think long term when it comes to computers - it should be more obvious now why I brought this up for discussion...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw
Quoted 24 lines · Dave Chinner
Quoted 9 lines · Emmanuel Florac

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

Quoted 4 lines

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but defaulted to v4. That means that unless you were extremely cautious (like I am :) 99% of RH/COs v7 will be running v4 volumes for the coming years. How many years, would you ask?

Largely irrelevant to the question at hand, as support is dependent on the distro lifecycle here. Essentially whatever is in RHEL7 is supported by RH until the end of it's life.

In RHEL8, we default to v5 filesystems, but fully support v4. That will be the case for the rest of it's life. Unless the user specifically asks for it, no new v4 filesystems are being created on current RHEL releases.

Quoted 6 lines

If we were to deprecate v4 now, then it will be marked as deprecated in the next major RHEL release. That means it's still fully supported in that release for it's entire life, but it will be removed in the next major release after that. So we are still talking about at least 15+ years of enterprise distro support for the format, even if upstream drops it sooner...

We probably ought to do it sooner than later though, particularly if we think/care about 5.9 turning into an LTS.

Quoted 12 lines
Quoted 4 lines

As for the lifecycle of a filesystem, I just ended support on a 40 TB archival server I set up back in 2007. I still have a number of supported systems from the years 2008-2010, and about a hundred from 2010-2013. That's how reliable XFS is, unfortunately :)

Yup, 10-15 years is pretty much the expected max life of storage systems before the hardware really needs to be retired. We made v5 the default 5 years ago, so give it another 10 years (the sort of timeframe we are talking about here) and just about everything will be running v5 and that's when v4 can likely be dropped.

Quoted 8 lines

The other thing to consider is that we need to drop v4 before we get to y2038 support issues as the format will never support dates beyond that. Essentially, we need to have the deprecation discussion and take action in the near future so that people have stopped using it before y2038 comes along and v4 filesystems break everything.

Not enough people think long term when it comes to computers - it should be more obvious now why I brought this up for discussion...

Ok then, who would like to help me get the y2038 timestamp patches reviewed for ~5.9? :D

(Anyone; not necessarily Dave)

--D

Quoted 6 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#8 lore raw
Quoted 54 lines · Darrick J. Wong
Quoted 24 lines · Dave Chinner
Quoted 9 lines · Emmanuel Florac

Le Wed, 20 May 2020 11:14:30 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's a difference between what a distro that heavily patches the upstream kernel is willing to support and what upstream supports. And, realistically, v4 is going to be around for at least one more major distro release, which means the distro support time window is still going to be in the order of 15 years.

Quoted 4 lines

IIRC, RedHat/CentOS v.7.x shipped with a v5-capable mkfs.xfs, but defaulted to v4. That means that unless you were extremely cautious (like I am :) 99% of RH/COs v7 will be running v4 volumes for the coming years. How many years, would you ask?

Largely irrelevant to the question at hand, as support is dependent on the distro lifecycle here. Essentially whatever is in RHEL7 is supported by RH until the end of it's life.

In RHEL8, we default to v5 filesystems, but fully support v4. That will be the case for the rest of it's life. Unless the user specifically asks for it, no new v4 filesystems are being created on current RHEL releases.

Quoted 6 lines

If we were to deprecate v4 now, then it will be marked as deprecated in the next major RHEL release. That means it's still fully supported in that release for it's entire life, but it will be removed in the next major release after that. So we are still talking about at least 15+ years of enterprise distro support for the format, even if upstream drops it sooner...

We probably ought to do it sooner than later though, particularly if we think/care about 5.9 turning into an LTS.

Quoted 18 lines
Quoted 4 lines

As for the lifecycle of a filesystem, I just ended support on a 40 TB archival server I set up back in 2007. I still have a number of supported systems from the years 2008-2010, and about a hundred from 2010-2013. That's how reliable XFS is, unfortunately :)

Yup, 10-15 years is pretty much the expected max life of storage systems before the hardware really needs to be retired. We made v5 the default 5 years ago, so give it another 10 years (the sort of timeframe we are talking about here) and just about everything will be running v5 and that's when v4 can likely be dropped.

The other thing to consider is that we need to drop v4 before we get to y2038 support issues as the format will never support dates beyond that. Essentially, we need to have the deprecation discussion and take action in the near future so that people have stopped using it before y2038 comes along and v4 filesystems break everything.

Quoted 6 lines
Quoted 2 lines

Not enough people think long term when it comes to computers - it should be more obvious now why I brought this up for discussion...

Ok then, who would like to help me get the y2038 timestamp patches reviewed for ~5.9? :D

I can help with review. Already looked at your branch.

Thanks,
Amir.

#9 lore raw
In reply to Dave Chinner

Le Mon, 25 May 2020 13:23:54 +1000
Dave Chinner <[email protected]> écrivait:

Quoted 2 lines

Not enough people think long term when it comes to computers - it should be more obvious now why I brought this up for discussion...

I'm thinking long term: I also run an Indy and an Octane and don't plan to retire them any time soon :)

Signature · 6 lines
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<[email protected]>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------