Skip to contentSkip to conversation

(Resend) 2038 Kernel Summit Discussion Fodder

DISCUSSION John Stultz 8 participants 11 messages Aug 13, 2014 – Sep 8, 2014

lore
Share & tools
lore
#0 lore raw

(Forgive the resend, somehow despite being in plain-text mode, the gmail compose interface somehow added some html-ism that caused lkml to bounce this)

Next week at Kernel Summit, I'm hoping to have a discussion on 2038 during one of the unconference slots, so I wanted to send out some background information about the various proposed approaches (as well as send this out for wider input, since apparently there won't be much libc representation at KS).

So 2038 brings the end of time for 32bit architectures. It being some twenty four years ahead, it may seem like there is plenty of time for folks to migrate to 64bit architectures that are (mostly) unaffected by this issue. However, 32bit processors are still being produced today in extremely high volumes, and many of those systems are being used in commercial, industrial and medical environments, where these systems may be quite literally embedded into the walls and machinery and are expected to run for 25 years or more. As these small systems become more and more pervasive, the risks of major trouble in 2038 grow. And that’s to say nothing of the impact on future classic-car resale prices for fancy cars like the Tesla when the high end in-dash display won’t work (gasp!).

Thus, the “just upgrade to 64bit” solution isn’t really sufficient, and we need to find a transition plan soon, which will allow 32bit cpus being sold today and in the future to function correctly past 2038.

Other OSes have already started rolling out solutions. NetBSD switched to 64bit time_t in 2012, providing a compatibility layer for old applications. Then in 2013, OpenBSD switched its 32bit systems time_t to use long long. You can read more about the OpenBSD transition here:
http://www.openbsd.org/papers/eurobsdcon_2013_time_t/. One item to note is that NetBSD preserved compatibility with 2038 unsafe applications, while OpenBSD did not (seeing the certainty of excluding unsafe applications as a feature over compatibility). For Linux, we obviously want to maintain compatibility, but I think its important we recognise we’re not the first movers here, and many of the discussions we’ll be having have already been had in those communities. I also think we should be sure to credit the NetBSD and OpenBSD devs for jumping in on this early and working to ensure issues in the userspace applications we share are already addressed.

Also, just to clarify, as time related discussions can bring out a laundry list of issues, I would like to focus this discussion on providing a 2038 solution for existing interfaces and applications in a way that ideally doesn't require modifying application source code. While there will be plenty of places where applications have cast or stored time_t values explicitly as longs, and for those applications, deep modifications will be necessary. But I’d like to avoid getting into new-interface discussions, like exporting ktime_t like nanosecond interfaces instead of timepsecs, unifying time-stamping formats, or methods for avoiding leapseconds. Those are all interesting issues, and I’d be up for discussing them separately, but those issue apply equally to 32bit and 64bit systems, and really aren't 2038 specific, so I think its best to separate them out.

I think any solution to the 2038 issue for Linux, from the kernel’s perspective, will have a number of phases:

1) In-kernel correctness: Making sure the kernel itself handles the 2038 rollover correctly. As noted in LWN (https://lwn.net/Articles/607741/) this work has begun and portions have been merged. Personally, I don’t really see this as that interesting of a discussion topic, and with the exception of dealing with external representations of time like on-disk filesystem formats, etc, it is for the most part a matter of just going through, finding and fixing things (with a few extra complications around ioctls). Mostly this will revolve around adding explicitly sized 64bit time_t types (and 32bit types for compatibility), and migrating in-kernel users over to the explicitly sized implementations, allowing us to validate conversion by eventually removing the in-kernel time_t type, of course finding non-time_t custom storage uses will be a long tail of work.

2) Userspace ABI modifications: This includes how we expose the new 64bit time_t and related structures to userland via syscalls and ioctls, and how we preserve compatibility to older applications. This is probably the most complex issue, as the different choices have large impacts to how the transition in userspace is done.

3) Aiding in validating userspace correctness: While we want to preserve compatibility, there is the very real aspect that 2038 unsafe applications (ie: almost all 32bit applications today) are terminally broken in 2038. While we can try to ensure that we don’t break those applications prematurely, we do want to ensure folks using unsafe applications are aware and motivated to upgrade to 2038-safe versions quickly. This might include warnings when unsafe syscalls are used, or options to disable unsafe syscalls, as well as maybe debug modes for testing where if you set the date to past 2038, the kernel will be extra verbose if any truncated absolute time_t values are observed (hinting that an application cast to a long and back).

So what I’d like to cover in this mail, are some discussion starters around ABI modifications that I think we should discuss at Kernel Summit, in order to make sure we have a clear path forward, as what we decide for the ABI modifications, has large impacts on how we help ensure userspace is correct or not.

Quoted 1 line

From discussions so far, it seems the preferred change to the userspace

interface is what I’ll call the “Large File” method, as it follows the approach used for large file support:

Create new 64bit time_t/timespec/timeval/etc variants for syscalls, while preserving existing interfaces. This has some complexity around IOCTLs, but that can mostly be handled by creating new ioctl numbers while preserving the old ones. Since we’re only modifying time types, we’ll also need to add compat versions for many of these syscalls for 64bit native systems.

Libc then introduces versioned symbols, and a new compile options to allow applications to be built for “large time”. New and old applications could then share the same libc.

The benefits of this approach is is simply and minimally extends the current 32 bit environment, without any effect on existing applications which continue to work. Most of the complexity is in the libc library and its build environment.

The downsides to this approach is that as it follows the large-file approach, it has many of the same problems as large-file support, in that the transition to large-file has been slow and is still ongoing. Also, since this solution focuses on libc, there is also the problem of existing 3rd party libraries, which have no way of knowing which sized time is being used, will break. So all libraries that do anything with time will then have to implement their own versioned interfaces. This approach also makes it a little more difficult to audit that a system is 2038 safe, without running it and looking for issues.

A potential alternative I’d like to also propose is the “Libc Version Bump” approach.

Basically this is the same as the above, where the kernel provides both legacy and new time_t related interfaces. However, the libc would make a version break, migrating to using 64bit time_t types and syscalls. Legacy applications would still work using the old glibc version, but this would provide a stronger line in the sand between 2038 safe and unsafe applications and libraries, making it easier to avoid mixing the two. NetBSD developers discussed this same approach back in 2008 here:
https://mail-index.netbsd.org/tech-userlevel/2008/03/22/msg000231.html

The downsides here is, for legacy application support, one would have to have all the requisite legacy libraries also installed, which will add a burden to distro vendors. However, this extra storage overhead would likely be a positive motivator to get applications rebuilt and migrated to new version. Additionally, for 3rd party libraries built against the new libc version, the libraries may need to do a version bump themselves, in order to be able to co-exist with versions built against the previous libc. This approach also assumes that libraries that use time_t related values would have a libc dependency.

A more aggressive version of the previous proposal is what I’m calling the “New Virtual-Architecture” approach, basically extending the versioning control from the linker down into the kernel as well. It would be adding a new “virtual-architecture” to the kernel, not entirely unlike how x32 is supported on x86_64 systems. We would create entirely new ABI and architecture name in the kernel (think something like “armllt” or “i386llt”). We would preserve compatibility for legacy applications via personalities, similar mechanism as the compat_ interface used to support 32bit applications on 64bit kernels. In this case, we wouldn’t introduce new 64 bit syscalls in the kernel, as the existing interfaces would just be typed correctly for our new virtual architecture, but we would have duplicate syscall interfaces via the compat interfaces. The extra complexity would also be that we would have to support new 32bit compat environment on 64bit systems. Userspace would be completely rebuilt to support the new -llt architecture, and compatibility for legacy applications would be done via the same multiarch packaging as is done now for running 32bit applications on 64bit systems.

The pros for this case is that it would be very easy to audit that applications have migrated to the new 64bit time_t ABI. Additionally since we know which type the application is in the kernel, it would make problematic compatability areas like IOCTLS easier to deal with utilizing a flag in the task structure.

The downsides here are many. The distros will probably hate this idea, as it requires rebuilding the world, and maintaining another legacy architecture support. I’m also not completely sure how robust multi-arch packaging is in the face of having to handle 3-4 architectures on one system. On the kernel side, it also adds more complexity, where we have to add even more complex compat support for 64bit systems to handle all the various 32bit applications possible.

That said the practical reality isn't much further from the the “Libc Version Bump” approach, since legacy support will need legacy versions of all dependent libraries there a well. Similarly the additional storage required to support legacy applications is a positive motivator to get folks to move away from unsafe legacy applications. I also personally like the clarity the new virtual architecture brings, and that it would allow a kernel option to disable 2038-unsafe legacy support.

With any of these approaches, we still have quite a bit of work to just get the kernel in-shape internally. And with the exception of the “virtual arch” approach, the changes on the kernel side are basically the same. The big thing we probably want to avoid is requiring any sort of flagday for distros, and instead allowing them each to migrate to the new solution individually (but hopefully not taking too long). Even so, I think having a clear vision for how userspace will make this transition is important, so hopefully during the Kernel Summit discussion we can come to consensus on what approach to take moving forward.

Anyway, this is probably more then enough to read and think about in the next week. I’d be very interested in further thoughts or alternative proposals to discuss.

Thanks to Thomas Gleixner, Arnd Bergmann, Mark Brown, Joseph Myers and others for their thoughts and proposals which I used to create this summary.

Also, Arnd has been keeping 2038 related details (most usefully on the non-time_t 2038 concerns) on the kenrelnewbies wiki here:
http://kernelnewbies.org/y2038

thanks
-john

#1 lore raw
Quoted 4 lines · John Stultz

Also, just to clarify, as time related discussions can bring out a laundry list of issues, I would like to focus this discussion on providing a 2038 solution for existing interfaces and applications in a way that ideally doesn't require modifying application source code. While there will be

Quoted 8 lines

plenty of places where applications have cast or stored time_t values explicitly as longs, and for those applications, deep modifications will be necessary. But I’d like to avoid getting into new-interface discussions, like exporting ktime_t like nanosecond interfaces instead of timepsecs, unifying time-stamping formats, or methods for avoiding leapseconds. Those are all interesting issues, and I’d be up for discussing them separately, but those issue apply equally to 32bit and 64bit systems, and really aren't 2038 specific, so I think its best to separate them out.

That's understandable. However, I wonder to what extent we could support unmodified source code via libc wrappers (since code calling syscalls directly can't work completely unmodified), while using better interfaces for new syscalls. Given syscalls written in terms of (for instance) nanoseconds rather than timespec values, it seems straightforward enough for libc to provide compatibility interfaces.

Quoted 19 lines

From discussions so far, it seems the preferred change to the userspace interface is what I’ll call the “Large File” method, as it follows the approach used for large file support:

Create new 64bit time_t/timespec/timeval/etc variants for syscalls, while preserving existing interfaces. This has some complexity around IOCTLs, but that can mostly be handled by creating new ioctl numbers while preserving the old ones. Since we’re only modifying time types, we’ll also need to add compat versions for many of these syscalls for 64bit native systems.

Libc then introduces versioned symbols, and a new compile options to allow applications to be built for “large time”. New and old applications could then share the same libc.

The benefits of this approach is is simply and minimally extends the current 32 bit environment, without any effect on existing applications which continue to work. Most of the complexity is in the libc library and its build environment.

Quoted 9 lines

The downsides to this approach is that as it follows the large-file approach, it has many of the same problems as large-file support, in that the transition to large-file has been slow and is still ongoing. Also, since this solution focuses on libc, there is also the problem of existing 3rd party libraries, which have no way of knowing which sized time is being used, will break. So all libraries that do anything with time will then have to implement their own versioned interfaces. This approach also makes it a little more difficult to audit that a system is 2038 safe, without running it and looking for issues.

If we go this route, we should also provide a "depends on EMBEDDED" Kconfig option that omits all of the compatibility support, for systems that have fully migrated to new userspace.

Quoted 12 lines

A potential alternative I’d like to also propose is the “Libc Version Bump” approach.

Basically this is the same as the above, where the kernel provides both legacy and new time_t related interfaces. However, the libc would make a version break, migrating to using 64bit time_t types and syscalls. Legacy applications would still work using the old glibc version, but this would provide a stronger line in the sand between 2038 safe and unsafe applications and libraries, making it easier to avoid mixing the two. NetBSD developers discussed this same approach back in 2008 here:
https://mail-index.netbsd.org/tech-userlevel/2008/03/22/msg000231.html

Quoted 9 lines

The downsides here is, for legacy application support, one would have to have all the requisite legacy libraries also installed, which will add a burden to distro vendors. However, this extra storage overhead would likely be a positive motivator to get applications rebuilt and migrated to new version. Additionally, for 3rd party libraries built against the new libc version, the libraries may need to do a version bump themselves, in order to be able to co-exist with versions built against the previous libc. This approach also assumes that libraries that use time_t related values would have a libc dependency.

The migration pain here doesn't seem worth it at all.

Quoted 9 lines

A more aggressive version of the previous proposal is what I’m calling the “New Virtual-Architecture” approach, basically extending the versioning control from the linker down into the kernel as well. It would be adding a new “virtual-architecture” to the kernel, not entirely unlike how x32 is supported on x86_64 systems. We would create entirely new ABI and architecture name in the kernel (think something like “armllt” or “i386llt”). We would preserve compatibility for legacy applications via personalities, similar mechanism as the compat_ interface used to support 32bit applications on 64bit kernels. In this case, we wouldn’t introduce

Quoted 8 lines

new 64 bit syscalls in the kernel, as the existing interfaces would just be typed correctly for our new virtual architecture, but we would have duplicate syscall interfaces via the compat interfaces. The extra complexity would also be that we would have to support new 32bit compat environment on 64bit systems. Userspace would be completely rebuilt to support the new -llt architecture, and compatibility for legacy applications would be done via the same multiarch packaging as is done now for running 32bit applications on 64bit systems.

I wonder: could we make this new architecture effectively use the signatures of the 64-bit syscalls (similar to x32), just with a 32-bit calling convention?

- Josh Triplett

#2 lore raw
Quoted 96 lines · On Tue, Aug 12, 2014 at 6:33 PM, <[email protected]> wrote:
Quoted 4 lines · John Stultz

Also, just to clarify, as time related discussions can bring out a laundry list of issues, I would like to focus this discussion on providing a 2038 solution for existing interfaces and applications in a way that ideally doesn't require modifying application source code. While there will be

Quoted 8 lines

plenty of places where applications have cast or stored time_t values explicitly as longs, and for those applications, deep modifications will be necessary. But I’d like to avoid getting into new-interface discussions, like exporting ktime_t like nanosecond interfaces instead of timepsecs, unifying time-stamping formats, or methods for avoiding leapseconds. Those are all interesting issues, and I’d be up for discussing them separately, but those issue apply equally to 32bit and 64bit systems, and really aren't 2038 specific, so I think its best to separate them out.

That's understandable. However, I wonder to what extent we could support unmodified source code via libc wrappers (since code calling syscalls directly can't work completely unmodified), while using better interfaces for new syscalls. Given syscalls written in terms of (for instance) nanoseconds rather than timespec values, it seems straightforward enough for libc to provide compatibility interfaces.

Quoted 19 lines

From discussions so far, it seems the preferred change to the userspace interface is what I’ll call the “Large File” method, as it follows the approach used for large file support:

Create new 64bit time_t/timespec/timeval/etc variants for syscalls, while preserving existing interfaces. This has some complexity around IOCTLs, but that can mostly be handled by creating new ioctl numbers while preserving the old ones. Since we’re only modifying time types, we’ll also need to add compat versions for many of these syscalls for 64bit native systems.

Libc then introduces versioned symbols, and a new compile options to allow applications to be built for “large time”. New and old applications could then share the same libc.

The benefits of this approach is is simply and minimally extends the current 32 bit environment, without any effect on existing applications which continue to work. Most of the complexity is in the libc library and its build environment.

Quoted 9 lines

The downsides to this approach is that as it follows the large-file approach, it has many of the same problems as large-file support, in that the transition to large-file has been slow and is still ongoing. Also, since this solution focuses on libc, there is also the problem of existing 3rd party libraries, which have no way of knowing which sized time is being used, will break. So all libraries that do anything with time will then have to implement their own versioned interfaces. This approach also makes it a little more difficult to audit that a system is 2038 safe, without running it and looking for issues.

If we go this route, we should also provide a "depends on EMBEDDED" Kconfig option that omits all of the compatibility support, for systems that have fully migrated to new userspace.

Quoted 12 lines

A potential alternative I’d like to also propose is the “Libc Version Bump” approach.

Basically this is the same as the above, where the kernel provides both legacy and new time_t related interfaces. However, the libc would make a version break, migrating to using 64bit time_t types and syscalls. Legacy applications would still work using the old glibc version, but this would provide a stronger line in the sand between 2038 safe and unsafe applications and libraries, making it easier to avoid mixing the two. NetBSD developers discussed this same approach back in 2008 here:
https://mail-index.netbsd.org/tech-userlevel/2008/03/22/msg000231.html

Quoted 9 lines

The downsides here is, for legacy application support, one would have to have all the requisite legacy libraries also installed, which will add a burden to distro vendors. However, this extra storage overhead would likely be a positive motivator to get applications rebuilt and migrated to new version. Additionally, for 3rd party libraries built against the new libc version, the libraries may need to do a version bump themselves, in order to be able to co-exist with versions built against the previous libc. This approach also assumes that libraries that use time_t related values would have a libc dependency.

The migration pain here doesn't seem worth it at all.

Quoted 9 lines

A more aggressive version of the previous proposal is what I’m calling the “New Virtual-Architecture” approach, basically extending the versioning control from the linker down into the kernel as well. It would be adding a new “virtual-architecture” to the kernel, not entirely unlike how x32 is supported on x86_64 systems. We would create entirely new ABI and architecture name in the kernel (think something like “armllt” or “i386llt”). We would preserve compatibility for legacy applications via personalities, similar mechanism as the compat_ interface used to support 32bit applications on 64bit kernels. In this case, we wouldn’t introduce

Quoted 8 lines

new 64 bit syscalls in the kernel, as the existing interfaces would just be typed correctly for our new virtual architecture, but we would have duplicate syscall interfaces via the compat interfaces. The extra complexity would also be that we would have to support new 32bit compat environment on 64bit systems. Userspace would be completely rebuilt to support the new -llt architecture, and compatibility for legacy applications would be done via the same multiarch packaging as is done now for running 32bit applications on 64bit systems.

Quoted 3 lines

I wonder: could we make this new architecture effectively use the signatures of the 64-bit syscalls (similar to x32), just with a 32-bit calling convention?

Doesn't x32 do the reverse? It invokes *compat* syscalls using a 64-bit calling convention.

--Andy

#4 lore raw
In reply to [email protected]
Quoted 13 lines · On 08/12/2014 06:33 PM, [email protected] wrote:
Quoted 4 lines · John Stultz

Also, just to clarify, as time related discussions can bring out a laundry list of issues, I would like to focus this discussion on providing a 2038 solution for existing interfaces and applications in a way that ideally doesn't require modifying application source code. While there will be

Quoted 8 lines

plenty of places where applications have cast or stored time_t values explicitly as longs, and for those applications, deep modifications will be necessary. But I’d like to avoid getting into new-interface discussions, like exporting ktime_t like nanosecond interfaces instead of timepsecs, unifying time-stamping formats, or methods for avoiding leapseconds. Those are all interesting issues, and I’d be up for discussing them separately, but those issue apply equally to 32bit and 64bit systems, and really aren't 2038 specific, so I think its best to separate them out.

Quoted 6 lines

That's understandable. However, I wonder to what extent we could support unmodified source code via libc wrappers (since code calling syscalls directly can't work completely unmodified), while using better interfaces for new syscalls. Given syscalls written in terms of (for instance) nanoseconds rather than timespec values, it seems straightforward enough for libc to provide compatibility interfaces.

So we'd be then introducing new syscalls not just for 32bit but 64bit systems as well, and its a fairly substantial list:

    http://kernelnewbies.org/y2038

Again, I worry that just getting time_t to be 64bits is a big enough problem to solve without trying to mix in interface preference discussions along, adding totally new interfaces, coordinating all the libcs out there to provide wrappers, and then deprecating the old interfaces.

If there are more desirable interfaces wanted, I really think that needs to be a separate topic from fixing the ones we have.

Quoted 29 lines
Quoted 20 lines

From discussions so far, it seems the preferred change to the userspace interface is what I’ll call the “Large File” method, as it follows the approach used for large file support:

Create new 64bit time_t/timespec/timeval/etc variants for syscalls, while preserving existing interfaces. This has some complexity around IOCTLs, but that can mostly be handled by creating new ioctl numbers while preserving the old ones. Since we’re only modifying time types, we’ll also need to add compat versions for many of these syscalls for 64bit native systems.

Libc then introduces versioned symbols, and a new compile options to allow applications to be built for “large time”. New and old applications could then share the same libc.

The benefits of this approach is is simply and minimally extends the current 32 bit environment, without any effect on existing applications which continue to work. Most of the complexity is in the libc library and its build environment.

Quoted 9 lines

The downsides to this approach is that as it follows the large-file approach, it has many of the same problems as large-file support, in that the transition to large-file has been slow and is still ongoing. Also, since this solution focuses on libc, there is also the problem of existing 3rd party libraries, which have no way of knowing which sized time is being used, will break. So all libraries that do anything with time will then have to implement their own versioned interfaces. This approach also makes it a little more difficult to audit that a system is 2038 safe, without running it and looking for issues.

Quoted 3 lines

If we go this route, we should also provide a "depends on EMBEDDED" Kconfig option that omits all of the compatibility support, for systems that have fully migrated to new userspace.

Right, though I think for any of the solutions, having methods to actually remove support for the 2038 unsafe interfaces is important to help ensure systems are completely converted.

Quoted 22 lines
Quoted 13 lines

A potential alternative I’d like to also propose is the “Libc Version Bump” approach.

Basically this is the same as the above, where the kernel provides both legacy and new time_t related interfaces. However, the libc would make a version break, migrating to using 64bit time_t types and syscalls. Legacy applications would still work using the old glibc version, but this would provide a stronger line in the sand between 2038 safe and unsafe applications and libraries, making it easier to avoid mixing the two. NetBSD developers discussed this same approach back in 2008 here:
https://mail-index.netbsd.org/tech-userlevel/2008/03/22/msg000231.html

Quoted 9 lines

The downsides here is, for legacy application support, one would have to have all the requisite legacy libraries also installed, which will add a burden to distro vendors. However, this extra storage overhead would likely be a positive motivator to get applications rebuilt and migrated to new version. Additionally, for 3rd party libraries built against the new libc version, the libraries may need to do a version bump themselves, in order to be able to co-exist with versions built against the previous libc. This approach also assumes that libraries that use time_t related values would have a libc dependency.

Quoted 1 line

The migration pain here doesn't seem worth it at all.

Hmm. I see this as a more mechanical process, rather then the large-file-style support, which requires libraries to all add versioned interfaces. This approach uses the library version number to in-effect version the interfaces. But it does require version bumps everywhere.

The basic point here is that for compatibility we have to pick a method to chose the new or old interfaces. That can be internal to the libraries at build time w/ versioned symbols, external to the libraries by the linker, using their named version numbers, or by the kernel loader itself with the virtual architecture.

Each step along the gradient has different trade-offs in how mechanical it can be vs the level of duplication needed to support it. I think the libc bump has the best balance, but think the distro folks might have better insight as to the pain of what I'm suggesting.

Quoted 18 lines
Quoted 10 lines

A more aggressive version of the previous proposal is what I’m calling the “New Virtual-Architecture” approach, basically extending the versioning control from the linker down into the kernel as well. It would be adding a new “virtual-architecture” to the kernel, not entirely unlike how x32 is supported on x86_64 systems. We would create entirely new ABI and architecture name in the kernel (think something like “armllt” or “i386llt”). We would preserve compatibility for legacy applications via personalities, similar mechanism as the compat_ interface used to support 32bit applications on 64bit kernels. In this case, we wouldn’t introduce

Quoted 8 lines

new 64 bit syscalls in the kernel, as the existing interfaces would just be typed correctly for our new virtual architecture, but we would have duplicate syscall interfaces via the compat interfaces. The extra complexity would also be that we would have to support new 32bit compat environment on 64bit systems. Userspace would be completely rebuilt to support the new -llt architecture, and compatibility for legacy applications would be done via the same multiarch packaging as is done now for running 32bit applications on 64bit systems.

Quoted 3 lines

I wonder: could we make this new architecture effectively use the signatures of the 64-bit syscalls (similar to x32), just with a 32-bit calling convention?

So if you mean on 64bit systems, having the 2038-safe 32bit compat use the same 64bit entry-points, I think that would be possible for some of the syscalls, but there's some where other data fields in structures won't be expected to be 64bit. But I might not be understanding what your suggesting.

thanks
-john

#3 lore raw
In reply to Andy Lutomirski
Quoted 20 lines · Andy Lutomirski
Quoted 19 lines · On Tue, Aug 12, 2014 at 6:33 PM, <[email protected]> wrote:
Quoted 17 lines · John Stultz

A more aggressive version of the previous proposal is what I’m calling the “New Virtual-Architecture” approach, basically extending the versioning control from the linker down into the kernel as well. It would be adding a new “virtual-architecture” to the kernel, not entirely unlike how x32 is supported on x86_64 systems. We would create entirely new ABI and architecture name in the kernel (think something like “armllt” or “i386llt”). We would preserve compatibility for legacy applications via personalities, similar mechanism as the compat_ interface used to support 32bit applications on 64bit kernels. In this case, we wouldn’t introduce new 64 bit syscalls in the kernel, as the existing interfaces would just be typed correctly for our new virtual architecture, but we would have duplicate syscall interfaces via the compat interfaces. The extra complexity would also be that we would have to support new 32bit compat environment on 64bit systems. Userspace would be completely rebuilt to support the new -llt architecture, and compatibility for legacy applications would be done via the same multiarch packaging as is done now for running 32bit applications on 64bit systems.

Quoted 6 lines
Quoted 3 lines

I wonder: could we make this new architecture effectively use the signatures of the 64-bit syscalls (similar to x32), just with a 32-bit calling convention?

Doesn't x32 do the reverse? It invokes *compat* syscalls using a 64-bit calling convention.

Not entirely. It uses 64-bit syscalls primarily with compat only when sharing structures containing pointers. We are working on something similar for arm64 (called the ILP32 ABI).

Signature · 1 line
Catalin
#5 lore raw
In reply to John Stultz
Quoted 25 lines · On Tuesday 12 August 2014 20:45:03 John Stultz wrote:
Quoted 13 lines · On 08/12/2014 06:33 PM, [email protected] wrote:
Quoted 4 lines · John Stultz

Also, just to clarify, as time related discussions can bring out a laundry list of issues, I would like to focus this discussion on providing a 2038 solution for existing interfaces and applications in a way that ideally doesn't require modifying application source code. While there will be

Quoted 8 lines

plenty of places where applications have cast or stored time_t values explicitly as longs, and for those applications, deep modifications will be necessary. But I’d like to avoid getting into new-interface discussions, like exporting ktime_t like nanosecond interfaces instead of timepsecs, unifying time-stamping formats, or methods for avoiding leapseconds. Those are all interesting issues, and I’d be up for discussing them separately, but those issue apply equally to 32bit and 64bit systems, and really aren't 2038 specific, so I think its best to separate them out.

Quoted 6 lines

That's understandable. However, I wonder to what extent we could support unmodified source code via libc wrappers (since code calling syscalls directly can't work completely unmodified), while using better interfaces for new syscalls. Given syscalls written in terms of (for instance) nanoseconds rather than timespec values, it seems straightforward enough for libc to provide compatibility interfaces.

So we'd be then introducing new syscalls not just for 32bit but 64bit systems as well, and its a fairly substantial list:

    http://kernelnewbies.org/y2038
Quoted 8 lines

Again, I worry that just getting time_t to be 64bits is a big enough problem to solve without trying to mix in interface preference discussions along, adding totally new interfaces, coordinating all the libcs out there to provide wrappers, and then deprecating the old interfaces.

If there are more desirable interfaces wanted, I really think that needs to be a separate topic from fixing the ones we have.

Agreed. I originally argued the same ways as Josh in the conversations we had a few weeks ago, but I think you are right.

It essentially comes down to the question what code paths we want to change. Introducing completely new syscall ABIs means we also have to change all 64-bit architectures to support those, while the plain 1:1 replacement means we only impact the 32-bit native and compat (under a 64-bit kernel) cases that actually want this change.

Quoted 7 lines
Quoted 3 lines

If we go this route, we should also provide a "depends on EMBEDDED" Kconfig option that omits all of the compatibility support, for systems that have fully migrated to new userspace.

Right, though I think for any of the solutions, having methods to actually remove support for the 2038 unsafe interfaces is important to help ensure systems are completely converted.

Yes. The way I'd do this is to introduce a new 'CONFIG_COMPAT_TIME' symbol that initially designates the architectures that are have assigned syscall numbers to the newly introduced calls, turning e.g.

SYSCALL_DEFINE4(clock_nanosleep, const clockid_t, which_clock, int, flags,
const struct timespec __user *, rqtp,
struct timespec __user *, rmtp)

{
	...
}

into this:

#if defined(CONFIG_COMPAT_TIME) || defined(CONFIG_64BIT) #define __kernel_timespec __kernel_timespec64 #else
#define __kernel_timespec __kernel_timespec32 #endif

#ifdef CONFIG_COMPAT_TIME
SYSCALL_DEFINE4(compat_clock_nanosleep, const clockid_t, which_clock, int, flags,
const struct __kernel_timespec32 __user *, rqtp, struct __kernel_timespec32 __user *, rmtp)

{
	...
}

#endif

SYSCALL_DEFINE4(clock_nanosleep, const clockid_t, which_clock, int, flags,
const struct __kernel_timespec __user *, rqtp, struct __kernel_timespec __user *, rmtp)

{
	...
}

Each 32-bit architecture (one at a time) has to change its syscall table so the __NR_clock_nanosleep entry points to the sys_compat_clock_nanosleep function, and it needs to assign a new number for sys_clock_nanosleep, and also do the same thing for all other syscalls involving time, and then set CONFIG_COMPAT_TIME unconditionally.

Once all architectures do this, we can turn CONFIG_COMPAT_TIME into a user-selectable option and turn it off for any system that wants to ensure it either works now or doesn't work, but does not break in 24 years from now.

Quoted 9 lines
Quoted 3 lines

I wonder: could we make this new architecture effectively use the signatures of the 64-bit syscalls (similar to x32), just with a 32-bit calling convention?

So if you mean on 64bit systems, having the 2038-safe 32bit compat use the same 64bit entry-points, I think that would be possible for some of the syscalls, but there's some where other data fields in structures won't be expected to be 64bit. But I might not be understanding what your suggesting.

Of the syscalls I've looked at, all would be able to do it this way:
you already have to introduce a new data structure at the libc-kernel interface, with the libc potentially translating it into a different format at least for legacy applications. An interesting case is 'struct stat': All architectures that don't use the asm-generic/stat.h file have their own incompatible definition of this structure, and the asm-generic one gets it wrong too (this is my fault I admit). For the existing 32-bit architectures that also come with a 64-bit counterpart, we have the choice to make each one use whatever their 64-bit ABI uses, or alternatively make them all use the same structure. Either way makes sense to me.

Arnd

#6 lore raw
In reply to John Stultz
Quoted 5 lines · On Tuesday 12 August 2014 17:08:53 John Stultz wrote:

Next week at Kernel Summit, I'm hoping to have a discussion on 2038 during one of the unconference slots, so I wanted to send out some background information about the various proposed approaches (as well as send this out for wider input, since apparently there won't be much libc representation at KS).

Thanks a lot for doing the writeup. I've been on vacation the past two weeks, so I couldn't really contribute much during that time, but a lot of what you write here is what we discussed earlier in private.

Quoted 4 lines

1) In-kernel correctness: Making sure the kernel itself handles the 2038 rollover correctly. As noted in LWN (https://lwn.net/Articles/607741/) this work has begun and portions have been merged. Personally, I don’t really see this as that interesting of a discussion topic, and with the exception

Quoted 8 lines

of dealing with external representations of time like on-disk filesystem formats, etc, it is for the most part a matter of just going through, finding and fixing things (with a few extra complications around ioctls). Mostly this will revolve around adding explicitly sized 64bit time_t types (and 32bit types for compatibility), and migrating in-kernel users over to the explicitly sized implementations, allowing us to validate conversion by eventually removing the in-kernel time_t type, of course finding non-time_t custom storage uses will be a long tail of work.

Some more thoughts on how I think this can proceed:

I think the most important tool here is compile-time validation, by ensuring no use of time_t or a derived structure (timespec, timeval, itimerspec, ...) can be compiled if you are building a kernel that you want to survive y2038.

Any drivers or file systems that we don't expect to need at that time (e.g. ext2/3 rather than ext4) can keep using the unsafe interfaces but will be disabled in Kconfig or removed from the code using #ifdef in case we are building a 64-bit-time-only kernel.

Quoted 9 lines

From discussions so far, it seems the preferred change to the userspace interface is what I’ll call the “Large File” method, as it follows the approach used for large file support:

Create new 64bit time_t/timespec/timeval/etc variants for syscalls, while preserving existing interfaces. This has some complexity around IOCTLs, but that can mostly be handled by creating new ioctl numbers while preserving the old ones. Since we’re only modifying time types, we’ll also need to add compat versions for many of these syscalls for 64bit native systems.

Josh Triplett already brought up the point of introducing completely new interfaces for some cases and having libc emulate the old calls. A variation of this would be to not add any 64-bit version of syscalls that are already outdated. For example, we don't really need a 64-bit gettimeofday system call, because we can easily implement it through clock_gettime.

I'd rather not introduce a 64-bit version of struct timeval if all syscalls using it already have a replacement using timespec.

Quoted 8 lines

The downsides here are many. The distros will probably hate this idea, as it requires rebuilding the world, and maintaining another legacy architecture support. I’m also not completely sure how robust multi-arch packaging is in the face of having to handle 3-4 architectures on one system. On the kernel side, it also adds more complexity, where we have to add even more complex compat support for 64bit systems to handle all the various 32bit applications possible.

Quoted 7 lines

That said the practical reality isn't much further from the the “Libc Version Bump” approach, since legacy support will need legacy versions of all dependent libraries there a well. Similarly the additional storage required to support legacy applications is a positive motivator to get folks to move away from unsafe legacy applications. I also personally like the clarity the new virtual architecture brings, and that it would allow a kernel option to disable 2038-unsafe legacy support.

As mentioned above, I think the last point is not unique to this approach, we can and should have that option in any case.

Between the "large file" approach and the "libc version bump" approach, there doesn't have to be such a strict distinction. As you write, they are the same from a kernel perspective, and I would expect most libc implementations other than glibc (i.e. uclibc, bionic, newlib, musl, ...) to only support the second approach for practical reasons.

In case of glibc, the preferred approach seems to be the "large file" method, which means that any function passing a time value will have at least two versions. However, you should be able to build a glibc binary that only supports the most recent version of these interfaces, effectively dropping support for the old ABI.

At a distro level, anybody can make the decision when to stop supporting the old ABI independently: Gentoo can do this as an option as soon as it works, while Debian can make a 10-year plan for a seamless migration and Fedora can decide to dodge the problem by never even enabling 64-bit time and just dropping support for 32-bit architectures at some point before 2038, following what RHEL7 already did.

Most importantly, the embedded distros that most care about really long term support (openembedded, buildroot, openwrt, ptxdist, ...) can just drop 32-bit time_t entirely by switching the default to 64-bit time_t in a new source release, independent of whether glibc gives them the option for backwards compatibility or not.

Arnd

#7 lore raw
In reply to John Stultz
Quoted 99 lines · John Stultz

So what I’d like to cover in this mail, are some discussion starters around ABI modifications that I think we should discuss at Kernel Summit, in order to make sure we have a clear path forward, as what we decide for the ABI modifications, has large impacts on how we help ensure userspace is correct or not.

From discussions so far, it seems the preferred change to the userspace interface is what I’ll call the “Large File” method, as it follows the approach used for large file support:

Create new 64bit time_t/timespec/timeval/etc variants for syscalls, while preserving existing interfaces. This has some complexity around IOCTLs, but that can mostly be handled by creating new ioctl numbers while preserving the old ones. Since we’re only modifying time types, we’ll also need to add compat versions for many of these syscalls for 64bit native systems.

Libc then introduces versioned symbols, and a new compile options to allow applications to be built for “large time”. New and old applications could then share the same libc.

The benefits of this approach is is simply and minimally extends the current 32 bit environment, without any effect on existing applications which continue to work. Most of the complexity is in the libc library and its build environment.

The downsides to this approach is that as it follows the large-file approach, it has many of the same problems as large-file support, in that the transition to large-file has been slow and is still ongoing. Also, since this solution focuses on libc, there is also the problem of existing 3rd party libraries, which have no way of knowing which sized time is being used, will break. So all libraries that do anything with time will then have to implement their own versioned interfaces. This approach also makes it a little more difficult to audit that a system is 2038 safe, without running it and looking for issues.

A potential alternative I’d like to also propose is the “Libc Version Bump” approach.

Basically this is the same as the above, where the kernel provides both legacy and new time_t related interfaces. However, the libc would make a version break, migrating to using 64bit time_t types and syscalls. Legacy applications would still work using the old glibc version, but this would provide a stronger line in the sand between 2038 safe and unsafe applications and libraries, making it easier to avoid mixing the two. NetBSD developers discussed this same approach back in 2008 here:
https://mail-index.netbsd.org/tech-userlevel/2008/03/22/msg000231.html

The downsides here is, for legacy application support, one would have to have all the requisite legacy libraries also installed, which will add a burden to distro vendors. However, this extra storage overhead would likely be a positive motivator to get applications rebuilt and migrated to new version. Additionally, for 3rd party libraries built against the new libc version, the libraries may need to do a version bump themselves, in order to be able to co-exist with versions built against the previous libc. This approach also assumes that libraries that use time_t related values would have a libc dependency.

A more aggressive version of the previous proposal is what I’m calling the “New Virtual-Architecture” approach, basically extending the versioning control from the linker down into the kernel as well. It would be adding a new “virtual-architecture” to the kernel, not entirely unlike how x32 is supported on x86_64 systems. We would create entirely new ABI and architecture name in the kernel (think something like “armllt” or “i386llt”). We would preserve compatibility for legacy applications via personalities, similar mechanism as the compat_ interface used to support 32bit applications on 64bit kernels. In this case, we wouldn’t introduce new 64 bit syscalls in the kernel, as the existing interfaces would just be typed correctly for our new virtual architecture, but we would have duplicate syscall interfaces via the compat interfaces. The extra complexity would also be that we would have to support new 32bit compat environment on 64bit systems. Userspace would be completely rebuilt to support the new -llt architecture, and compatibility for legacy applications would be done via the same multiarch packaging as is done now for running 32bit applications on 64bit systems.

The pros for this case is that it would be very easy to audit that applications have migrated to the new 64bit time_t ABI. Additionally since we know which type the application is in the kernel, it would make problematic compatability areas like IOCTLS easier to deal with utilizing a flag in the task structure.

The downsides here are many. The distros will probably hate this idea, as it requires rebuilding the world, and maintaining another legacy architecture support. I’m also not completely sure how robust multi-arch packaging is in the face of having to handle 3-4 architectures on one system. On the kernel side, it also adds more complexity, where we have to add even more complex compat support for 64bit systems to handle all the various 32bit applications possible.

That said the practical reality isn't much further from the the “Libc Version Bump” approach, since legacy support will need legacy versions of all dependent libraries there a well. Similarly the additional storage required to support legacy applications is a positive motivator to get folks to move away from unsafe legacy applications. I also personally like the clarity the new virtual architecture brings, and that it would allow a kernel option to disable 2038-unsafe legacy support.

Quoted 9 lines

With any of these approaches, we still have quite a bit of work to just get the kernel in-shape internally. And with the exception of the “virtual arch” approach, the changes on the kernel side are basically the same. The big thing we probably want to avoid is requiring any sort of flagday for distros, and instead allowing them each to migrate to the new solution individually (but hopefully not taking too long). Even so, I think having a clear vision for how userspace will make this transition is important, so hopefully during the Kernel Summit discussion we can come to consensus on what approach to take moving forward.

So I just wanted to send a (late) follow up here to summarize my recollection of the discussion from the Kernel Summit workshop session. Since there wasn't representation from the userspace world, there was only a brief 30 minute session to try to get feedback on the above proposals, and have a bit of open discussion to allow for folks to air any ideas that hadn't already been proposed.

To start, I expresses what I felt to be the need for whatever solution we choose, it should ideally be easy to audit in order for folks to be able to validate that a system has or has not been updated to use 64bit time. The counterpoint raised is that userspace may still do problematic things like storing timestamps on disk or elsewhere in 32bit structures, so the value of any trivial auditing may not be that great.

Quoted 1 line

From the discussion on the list, the "libc version bump" approach is

unlikely, and consensus in the session is that it would be a userspace decision anyway. The "virtual architecture" approach, didn't seem to have many fans either. Linus cut through the discussion with the request that we not over-design this, just provide the 64bit interfaces and let userspace do what it wants. So that resolved my main open questions, and gave a clear way forward from the kernel's perspective, with userland (or atleast glibc) likely taking the "large-file style" approach.

There was also some pre-discussion before the sessions started around why we don't just change time_t to be unsigned. From the kernel's point of view this would be mostly fine, since dates before 1970 are not considered valid for any internal uses (with the exception of some filesystem timestamps). However, the problem with this approach is that userspace may want to handle dates prior to 1970, so this would eliminate that possibility. And removing the sign could cause problems with existing comparison logic. There is also the fact that having the same time unit range on 32bit and 64bit systems avoids complicating how timestamps are interpreted between architectures, where as having it be unsigned on 32bit but signed on 64bit would likely cause confusion. It is quite likely that using unsigned timestamps will be a useful solution for cases where timestamps cannot be converted to 64bits, but from the kernel perspective if we are going to change the abi, we should probably go all the way to 64bits. There seemed to be no disagreement here.

For the rest of the session, I opened it up for further thoughts or ideas. While there wasn't any new proposals, there was a question as to if anyone will really be running 32bit hardware in 2038, which made some folks point out that as systems get smaller there are likely to be tiny embedded platforms using Linux. The point that these systems are likely never to be updated brought out a discussion as to if 2038 is even the least of our worries, given the security issues around these abandoned systems. Debate of the risks of light-bulb hacks ensued. This also raised an interesting but further tangential point that apparently there is talk of legislative enforced obsolescence on devices that don't get updates.

<sarcasm> So clearly the solution is to simply make it illegal to hit any 2038 issue. Then there's nothing to worry about. :) </sarcasm>

So that's everything I recall. It was a fairly short session. I'm hoping Thomas will be able to have a similar session with more userspace representation at Plumbers, but given the attendance caps were hit early, I'm suspecting will have to have continued discussions on the list and wherever we can next year.

I'll be continuing to work (along with Arnd and others) on getting the kernel internals migrated to 64bit time, and then eventually exporting 64bit syscalls for userspace to use. Further discussion, feedback and help is of course welcome!

thanks
-john

#8 lore raw

Hi!

Quoted 17 lines

There was also some pre-discussion before the sessions started around why we don't just change time_t to be unsigned. From the kernel's point of view this would be mostly fine, since dates before 1970 are not considered valid for any internal uses (with the exception of some filesystem timestamps). However, the problem with this approach is that userspace may want to handle dates prior to 1970, so this would eliminate that possibility. And removing the sign could cause problems with existing comparison logic. There is also the fact that having the same time unit range on 32bit and 64bit systems avoids complicating how timestamps are interpreted between architectures, where as having it be unsigned on 32bit but signed on 64bit would likely cause confusion. It is quite likely that using unsigned timestamps will be a useful solution for cases where timestamps cannot be converted to 64bits, but from the kernel perspective if we are going to change the abi, we should probably go all the way to 64bits. There seemed to be no disagreement here.

Quoted 5 lines

For the rest of the session, I opened it up for further thoughts or ideas. While there wasn't any new proposals, there was a question as to if anyone will really be running 32bit hardware in 2038, which made some folks point out that as systems get smaller there are likely to be tiny embedded platforms using Linux. The point that these systems

I'm pretty sure people will run 32bit hardware in 2038... that is not even a question.

Today, there's good chance there's linux somewhere in your car. (Dashboard, entertainment system). People like to keep cars from 1910 working, and I suspect that is not going to change.

So yes, in 2038 people will be running 32bit linux.

Whether there will be people putting 32bit linux into new devices is a question, but I suspect answer is still yes.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#9 lore raw
Quoted 8 lines

Today, there's good chance there's linux somewhere in your car. (Dashboard, entertainment system). People like to keep cars from 1910 working, and I suspect that is not going to change.

So yes, in 2038 people will be running 32bit linux.

Whether there will be people putting 32bit linux into new devices is a question, but I suspect answer is still yes.

I'm currently trying to add 64bit longlong support to an initial PCC 8086 compiler port so I can fix that for some 16bit projects 8-)

My 8bit machines are mostly 2038 safe already.

Alan

#10 lore raw
Quoted 6 lines · One Thousand Gnomes
Quoted 6 lines

Today, there's good chance there's linux somewhere in your car. (Dashboard, entertainment system). People like to keep cars from 1910 working, and I suspect that is not going to change.

So yes, in 2038 people will be running 32bit linux.

Quoted 8 lines
Quoted 2 lines

Whether there will be people putting 32bit linux into new devices is a question, but I suspect answer is still yes.

I'm currently trying to add 64bit longlong support to an initial PCC 8086 compiler port so I can fix that for some 16bit projects 8-)

My 8bit machines are mostly 2038 safe already.

Back in 1985 I decided to Y2K-test my ABC800 8-bit computer. It passed (they had implemented a windowed year solution which breaks in 2070 or 2080, as I recall.)

-hpa