Skip to contentSkip to conversation

Phasing out kernel thread freezing

LSF/MM TOPIC Luis R. Rodriguez 5 participants 10 messages Jan 26, 2018 – Feb 26, 2018 VFS

lore
Share & tools
lore
#0 lore raw

Since the 2015 Kernel summit in South Korea we agreed that we should phase out the kernel thread freezer. This was due to the fact that filesystem freezing was originally added into the kernel to aid in going to suspend to ensure no unwanted IO activity would cause filesystem corruption, and we could instead replace this by using the already implemented filesystem suspend/thaw calls.

Filesystems are not the only users of the freezer API now though. Although most uses outside of filesystems might be bogus, we're prone to hit many regressions with a wide sweep removal. Actually phasing out kernel thread freezing turns out to be trickier than expected even just in filesystems alone, so the current approach is to slowly phase this out one step at time. One subsystem and driver type at a time. Clearly the first subsystem we should tackle is filesystems.

We now seems to have reached consensus on how to do this now for a few filesystems which implement freeze_fs() only. The outstanding work I have has to just do evaluation of the prospect use of sharing the same semantics to freeze as with freeze_bdev(), initiated by dm, and a proper way to address reference counting in a generic form for sb freezing. The only filesystems which implement freeze_fs():

o xfs
o reiserfs
o nilfs2
o jfs
o f2fs
o ext4
o ext2
o btrfs

Of these, the following have freezer helpers, which can then be removed after the kernel automaticaly calls freeze_fs for us on suspend:

o xfs
o nilfs2
o jfs
o f2fs
o ext4

Long term we need to decide what to do with filesystem which do not implement freeze_fs(), or for instance filesystems which implement freeze_super(). Jan Kara made a few suggestions I'll be evaluating soon to this regards, however there are others special filesystem with other considerations though. As an example, for NFS Jeff Layton has suggested to have freeze_fs() make the RPC engine "park" newly issued RPCs for that fs' client onto a rpc_wait_queue. Any RPC that has already been sent however, we need to wait for a reply. Once everything is quiesced we can return and call it frozen. unfreeze_fs can then just have the engine stop parking RPCs and wake up the waitq. He however points out that if we're interested in making the cgroup freezer also work, then we may need to do a bit more work to ensure that we don't end up with frozen tasks squatting on VFS locks. Dave Chinner however notes that cgroup is broken by design *if* it requires tasks to be frozen without holding any VFS/filesystem lock context, and as such we *should* be able to ignore it.

We also need to decide what to do with complex layered situations, for example Bart Van Assche suggested considering the case of a filesystem that exists on top of an md device where the md device uses one or more files as backing store and with the loop driver between the md device and the files. Chinner has suggested to allow block devices to freez superblocks on the block device, however some *may* prefer to have a call to allow a superblock to quiesce the underlying block device which would allow md/dm to suspend whatever on-going maintenance operations it has in progress until the filesystem suggests it needs to thaw. The pros / cons of both approaches should probably be discussed unless its already crystal clear what path to take.

Finally, we should evaluate any other potential uses of the kernel freezer API which now have grown dependent on it, even though the design for it was only to help avoid filesystem corruption on our way to suspend. If none have really become dependent on them, then great, we can just remove them one at a time subsystem at a time to avoid regressions.

Luis

#1 lore raw
Quoted 54 lines · Luis R. Rodriguez

Since the 2015 Kernel summit in South Korea we agreed that we should phase out the kernel thread freezer. This was due to the fact that filesystem freezing was originally added into the kernel to aid in going to suspend to ensure no unwanted IO activity would cause filesystem corruption, and we could instead replace this by using the already implemented filesystem suspend/thaw calls.

Filesystems are not the only users of the freezer API now though. Although most uses outside of filesystems might be bogus, we're prone to hit many regressions with a wide sweep removal. Actually phasing out kernel thread freezing turns out to be trickier than expected even just in filesystems alone, so the current approach is to slowly phase this out one step at time. One subsystem and driver type at a time. Clearly the first subsystem we should tackle is filesystems.

We now seems to have reached consensus on how to do this now for a few filesystems which implement freeze_fs() only. The outstanding work I have has to just do evaluation of the prospect use of sharing the same semantics to freeze as with freeze_bdev(), initiated by dm, and a proper way to address reference counting in a generic form for sb freezing. The only filesystems which implement freeze_fs():

o xfs
o reiserfs
o nilfs2
o jfs
o f2fs
o ext4
o ext2
o btrfs

Of these, the following have freezer helpers, which can then be removed after the kernel automaticaly calls freeze_fs for us on suspend:

o xfs
o nilfs2
o jfs
o f2fs
o ext4

Long term we need to decide what to do with filesystem which do not implement freeze_fs(), or for instance filesystems which implement freeze_super(). Jan Kara made a few suggestions I'll be evaluating soon to this regards, however there are others special filesystem with other considerations though. As an example, for NFS Jeff Layton has suggested to have freeze_fs() make the RPC engine "park" newly issued RPCs for that fs' client onto a rpc_wait_queue. Any RPC that has already been sent however, we need to wait for a reply. Once everything is quiesced we can return and call it frozen. unfreeze_fs can then just have the engine stop parking RPCs and wake up the waitq. He however points out that if we're interested in making the cgroup freezer also work, then we may need to do a bit more work to ensure that we don't end up with frozen tasks squatting on VFS locks. Dave Chinner however notes that cgroup is broken by design *if* it requires tasks to be frozen without holding any VFS/filesystem lock context, and as such we *should* be able to ignore it.

Quoted 10 lines

We also need to decide what to do with complex layered situations, for example Bart Van Assche suggested considering the case of a filesystem that exists on top of an md device where the md device uses one or more files as backing store and with the loop driver between the md device and the files. Chinner has suggested to allow block devices to freez superblocks on the block device, however some *may* prefer to have a call to allow a superblock to quiesce the underlying block device which would allow md/dm to suspend whatever on-going maintenance operations it has in progress until the filesystem suggests it needs to thaw. The pros / cons of both approaches should probably be discussed unless its already crystal clear what path to take.

For a brief moment I pondered whether it would make sense to make filesystems part of the device model so that the suspend code could work out fs <-> bdev dependencies and know in which order to freeze filesystems and quiesce devices, but every time I go digging into how all those macros work I get confused and my eyes glaze over, so I don't know if this is at all a good idea or just confused ramblings.

Maybe it would suffice to start freezing in reverse order of mount and have some way to tell the underlying bdev that it should flush/quiesce/whatever itself?

--D

Quoted 7 lines

Finally, we should evaluate any other potential uses of the kernel freezer API which now have grown dependent on it, even though the design for it was only to help avoid filesystem corruption on our way to suspend. If none have really become dependent on them, then great, we can just remove them one at a time subsystem at a time to avoid regressions.

Luis

#2 lore raw
Quoted 6 lines · Darrick J. Wong

For a brief moment I pondered whether it would make sense to make filesystems part of the device model so that the suspend code could work out fs <-> bdev dependencies and know in which order to freeze filesystems and quiesce devices, but every time I go digging into how all those macros work I get confused and my eyes glaze over, so I don't know if this is at all a good idea or just confused ramblings.

If we have to go this way: shouldn't we introduce a new abstraction ("storage stack element" or similar) rather than making filesystems part of the device model?

Thanks,

Bart.

#3 lore raw
Quoted 11 lines · Bart Van Assche
Quoted 6 lines · Darrick J. Wong

For a brief moment I pondered whether it would make sense to make filesystems part of the device model so that the suspend code could work out fs <-> bdev dependencies and know in which order to freeze filesystems and quiesce devices, but every time I go digging into how all those macros work I get confused and my eyes glaze over, so I don't know if this is at all a good idea or just confused ramblings.

If we have to go this way: shouldn't we introduce a new abstraction ("storage stack element" or similar) rather than making filesystems part of the device model?

That would be my approach.

Trying to "suspend" filesystems at the same time as I/O devices (and all of that asynchronously) may be problematic for ordering reasons and similar.

Moreover, during hibernation devices are suspended for two times (and resumed in between, of course) whereas filesystems only need to be "suspended" once.

With that in mind, I would add a mechanism allowing filesystems (and possibly other components of the storage stack) to register a set of callbacks for suspend and resume and then invoking those callbacks in a specific order.

#4 lore raw
Quoted 13 lines · Rafael J. Wysocki
Quoted 11 lines · Bart Van Assche
Quoted 6 lines · Darrick J. Wong

For a brief moment I pondered whether it would make sense to make filesystems part of the device model so that the suspend code could work out fs <-> bdev dependencies and know in which order to freeze filesystems and quiesce devices, but every time I go digging into how all those macros work I get confused and my eyes glaze over, so I don't know if this is at all a good idea or just confused ramblings.

If we have to go this way: shouldn't we introduce a new abstraction ("storage stack element" or similar) rather than making filesystems part of the device model?

Quoted 5 lines

That would be my approach.

Trying to "suspend" filesystems at the same time as I/O devices (and all of that asynchronously) may be problematic for ordering reasons and similar.

Oh look, another ordering issue. And this is why I was not a fan of the device link API even though that is what we got merged. Moving on...

Quoted 5 lines

Moreover, during hibernation devices are suspended for two times (and resumed in between, of course) whereas filesystems only need to be "suspended" once.

From your point of view yes, but actually internally the VFS layer or

filesystems themselves may end up re-using this mechanism later for other things like -- snapshotting. And if some folks have it the way they want it, we may need a dependency map between filesystems anyway for filesystem specific reasons.

Quoted 4 lines

With that in mind, I would add a mechanism allowing filesystems (and possibly other components of the storage stack) to register a set of callbacks for suspend and resume and then invoking those callbacks in a specific order.

That's what I had done in my series, the issue here is order. Order in my series is simple but should work for starters, later however I suspect we'll need something more robust to help.

Luis

#5 lore raw
Quoted 27 lines · Luis R. Rodriguez
Quoted 13 lines · Rafael J. Wysocki
Quoted 11 lines · Bart Van Assche
Quoted 6 lines · Darrick J. Wong

For a brief moment I pondered whether it would make sense to make filesystems part of the device model so that the suspend code could work out fs <-> bdev dependencies and know in which order to freeze filesystems and quiesce devices, but every time I go digging into how all those macros work I get confused and my eyes glaze over, so I don't know if this is at all a good idea or just confused ramblings.

If we have to go this way: shouldn't we introduce a new abstraction ("storage stack element" or similar) rather than making filesystems part of the device model?

Quoted 5 lines

That would be my approach.

Trying to "suspend" filesystems at the same time as I/O devices (and all of that asynchronously) may be problematic for ordering reasons and similar.

Oh look, another ordering issue. And this is why I was not a fan of the device link API even though that is what we got merged. Moving on...

Quoted 3 lines

Moreover, during hibernation devices are suspended for two times (and resumed in between, of course) whereas filesystems only need to be "suspended" once.

Quoted 5 lines

From your point of view yes, but actually internally the VFS layer or filesystems themselves may end up re-using this mechanism later for other things like -- snapshotting. And if some folks have it the way they want it, we may need a dependency map between filesystems anyway for filesystem specific reasons.

That's orthogonal to what I said.

A dependency map between filesystems and other components of the block layer (like md, dm etc) will be necessary going forward (if all of the suspending and resuming of them is expected to be reliable anyway), but that doesn't change hibernation-related requirements one whit.

Filesystems need to be suspended (or frozen or whatever terminology ends up being used for that) *before* creating a hibernation image and they *cannot* be resumed (unfrozen etc) after that until the system is off or the kernel decides that the hibernation has failed and rolls back. Whatever data/metadata are there in persistent storage before the image is created, changing them after that point is potentially critically harmful, so (in the hibernation case) all of the in-flight I/O that may end up being written to persistent storage needs to be flushed before creating the image.

However, *devices* are resumed after creating the image so that the image itself can be written to persistent storage and are suspended after that again before putting the system to sleep (for wakeup to work, among other things).

That's why suspend/resume of filesystems cannot be tied to suspend/resume of devices.

Note that this isn't the case for system suspend/resume (suspend-to-RAM or suspend-to-idle).

Quoted 8 lines
Quoted 4 lines

With that in mind, I would add a mechanism allowing filesystems (and possibly other components of the storage stack) to register a set of callbacks for suspend and resume and then invoking those callbacks in a specific order.

That's what I had done in my series, the issue here is order. Order in my series is simple but should work for starters, later however I suspect we'll need something more robust to help.

Quite likely.

Thanks,
Rafael

#6 lore raw
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Luis R. Rodriguez
Quoted 3 lines · Rafael J. Wysocki

Moreover, during hibernation devices are suspended for two times (and resumed in between, of course) whereas filesystems only need to be "suspended" once.

Quoted 7 lines
Quoted 5 lines

From your point of view yes, but actually internally the VFS layer or filesystems themselves may end up re-using this mechanism later for other things like -- snapshotting. And if some folks have it the way they want it, we may need a dependency map between filesystems anyway for filesystem specific reasons.

That's orthogonal to what I said.

<-- snip -->

Quoted 7 lines

However, *devices* are resumed after creating the image so that the image itself can be written to persistent storage and are suspended after that again before putting the system to sleep (for wakeup to work, among other things).

That's why suspend/resume of filesystems cannot be tied to suspend/resume of devices.

Ah, yes, I see your point now. So for filesystems we really don't care if its suspend or hibernation, we just need to freeze in the right order. So long as we get that order right we should be OK.

Curious -- do we resume *all* devices after creating the image for hibernation today? Not that I am advocating using devices for this mechanism or resolution for filesystems, I'm just curious as we're on the topic.

Luis

#7 lore raw
Quoted 22 lines · Luis R. Rodriguez
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Luis R. Rodriguez
Quoted 3 lines · Rafael J. Wysocki

Moreover, during hibernation devices are suspended for two times (and resumed in between, of course) whereas filesystems only need to be "suspended" once.

Quoted 7 lines
Quoted 5 lines

From your point of view yes, but actually internally the VFS layer or filesystems themselves may end up re-using this mechanism later for other things like -- snapshotting. And if some folks have it the way they want it, we may need a dependency map between filesystems anyway for filesystem specific reasons.

That's orthogonal to what I said.

<-- snip -->

Quoted 5 lines

However, *devices* are resumed after creating the image so that the image itself can be written to persistent storage and are suspended after that again before putting the system to sleep (for wakeup to work, among other things).

Quoted 6 lines
Quoted 2 lines

That's why suspend/resume of filesystems cannot be tied to suspend/resume of devices.

Ah, yes, I see your point now. So for filesystems we really don't care if its suspend or hibernation, we just need to freeze in the right order. So long as we get that order right we should be OK.

Generally, yes.

System suspend/resume (S2R, S2I), however, doesn't really require the flushing part, so in principle the full-blown fs freezing is not necessary in that case, strictly speaking. The ordering of writes still has to be preserved in this case (that is, writes to persistent storage in the presence of a suspend-resume cycle must occur in the same order as without the suspend-resume), but it doesn't matter too much when the writes actually happen. They may be carried out before the suspend or after the resume or somewhere during one of them: all should be fine so long as the ordering of writes doesn't change as a result of the suspend-resume (and suspend-resume failures are like surprise resets from the fs perspective in that case, so journaling should be sufficient to recover from them).

Of course, the full-blown freezing will work for system suspend too, but it may be high-latency which is not desirable in some scenarios utilizing system suspend/resume ("dark resume" or "lucid sleep" scenarios, for example). That will be a concern in the long run (it kind of is a concern already today), so I would consider special-casing it from the outset.

Quoted 2 lines

Curious -- do we resume *all* devices after creating the image for hibernation today?

Yes, we do.

It is not straightforward to determine which devices will be necessary to save the image in general and we would need to resume a significant subset of the device hierarchy for this purpose anyway.

Thanks,
Rafael

reviewer
#8 lore raw
Quoted 7 lines · Rafael J. Wysocki
Quoted 3 lines · Luis R. Rodriguez

Ah, yes, I see your point now. So for filesystems we really don't care if its suspend or hibernation, we just need to freeze in the right order. So long as we get that order right we should be OK.

Generally, yes.

Quoted 7 lines

System suspend/resume (S2R, S2I), however, doesn't really require the flushing part, so in principle the full-blown fs freezing is not necessary in that case, strictly speaking. The ordering of writes still has to be preserved in this case (that is, writes to persistent storage in the presence of a suspend-resume cycle must occur in the same order as without the suspend-resume), but it doesn't matter too much when the writes actually happen.

I agree that in principle we don't have to flush all dirty data from page cache before system suspend (I believe this is what you are speaking about here, isn't it?). However from implementation POV it is much simpler that way as otherwise processes get blocked in the kernel in unexpected places waiting for locks held by blocked flusher threads etc.

Quoted 6 lines

They may be carried out before
the suspend or after the resume or somewhere during one of them: all should be fine so long as the ordering of writes doesn't change as a result of the suspend-resume (and suspend-resume failures are like surprise resets from the fs perspective in that case, so journaling should be sufficient to recover from them).

Err, I don't follow how suspend failure looks like surprise reset to the fs. If suspend fails, we just thaw the filesystem and off we go, no IO lost. If you speak about a situation where you suspend but then boot without resuming - yes, that looks like a surprise reset and it behaves that way already today.

Quoted 6 lines

Of course, the full-blown freezing will work for system suspend too, but it may be high-latency which is not desirable in some scenarios utilizing system suspend/resume ("dark resume" or "lucid sleep" scenarios, for example). That will be a concern in the long run (it kind of is a concern already today), so I would consider special-casing it from the outset.

Understood but that would require considerable amount of work on the fs side and the problem is hard enough as is :) And switching the freeze implementation to avoid sync(2) if asked to is quite independent from implementing system suspend to use fs freezing.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
Quoted 16 lines · Jan Kara
Quoted 7 lines · Rafael J. Wysocki
Quoted 3 lines · Luis R. Rodriguez

Ah, yes, I see your point now. So for filesystems we really don't care if its suspend or hibernation, we just need to freeze in the right order. So long as we get that order right we should be OK.

Generally, yes.

Quoted 7 lines

System suspend/resume (S2R, S2I), however, doesn't really require the flushing part, so in principle the full-blown fs freezing is not necessary in that case, strictly speaking. The ordering of writes still has to be preserved in this case (that is, writes to persistent storage in the presence of a suspend-resume cycle must occur in the same order as without the suspend-resume), but it doesn't matter too much when the writes actually happen.

Quoted 3 lines

I agree that in principle we don't have to flush all dirty data from page cache before system suspend (I believe this is what you are speaking about here, isn't it?).

Yes, it is.

Quoted 3 lines

However from implementation POV it is much simpler that
way as otherwise processes get blocked in the kernel in unexpected places waiting for locks held by blocked flusher threads etc.

Understood.

Quoted 10 lines
Quoted 6 lines

They may be carried out before
the suspend or after the resume or somewhere during one of them: all should be fine so long as the ordering of writes doesn't change as a result of the suspend-resume (and suspend-resume failures are like surprise resets from the fs perspective in that case, so journaling should be sufficient to recover from them).

Err, I don't follow how suspend failure looks like surprise reset to the fs. If suspend fails, we just thaw the filesystem and off we go, no IO lost.

Right.

Quoted 3 lines

If you speak about a situation where you suspend but then boot without resuming - yes, that looks like a surprise reset and it behaves that way already today.

I was talking about the latter.

Quoted 9 lines
Quoted 6 lines

Of course, the full-blown freezing will work for system suspend too, but it may be high-latency which is not desirable in some scenarios utilizing system suspend/resume ("dark resume" or "lucid sleep" scenarios, for example). That will be a concern in the long run (it kind of is a concern already today), so I would consider special-casing it from the outset.

Understood but that would require considerable amount of work on the fs side and the problem is hard enough as is :)

Fair enough. :-)

Quoted 2 lines

And switching the freeze implementation to avoid sync(2) if asked to is quite independent from implementing system suspend to use fs freezing.

Agreed.

Thanks,
Rafael