Remove process freezer from suspend to RAM pathway

Suspend to RAM on a machine with / on a fuse filesystem turns out to be a screaming nightmare - either the suspend fails because syslog (for instance) can't be frozen, or the machine deadlocks for some other reason I haven't tracked down. We could "fix" fuse, or alternatively we could do what we do for suspend to RAM on other platforms (PPC and APM) and just not use the freezer.

Signed-off-by Matthew Garrett <[email protected]>

Suspend to RAM on a machine with / on a fuse filesystem turns out to be a screaming nightmare - either the suspend fails because syslog (for instance) can't be frozen, or the machine deadlocks for some other reason I haven't tracked down. We could "fix" fuse, or alternatively we could do what we do for suspend to RAM on other platforms (PPC and APM) and just not use the freezer.

Signed-off-by Matthew Garrett <[email protected]>

Note, though, that this won't help at all when people use the "suspend-to-ram instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

Note, though, that this won't help at all when people use the "suspend-to-ram instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

Quoted 1 line · Nigel Cunningham

Note, though, that this won't help at all when people use

Quoted 3 lines

instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

If / is on fuse it's unlikely that hibernation is high on your list of priorities right now.

Quoted 1 line · Nigel Cunningham

Note, though, that this won't help at all when people use

Quoted 3 lines

instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

If / is on fuse it's unlikely that hibernation is high on your list of priorities right now.

Note, though, that this won't help at all when people use the "suspend-to-ram instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

Note, though, that this won't help at all when people use the "suspend-to-ram instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

Quoted 2 lines

Note, though, that this won't help at all when people use

Quoted 3 lines

instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

No, Fuse is not a broken idea in the first place. It's the freezer that is a totally broken idea. It has proven many times to be racy by design and cannot be made right. Ther usermode helper mess is just part of that, fuse is another example, etc etc ...

So I think Matthew is totally right. In fact, the presence of the freezer is the main reason why Paulus so far NACKed Johannes attempts at merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

As for suspend-to-disk, I refer you to the discussions we had in the past with Linus, where he explains I think quite clearly how wrong the current implementation of STR is :-)

Thing is, if you're going to do snapshots, you should probably not sync after you have "frozen" anyway.

Quoted 2 lines

Note, though, that this won't help at all when people use

Quoted 3 lines

instead of powering down after writing a hibernation image" feature in (uswsusp | tuxonice). Fuse is just a broken idea in the first place, but given that it exists, we still need to find the underlying cause.

No, Fuse is not a broken idea in the first place. It's the freezer that is a totally broken idea. It has proven many times to be racy by design and cannot be made right. Ther usermode helper mess is just part of that, fuse is another example, etc etc ...

So I think Matthew is totally right. In fact, the presence of the freezer is the main reason why Paulus so far NACKed Johannes attempts at merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

As for suspend-to-disk, I refer you to the discussions we had in the past with Linus, where he explains I think quite clearly how wrong the current implementation of STR is :-)

Thing is, if you're going to do snapshots, you should probably not sync after you have "frozen" anyway.

Quoted 6 lines

So I think Matthew is totally right. In fact, the presence of the freezer is the main reason why Paulus so far NACKed Johannes attempts at merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

Fuse depends on !PPC?

Quoted 3 lines

As for suspend-to-disk, I refer you to the discussions we had in the past with Linus, where he explains I think quite clearly how wrong the current implementation of STR is :-)

I assume you mean STD.

The problem there is that Linus doesn't care about STD.
If he did, I dare say he'd think through the issues more thoroughly than he apparently has.

Quoted 2 lines

Thing is, if you're going to do snapshots, you should probably not sync after you have "frozen" anyway.

Fully agree. But how do you stop things syncing while you're writing the image if you don't have a freezer or equivalent? (scheduler based, kexec.. they're all workarounds for this issue).

Quoted 6 lines

So I think Matthew is totally right. In fact, the presence of the freezer is the main reason why Paulus so far NACKed Johannes attempts at merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

Fuse depends on !PPC?

Quoted 3 lines

As for suspend-to-disk, I refer you to the discussions we had in the past with Linus, where he explains I think quite clearly how wrong the current implementation of STR is :-)

I assume you mean STD.

The problem there is that Linus doesn't care about STD.
If he did, I dare say he'd think through the issues more thoroughly than he apparently has.

Quoted 2 lines

Thing is, if you're going to do snapshots, you should probably not sync after you have "frozen" anyway.

Fully agree. But how do you stop things syncing while you're writing the image if you don't have a freezer or equivalent? (scheduler based, kexec.. they're all workarounds for this issue).

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

Quoted 13 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 5 lines

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with. Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

Quoted 13 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 5 lines

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with. Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

Quoted 19 lines

Quoted 18 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

Quoted 5 lines

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

You are talking about hibernate, right?  Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

Quoted 19 lines

Quoted 18 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

Quoted 5 lines

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

You are talking about hibernate, right?  Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

Quoted 2 lines

You are talking about hibernate, right? Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

If that is so, why do you care? If it is really atomic, fuse has no chance
to call out to its component in user space either. Removing the freezer
cannot make a difference.

Quoted 2 lines

You are talking about hibernate, right? Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

If that is so, why do you care? If it is really atomic, fuse has no chance
to call out to its component in user space either. Removing the freezer
cannot make a difference.

Quoted 8 lines

Quoted 2 lines

You are talking about hibernate, right?  Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

If that is so, why do you care? If it is really atomic, fuse has no chance
to call out to its component in user space either. Removing the freezer
cannot make a difference.

It's not atomic. You will get called after suspend() in drivers. The thing is ... you just have to deal with it :-)

Quoted 4 lines

Quoted 3 lines

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

Quoted 5 lines

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

You are talking about hibernate, right? Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

Quoted 4 lines

Quoted 3 lines

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

Quoted 5 lines

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

You are talking about hibernate, right? Suspending (to ram) is instantaneous, in that _after_ suspend no CPU is active obviously.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with. Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

Quoted 3 lines

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

No, that's always been bullshit. You can have IOs emitted by kernel threads (think knfsd, and that's just one among many others). Beside,

relying on having userland frozen means that your driver will be unable to be "live" suspended/resumed for more ambitious dynamic power management schemes.

So it's always been wrong, imho, to rely on that. I've had powermac STR work fine without the freezer for years, and few drivers have been a problem, and we just fixed them.

Quoted 3 lines

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

No, that's always been bullshit. You can have IOs emitted by kernel threads (think knfsd, and that's just one among many others). Beside,

relying on having userland frozen means that your driver will be unable to be "live" suspended/resumed for more ambitious dynamic power management schemes.

So it's always been wrong, imho, to rely on that. I've had powermac STR work fine without the freezer for years, and few drivers have been a problem, and we just fixed them.

That's why we have the problem of freezing the kernel threads or not.

You want to have all that pain for fuse?

That's why we have the problem of freezing the kernel threads or not.

You want to have all that pain for fuse?

Quoted 1 line

That's why we have the problem of freezing the kernel threads or not.

That problem is a symptom of the deeper conceptual problem, as is the problem with FUSE.

Quoted 1 line

You want to have all that pain for fuse?

I'd certainly rather get the drivers right, and maybe have an occasional deadlock if I miss something, than have a GUARANTEED deadlock every time I suspend with a FUSE filesystem mounted (which is pretty much every time, since I use encfs regularly).

That's weird, I never had a suspend problem due to a fuse mount, though I have them all the time. And I suspect, that even the sync()

thing that suspend does is not the real cause, because sync() actually does nothing in fuse filesystems.

Quoted 2 lines

That's weird, I never had a suspend problem due to a fuse mount, though I have them all the time. And I suspect, that even the sync()

Well, I don't either, because we don't freeze processes on powerbooks. But I have heard that other people have problems with suspending with a fuse filesystem mounted. Maybe the difference is whether or not the filesystem is writable?

Quoted 2 lines

thing that suspend does is not the real cause, because sync() actually does nothing in fuse filesystems.

It's not the filesystem sync method, as I understand it, it's that if there are dirty pages in the page cache for files on the fuse filesystem,

the system will initiate a write-out on them and wait for it to finish. But if the fuse userspace is frozen, the write-out will never complete.

Quoted 2 lines

That's weird, I never had a suspend problem due to a fuse mount, though I have them all the time. And I suspect, that even the sync()

Well, I don't either, because we don't freeze processes on powerbooks. But I have heard that other people have problems with suspending with a fuse filesystem mounted. Maybe the difference is whether or not the filesystem is writable?

Quoted 2 lines

thing that suspend does is not the real cause, because sync() actually does nothing in fuse filesystems.

It's not the filesystem sync method, as I understand it, it's that if there are dirty pages in the page cache for files on the fuse filesystem,

the system will initiate a write-out on them and wait for it to finish. But if the fuse userspace is frozen, the write-out will never complete.

Quoted 8 lines

Quoted 2 lines

That's weird, I never had a suspend problem due to a fuse mount, though I have them all the time. And I suspect, that even the sync()

Well, I don't either, because we don't freeze processes on powerbooks. But I have heard that other people have problems with suspending with a fuse filesystem mounted. Maybe the difference is whether or not the filesystem is writable?

Quoted 6 lines

Quoted 2 lines

thing that suspend does is not the real cause, because sync() actually does nothing in fuse filesystems.

It's not the filesystem sync method, as I understand it, it's that if there are dirty pages in the page cache for files on the fuse filesystem,

Currently fuse doesn't produce dirty pages. Normal writes are done synchronously, and writable mmap is not supported. So sync() should really be a no-op for fuse.

Quoted 3 lines

the system will initiate a write-out on them and wait for it to finish. But if the fuse userspace is frozen, the write-out will never complete.

Maybe there is some other fs operation being done, possibly not directly, but by waiting for a kernel thread, that does that.

It would be nice, if someone who can reproduce the deadlock could debug it.

Does sysrq still work during suspend?

Quoted 8 lines

Quoted 2 lines

That's weird, I never had a suspend problem due to a fuse mount, though I have them all the time. And I suspect, that even the sync()

Well, I don't either, because we don't freeze processes on powerbooks. But I have heard that other people have problems with suspending with a fuse filesystem mounted. Maybe the difference is whether or not the filesystem is writable?

Quoted 6 lines

Quoted 2 lines

thing that suspend does is not the real cause, because sync() actually does nothing in fuse filesystems.

It's not the filesystem sync method, as I understand it, it's that if there are dirty pages in the page cache for files on the fuse filesystem,

Currently fuse doesn't produce dirty pages. Normal writes are done synchronously, and writable mmap is not supported. So sync() should really be a no-op for fuse.

Quoted 3 lines

the system will initiate a write-out on them and wait for it to finish. But if the fuse userspace is frozen, the write-out will never complete.

Maybe there is some other fs operation being done, possibly not directly, but by waiting for a kernel thread, that does that.

It would be nice, if someone who can reproduce the deadlock could debug it.

Does sysrq still work during suspend?

Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:

Quoted 13 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 5 lines

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with. Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

Am Dienstag, 3. Juli 2007 schrieb Miklos Szeredi:

Quoted 13 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 5 lines

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with. Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

Quoted 13 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 5 lines

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with.

Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

Quoted 13 lines

Quoted 4 lines

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

 - For STR, don't do the freezer thing.

Quoted 8 lines

Quoted 3 lines

 - For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 5 lines

Oliver, can you please explain your worries in a bit more detail?

I don't claim to know anything about how STR or hibernate works, but neither seem to have any problem with I/O on the fuse device "racing" with them.

The problem is not with fuse. The problem is generic in nature.

If you remove the freezer, user space remains active until the last CPU goes into suspend. It can do syscalls. Or do you know a clean way to exempt only the tasks fuse might use?

Now device drivers have a guaranteed temporal sequence:

last io -> suspend() -> resume() [or disconnect()] -> new io

This is because suspend() is called after the freezer goes into action. If you remove the freezer, you need to deal with

1. io to suspended devices
2. resume() assuming that the device is in the state suspend() left it
3. io changing a device's state while suspend is saving it

and you need to fix this for all device drivers, not just those fuse is involved with.

Removing the freezer means doing a more or less full audit of every driver and additional locking in many drivers.

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

I'm actually having a hard time thinking of how to test your assertion since there are so few things on a typical computer that are plain character devices driving real hardware.  A serial port would be about the only one; keyboards and mice (and serial ports :) are USB these days, or ADB on older powerbooks.

USB devices certainly have suspend methods.

USB devices certainly have suspend methods.

Quoted 1 line · Oliver Neukum

USB devices certainly have suspend methods.

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case.  Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

Quoted 1 line · Oliver Neukum

USB devices certainly have suspend methods.

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case.  Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

Quoted 5 lines

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case.  Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

And what happens to that IO? I suspended and lost data is not an acceptable behavior.

Quoted 5 lines

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case.  Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

And what happens to that IO? I suspended and lost data is not an acceptable behavior.

Quoted 6 lines · Oliver Neukum

Quoted 5 lines

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case.  Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

Quoted 2 lines

And what happens to that IO? I suspended and lost data is not an acceptable behavior.

It's not lost, it's sitting in RAM, and will be sent out when you resume.

Quoted 6 lines · Oliver Neukum

Quoted 5 lines

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case.  Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

Quoted 2 lines

And what happens to that IO? I suspended and lost data is not an acceptable behavior.

It's not lost, it's sitting in RAM, and will be sent out when you resume.

Quoted 2 lines

It's not lost, it's sitting in RAM, and will be sent out when you resume.

Unfortunately this is not the case. The URB will error out.

Quoted 2 lines

It's not lost, it's sitting in RAM, and will be sent out when you resume.

Unfortunately this is not the case. The URB will error out.

Quoted 4 lines · Oliver Neukum

Quoted 2 lines

It's not lost, it's sitting in RAM, and will be sent out when you resume.

Unfortunately this is not the case. The URB will error out.

So the higher-level driver needs to do the sensible thing, i.e., resubmit the URB after resume.  It's not rocket science.  The data is not lost, it's sitting in RAM, and the higher-level driver will send it out when you resume.  If not, then we fix the higher-level driver.

Quoted 4 lines · Oliver Neukum

Quoted 2 lines

It's not lost, it's sitting in RAM, and will be sent out when you resume.

Unfortunately this is not the case. The URB will error out.

So the higher-level driver needs to do the sensible thing, i.e., resubmit the URB after resume.  It's not rocket science.  The data is not lost, it's sitting in RAM, and the higher-level driver will send it out when you resume.  If not, then we fix the higher-level driver.

You cannot simply restart the URB without thinking. The device after resumption may or may not be in the stage you left it. It needs to be rechecked and some settings must be renewed. You cannot simple throw an URB from an arbitrary stage of the protocol at it.
Suspension of devices can only happen at some points in the protocol.

Quoted 7 lines · Oliver Neukum

You cannot simply restart the URB without thinking. The device after resumption may or may not be in the stage you left it. It needs to be rechecked and some settings must be renewed. You cannot simple throw an URB from an arbitrary stage of the protocol at it.
Suspension of devices can only happen at some points in the protocol.

Yeah, and?

I said "the higher-level driver needs to do the sensible thing".

They can't. Device specific protocols are known to the drivers only. The fact remains, remove the freezer and you need to go through all drivers.

They can't. Device specific protocols are known to the drivers only. The fact remains, remove the freezer and you need to go through all drivers.

Quoted 3 lines · Oliver Neukum

They can't. Device specific protocols are known to the drivers only. The fact remains, remove the freezer and you need to go through all drivers.

The freezer does not actually mean that you don't have to get the drivers right, because kernel threads can issue I/O requests.

Quoted 3 lines · Oliver Neukum

They can't. Device specific protocols are known to the drivers only. The fact remains, remove the freezer and you need to go through all drivers.

The freezer does not actually mean that you don't have to get the drivers right, because kernel threads can issue I/O requests.

Quoted 1 line · Oliver Neukum

USB devices certainly have suspend methods.

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case. Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

Quoted 1 line · Oliver Neukum

USB devices certainly have suspend methods.

Indeed, and the USB framework has code to know when the host controller is suspended and avoid trying to send out urbs in that case. Or at least it did last time I looked at it in any detail; it's been "just working" - including suspending and resuming, without the freezer - for quite a while now.

Going back to the old powerbook sleep code, we had a two-phase
suspend: drivers got notified once when userspace is still running, with interrupts enabled, in process context; and then a second time with interrupts disabled and with only one CPU up, so the process that is initiating the suspend is the only process running (since interrupts are disabled and nothing it does can sleep, no other process can get to run).

I still believe that is the right way to go, although we currently only have a single-phase suspend.

Most drivers suspended their hardware in the second call. If they are in the middle of a conversation with their device that *has* to be completed, they can do that by polling.

If it's a character device, a
better approach would be to set a flag or whatever in the first suspend call to make sure that no new conversations get started with the device, sleeping if necessary.

I'm actually having a hard time thinking of how to test your assertion since there are so few things on a typical computer that are plain character devices driving real hardware. A serial port would be about the only one; keyboards and mice (and serial ports :) are USB these days, or ADB on older powerbooks.

Quoted 3 lines

Most drivers suspended their hardware in the second call. If they are in the middle of a conversation with their device that *has* to be completed, they can do that by polling.

Ugh. That will cause problems when you try to integrate runtime suspend. In fact this whole approach is unsuitable for runtime PM and it obscures the similarities between runtime PM and STR.

Quoted 3 lines

Most drivers suspended their hardware in the second call. If they are in the middle of a conversation with their device that *has* to be completed, they can do that by polling.

Ugh. That will cause problems when you try to integrate runtime suspend. In fact this whole approach is unsuitable for runtime PM and it obscures the similarities between runtime PM and STR.

Quoted 4 lines · Alan Stern

Quoted 3 lines

Most drivers suspended their hardware in the second call. If they are in the middle of a conversation with their device that *has* to be completed, they can do that by polling.

Quoted 3 lines

Ugh. That will cause problems when you try to integrate runtime suspend. In fact this whole approach is unsuitable for runtime PM and it obscures the similarities between runtime PM and STR.

Yes there are similarities, but it would be a big mistake to say that a requirement for STR is that all drivers do runtime PM.

The main attraction of the late-suspend call is that it really does, reliably, guarantee that the driver's I/O request methods won't get called between the late-suspend call and the early-resume call.

Quoted 4 lines · Alan Stern

Quoted 3 lines

Most drivers suspended their hardware in the second call. If they are in the middle of a conversation with their device that *has* to be completed, they can do that by polling.

Quoted 3 lines

Ugh. That will cause problems when you try to integrate runtime suspend. In fact this whole approach is unsuitable for runtime PM and it obscures the similarities between runtime PM and STR.

Yes there are similarities, but it would be a big mistake to say that a requirement for STR is that all drivers do runtime PM.

The main attraction of the late-suspend call is that it really does, reliably, guarantee that the driver's I/O request methods won't get called between the late-suspend call and the early-resume call.

That's not what I'm saying. What I'm saying is that it would be a big mistake to force all drivers which implement runtime PM to do it using a separate code path from system PM.

Quoted 3 lines

The main attraction of the late-suspend call is that it really does, reliably, guarantee that the driver's I/O request methods won't get called between the late-suspend call and the early-resume call.

For some drivers (like USB), carrying out an actual suspend requires a delay. Right now we implement those delays using wait_event(), wait_for_completion(), and so on. Would you have us check at runtime whether or not a system suspend is underway and in each case use a busy-loop instead if it is?

What happens if, in order to carry out the late-suspend, a driver needs to acquire a mutex which happens to be held by some other task? That other task won't be able to run and release the mutex, so you will deadlock.

That's not what I'm saying. What I'm saying is that it would be a big mistake to force all drivers which implement runtime PM to do it using a separate code path from system PM.

Quoted 3 lines

The main attraction of the late-suspend call is that it really does, reliably, guarantee that the driver's I/O request methods won't get called between the late-suspend call and the early-resume call.

For some drivers (like USB), carrying out an actual suspend requires a delay. Right now we implement those delays using wait_event(), wait_for_completion(), and so on. Would you have us check at runtime whether or not a system suspend is underway and in each case use a busy-loop instead if it is?

What happens if, in order to carry out the late-suspend, a driver needs to acquire a mutex which happens to be held by some other task? That other task won't be able to run and release the mutex, so you will deadlock.

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:

Quoted 4 lines

So to summarize, the plan that makes things work with fuse is:

- For STR, don't do the freezer thing.

Quoted 3 lines

- For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Quoted 9 lines

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:

Quoted 7 lines

So to summarize, the plan that makes things work with fuse is:

- For STR, don't do the freezer thing.

- For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

Quoted 4 lines

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Ugh ... "character devices" ... that's a pretty wide statement... there's lots of those and very different one from the other...

Any sane device-driver will have to cope with being suspended in a "live" system. I've demonstrated multiple times in the past why this is necessary anyway, for things like dynamic power management, among others.

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:

Quoted 10 lines · Oliver Neukum

Quoted 9 lines

Am Dienstag, 3. Juli 2007 schrieb Benjamin Herrenschmidt:

Quoted 7 lines

So to summarize, the plan that makes things work with fuse is:

- For STR, don't do the freezer thing.

- For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of

Quoted 7 lines

Quoted 4 lines

At the risk of repeating myself. Character device drivers are written with the assumption that normal io and suspend/resume do not race with each other due to the freezer.
What do you intend to do about that?

Ugh ... "character devices" ... that's a pretty wide statement... there's lots of those and very different one from the other...

That is a good summary of the problem ;-(

Quoted 4 lines

Any sane device-driver will have to cope with being suspended in a "live" system. I've demonstrated multiple times in the past why this is necessary anyway, for things like dynamic power management, among others.

That is an interesting notion. I'd rather see device drivers reporting their devices idle and requsting to be suspended.
But in any case it doesn't solve the problem.

Quoted 8 lines · Nigel Cunningham

Quoted 6 lines

So I think Matthew is totally right. In fact, the presence of the freezer is the main reason why Paulus so far NACKed Johannes attempts at merging the PPC PM code with the generic code in kernel/power.c

We've been doing fine without it so far and intend to continue to do so.

Fuse depends on !PPC?

No, that's not what I'm saying. I'm saying we've been doing STR without the freezer and that's the way to go imho.

Quoted 5 lines

Quoted 3 lines

As for suspend-to-disk, I refer you to the discussions we had in the past with Linus, where he explains I think quite clearly how wrong the current implementation of STR is :-)

I assume you mean STD.

Oops, yeah, sorry.

Quoted 3 lines

The problem there is that Linus doesn't care about STD.
If he did, I dare say he'd think through the issues more thoroughly than he apparently has.

Heh, that might be the case :-)

Quoted 6 lines

Quoted 2 lines

Thing is, if you're going to do snapshots, you should probably not sync after you have "frozen" anyway.

Fully agree. But how do you stop things syncing while you're writing the image if you don't have a freezer or equivalent? (scheduler based, kexec.. they're all workarounds for this issue).

Well, I was saying that in the context of the -current- snapshotting mechanism which is based on the freezer, then you should not sys_sync().

Some random user or kernel thread doing a sync is not a problem. It will stop in the middle of sync and resume on wakeup.

The problem is currently because STD -itself- attempts to sync after it has frozen things.

I think that should be changed. If you want to sync for whatever reason, (mostly save RAM ?) do it before the freeze. That means you may get new dirty data in memory that isn't written out by the sync before you freeze, but that's allright, that data will be in the suspend image anyway. If you fail to wakeup, that's akin to a normal crash, the user will only lose the last data written at the time of the suspend and journaling fs'es should take care of fs metadata integrity.

So to summarize, the plan that makes things work with fuse is:

- For STR, don't do the freezer thing.

- For STD, don't sys_sync() after you froze

There might be -other- issues, but that should get you through some of them at least. Of course, you'll be in trouble if you try to do things like STD-to-a-file which sits on a fuse FS but there's a limit to insanity :-)

Still, can you please read this post from Alan Stern:

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

? I don't think I'm able to repeat the arguments given in there in a convincing way.

Still, can you please read this post from Alan Stern:

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

? I don't think I'm able to repeat the arguments given in there in a convincing way.

Quoted 6 lines · Rafael J. Wysocki

Still, can you please read this post from Alan Stern:

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

? I don't think I'm able to repeat the arguments given in there in a convincing way.

As I read it, Alan basically has two objections:
(1) drivers shouldn't need to worry about this
(2) suspend should be transparent to userspace

His proposed solution (freezing tasks when they cross the kernel boundary) helps for the s-t-r case, but in fact doesn't solve (1) because devices can be suspended at runtime

and then you certainly do not want to freeze tasks that try to access the device.

(2) is related but not identical, what if you have a device suspended at runtime and some tasks tries to access it; should the task block until you wake up that device?

I think the core of the discussion isn't appreciated by everybody here yet---we need to solve both run-time and suspend-to-ram-time device suspend, not just one of them.

Quoted 3 lines

His proposed solution (freezing tasks when they cross the kernel boundary) helps for the s-t-r case, but in fact doesn't solve (1) because devices can be suspended at runtime

This is a different thing and a different infrastructure is needed for it (not present at the moment).

Quoted 6 lines

and then you certainly do not want to freeze tasks that try to access the device.

(2) is related but not identical, what if you have a device suspended at runtime and some tasks tries to access it; should the task block until you wake up that device?

I think the device should be woken up in that case.

Quoted 3 lines

I think the core of the discussion isn't appreciated by everybody here yet---we need to solve both run-time and suspend-to-ram-time device suspend, not just one of them.

For now, we're discussing the suspend-to-ram-time suspend only, for which we have (some) infrastrcuture (and which should be supported by all drivers, IMO).

Quoted 6 lines · Rafael J. Wysocki

Still, can you please read this post from Alan Stern:

https://lists.linux-foundation.org/pipermail/linux-pm/2007-June/012847.html

? I don't think I'm able to repeat the arguments given in there in a convincing way.

As I read it, Alan basically has two objections:
(1) drivers shouldn't need to worry about this
(2) suspend should be transparent to userspace

His proposed solution (freezing tasks when they cross the kernel boundary) helps for the s-t-r case, but in fact doesn't solve (1) because devices can be suspended at runtime and then you certainly do not want to freeze tasks that try to access the device.

(2) is related but not identical, what if you have a device suspended at runtime and some tasks tries to access it; should the task block until you wake up that device?

I think the core of the discussion isn't appreciated by everybody here yet---we need to solve both run-time and suspend-to-ram-time device suspend, not just one of them.

Time for me to jump in.

USB already implements runtime PM. If a device is suspended at runtime and a task tries to access it, the device is automatically resumed. No problem there.

The problem comes when the system is doing a STR. Right now the code doesn't keep track of the difference between a runtime suspend and a system suspend -- once the device is suspended, it's suspended, period. Consequently, a non-frozen user task trying to do I/O to a suspended device during STR will cause that device to resume, thereby forcing the system suspend to abort. Something much like this has actually happened and been reported as a bug on LKML (I don't have a URL handy, and it was actually a non-frozen kernel thread interfering with hibernate rather than a non-frozen user task interfering with STR, but the principle is the same).

Yes, the code could be changed to keep track of the reason for a device suspend. But that just raises the old problem of what to do when there's an I/O request for a suspended device during STR.

Quoted 3 lines

I think the core of the discussion isn't appreciated by everybody here yet---we need to solve both run-time and suspend-to-ram-time device suspend, not just one of them.

Runtime suspend isn't a problem. Only STR.

Consider a particularly troublesome case: During STR, a non-frozen task writes to /sys/bus/BBB/drivers/DDD/bind. The sysfs core grabs the device semaphore and calls the driver's probe routine. If the driver isn't PM-aware it simply tries to initialize the device and fails because the device is already suspended. That's no good; it isn't transparent.

So assume the driver is PM-aware. It tries to resume the device, which fails because STR is underway. Now what can it do? There's only one
possibility: It must block until the resume call can succeed. But when is that?

It has to be before the PM core tries to resume the device, because the core will try to acquire the device semaphore and will block waiting for the probe call to complete. But it has to be after the PM core resumes the device's parent, because obviously the device can't resume until its parent is awake.

As you can see, this is a very difficult problem to solve.

Time for me to jump in.

USB already implements runtime PM. If a device is suspended at runtime and a task tries to access it, the device is automatically resumed. No problem there.

The problem comes when the system is doing a STR. Right now the code doesn't keep track of the difference between a runtime suspend and a system suspend -- once the device is suspended, it's suspended, period. Consequently, a non-frozen user task trying to do I/O to a suspended device during STR will cause that device to resume, thereby forcing the system suspend to abort. Something much like this has actually happened and been reported as a bug on LKML (I don't have a URL handy, and it was actually a non-frozen kernel thread interfering with hibernate rather than a non-frozen user task interfering with STR, but the principle is the same).

Yes, the code could be changed to keep track of the reason for a device suspend. But that just raises the old problem of what to do when there's an I/O request for a suspended device during STR.

Quoted 3 lines

I think the core of the discussion isn't appreciated by everybody here yet---we need to solve both run-time and suspend-to-ram-time device suspend, not just one of them.

Runtime suspend isn't a problem. Only STR.

Consider a particularly troublesome case: During STR, a non-frozen task writes to /sys/bus/BBB/drivers/DDD/bind. The sysfs core grabs the device semaphore and calls the driver's probe routine. If the driver isn't PM-aware it simply tries to initialize the device and fails because the device is already suspended. That's no good; it isn't transparent.

So assume the driver is PM-aware. It tries to resume the device, which fails because STR is underway. Now what can it do? There's only one
possibility: It must block until the resume call can succeed. But when is that?

It has to be before the PM core tries to resume the device, because the core will try to acquire the device semaphore and will block waiting for the probe call to complete. But it has to be after the PM core resumes the device's parent, because obviously the device can't resume until its parent is awake.

As you can see, this is a very difficult problem to solve.

Quoted 1 line · Alan Stern

Time for me to jump in.

:)

Quoted 3 lines

USB already implements runtime PM. If a device is suspended at runtime and a task tries to access it, the device is automatically resumed. No problem there.

Right.

Quoted 10 lines

The problem comes when the system is doing a STR. Right now the code doesn't keep track of the difference between a runtime suspend and a system suspend -- once the device is suspended, it's suspended, period. Consequently, a non-frozen user task trying to do I/O to a suspended device during STR will cause that device to resume, thereby forcing the system suspend to abort. Something much like this has actually happened and been reported as a bug on LKML (I don't have a URL handy, and it was actually a non-frozen kernel thread interfering with hibernate rather than a non-frozen user task interfering with STR, but the principle is the same).

Yeah, I can see that happen.

Quoted 9 lines

Yes, the code could be changed to keep track of the reason for a device suspend. But that just raises the old problem of what to do when there's an I/O request for a suspended device during STR.

Quoted 3 lines

I think the core of the discussion isn't appreciated by everybody here yet---we need to solve both run-time and suspend-to-ram-time device suspend, not just one of them.

Runtime suspend isn't a problem. Only STR.

Ah but for all those character devices people were saying are the problem we haven't even solved runtime suspend as far as I can tell from the discussion.

Quoted 18 lines

Consider a particularly troublesome case: During STR, a non-frozen task writes to /sys/bus/BBB/drivers/DDD/bind. The sysfs core grabs the device semaphore and calls the driver's probe routine. If the driver isn't PM-aware it simply tries to initialize the device and fails because the device is already suspended. That's no good; it isn't transparent.

So assume the driver is PM-aware. It tries to resume the device, which fails because STR is underway. Now what can it do? There's only one
possibility: It must block until the resume call can succeed. But when is that?

It has to be before the PM core tries to resume the device, because the core will try to acquire the device semaphore and will block waiting for the probe call to complete. But it has to be after the PM core resumes the device's parent, because obviously the device can't resume until its parent is awake.

Quoted 1 line

As you can see, this is a very difficult problem to solve.

Indeed. Actually, one could argue that it's impossible to solve the problem as long as we try to call out to userspace during suspend and need to wait until that's finished, like in the case of sys_sync() and fuse filesystems, and probably other cases. Maybe we should make *those* calls return a failure so that the suspend isn't transparent inside the kernel but is transparent to userspace.

Quoted 6 lines

Indeed. Actually, one could argue that it's impossible to solve the problem as long as we try to call out to userspace during suspend and need to wait until that's finished, like in the case of sys_sync() and fuse filesystems, and probably other cases. Maybe we should make *those* calls return a failure so that the suspend isn't transparent inside the kernel but is transparent to userspace.

Well, it generally needs more consideration. :-)

I think that we should introduce mechanisms that will allow us to notify all kernel subsystems, including FUSE and similar, that the system is going to enter a sleep state (one of those is the notifier chain introduced recently).

Then, they may react to such a notification by entering a "suspend" mode of operation in which they will return errors from some callbacks that otherwise should have succeeded etc. That depends on the subsystem in question.

Quoted 7 lines

Quoted 6 lines

Indeed. Actually, one could argue that it's impossible to solve the problem as long as we try to call out to userspace during suspend and need to wait until that's finished, like in the case of sys_sync() and fuse filesystems, and probably other cases. Maybe we should make *those* calls return a failure so that the suspend isn't transparent inside the kernel but is transparent to userspace.

Quoted 5 lines

Well, it generally needs more consideration. :-)

I think that we should introduce mechanisms that will allow us to notify all kernel subsystems, including FUSE and similar, that the system is going to enter a sleep state (one of those is the notifier chain introduced recently).

Ugh, please no.

Believe me, fuse is doing _nothing_ out of the ordinary, and should not need special treatment during suspend/resume. If suspend itself is doing something that triggers fuse activity, then that's a bug, such as the sync() thing that started this thread.

Quoted 4 lines

Then, they may react to such a notification by entering a "suspend" mode of operation in which they will return errors from some callbacks that otherwise should have succeeded etc. That depends on the subsystem in question.

Sounds horrible.

Why do we need to deal with subsystem interdependencies during suspend? Isn't it about saving device state to ram?

That definitely _should not_ need to trigger anything that touches filesystems or other subsystems.

Quoted 7 lines

Quoted 6 lines

Indeed. Actually, one could argue that it's impossible to solve the problem as long as we try to call out to userspace during suspend and need to wait until that's finished, like in the case of sys_sync() and fuse filesystems, and probably other cases. Maybe we should make *those* calls return a failure so that the suspend isn't transparent inside the kernel but is transparent to userspace.

Quoted 5 lines

Well, it generally needs more consideration. :-)

I think that we should introduce mechanisms that will allow us to notify all kernel subsystems, including FUSE and similar, that the system is going to enter a sleep state (one of those is the notifier chain introduced recently).

Ugh, please no.

Believe me, fuse is doing _nothing_ out of the ordinary, and should not need special treatment during suspend/resume. If suspend itself is doing something that triggers fuse activity, then that's a bug, such as the sync() thing that started this thread.

Quoted 4 lines

Then, they may react to such a notification by entering a "suspend" mode of operation in which they will return errors from some callbacks that otherwise should have succeeded etc. That depends on the subsystem in question.

Sounds horrible.

Why do we need to deal with subsystem interdependencies during suspend? Isn't it about saving device state to ram?

That definitely _should not_ need to trigger anything that touches filesystems or other subsystems.

Quoted 1 line

Runtime suspend isn't a problem. Only STR.

Ah but for all those character devices people were saying are the problem we haven't even solved runtime suspend as far as I can tell from the discussion.

Quoted 1 line

As you can see, this is a very difficult problem to solve.

Indeed. Actually, one could argue that it's impossible to solve the problem as long as we try to call out to userspace during suspend and need to wait until that's finished, like in the case of sys_sync() and fuse filesystems, and probably other cases. Maybe we should make *those* calls return a failure so that the suspend isn't transparent inside the kernel but is transparent to userspace.

Quoted 1 line

Runtime suspend isn't a problem. Only STR.

Ah but for all those character devices people were saying are the problem we haven't even solved runtime suspend as far as I can tell from the discussion.

Quoted 1 line

As you can see, this is a very difficult problem to solve.

Indeed. Actually, one could argue that it's impossible to solve the problem as long as we try to call out to userspace during suspend and need to wait until that's finished, like in the case of sys_sync() and fuse filesystems, and probably other cases. Maybe we should make *those* calls return a failure so that the suspend isn't transparent inside the kernel but is transparent to userspace.

I disagree. The problem isn't the kernel calling userspace; it's userspace trying to do I/O at a time when everything is supposed to be quiescing. Detecting that and blocking it in drivers is hard and error-prone; preventing it by freezing userspace is easy and cheap.

The reasons why the PPC people dislike the whole idea aren't clear to me.

If it were necessary to have some user task running in order to carry out the STR then their objection would make sense -- obviously that task couldn't do its job if it were frozen. But it isn't necessary, or at least it should not be.

Userspace will be effectively "frozen" while the system as a whole is suspended. So what's wrong with freezing it a little early? Despite Ben's comments, it seems to me that the freezer doesn't hide problems -- it prevents them.

Now people may claim that the freezer implementation itself is buggy. I wouldn't dispute it. But the bugs should be fixable; nobody has pointed out anything fundamentally wrong with the idea AFAICT.

Quoted 4 lines · Alan Stern

I disagree. The problem isn't the kernel calling userspace; it's userspace trying to do I/O at a time when everything is supposed to be quiescing. Detecting that and blocking it in drivers is hard and error-prone; preventing it by freezing userspace is easy and cheap.

And unreliable, and prone to deadlocks, and invasive - requiring changes to kernel threads that have nothing to do with drivers or suspend/resume.

Quoted 2 lines

The reasons why the PPC people dislike the whole idea aren't clear to me.

Our experience is that it isn't necessary. It's extra code that in practice causes deadlocks and added maintenance burden for no discernable benefit.

The freezer doesn't achieve its stated goal of preventing drivers from getting I/O requests after suspend, since kernel threads can (and do) initiate I/O. So then we say that some kernel threads need to be frozen and others don't, but making that decision is difficult and error-prone.

In fact I believe that making a distinction between user and kernel threads is wrong and likely to lead to problems, since userspace can be involved in doing I/O (e.g. FUSE or the user-space driver framework). So the argument of the previous paragraph also applies to some userspace processes.

I remain convinced that the right approach is to fix the drivers to do one of two things; either do something in the suspend call to block further requests to the device, or use a late-suspend call to put their device into a low-power state. Of course, correctly-written frameworks can do a lot to help the chipset drivers here.

Quoted 4 lines · Alan Stern

I disagree. The problem isn't the kernel calling userspace; it's userspace trying to do I/O at a time when everything is supposed to be quiescing. Detecting that and blocking it in drivers is hard and error-prone; preventing it by freezing userspace is easy and cheap.

And unreliable, and prone to deadlocks, and invasive - requiring changes to kernel threads that have nothing to do with drivers or suspend/resume.

Quoted 2 lines

The reasons why the PPC people dislike the whole idea aren't clear to me.

Our experience is that it isn't necessary. It's extra code that in practice causes deadlocks and added maintenance burden for no discernable benefit.

The freezer doesn't achieve its stated goal of preventing drivers from getting I/O requests after suspend, since kernel threads can (and do) initiate I/O. So then we say that some kernel threads need to be frozen and others don't, but making that decision is difficult and error-prone.

In fact I believe that making a distinction between user and kernel threads is wrong and likely to lead to problems, since userspace can be involved in doing I/O (e.g. FUSE or the user-space driver framework). So the argument of the previous paragraph also applies to some userspace processes.

I remain convinced that the right approach is to fix the drivers to do one of two things; either do something in the suspend call to block further requests to the device, or use a late-suspend call to put their device into a low-power state. Of course, correctly-written frameworks can do a lot to help the chipset drivers here.

Let's agree the kernel threads and the freezer are a separate issue.

In the most recent kernels, the freezer does not suspend kernel threads by default.

I agree the kernel threads which try to do I/O during a suspend will need extra attention. However if these threads are necessary for the suspend procedure, then blocking them (which is how people on this thread have been saying driver should treat I/O requests during a suspend) will cause additional problems. There's no way around it; these threads _will_ require more work.

Quoted 6 lines

Quoted 2 lines

The reasons why the PPC people dislike the whole idea aren't clear to me.

Our experience is that it isn't necessary. It's extra code that in practice causes deadlocks and added maintenance burden for no discernable benefit.

I have discussed the benefits elsewhere. As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

Userspace cannot do I/O directly on its own, apart from some exceptional situations where a privileged task directly twiddles some I/O ports or the equivalent.

There remains the problem of user tasks whose assistance is required to carry out some I/O (as with FUSE). If the I/O can be deferred until after the resume, then there's no problem. If the I/O can be carried out before the suspend, then it should be. And finally, if the I/O must be done during the suspend, you're in real trouble -- how do you do I/O to a suspended device?

Quoted 5 lines

I remain convinced that the right approach is to fix the drivers to do one of two things; either do something in the suspend call to block further requests to the device, or use a late-suspend call to put their device into a low-power state. Of course, correctly-written frameworks can do a lot to help the chipset drivers here.

The first alternative is a possibility. My argument all along has been that it is difficult and error-prone, and it adds more overhead to system operation (even when not suspending!) than simply freezing userspace.

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Now process A is unfreezable.  We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Now process A is unfreezable.  We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 16 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Quoted 2 lines

Now process A is unfreezable.  We cannot make rename() restartable, hence it cannot be interruptible.

Then this is a problem specific to fuse. You should teach fuse to block suspension while such operations are being performed.

Quoted 16 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Quoted 2 lines

Now process A is unfreezable.  We cannot make rename() restartable, hence it cannot be interruptible.

Then this is a problem specific to fuse. You should teach fuse to block suspension while such operations are being performed.

Quoted 19 lines

Quoted 18 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Now process A is unfreezable.  We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 2 lines

Then this is a problem specific to fuse. You should teach fuse to block suspension while such operations are being performed.

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

Quoted 19 lines

Quoted 18 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

 - process A calls rename() on a fuse fs
 - process B, the fuse server, starts to process the rename request
 - process B is frozen before it can reply

Now process A is unfreezable.  We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 2 lines

Then this is a problem specific to fuse. You should teach fuse to block suspension while such operations are being performed.

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Quoted 11 lines

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

And flag every other process, that the flagged process is communicating with? How are you proposing to do that?

Quoting Paul:

"1. The freezer cannot be guaranteed deadlock-free without constructing
a dependency graph between tasks (both user and kernel), which is virtually impossible since the dependencies are not externally observable."

Quoted 11 lines

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

And flag every other process, that the flagged process is communicating with? How are you proposing to do that?

Quoting Paul:

"1. The freezer cannot be guaranteed deadlock-free without constructing
a dependency graph between tasks (both user and kernel), which is virtually impossible since the dependencies are not externally observable."

Quoted 15 lines

Quoted 11 lines

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

And flag every other process, that the flagged process is communicating with? How are you proposing to do that?

Quoted 6 lines

Quoting Paul:

"1. The freezer cannot be guaranteed deadlock-free without constructing
a dependency graph between tasks (both user and kernel), which is virtually impossible since the dependencies are not externally observable."

A deadlock requires that the circular wait is uninterruptible. Normal IPC isn't.

What are you doing in the userland portions of fuse? Some kind of IPC with other tasks?

There is a limit to which you can push kernel functionality into user space.

Quoted 15 lines

Quoted 15 lines

Quoted 11 lines

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

And flag every other process, that the flagged process is communicating with? How are you proposing to do that?

Quoted 6 lines

Quoted 6 lines

Quoting Paul:

"1. The freezer cannot be guaranteed deadlock-free without constructing
a dependency graph between tasks (both user and kernel), which is virtually impossible since the dependencies are not externally observable."

Quoted 5 lines

A deadlock requires that the circular wait is uninterruptible. Normal IPC isn't.

What are you doing in the userland portions of fuse? Some kind of IPC with other tasks?

Anything, writing to a file, writing to shared memory, sending things over the network. There's no limit to what a filesystem daemon may do. It's a perfectly ordinary unprivileged userspace process. And this is a feature not a bug.

Quoted 2 lines

There is a limit to which you can push kernel functionality into user space.

Limiting what a userspace filesystem can do would defeat the whole purpose of the bloody thing. This is not negotiable ;)

Quoted 2 lines

Limiting what a userspace filesystem can do would defeat the whole purpose of the bloody thing. This is not negotiable ;)

Which doesn't change the fact that FUSE _is_ special, because it adds dependencies between processed that were not present before.

Quoted 5 lines

Quoted 2 lines

Limiting what a userspace filesystem can do would defeat the whole purpose of the bloody thing. This is not negotiable ;)

Which doesn't change the fact that FUSE _is_ special, because it adds dependencies between processed that were not present before.

OK, fuse is special. So is the userspace driver framework (UIO) proposed by Greg KH and co. Now what can be done about these?

- making them not-special is not an option due to the established interfaces, which don't allow restartability.

- fixing the freezer is pretty much impossible because the dependencies between the tasks cannot be known.

- removing the freezer and fixing the drivers seems workable, we already have a prototype in the form of the powermac architecture.

It seems pretty clear cut. Whining about how much problems this will cause won't get us nearer to a solution.

Quoted 2 lines

It seems pretty clear cut.  Whining about how much problems this will cause won't get us nearer to a solution.

Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)

Quoted 2 lines

It seems pretty clear cut.  Whining about how much problems this will cause won't get us nearer to a solution.

Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)

If, at a minimum, we can determine that we can STD without a freezer. It makes no sense to invest a lot of work to face the same problem again with STD.

If, at a minimum, we can determine that we can STD without a freezer. It makes no sense to invest a lot of work to face the same problem again with STD.

Quoted 3 lines · Oliver Neukum

If, at a minimum, we can determine that we can STD without a freezer. It makes no sense to invest a lot of work to face the same problem again with STD.

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

Quoted 5 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

The basic model is that nobody's really described a use-case where we actually care about restoring system state. What people want is to be able to restore application state.

Quoted 5 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

The basic model is that nobody's really described a use-case where we actually care about restoring system state. What people want is to be able to restore application state.

Hmm, careful. There are a bunch of people who use suspend2 exactly because it saves and restores the page cache, leaving the system in a usable state without waiting for the universe to swap back in from disk. It makes a big difference on older laptops with slow drives. While the other advantages you list for process cryogenics are pretty neat, let's remember that the 99% use case for STD is laptops.

Hmm, careful. There are a bunch of people who use suspend2 exactly because it saves and restores the page cache, leaving the system in a usable state without waiting for the universe to swap back in from disk. It makes a big difference on older laptops with slow drives. While the other advantages you list for process cryogenics are pretty neat, let's remember that the 99% use case for STD is laptops.

Quoted 4 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

Quoted 1 line

You can just describe it, as far as I'm concerned. :-)

The basic model is that nobody's really described a use-case where we actually care about restoring system state. What people want is to be able to restore application state. So, arguably, what we want isn't to save the entire kernel state and application state in one go because we can reconstruct a huge amount of that afterwards.

This isn't too much of a problem. All we actually need to be able to do is to atomically dump process state (which requires the freezer, but doesn't require freezing the entire system),

shut down, get the system back into approximately the correct state (remount filesystems, start X, whatever) and then restore the processes.

Now, obviously, there's actually quite a lot of complexity here that I'm neatly eliding :) The biggest issue is restoring hardware state. We'd require quite a different model to the existing one, but I think there are arguments there for it being helpful anywy. Keeping state in the midlevels rather than the low-level drivers would give us much more ability to deal with hardware issues, and potentially allow the replacement of faulty hardware without userspace caring (freeze your mission-critical application, hotplug the network card, let the kernel restore state and resume it)

There's other advantages to this. As long as the kernel hasn't changed too much it would be possible to restore userspace across kernel security upgrades. You end up saving less to disk so performance should be better.

Touching filesystems between suspend and resume doesn't result in the entire world ending.

I've mocked up a basic implementation using cryopid, but it's somewhat limited by the lack of support for sockets. I'd like to move more of the smarts into the kernel (Hurray, checkpointing!) and then see how much hardware support ends up horifically broken.

Quoted 4 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

Quoted 1 line

You can just describe it, as far as I'm concerned. :-)

The basic model is that nobody's really described a use-case where we actually care about restoring system state. What people want is to be able to restore application state. So, arguably, what we want isn't to save the entire kernel state and application state in one go because we can reconstruct a huge amount of that afterwards.

This isn't too much of a problem. All we actually need to be able to do is to atomically dump process state (which requires the freezer, but doesn't require freezing the entire system),

shut down, get the system back into approximately the correct state (remount filesystems, start X, whatever) and then restore the processes.

Now, obviously, there's actually quite a lot of complexity here that I'm neatly eliding :) The biggest issue is restoring hardware state. We'd require quite a different model to the existing one, but I think there are arguments there for it being helpful anywy. Keeping state in the midlevels rather than the low-level drivers would give us much more ability to deal with hardware issues, and potentially allow the replacement of faulty hardware without userspace caring (freeze your mission-critical application, hotplug the network card, let the kernel restore state and resume it)

There's other advantages to this. As long as the kernel hasn't changed too much it would be possible to restore userspace across kernel security upgrades. You end up saving less to disk so performance should be better.

Touching filesystems between suspend and resume doesn't result in the entire world ending.

I've mocked up a basic implementation using cryopid, but it's somewhat limited by the lack of support for sockets. I'd like to move more of the smarts into the kernel (Hurray, checkpointing!) and then see how much hardware support ends up horifically broken.

Quoted 5 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

Quoted 5 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

Quoted 5 lines · Rafael J. Wysocki

Quoted 2 lines · Matthew Garrett

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

You can just describe it, as far as I'm concerned. :-)

The basic model is that nobody's really described a use-case where we actually care about restoring system state. What people want is to be able to restore application state. So, arguably, what we want isn't to save the entire kernel state and application state in one go because we can reconstruct a huge amount of that afterwards.

I've mocked up a basic implementation using cryopid, but it's somewhat limited by the lack of support for sockets. I'd like to move more of the smarts into the kernel (Hurray, checkpointing!) and then see how much hardware support ends up horifically broken.

Quoted 3 lines · Oliver Neukum

If, at a minimum, we can determine that we can STD without a freezer. It makes no sense to invest a lot of work to face the same problem again with STD.

I have a model for STD that avoids the need to freeze the entirity of userspace, but I need to find some more time to flesh it out.

Am Donnerstag, 5. Juli 2007 schrieb Rafael J. Wysocki:

Quoted 4 lines

Quoted 2 lines

It seems pretty clear cut.  Whining about how much problems this will cause won't get us nearer to a solution.

Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)

If, at a minimum, we can determine that we can STD without a freezer.

It makes no sense to invest a lot of work to face the same problem again with STD.

Am Donnerstag, 5. Juli 2007 schrieb Rafael J. Wysocki:

Quoted 4 lines

Quoted 2 lines

It seems pretty clear cut.  Whining about how much problems this will cause won't get us nearer to a solution.

Yes, that's pretty clear cut, but we should start from fixing the drivers. :-)

If, at a minimum, we can determine that we can STD without a freezer.

It makes no sense to invest a lot of work to face the same problem again with STD.

Quoted 5 lines

Quoted 2 lines

Limiting what a userspace filesystem can do would defeat the whole purpose of the bloody thing. This is not negotiable ;)

Which doesn't change the fact that FUSE _is_ special, because it adds dependencies between processed that were not present before.

OK, fuse is special. So is the userspace driver framework (UIO) proposed by Greg KH and co. Now what can be done about these?

- making them not-special is not an option due to the established interfaces, which don't allow restartability.

- fixing the freezer is pretty much impossible because the dependencies between the tasks cannot be known.

- removing the freezer and fixing the drivers seems workable, we already have a prototype in the form of the powermac architecture.

It seems pretty clear cut. Whining about how much problems this will cause won't get us nearer to a solution.

Quoted 15 lines

Quoted 15 lines

Quoted 11 lines

Quoted 5 lines

And teach VFS to block suspension, while waiting on a mutex held by another process performing a fuse operation.

I can already hear the beautiful praise from Al Viro at the sight of that ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

And flag every other process, that the flagged process is communicating with? How are you proposing to do that?

Quoted 6 lines

Quoted 6 lines

Quoting Paul:

"1. The freezer cannot be guaranteed deadlock-free without constructing
a dependency graph between tasks (both user and kernel), which is virtually impossible since the dependencies are not externally observable."

Quoted 5 lines

A deadlock requires that the circular wait is uninterruptible. Normal IPC isn't.

What are you doing in the userland portions of fuse? Some kind of IPC with other tasks?

Anything, writing to a file, writing to shared memory, sending things over the network. There's no limit to what a filesystem daemon may do. It's a perfectly ordinary unprivileged userspace process. And this is a feature not a bug.

Quoted 2 lines

There is a limit to which you can push kernel functionality into user space.

Limiting what a userspace filesystem can do would defeat the whole purpose of the bloody thing. This is not negotiable ;)

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Quoted 6 lines

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Yet another ugly hack to work around the fact that the freezer cannot work reliably ... yuck

Why not spend that energy fixing drivers to properly block requests instead ?

Quoted 6 lines

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Yet another ugly hack to work around the fact that the freezer cannot work reliably ... yuck

Why not spend that energy fixing drivers to properly block requests instead ?

Quoted 6 lines

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Yet another ugly hack to work around the fact that the freezer cannot work reliably ... yuck

Why not spend that energy fixing drivers to properly block requests instead ?

Quoted 6 lines

There is that.

OK, bite the bullet. Tasks involved in fuse are special. Give them a flag and teach the freezer to put them on ice only after all other task are frozen. In a way they are kernel, there's no use denying that.

Yet another ugly hack to work around the fact that the freezer cannot work reliably ... yuck

Why not spend that energy fixing drivers to properly block requests instead ?

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"ON" == Oliver Neukum <[email protected]> writes:

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"ON" == Oliver Neukum <[email protected]> writes:

ON> Because we will be unable to escape that job. Let's assume that we ON> remove the freezer from the STR path. The next complaint would be ON> that we cannot do STD with fuse. "Then don't do that" would not be ON> taken kindly as answer.

Ah, we are back to keeping STR broken in order to maybe get STD working. STR is much more interesting than STD.

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 16 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Quoted 2 lines

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Yes, we are claiming fuse is very special in this regard, and perhaps even broken.

Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks unkillable (even for SIGKILL!) forever.

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

So no, you can't claim "FUSE is just IPC". It is very special IPC.

Quoted 16 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Quoted 2 lines

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Yes, we are claiming fuse is very special in this regard, and perhaps even broken.

Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks unkillable (even for SIGKILL!) forever.

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

So no, you can't claim "FUSE is just IPC". It is very special IPC.

Quoted 16 lines

Quoted 16 lines

Quoted 11 lines

Quoted 8 lines

Quoted 3 lines

I have discussed the benefits elsewhere.  As for the deadlocks -- do you still observe them if you use the version of the freezer which doesn't freeze kernel threads?

In general the only way to guarantee there are no deadlocks is to construct the graph of dependencies between tasks.  Those dependencies are not in practice observable from outside the tasks, so it is virtually impossible to construct the graph.

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Quoted 8 lines

Quoted 2 lines

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Yes, we are claiming fuse is very special in this regard, and perhaps even broken.

Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks unkillable (even for SIGKILL!) forever.

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

Quoted 2 lines

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous and non-restartable. That's just how the UNIX filesystem API works and is hardly a bug in fuse.

Quoted 1 line

So no, you can't claim "FUSE is just IPC". It is very special IPC.

I did say it's special. Sure, it has some "interesting" properties, and with a bit of malice you can do very ugly things with it. If you are interested, read Documentation/filesystems/fuse.txt, especially the "Tricky deadlock" section ;)

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

Quoted 2 lines

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would make suspend failure less likely. But making problems less easy to reproduce is not a good thing.

Quoted 6 lines

Quoted 2 lines

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would make suspend failure less likely. But making problems less easy to reproduce is not a good thing.

So, how about eliminating them?

Quoted 6 lines

Quoted 2 lines

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would make suspend failure less likely. But making problems less easy to reproduce is not a good thing.

So, how about eliminating them?

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would make suspend failure less likely. But making problems less easy to reproduce is not a good thing.

So, how about eliminating them?

That can't be done just within fuse, a process might be sleeping on a VFS mutex. Do we want to hack VFS as well?

I guess I know your answer. But it ain't gonna work. Suspend code really doesn't belong in VFS, and I'm pretty sure the maintainers of that little piece of code would agree with me on this.

Quoted 4 lines

I guess I know your answer. But it ain't gonna work. Suspend code really doesn't belong in VFS, and I'm pretty sure the maintainers of that little piece of code would agree with me on this.

Surprise, surprise. Not that I'm scared of the VFS maintainers, though. ;-)

Quoted 4 lines

I guess I know your answer. But it ain't gonna work. Suspend code really doesn't belong in VFS, and I'm pretty sure the maintainers of that little piece of code would agree with me on this.

Surprise, surprise. Not that I'm scared of the VFS maintainers, though. ;-)

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would make suspend failure less likely. But making problems less easy to reproduce is not a good thing.

So, how about eliminating them?

That can't be done just within fuse, a process might be sleeping on a VFS mutex. Do we want to hack VFS as well?

I guess I know your answer. But it ain't gonna work. Suspend code really doesn't belong in VFS, and I'm pretty sure the maintainers of that little piece of code would agree with me on this.

Quoted 2 lines

Don't you think, however, that it can be modified a little to play well, for example, with the freezer?

I could stick a couple of try_to_freeze()s into fuse, and that would make suspend failure less likely. But making problems less easy to reproduce is not a good thing.

Quoted 10 lines

Quoted 9 lines

Quoted 2 lines

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 5 lines

Yes, we are claiming fuse is very special in this regard, and perhaps even broken.

Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks unkillable (even for SIGKILL!) forever.

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

Quoted 2 lines

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous and non-restartable. That's just how the UNIX filesystem API works and is hardly a bug in fuse.

Quoted 10 lines

Quoted 9 lines

Quoted 2 lines

In which way can user space tasks depend on each other in a way that allows a them members of that cycle to be in uninterruptible sleep?

- process A calls rename() on a fuse fs
- process B, the fuse server, starts to process the rename request
- process B is frozen before it can reply

Now process A is unfreezable. We cannot make rename() restartable, hence it cannot be interruptible.

Quoted 5 lines

Yes, we are claiming fuse is very special in this regard, and perhaps even broken.

Let's see. If I SIGSTOP the fuse server, I can get unrelated tasks unkillable (even for SIGKILL!) forever.

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

Quoted 2 lines

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous and non-restartable. That's just how the UNIX filesystem API works and is hardly a bug in fuse.

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

Plus, it would be nice to find out where suspend/hibernation is triggering fuse activity. We can then decide where to fix it -- in fuse or in suspend parts. You said sys_sync is not implemented... so where is the problem?

Quoted 6 lines

Quoted 2 lines

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous and non-restartable. That's just how the UNIX filesystem API works and is hardly a bug in fuse.

Well, unix is not plan9, and maybe userland filesystems are impossible in unix. But that is hardly a bug in unix :-).

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem. Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen. And it is not possible to know this ordering.

And even if the ordering were solved, the freezer would still not work if the filesystem is not responding due to external events, such as a lost network (this affects NFS, CIFS, whatever just the same as fuse).

Quoted 4 lines

Plus, it would be nice to find out where suspend/hibernation is triggering fuse activity. We can then decide where to fix it -- in fuse or in suspend parts. You said sys_sync is not implemented... so where is the problem?

I cannot say without having a sysrq-t of the situation.

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous and non-restartable. That's just how the UNIX filesystem API works and is hardly a bug in fuse.

Quoted 2 lines

Well, unix is not plan9, and maybe userland filesystems are impossible in unix. But that is hardly a bug in unix :-).

I'd rather say, reliable suspend to ram is impossible in the presense of userspace filesystems, iff people are too lazy to fix the suspend framework and the drivers to work without the freezer.

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem. Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen. And it is not possible to know this ordering.

And even if the ordering were solved, the freezer would still not work if the filesystem is not responding due to external events, such as a lost network (this affects NFS, CIFS, whatever just the same as fuse).

Quoted 4 lines

Plus, it would be nice to find out where suspend/hibernation is triggering fuse activity. We can then decide where to fix it -- in fuse or in suspend parts. You said sys_sync is not implemented... so where is the problem?

I cannot say without having a sysrq-t of the situation.

Quoted 7 lines

Quoted 6 lines

Quoted 2 lines

That's very special, and maybe even a FUSE bug. And that is also what makes FUSE special w.r.t. s2ram.

What makes fuse special is that some file operations are synchronous and non-restartable. That's just how the UNIX filesystem API works and is hardly a bug in fuse.

Quoted 2 lines

Well, unix is not plan9, and maybe userland filesystems are impossible in unix. But that is hardly a bug in unix :-).

I'd rather say, reliable suspend to ram is impossible in the presense of userspace filesystems, iff people are too lazy to fix the suspend framework and the drivers to work without the freezer.

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Quoted 12 lines

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Quoted 7 lines

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

Quoted 19 lines

Quoted 18 lines

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Quoted 5 lines

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

How probable is this situation?

Quoted 19 lines

Quoted 19 lines

Quoted 18 lines

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Quoted 7 lines

Quoted 5 lines

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

How probable is this situation?

I guess it depends on usage patterns.

I don't remember seeing any such cases, even though I have a permanent fuse mount, and I suspend regularly. But the fs is probably totally idle during suspend in my case.

But some people use fuse as a home or root filesystem and in those cases it can become quite likely to cause problems.

Quoted 19 lines

Quoted 19 lines

Quoted 18 lines

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Quoted 7 lines

Quoted 5 lines

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

How probable is this situation?

I guess it depends on usage patterns.

I don't remember seeing any such cases, even though I have a permanent fuse mount, and I suspend regularly. But the fs is probably totally idle during suspend in my case.

But some people use fuse as a home or root filesystem and in those cases it can become quite likely to cause problems.

Quoted 12 lines

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Quoted 7 lines

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

Quoted 12 lines

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

Actually fuse allows SIGKILL, because it's always fatal, and the syscall may not be restarted.

I think you want to stick try_to_freeze() at the same places where you do SIGKILL handling. That should solve the 'syslogd is unfreezeable' problem.

I could, but it would not solve the general problem.  Namely, that the presence of fuse imposes a certain ordering in which userspace tasks have to be frozen.  And it is not possible to know this ordering.

Quoted 7 lines

Actually, why do you need this? There is no absolute need that you finish the request. You must either finish the request or let yourself be frozen.

A quick look through fuse reveals principally request_wait_answer() And maybe a few other places. Is there some hidden reason you cannot handle being frozen here?

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

Quoted 5 lines

Yes, fuse could handle being frozen there. However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited. You must never ever block on a lock one of your clients is holding. In this case the limitation is not influenced by the freezer.

The freezer introduces a further limitation in that the server can freeze before the client, which must not be. You can prevent that by freezing the servers last.

In principle you might have dependencies between servers and you won't catch that, true. You won't catch servers blocking on IPC, but you are balancing on the edge of deadlock with fuse anyway.

Quoted 11 lines

Quoted 5 lines

Yes, fuse could handle being frozen there.  However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited. You must never ever block on a lock one of your clients is holding. In this case the limitation is not influenced by the freezer.

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

Quoted 11 lines

Quoted 5 lines

Yes, fuse could handle being frozen there.  However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited. You must never ever block on a lock one of your clients is holding. In this case the limitation is not influenced by the freezer.

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

Am Donnerstag, 5. Juli 2007 schrieb Miklos Szeredi:

Quoted 16 lines

Quoted 11 lines

Quoted 5 lines

Yes, fuse could handle being frozen there.  However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited. You must never ever block on a lock one of your clients is holding. In this case the limitation is not influenced by the freezer.

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

Quoted 3 lines

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Quoted 7 lines

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Quoted 7 lines

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Quoted 9 lines

Quoted 8 lines

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

Quoted 3 lines

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Just to make things more complicated... Since a server isn't restricted in what it can do, what happens when one server depends on another server?

Quoted 9 lines

Quoted 8 lines

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

Quoted 3 lines

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Just to make things more complicated... Since a server isn't restricted in what it can do, what happens when one server depends on another server?

Quoted 14 lines · Oliver Neukum

Quoted 12 lines

Quoted 8 lines

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Quoted 3 lines

Just to make things more complicated... Since a server isn't restricted in what it can do, what happens when one server depends on another server?

The same principle applies. If you really want that you can solve this by freezing servers in the reverse sequence they were started.

The main point remains. If you have a circular dependency anywhere among the servers you can deadlock independent of the freezer.

Quoted 16 lines

Quoted 11 lines

Quoted 5 lines

Yes, fuse could handle being frozen there.  However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited. You must never ever block on a lock one of your clients is holding. In this case the limitation is not influenced by the freezer.

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

Quoted 3 lines

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

Quoted 16 lines

Quoted 11 lines

Quoted 5 lines

Yes, fuse could handle being frozen there.  However that would only solve part of the problem: an operation waiting for a reply could be holding a VFS mutex and some other task may be blocked on that mutex.

How would you solve freezing those tasks?

OK, you made me reach for literatur on theoretical computer science.

IMHO the range of actions a fuse server is inherently limited. You must never ever block on a lock one of your clients is holding. In this case the limitation is not influenced by the freezer.

Obviously.  But I wasn't about the server trying to acquire a lock held by a client.  I was talking about a client trying to acquire a lock held by _another_ client.

Quoted 3 lines

If this coincides with the server (or some other task which the server is depending on) being frozen before the clients, the freezer has a problem.

True, but that case can only happen if servers are frozen before clients. You don't need a full dependency graph. A simple set sequence of two classes of tasks will do.

I fear, that your efforts to "save" the freezer are in vain. It is already moderately hackish with that PF_FREEZER_SKIP and the kernel dotted randomly with try_to_freeze() calls, but adding bandaids to try to order freezing userspace processes in the right order would just make it a horrible mess.

I fear, that your efforts to "save" the freezer are in vain. It is already moderately hackish with that PF_FREEZER_SKIP and the kernel dotted randomly with try_to_freeze() calls, but adding bandaids to try to order freezing userspace processes in the right order would just make it a horrible mess.

Quoted 5 lines

I fear, that your efforts to "save" the freezer are in vain. It is already moderately hackish with that PF_FREEZER_SKIP and the kernel dotted randomly with try_to_freeze() calls, but adding bandaids to try to order freezing userspace processes in the right order would just make it a horrible mess.

I agree that bandaids won't work. What's needed is something more radical. Things like FUSE must be written so that the kernel parts _can_ freeze even while they are waiting for a response from a user thread.

Quoted 6 lines

I fear, that your efforts to "save" the freezer are in vain. It is already moderately hackish with that PF_FREEZER_SKIP and the kernel dotted randomly with try_to_freeze() calls, but adding bandaids to try to order freezing userspace processes in the right order would just make it a horrible mess.

I agree that bandaids won't work. What's needed is something more radical. Things like FUSE must be written so that the kernel parts _can_ freeze even while they are waiting for a response from a user thread.

Quoted 6 lines

I fear, that your efforts to "save" the freezer are in vain. It is already moderately hackish with that PF_FREEZER_SKIP and the kernel dotted randomly with try_to_freeze() calls, but adding bandaids to try to order freezing userspace processes in the right order would just make it a horrible mess.

I agree that bandaids won't work. What's needed is something more radical. Things like FUSE must be written so that the kernel parts _can_ freeze even while they are waiting for a response from a user thread.

Am Donnerstag, 5. Juli 2007 schrieb Alan Stern:

Quoted 8 lines · Miklos Szeredi

Quoted 6 lines

I fear, that your efforts to "save" the freezer are in vain. It is already moderately hackish with that PF_FREEZER_SKIP and the kernel dotted randomly with try_to_freeze() calls, but adding bandaids to try to order freezing userspace processes in the right order would just make it a horrible mess.

Quoted 4 lines

I agree that bandaids won't work. What's needed is something more radical. Things like FUSE must be written so that the kernel parts _can_ freeze even while they are waiting for a response from a user thread.

OK, some radical ideas.

In principle we want a deadlock here. Tasks that are frozen due to fuse are as good as frozen, there's no need to formally freeze them. The bad uninterruptible tasks are those waiting for hardware.

Can we detect the difference? Perhaps we can label those crucial locks and when only tasks in the refrigerator and tasks waiting on these locks are left we are content and go on to suspending the device tree.

The only reason (I know of) why we don't handle uninterruptible tasks in the freezer is that we're afraid of the suspend process deadlocking with an uninterruptible task holding a lock, but AFAICS the probability of such an event is extremely small.

Also, the powermac suspend will deadlock in such cases, so the fact that it doesn't deadlock means that they don't occur very often (if at all).

Quoted 5 lines

The only reason (I know of) why we don't handle uninterruptible tasks in the freezer is that we're afraid of the suspend process deadlocking with an uninterruptible task holding a lock, but AFAICS the probability of such an event is extremely small.

What would deadlock specifically ? One of the drivers trying to acquire that lock ? It would be a driver bug then.

Quoted 5 lines

The only reason (I know of) why we don't handle uninterruptible tasks in the freezer is that we're afraid of the suspend process deadlocking with an uninterruptible task holding a lock, but AFAICS the probability of such an event is extremely small.

What would deadlock specifically ? One of the drivers trying to acquire that lock ? It would be a driver bug then.

Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:

Quoted 9 lines · Rafael J. Wysocki

Quoted 5 lines

The only reason (I know of) why we don't handle uninterruptible tasks in the freezer is that we're afraid of the suspend process deadlocking with an uninterruptible task holding a lock, but AFAICS the probability of such an event is extremely small.

What would deadlock specifically ? One of the drivers trying to acquire that lock ? It would be a driver bug then.

Your driver's write method looks like:

mutex_lock();
poke_some_hardware();
wait_event_uninterruptible(); //for result
res = evaluate_result();
mutex_unlock();
return res;

If you put a task into the refrigerator at wait_event_interruptible() you will deadlock if you need this lock for the driver to go to suspend. The suspend method then must not take the lock _and_ it must be aware that there may be an ongoing operation.

you will deadlock if you need this lock for the driver to go to suspend. The suspend method then must not take the lock _and_ it must be aware that there may be an ongoing operation.

Moreover, I claim that, in the context of your example, _if_ the task is stuck
at the wait_event_uninterruptible(), _then_ the freezerless suspend will
deadlock with the task.

Am Freitag, 6. Juli 2007 schrieb Benjamin Herrenschmidt:

Quoted 9 lines · Rafael J. Wysocki

Quoted 5 lines

The only reason (I know of) why we don't handle uninterruptible tasks in the freezer is that we're afraid of the suspend process deadlocking with an uninterruptible task holding a lock, but AFAICS the probability of such an event is extremely small.

What would deadlock specifically ? One of the drivers trying to acquire that lock ? It would be a driver bug then.

Your driver's write method looks like:

mutex_lock();
poke_some_hardware();
wait_event_uninterruptible(); //for result
res = evaluate_result();
mutex_unlock();
return res;

If you put a task into the refrigerator at wait_event_interruptible() you will deadlock if you need this lock for the driver to go to suspend. The suspend method then must not take the lock _and_ it must be aware that there may be an ongoing operation.