Skip to contentSkip to conversation

freeze vs freezer

DISCUSSION Jeremy Fitzhardinge 14 participants 50 messages Nov 22, 2007 – Jul 3, 2008

lore
Share & tools
lore
#0 lore raw

It seems that a process blocked in a write to an xfs filesystem due to xfs_freeze cannot be frozen by the freezer.

I see this if I suspend my laptop while doing something xfs-filesystem intensive, like a kernel build. My suspend scripts freeze the XFS filesystem (as Dave said I should), which presumably blocks some writer, and then the freezer times out and fails to complete.

Here's part of the process dump the freezer does when it times out:

cc1 D 00000000 0 18138 18137

       dd5f1e24 00200082 00000002 00000000 ecdeeb00 ecdeec64 c200f280 00000001 
       009c09a0 dd5f1e0c dd5f1e0c 0000000f 00000000 00000000 00000000 dd5f1e74 
       c7beb480 dd5f1e88 dd5f1ea8 c0228d97 e8889540 dd5f1e38 c015b75d dd5f1e44 
Call Trace:
 [<c0228d97>] xfs_write+0xf4/0x6d9
 [<c0226038>] xfs_file_aio_write+0x53/0x5b
 [<c0171c15>] do_sync_write+0xae/0xec
 [<c0172343>] vfs_write+0xa4/0x120
 [<c01728d7>] sys_write+0x3b/0x60
 [<c0106fae>] sysenter_past_esp+0x6b/0xa1
 =======================

I haven't looked at how to fix this yet. I only just worked out why I was getting suspend failures.

J

#1 lore raw
Quoted 2 lines · Jeremy Fitzhardinge

It seems that a process blocked in a write to an xfs filesystem due to xfs_freeze cannot be frozen by the freezer.

The freezer doesn't handle tasks in TASK_UNINTERRUPTIBLE and I don't know how to make it handle them without at least partially defeating its purpose.

Quoted 21 lines

I see this if I suspend my laptop while doing something xfs-filesystem intensive, like a kernel build. My suspend scripts freeze the XFS filesystem (as Dave said I should), which presumably blocks some writer, and then the freezer times out and fails to complete.

Here's part of the process dump the freezer does when it times out:

cc1 D 00000000 0 18138 18137

       dd5f1e24 00200082 00000002 00000000 ecdeeb00 ecdeec64 c200f280 00000001 
       009c09a0 dd5f1e0c dd5f1e0c 0000000f 00000000 00000000 00000000 dd5f1e74 
       c7beb480 dd5f1e88 dd5f1ea8 c0228d97 e8889540 dd5f1e38 c015b75d dd5f1e44 
Call Trace:
 [<c0228d97>] xfs_write+0xf4/0x6d9
 [<c0226038>] xfs_file_aio_write+0x53/0x5b
 [<c0171c15>] do_sync_write+0xae/0xec
 [<c0172343>] vfs_write+0xa4/0x120
 [<c01728d7>] sys_write+0x3b/0x60
 [<c0106fae>] sysenter_past_esp+0x6b/0xa1
 =======================
Quoted 2 lines

I haven't looked at how to fix this yet. I only just worked out why I was getting suspend failures.

Well, you can add freezer_do_not_count()/freezer_count() annotations to xfs_write() (and whatever else is blocked as a result of the XFS being frozen).

Generally, that would be risky without the freezing of XFS, however, because it might leak us filesystem data to a storage device after creating a hibernation image which would result in the filesystem corruption after the resume.

Still, if you only suspend to RAM, that should be safe.

Greetings,
Rafael

#2 lore raw
Quoted 9 lines · Rafael J. Wysocki
Quoted 3 lines · Jeremy Fitzhardinge

It seems that a process blocked in a write to an xfs filesystem due to xfs_freeze cannot be frozen by the freezer.

The freezer doesn't handle tasks in TASK_UNINTERRUPTIBLE and I don't know how to make it handle them without at least partially defeating its purpose.

Well, I guess the question is whether an xfs-frozen writer really needs to be UNINTERRUPTIBLE from the freezer's perspective (clearly it does from usermode's perspective - filesystem writes just don't return EINTR).

Quoted 1 line

From a quick poke around, it looks to me like freezing is actually

implemented in the VFS layer rather than in XFS itself: is that right? Could vfs_check_frozen() be changed to something that is freezer-compatible?

Quoted 21 lines
Quoted 21 lines

I see this if I suspend my laptop while doing something xfs-filesystem intensive, like a kernel build. My suspend scripts freeze the XFS filesystem (as Dave said I should), which presumably blocks some writer, and then the freezer times out and fails to complete.

Here's part of the process dump the freezer does when it times out:

cc1 D 00000000 0 18138 18137

       dd5f1e24 00200082 00000002 00000000 ecdeeb00 ecdeec64 c200f280 00000001 
       009c09a0 dd5f1e0c dd5f1e0c 0000000f 00000000 00000000 00000000 dd5f1e74 
       c7beb480 dd5f1e88 dd5f1ea8 c0228d97 e8889540 dd5f1e38 c015b75d dd5f1e44 
Call Trace:
 [<c0228d97>] xfs_write+0xf4/0x6d9
 [<c0226038>] xfs_file_aio_write+0x53/0x5b
 [<c0171c15>] do_sync_write+0xae/0xec
 [<c0172343>] vfs_write+0xa4/0x120
 [<c01728d7>] sys_write+0x3b/0x60
 [<c0106fae>] sysenter_past_esp+0x6b/0xa1
 =======================
Quoted 7 lines
Quoted 3 lines

I haven't looked at how to fix this yet. I only just worked out why I was getting suspend failures.

Well, you can add freezer_do_not_count()/freezer_count() annotations to xfs_write() (and whatever else is blocked as a result of the XFS being frozen).

What would be the implications of that? Would that just prevent freezing while there's something blocked there?

Quoted 6 lines

Generally, that would be risky without the freezing of XFS, however, because it might leak us filesystem data to a storage device after creating a hibernation image which would result in the filesystem corruption after the resume.

Still, if you only suspend to RAM, that should be safe.

I specifically added it because I was getting data loss due to crashes during suspend/resume problems. It's been pretty stable lately, but I may as well remove the xfs_freeze from my suspend scripts if this is the solution.

I think the broader issue is that there's no reason in principle why something blocked due to xfs-freezing (or vfs freezing) should prevent the freezer from completing.

J

#3 lore raw
Quoted 11 lines · Jeremy Fitzhardinge
Quoted 9 lines · Rafael J. Wysocki
Quoted 3 lines · Jeremy Fitzhardinge

It seems that a process blocked in a write to an xfs filesystem due to xfs_freeze cannot be frozen by the freezer.

The freezer doesn't handle tasks in TASK_UNINTERRUPTIBLE and I don't know how to make it handle them without at least partially defeating its purpose.

Quoted 6 lines

Well, I guess the question is whether an xfs-frozen writer really needs to be UNINTERRUPTIBLE from the freezer's perspective (clearly it does from usermode's perspective - filesystem writes just don't return EINTR).

From a quick poke around, it looks to me like freezing is actually implemented in the VFS layer rather than in XFS itself: is that right?

I don't know the details.

Quoted 1 line

Could vfs_check_frozen() be changed to something that is freezer-compatible?

That seems doable in principle. I'll have a closer look at it.

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines

I see this if I suspend my laptop while doing something xfs-filesystem intensive, like a kernel build. My suspend scripts freeze the XFS filesystem (as Dave said I should), which presumably blocks some writer, and then the freezer times out and fails to complete.

Here's part of the process dump the freezer does when it times out:

cc1 D 00000000 0 18138 18137

       dd5f1e24 00200082 00000002 00000000 ecdeeb00 ecdeec64 c200f280 00000001 
       009c09a0 dd5f1e0c dd5f1e0c 0000000f 00000000 00000000 00000000 dd5f1e74 
       c7beb480 dd5f1e88 dd5f1ea8 c0228d97 e8889540 dd5f1e38 c015b75d dd5f1e44 
Call Trace:
 [<c0228d97>] xfs_write+0xf4/0x6d9
 [<c0226038>] xfs_file_aio_write+0x53/0x5b
 [<c0171c15>] do_sync_write+0xae/0xec
 [<c0172343>] vfs_write+0xa4/0x120
 [<c01728d7>] sys_write+0x3b/0x60
 [<c0106fae>] sysenter_past_esp+0x6b/0xa1
 =======================

I haven't looked at how to fix this yet. I only just worked out why I was getting suspend failures.

Quoted 6 lines
Quoted 3 lines

Well, you can add freezer_do_not_count()/freezer_count() annotations to xfs_write() (and whatever else is blocked as a result of the XFS being frozen).

What would be the implications of that? Would that just prevent freezing while there's something blocked there?

The freezer will not wait for this particular task. Still, the task will have TIF_FREEZE set, so it will freeze as soon as freezer_count() is reached, unless the thawing of tasks is carried out first.

If used in the right place, it's reasonably safe, but we need to know what the right place is. [That's how we handle vfork(), BTW.]

Quoted 11 lines
Quoted 6 lines

Generally, that would be risky without the freezing of XFS, however, because it might leak us filesystem data to a storage device after creating a hibernation image which would result in the filesystem corruption after the resume.

Still, if you only suspend to RAM, that should be safe.

I specifically added it because I was getting data loss due to crashes during suspend/resume problems. It's been pretty stable lately, but I may as well remove the xfs_freeze from my suspend scripts if this is the solution.

Not exactly. :-)

Quoted 3 lines

I think the broader issue is that there's no reason in principle why something blocked due to xfs-freezing (or vfs freezing) should prevent the freezer from completing.

Agreed, but the only way to tell the freezer "don't wait for this task", if the task in question is in TASK_UNINTERRUPTIBLE, is to annotate it.

Greetings,
Rafael

#4 lore raw
In reply to Rafael J. Wysocki
Quoted 6 lines · Rafael J. Wysocki
Quoted 2 lines · Jeremy Fitzhardinge

It seems that a process blocked in a write to an xfs filesystem due to xfs_freeze cannot be frozen by the freezer.

The freezer doesn't handle tasks in TASK_UNINTERRUPTIBLE and I don't know how to make it handle them without at least partially defeating its purpose.

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

Quoted 21 lines
Quoted 21 lines

I see this if I suspend my laptop while doing something xfs-filesystem intensive, like a kernel build. My suspend scripts freeze the XFS filesystem (as Dave said I should), which presumably blocks some writer, and then the freezer times out and fails to complete.

Here's part of the process dump the freezer does when it times out:

cc1 D 00000000 0 18138 18137

       dd5f1e24 00200082 00000002 00000000 ecdeeb00 ecdeec64 c200f280 00000001 
       009c09a0 dd5f1e0c dd5f1e0c 0000000f 00000000 00000000 00000000 dd5f1e74 
       c7beb480 dd5f1e88 dd5f1ea8 c0228d97 e8889540 dd5f1e38 c015b75d dd5f1e44 
Call Trace:
 [<c0228d97>] xfs_write+0xf4/0x6d9
 [<c0226038>] xfs_file_aio_write+0x53/0x5b
 [<c0171c15>] do_sync_write+0xae/0xec
 [<c0172343>] vfs_write+0xa4/0x120
 [<c01728d7>] sys_write+0x3b/0x60
 [<c0106fae>] sysenter_past_esp+0x6b/0xa1
 =======================
Quoted 5 lines
Quoted 2 lines

I haven't looked at how to fix this yet. I only just worked out why I was getting suspend failures.

Well, you can add freezer_do_not_count()/freezer_count() annotations to xfs_write() (and whatever else is blocked as a result of the XFS being frozen).

May as well annotate the whole VFS, then, because once the transaction subsystem is frozen any operation that modifies the filesystem will get blocked like this.

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#5 lore raw
Quoted 10 lines · David Chinner
Quoted 6 lines · Rafael J. Wysocki
Quoted 2 lines · Jeremy Fitzhardinge

It seems that a process blocked in a write to an xfs filesystem due to xfs_freeze cannot be frozen by the freezer.

The freezer doesn't handle tasks in TASK_UNINTERRUPTIBLE and I don't know how to make it handle them without at least partially defeating its purpose.

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines

I see this if I suspend my laptop while doing something xfs-filesystem intensive, like a kernel build. My suspend scripts freeze the XFS filesystem (as Dave said I should), which presumably blocks some writer, and then the freezer times out and fails to complete.

Here's part of the process dump the freezer does when it times out:

cc1 D 00000000 0 18138 18137

       dd5f1e24 00200082 00000002 00000000 ecdeeb00 ecdeec64 c200f280 00000001 
       009c09a0 dd5f1e0c dd5f1e0c 0000000f 00000000 00000000 00000000 dd5f1e74 
       c7beb480 dd5f1e88 dd5f1ea8 c0228d97 e8889540 dd5f1e38 c015b75d dd5f1e44 
Call Trace:
 [<c0228d97>] xfs_write+0xf4/0x6d9
 [<c0226038>] xfs_file_aio_write+0x53/0x5b
 [<c0171c15>] do_sync_write+0xae/0xec
 [<c0172343>] vfs_write+0xa4/0x120
 [<c01728d7>] sys_write+0x3b/0x60
 [<c0106fae>] sysenter_past_esp+0x6b/0xa1
 =======================

I haven't looked at how to fix this yet. I only just worked out why I was getting suspend failures.

Quoted 6 lines
Quoted 2 lines

Well, you can add freezer_do_not_count()/freezer_count() annotations to xfs_write() (and whatever else is blocked as a result of the XFS being frozen).

May as well annotate the whole VFS, then, because once the transaction subsystem is frozen any operation that modifies the filesystem will get blocked like this.

Well, I don't know how this mechanism actually works, so I can't comment.

Is there a mutex on which tasks block if the filesystem is frozen?

Greetings,
Rafael

#6 lore raw
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines · David Chinner

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

Signature · 1 line
Matthew Garrett | [email protected]
#7 lore raw
Quoted 9 lines · Matthew Garrett
Quoted 5 lines · Rafael J. Wysocki
Quoted 2 lines · David Chinner

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Greetings,
Rafael

#8 lore raw
Quoted 5 lines · Rafael J. Wysocki
Quoted 4 lines · Matthew Garrett
Quoted 3 lines · Rafael J. Wysocki
Quoted 1 line · David Chinner

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

When all tasks are preemptable we can very trivially rely on the drivers to enforce the "Stop new IO submission" with a dirt-simple semaphore or waitqueue. The sleep itself will be TASK_UNINTERRUPTIBLE, but it will be done from a preemptible context.

That way the system suspend time is the sum of the suspend times of the devices on the system, and the suspend time of any given device is the sum of its maximum non-preemptible critical section and the time to flush all of its remaining pending DMA/etc. This is almost completely independent of the load-level of the machine, and it does not depend on things like NFS filesystems. The one gotcha is that it does not flush dirty filesystem pages to disk first, although that could be fixed with a few VFS and blockdev hooks which hierarchically flush and "freeze" block devices and filesystems before actually disabling devices much the way that device-mapper can pause a device to take a snapshot and end up with a clean journal on the filesystem afterwards.

Cheers,
Kyle Moffett

#9 lore raw
Quoted 16 lines · Kyle Moffett
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Matthew Garrett
Quoted 4 lines · Rafael J. Wysocki
Quoted 2 lines · David Chinner

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Quoted 8 lines

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Well, this is more-or-less how we all imagine that should be done eventually.

The main problem is how to implement it without causing too much breakage. Also, there are some dirty details that need to be taken into consideration.

Quoted 4 lines

When all tasks are preemptable we can very trivially rely on the drivers to enforce the "Stop new IO submission" with a dirt-simple semaphore or waitqueue. The sleep itself will be TASK_UNINTERRUPTIBLE, but it will be done from a preemptible context.

If there are any drivers that make their devices available via mmap(), that won't be sufficient.

Probably, we'll need a two iterations over devices to handle all corner cases.

Moreover, for hibernation we need to resume at least some devices in order to save the image, which shouldn't result in unblocking the waiting tasks.

Quoted 4 lines

That way the system suspend time is the sum of the suspend times of the devices on the system, and the suspend time of any given device is the sum of its maximum non-preemptible critical section and the time to flush all of its remaining pending DMA/etc. This is almost

Quoted 8 lines

completely independent of the load-level of the machine, and it does not depend on things like NFS filesystems. The one gotcha is that it does not flush dirty filesystem pages to disk first, although that could be fixed with a few VFS and blockdev hooks which hierarchically flush and "freeze" block devices and filesystems before actually disabling devices much the way that device-mapper can pause a device to take a snapshot and end up with a clean journal on the filesystem afterwards.

Yes, I generally agree.

Greetings,
Rafael

#10 lore raw
Quoted 5 lines · Rafael J. Wysocki

Well, this is more-or-less how we all imagine that should be done eventually.

The main problem is how to implement it without causing too much breakage. Also, there are some dirty details that need to be taken into consideration.

For Xen suspend/resume, I'd like to use the freezer to get all threads into a known consistent state (where, specifically, they don't have any outstanding pagetable updates pending). In other words, the freezer as it currently stands is what I want, modulo some of these issues where it gets caught up unexpectedly. If threads end up getting frozen anywhere preempt isn't explicitly disabled, it wouldn't work for me.

J

#11 lore raw
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 3 lines · Rafael J. Wysocki

Well, this is more-or-less how we all imagine that should be done eventually.

The main problem is how to implement it without causing too much breakage. Also, there are some dirty details that need to be taken into consideration.

For Xen suspend/resume, I'd like to use the freezer to get all threads into a known consistent state (where, specifically, they don't have any outstanding pagetable updates pending). In other words, the freezer as it currently stands is what I want, modulo some of these issues where it gets caught up unexpectedly. If threads end up getting frozen anywhere preempt isn't explicitly disabled, it wouldn't work for me.

The problem with "one freezer" is that "known consistent state" means something completely different to every single driver and subsystem. Xen wants it to mean "No pending page table updates and no more updates from this point forward". A network driver wants it to mean "All pending network packets DMAed out or in and the device shut down with all remaining packets queued. A SATA controller wants it to mean "All DMA quiesced and no more commands", etc.

The only way to have that work is to put minimal definitions of what state you care about in the drivers themselves. For Xen this means that you need to have an appropriately-timed suspend handler which hooks into Xen code very precisely to create and preserve the "No pending page table updates" state that you care about. It will be more work in the short term but it's the only maintainable solution in the long term IMO.

Cheers,
Kyle Moffett

#12 lore raw
Quoted 17 lines · Kyle Moffett
Quoted 8 lines · Jeremy Fitzhardinge
Quoted 6 lines · Rafael J. Wysocki

Well, this is more-or-less how we all imagine that should be done eventually.

The main problem is how to implement it without causing too much breakage. Also, there are some dirty details that need to be taken into consideration.

Quoted 7 lines

For Xen suspend/resume, I'd like to use the freezer to get all threads into a known consistent state (where, specifically, they don't have any outstanding pagetable updates pending). In other words, the freezer as it currently stands is what I want, modulo some of these issues where it gets caught up unexpectedly. If threads end up getting frozen anywhere preempt isn't explicitly disabled, it wouldn't work for me.

Quoted 2 lines

The problem with "one freezer" is that "known consistent state" means something completely different to every single driver and subsystem.

Not really. The freezer puts tasks into a particular well-understood
state: they're either in usermode, or in the kernel in the refrigerator. And since the places which call into the refrigerator are explicit in the source, and not terribly numerous, its easy to audit exactly what the state is at each call.

Quoted 5 lines

Xen wants it to mean "No pending page table updates and no more updates from this point forward". A network driver wants it to mean "All pending network packets DMAed out or in and the device shut down with all remaining packets queued. A SATA controller wants it to mean "All DMA quiesced and no more commands", etc.

Well, those are somewhat different. The existing suspend/resume driver callbacks are sufficient for a device to be in that state. What I want for Xen is more global: I just want to make sure tasks are not preempted in the middle of a state which can't be suspended. The specific details of the state I want are moderately complex, but short lived. The problem with other mechanisms - like stop_machine - is that they can leave threads preempted in one of the states I can't handle, whereas the the freezer is more deterministic.

Quoted 7 lines

The only way to have that work is to put minimal definitions of what state you care about in the drivers themselves. For Xen this means that you need to have an appropriately-timed suspend handler which hooks into Xen code very precisely to create and preserve the "No pending page table updates" state that you care about. It will be more work in the short term but it's the only maintainable solution in the long term IMO.

No, that doesn't really work. Aside from scattering hooks everywhere there's pagetable updates, there's no real existing place to hook into. While I could put those hooks in, they would amount to changing the kernel-internal pagetable update interface for everyone to deal with a corner case of a fairly obscure user - I don't think its a good tradeoff.

The freezer is nice because the state it puts each task into is well-defined, and is well-suited for Xen's use. In fact, I would agree with you that the use I want to put the freezer to better suits it than its current use in suspend/resume.

J

#13 lore raw
In reply to Kyle Moffett

Hi!

Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Quoted 8 lines

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

"each driver" means this is a lot of work. But yes, that is probably way to go, and patch would be welcome.

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#14 lore raw

Hi.

Quoted 19 lines · Pavel Machek

Hi!

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Quoted 8 lines

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Quoted 2 lines

"each driver" means this is a lot of work. But yes, that is probably way to go, and patch would be welcome.

Yes, that does work. It's what I've done in my (preliminary) support for fuse.

Regards,

Nigel

#15 lore raw
Quoted 22 lines · Nigel Cunningham

Hi.

Quoted 19 lines · Pavel Machek

Hi!

Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Quoted 5 lines
Quoted 2 lines

"each driver" means this is a lot of work. But yes, that is probably way to go, and patch would be welcome.

Yes, that does work. It's what I've done in my (preliminary) support for fuse.

Hmm, can you please elaborate a bit?

Rafael

#16 lore raw

Hi.

Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines · Nigel Cunningham
Quoted 16 lines · Pavel Machek
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

"each driver" means this is a lot of work. But yes, that is probably way to go, and patch would be welcome.

Yes, that does work. It's what I've done in my (preliminary) support for fuse.

Hmm, can you please elaborate a bit?

Sorry. I wasn't very unambiguous, was I? And I'm not sure now whether you're meaning "How does fuse support relate to freezing block devices?" or "What's this about fuse support?". Let me therefore seek to answer both questions:

Higher level, I know (filesystems rather than block devices), but I was meaning the general concept of blocking new requests and completing existing ones worked fine for the supposedly impossible fuse support.

Re fuse support, let me start by saying "I know this doesn't handle all situations, but I think it's a good enough proof-of-concept implementation".

I added some simple hooks to the code for submitting new work to fuse threads.

#define FUSE_MIGHT_FREEZE(superblock, desc) \

do { \
       int printed = 0; \
       while(superblock->s_frozen != SB_UNFROZEN) { \
               if (!printed) { \
                       printk("%d frozen in " desc ".\n", current->pid); \
                       printed = 1; \
               } \
               try_to_freeze(); \
               yield(); \
       } \
} while (0)

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

The end result is:

int freeze_processes(void)

{
        int error;

        printk(KERN_INFO "Stopping fuse filesystems.\n");
        freeze_filesystems(FS_FREEZER_FUSE);
        freezer_state = FREEZER_FILESYSTEMS_FROZEN;
        printk(KERN_INFO "Freezing user space processes ... ");
        error = try_to_freeze_tasks(FREEZER_USER_SPACE);
        if (error)
                goto Exit;
        printk(KERN_INFO "done.\n");

        sys_sync();
        printk(KERN_INFO "Stopping normal filesystems.\n");
        freeze_filesystems(FS_FREEZER_NORMAL);
        freezer_state = FREEZER_USERSPACE_FROZEN;
        printk(KERN_INFO "Freezing remaining freezable tasks ... ");
        error = try_to_freeze_tasks(FREEZER_KERNEL_THREADS);
        if (error)
                goto Exit;
        printk(KERN_INFO "done.");
        freezer_state = FREEZER_FULLY_ON;
 Exit:
        BUG_ON(in_atomic());
        printk("\n");
        return error;
}

Sorry if that's more info than you wanted.

Nigel

#17 lore raw

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

How do you deal with fuse server tasks using other fuse filesystems? How does freeze_filesystems() look?

Regards
Oliver

#18 lore raw

Hi.

Quoted 8 lines · Oliver Neukum

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

How do you deal with fuse server tasks using other fuse filesystems?

Since they're frozen in reverse order, the dependant one would be frozen first.

Quoted 1 line

How does freeze_filesystems() look?

Removing my ugly debugging statements, it's currently:

/**
* freeze_filesystems - lock all filesystems and force them into a
consistent
* state
*/
void freeze_filesystems(int which)

{
        struct super_block *sb;

        lockdep_off();

        /*
         * Freeze in reverse order so filesystems dependant upon others are
         * frozen in the right order (eg. loopback on ext3).
         */
        list_for_each_entry_reverse(sb, &super_blocks, s_list) {
                if (sb->s_type->fs_flags & FS_IS_FUSE &&
                    sb->s_frozen == SB_UNFROZEN &&
                    which & FS_FREEZER_FUSE) {
                        sb->s_frozen = SB_FREEZE_TRANS;
                        sb->s_flags |= MS_FROZEN;
                        continue;
                }

                if (!sb->s_root || !sb->s_bdev ||
                    (sb->s_frozen == SB_FREEZE_TRANS) ||
                    (sb->s_flags & MS_RDONLY) ||
                    (sb->s_flags & MS_FROZEN) ||
                    !(which & FS_FREEZER_NORMAL))
                        continue;
                freeze_bdev(sb->s_bdev);
                sb->s_flags |= MS_FROZEN;
        }

        lockdep_on();
}

Nigel

#19 lore raw

Am Donnerstag, 3. Januar 2008 10:52:53 schrieb Nigel Cunningham:

Quoted 10 lines

Hi.

Quoted 7 lines · Oliver Neukum

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

Quoted 4 lines
Quoted 1 line

How do you deal with fuse server tasks using other fuse filesystems?

Since they're frozen in reverse order, the dependant one would be frozen first.

Say I do:

a) mount fuse on /tmp/first
b) mount fuse on /tmp/second

Then the server task for (a) does "ls /tmp/second". So it will be frozen, right? How do you then freeze (a)? And keep in mind that the server task may have forked.

Regards
Oliver

#20 lore raw

Hi.

Quoted 14 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 10:52:53 schrieb Nigel Cunningham:

Quoted 4 lines

Hi.

Quoted 1 line · Oliver Neukum

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

How do you deal with fuse server tasks using other fuse filesystems?

Since they're frozen in reverse order, the dependant one would be frozen first.

Quoted 8 lines

Say I do:

a) mount fuse on /tmp/first
b) mount fuse on /tmp/second

Then the server task for (a) does "ls /tmp/second". So it will be frozen, right? How do you then freeze (a)? And keep in mind that the server task may have forked.

I guess I should first ask, is this a real life problem or a hypothetical twisted web? I don't see why you would want to make two filesystems interdependent - it sounds like the way to create livelock and deadlocks in normal use, before we even begin to think about hibernating.

Regards,

Nigel

#25 lore raw
In reply to Nigel Cunningham
Quoted 93 lines · Nigel Cunningham

Hi.

Quoted 18 lines · Rafael J. Wysocki
Quoted 17 lines · Nigel Cunningham
Quoted 16 lines · Pavel Machek
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

So how do you handle threads that are blocked on I/O or a lock during the system freeze process, then?

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

"each driver" means this is a lot of work. But yes, that is probably way to go, and patch would be welcome.

Yes, that does work. It's what I've done in my (preliminary) support for fuse.

Hmm, can you please elaborate a bit?

Sorry. I wasn't very unambiguous, was I? And I'm not sure now whether you're meaning "How does fuse support relate to freezing block devices?" or "What's this about fuse support?". Let me therefore seek to answer both questions:

Higher level, I know (filesystems rather than block devices), but I was meaning the general concept of blocking new requests and completing existing ones worked fine for the supposedly impossible fuse support.

Re fuse support, let me start by saying "I know this doesn't handle all situations, but I think it's a good enough proof-of-concept implementation".

I added some simple hooks to the code for submitting new work to fuse threads.

#define FUSE_MIGHT_FREEZE(superblock, desc) \

do { \
       int printed = 0; \
       while(superblock->s_frozen != SB_UNFROZEN) { \
               if (!printed) { \
                       printk("%d frozen in " desc ".\n", current->pid); \
                       printed = 1; \
               } \
               try_to_freeze(); \
               yield(); \
       } \
} while (0)

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

The end result is:

int freeze_processes(void)

{
        int error;

        printk(KERN_INFO "Stopping fuse filesystems.\n");
        freeze_filesystems(FS_FREEZER_FUSE);
        freezer_state = FREEZER_FILESYSTEMS_FROZEN;
        printk(KERN_INFO "Freezing user space processes ... ");
        error = try_to_freeze_tasks(FREEZER_USER_SPACE);
        if (error)
                goto Exit;
        printk(KERN_INFO "done.\n");

        sys_sync();
        printk(KERN_INFO "Stopping normal filesystems.\n");
        freeze_filesystems(FS_FREEZER_NORMAL);
        freezer_state = FREEZER_USERSPACE_FROZEN;
        printk(KERN_INFO "Freezing remaining freezable tasks ... ");
        error = try_to_freeze_tasks(FREEZER_KERNEL_THREADS);
        if (error)
                goto Exit;
        printk(KERN_INFO "done.");
        freezer_state = FREEZER_FULLY_ON;
 Exit:
        BUG_ON(in_atomic());
        printk("\n");
        return error;
}
Quoted 1 line

Sorry if that's more info than you wanted.

No, that's fine, thanks.

Greetings,
Rafael

#21 lore raw
In reply to Nigel Cunningham

Am Donnerstag, 3. Januar 2008 23:06:07 schrieb Nigel Cunningham:

Quoted 26 lines

Hi.

Quoted 14 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 10:52:53 schrieb Nigel Cunningham:

Quoted 4 lines

Hi.

Quoted 1 line · Oliver Neukum

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

How do you deal with fuse server tasks using other fuse filesystems?

Since they're frozen in reverse order, the dependant one would be frozen first.

Quoted 8 lines

Say I do:

a) mount fuse on /tmp/first
b) mount fuse on /tmp/second

Then the server task for (a) does "ls /tmp/second". So it will be frozen, right? How do you then freeze (a)? And keep in mind that the server task may have forked.

Quoted 5 lines

I guess I should first ask, is this a real life problem or a hypothetical twisted web? I don't see why you would want to make two filesystems interdependent - it sounds like the way to create livelock and deadlocks in normal use, before we even begin to think about hibernating.

Good questions. I personally don't use fuse, but I do care about power management. The problem I see is that an unprivileged user could make that dependency, even inadvertedly.

Regards
Oliver

#22 lore raw
Quoted 10 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 23:06:07 schrieb Nigel Cunningham:

Quoted 7 lines

Hi.

Quoted 4 lines

a) mount fuse on /tmp/first
b) mount fuse on /tmp/second

Then the server task for (a) does "ls /tmp/second". So it will be frozen, right? How do you then freeze (a)? And keep in mind that the server task may have forked.

I guess I should first ask, is this a real life problem or a hypothetical twisted web? I don't see why you would want to make two filesystems interdependent - it sounds like the way to create livelock and deadlocks in normal use, before we even begin to think about hibernating.

Good questions. I personally don't use fuse, but I do care about power management. The problem I see is that an unprivileged user could make that dependency, even inadvertedly.

I don't think it makes sense for the kernel to try to keep track of hard data dependencies for FUSE filesystems, or to even *attempt* to auto-suspend them. You should instead allow a privileged program to initiate a "freeze-and-flush" operation on a particular FUSE filesystem and optionally wait for it to finish. Then your userspace would be configured with the appropriate data dependencies and would stop FUSE filesystems in the appropriate order.

In addition, the kernel would automatically understand ext3=>loopback=>fuse, and when asked to freeze the "fuse" part, it would first freeze the "ext3" and the "loopback" parts using similar mechanisms as device-mapper currently uses when you do "dmsetup suspend mydev" followed by "echo 0 $SIZE snapshot /dev/mapper/mydev-base /dev/mapper/mydev-snap-back p 8 | dmsetup load mydev" (IE: when you create a snapshot of a given device).

Naturally userspace could deadlock itself (although not the kernel) by freezing a block device and then attempting to access it, but since the "freeze" operation is limited to root this is not a big issue. The way to freeze all filesystems safely would be to clone a new mount namespace, mlockall(), mount a tmpfs, pivot_root() into the tmpfs, bind-mount the filesystems you want to freeze directly onto subdirectories of the tmpfs, and then freeze them in an appropriate order.

Besides which the worst-case is a pretty straightforward non-critical failure; you might fail to fully sync a FUSE filesystem because its daemon is asleep waiting on something (possibly even just sitting in a "sleep(10000)" call with all signals masked). You simply need to make sure that all tasks are asleep outside of driver critical sections so that you can properly suspend your device tree.

Cheers,
Kyle Moffett

#23 lore raw
In reply to Oliver Neukum
Quoted 33 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 23:06:07 schrieb Nigel Cunningham:

Quoted 26 lines

Hi.

Quoted 14 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 10:52:53 schrieb Nigel Cunningham:

Quoted 4 lines

Hi.

Quoted 1 line · Oliver Neukum

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

How do you deal with fuse server tasks using other fuse filesystems?

Since they're frozen in reverse order, the dependant one would be frozen first.

Quoted 8 lines

Say I do:

a) mount fuse on /tmp/first
b) mount fuse on /tmp/second

Then the server task for (a) does "ls /tmp/second". So it will be frozen, right? How do you then freeze (a)? And keep in mind that the server task may have forked.

Quoted 5 lines

I guess I should first ask, is this a real life problem or a hypothetical twisted web? I don't see why you would want to make two filesystems interdependent - it sounds like the way to create livelock and deadlocks in normal use, before we even begin to think about hibernating.

Quoted 3 lines

Good questions. I personally don't use fuse, but I do care about power management. The problem I see is that an unprivileged user could make that dependency, even inadvertedly.

Other problem is that unprivileged user can do it with evil intent. So called "denial-of-service" attack.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#24 lore raw

Hi.

Quoted 27 lines · Pavel Machek
Quoted 26 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 23:06:07 schrieb Nigel Cunningham:

Quoted 25 lines
Quoted 11 lines · Oliver Neukum

Am Donnerstag, 3. Januar 2008 10:52:53 schrieb Nigel Cunningham:

Quoted 10 lines
Quoted 7 lines · Oliver Neukum

Am Donnerstag 03 Januar 2008 schrieb Nigel Cunningham:

Quoted 5 lines

On top of this, I made a (too simple at the moment) freeze_filesystems function which iterates through &super_blocks in reverse order, freezing fuse filesystems or ordinary ones. I say 'too simple' because it doesn't currently allow for the possibility of someone mounting (say) ext3 on fuse, but that would just be an extension of what's already done.

How do you deal with fuse server tasks using other fuse filesystems?

Since they're frozen in reverse order, the dependant one would be frozen first.

Quoted 8 lines

Say I do:

a) mount fuse on /tmp/first
b) mount fuse on /tmp/second

Then the server task for (a) does "ls /tmp/second". So it will be frozen, right? How do you then freeze (a)? And keep in mind that the server task may have forked.

I guess I should first ask, is this a real life problem or a hypothetical twisted web? I don't see why you would want to make two filesystems interdependent - it sounds like the way to create livelock and deadlocks in normal use, before we even begin to think about hibernating.

Quoted 6 lines
Quoted 3 lines

Good questions. I personally don't use fuse, but I do care about power management. The problem I see is that an unprivileged user could make that dependency, even inadvertedly.

Other problem is that unprivileged user can do it with evil intent. So called "denial-of-service" attack.

Only in this case it would be a denial-of-denial-of-service attack, since it would stop you hibernating or suspending :).

This is still all hypothetical. If I could have a real life case where this could actually happen, it would help a lot.

Nigel

#26 lore raw
In reply to Kyle Moffett

Hi!

(replying to *very* old mail).

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Quoted 7 lines

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Patch would be welcome, actually. It turns out blocking new IO-requests is not completely trivial.
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#27 lore raw
Quoted 19 lines · Pavel Machek

(replying to *very* old mail).

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Quoted 7 lines

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Quoted 2 lines

Patch would be welcome, actually. It turns out blocking new IO-requests is not completely trivial.

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#28 lore raw
Quoted 21 lines · Henrique de Moraes Holschuh
Quoted 20 lines · Pavel Machek

(replying to *very* old mail).

Quoted 16 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

We wait until they can continue.

So if I have a process blocked on an unavilable NFS mount, I can't suspend?

That's correct, you can't.

[And I know what you're going to say. ;-)]

Why exactly does suspend/hibernation depend on "TASK_INTERRUPTIBLE" instead of a zero preempt_count()? Really what we should do is just iterate over all of the actual physical devices and tell each one "Block new IO requests preemptably, finish pending DMA, put the hardware in low-power mode, and prepare for suspend/hibernate". As long as each driver knows how to do those simple things we can have an entirely consistent kernel image for both suspend and for hibernation.

Quoted 2 lines
Quoted 2 lines

Patch would be welcome, actually. It turns out blocking new IO-requests is not completely trivial.

Quite. But I'm not sure I see what this is all about yet. From the IDE and SCSI subsystems I remember that they block all I/O from higher levels once the suspend callbacks have been executed. I haven't made an effort to understand the freezer (or indeed anything related to hibernation) yet since I don't even use hibernation myself (only s2ram). Do you have any suggestion where to start reading up on things so I can get an idea what the issues are and what you would like IDE / SCSI / ... to do?

Quoted 5 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Regards,

Elias

#29 lore raw

Hi!

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Patch would be welcome, actually. It turns out blocking new IO-requests is not completely trivial.

Quite. But I'm not sure I see what this is all about yet. From the IDE and SCSI subsystems I remember that they block all I/O from higher levels once the suspend callbacks have been executed. I haven't made an effort to understand the freezer (or indeed anything related to hibernation) yet since I don't even use hibernation myself (only s2ram). Do you have

s2ram also uses freezer these days. Difference is s2ram does not really need it.

Quoted 2 lines

any suggestion where to start reading up on things so I can get an idea what the issues are and what you would like IDE / SCSI / ... to do?

I'd like block layer to block any process that tries to do I/O.

Quoted 5 lines
Quoted 4 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 7 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Actually, I believe requirements are same.

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#30 lore raw
Quoted 13 lines · Pavel Machek
Quoted 5 lines
Quoted 4 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 7 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Quoted 6 lines

Actually, I believe requirements are same.

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#31 lore raw
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Pavel Machek
Quoted 5 lines
Quoted 4 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 7 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Actually, I believe requirements are same.

Quoted 8 lines
Quoted 4 lines

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

I talked with Jens about it on a very general level, but it seems doable at first sight.

Thanks,
Rafael

#32 lore raw
Quoted 11 lines · Rafael J. Wysocki
Quoted 10 lines
Quoted 6 lines

Actually, I believe requirements are same.

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Quoted 6 lines

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

I talked with Jens about it on a very general level, but it seems doable at first sight.

Why would you hack the blok layer when we already have a perfectly fine facility to archive what you want? freeze_bdev is there exactly for the purpose to make the filesystem consistant on disk and then freeze all I/O.

#35 lore raw
In reply to Rafael J. Wysocki
Quoted 22 lines · Rafael J. Wysocki
Quoted 21 lines · Dave Chinner
Quoted 19 lines · Pavel Machek
Quoted 5 lines
Quoted 4 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 7 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Actually, I believe requirements are same.

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

Quoted 7 lines
Quoted 3 lines

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Why? What part of freeze_bdev() doesn't work for you?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#36 lore raw
Quoted 30 lines · Dave Chinner
Quoted 28 lines · Rafael J. Wysocki
Quoted 26 lines · Dave Chinner
Quoted 13 lines · Pavel Machek
Quoted 5 lines
Quoted 5 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 8 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Quoted 7 lines

Actually, I believe requirements are same.

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Quoted 6 lines
Quoted 4 lines

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

J

#37 lore raw
Quoted 32 lines · Jeremy Fitzhardinge
Quoted 31 lines · Dave Chinner
Quoted 29 lines · Rafael J. Wysocki
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Pavel Machek
Quoted 5 lines
Quoted 5 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 8 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Actually, I believe requirements are same.

Quoted 9 lines
Quoted 5 lines

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that? What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#33 lore raw
In reply to Christoph Hellwig
Quoted 16 lines · Christoph Hellwig
Quoted 15 lines · Rafael J. Wysocki
Quoted 10 lines
Quoted 6 lines

Actually, I believe requirements are same.

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 7 lines
Quoted 2 lines

I talked with Jens about it on a very general level, but it seems doable at first sight.

Why would you hack the blok layer when we already have a perfectly fine facility to archive what you want? freeze_bdev is there exactly for the purpose to make the filesystem consistant on disk and then freeze all I/O.

We tried that in the past and it didn't work very well due to some bad interactions with the md layer that we wanted to stay functional while we were saving the image.

Also, do all of the supported filesystems implement this feature?

#38 lore raw
In reply to Dave Chinner
Quoted 33 lines · Dave Chinner
Quoted 32 lines · Jeremy Fitzhardinge
Quoted 31 lines · Dave Chinner
Quoted 29 lines · Rafael J. Wysocki
Quoted 16 lines · Dave Chinner
Quoted 15 lines · Pavel Machek
Quoted 5 lines
Quoted 5 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 8 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Actually, I believe requirements are same.

Quoted 9 lines
Quoted 5 lines

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 5 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that?

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

Quoted 5 lines

What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

This might work too, but it would be nice to do something independent of the freezer, so that we can drop the freezer when we want and not when we are forced to.

Thanks,
Rafael

#39 lore raw
Quoted 16 lines · Rafael J. Wysocki
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 8 lines
Quoted 5 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that?

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

You're suggesting that you let processes trying to do I/O continue until *after* the memory image is taken? How is that going to work? You've got to quiesce the filesystems totally *before* taking an image of memory - it's the only way to guarantee that they are the in-memory state and on disk state are consistent state on resume.

Don't re-invent the wheel - use the API we already have that does exactly what needs to be done.

Quoted 9 lines
Quoted 5 lines

What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

This might work too, but it would be nice to do something independent of the freezer, so that we can drop the freezer when we want and not when we are forced to.

vfs_check_frozen() is completely independent of the process freezer.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#40 lore raw
Quoted 20 lines · Dave Chinner
Quoted 19 lines · Rafael J. Wysocki
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 3 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Seriously, though, how is stopping I/O in the elevator is going to change that?

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

You're suggesting that you let processes trying to do I/O continue until *after* the memory image is taken?

I'm not going to let the data get to the disk.

Quoted 4 lines

How is that going to work?
You've got to quiesce the filesystems totally *before* taking an image of memory - it's the only way to guarantee that they are the in-memory state and on disk state are consistent state on resume.

No, it's not the only way.

We have to ensure that the fs data that did't make it to the disk(s) before creating the snapshot image will not be written to the disk(s) after the image has been created. In theory one can think of many ways to achieve that and the freezing of filesystems is certainly one of those.

Quoted 9 lines

Don't re-invent the wheel - use the API we already have that does exactly what needs to be done.

Quoted 6 lines
Quoted 5 lines

What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

Quoted 5 lines
Quoted 3 lines

This might work too, but it would be nice to do something independent of the freezer, so that we can drop the freezer when we want and not when we are forced to.

vfs_check_frozen() is completely independent of the process freezer.

Well, can you please tell me how exactly that works, then?

Thanks,
Rafael

#41 lore raw
Quoted 24 lines · Rafael J. Wysocki
Quoted 23 lines · Dave Chinner
Quoted 22 lines · Rafael J. Wysocki
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 5 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

You're suggesting that you let processes trying to do I/O continue until *after* the memory image is taken?

I'm not going to let the data get to the disk.

Yes, but you still haven't answered the original question - What are you going to do with sync I/O that leaves a process in D state because you've prevented the I/O from being completed?

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This might work too, but it would be nice to do something independent of the freezer, so that we can drop the freezer when we want and not when we are forced to.

vfs_check_frozen() is completely independent of the process freezer.

Well, can you please tell me how exactly that works, then?

Try looking at the code. When we freeze a filesystem sb->s_frozen changes state depending on the level of freeze currently obtained by the filesystem. And:

#define vfs_check_frozen(sb, level) \
        wait_event((sb)->s_wait_unfrozen, ((sb)->s_frozen < (level)))

Pretty bloody simple, really.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#49 lore raw
In reply to Jeremy Fitzhardinge
Quoted 36 lines · Jeremy Fitzhardinge
Quoted 35 lines · Dave Chinner
Quoted 32 lines · Rafael J. Wysocki
Quoted 17 lines · Dave Chinner
Quoted 15 lines · Pavel Machek
Quoted 5 lines
Quoted 5 lines

Is this the same thing the per-device IO-queue-freeze patches for HDAPS also
need to do? If so, you may want to talk to Elias Oltmanns <[email protected]> about it. Added to CC.

Quoted 8 lines

Thanks for the heads up Henrique. Even though these issues seem to be related up to a certain degree, there probably are some important differences. When suspending a system, the emphasis is on leaving the system in a consistent state (think of journalled file systems), whereas disk shock protection is mainly concerned with stopping I/O as soon as possible. As yet, I cannot possibly say to what extend these two concepts can be reconciled in the sense of sharing some common code.

Actually, I believe requirements are same.

Quoted 9 lines
Quoted 5 lines

'don't do i/o in dangerous period'.

swsusp will just do sync() before entering dangerous period. That provides consistent-enough state...

As I've said many times before - if the requirement is "don't do I/O" then you have to freeze the filesystem. In no way does 'sync' prevent filesystems from doing I/O.....

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

Well, if it is in D state but does not hold any important locks, you can just add "try_to_freeze()" in the place where it is sleeping, right?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#42 lore raw
In reply to Dave Chinner
Quoted 28 lines · Dave Chinner
Quoted 27 lines · Rafael J. Wysocki
Quoted 26 lines · Dave Chinner
Quoted 24 lines · Rafael J. Wysocki
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 5 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that?

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

You're suggesting that you let processes trying to do I/O continue until *after* the memory image is taken?

I'm not going to let the data get to the disk.

Yes, but you still haven't answered the original question - What are you going to do with sync I/O that leaves a process in D state because you've prevented the I/O from being completed?

I don't want to intercept those processes, just allow them to block on that I/O.

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This might work too, but it would be nice to do something independent of the freezer, so that we can drop the freezer when we want and not when we are forced to.

vfs_check_frozen() is completely independent of the process freezer.

Well, can you please tell me how exactly that works, then?

Quoted 8 lines

Try looking at the code. When we freeze a filesystem sb->s_frozen changes state depending on the level of freeze currently obtained by the filesystem. And:

#define vfs_check_frozen(sb, level) \
        wait_event((sb)->s_wait_unfrozen, ((sb)->s_frozen < (level)))

Pretty bloody simple, really.

OK

Do all of the filesystems implement the freezing?

Rafael

#43 lore raw
Quoted 6 lines · Rafael J. Wysocki
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki
Quoted 2 lines · Dave Chinner
Quoted 1 line · Rafael J. Wysocki

On Monday, 30 of June 2008, Dave Chinner wrote:

[...]

Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

What do you do with a sync I/O (read or write)? The process is going to have to go to sleep somewhere in D state waiting for that I/O to complete. If you're going to intercept such processes somewhere else to do something magic, then why not put that magic in vfs_check_frozen()?

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This might work too, but it would be nice to do something independent of the freezer, so that we can drop the freezer when we want and not when we are forced to.

vfs_check_frozen() is completely independent of the process freezer.

Well, can you please tell me how exactly that works, then?

Try looking at the code. When we freeze a filesystem sb->s_frozen changes state depending on the level of freeze currently obtained by the filesystem. And:

Quoted 8 lines
Quoted 4 lines
#define vfs_check_frozen(sb, level) \
        wait_event((sb)->s_wait_unfrozen, ((sb)->s_frozen < (level)))

Pretty bloody simple, really.

OK

Do all of the filesystems implement the freezing?

There is some work in progress [1]. If you think this will help you to address this issue on the fs level, where I think it should be done, you may even be able to request some changes to fit your needs before it gets merged into mainline.

Regards,

Elias

[1] http://permalink.gmane.org/gmane.linux.file-systems/24716

#44 lore raw
Quoted 4 lines · Elias Oltmanns

There is some work in progress [1]. If you think this will help you to address this issue on the fs level, where I think it should be done, you may even be able to request some changes to fit your needs before it gets merged into mainline.

That is just a direct user interface to this functionality. The functionality has been around for a long time.

#47 lore raw
In reply to Rafael J. Wysocki
Quoted 32 lines · Rafael J. Wysocki
Quoted 31 lines · Dave Chinner
Quoted 30 lines · Rafael J. Wysocki
Quoted 28 lines · Dave Chinner
Quoted 16 lines · Rafael J. Wysocki
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 8 lines
Quoted 5 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that?

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

You're suggesting that you let processes trying to do I/O continue until *after* the memory image is taken?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I'm not going to let the data get to the disk.

Yes, but you still haven't answered the original question - What are you going to do with sync I/O that leaves a process in D state because you've prevented the I/O from being completed?

I don't want to intercept those processes, just allow them to block on that I/O.

So you're going to allow them to go to D state somewhere. Ok, so what's the problem with blocking them in vfs_check_frozen(), then?

Quoted 1 line

Do all of the filesystems implement the freezing?

Most of the major ones - those that implement ->write_super_lockfs() should work just fine.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#45 lore raw
In reply to Elias Oltmanns
Quoted 4 lines · Elias Oltmanns
Quoted 1 line · Rafael J. Wysocki

Do all of the filesystems implement the freezing?

There is some work in progress [1].

[....]

Quoted 1 line

[1] http://permalink.gmane.org/gmane.linux.file-systems/24716

No, that's the userspace ioctl interface to enable freezing from something other than dm-snapshot. The filesystems that is aimed at already support freezing via freeze_bdev().

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#48 lore raw
In reply to Dave Chinner
Quoted 41 lines · Dave Chinner
Quoted 32 lines · Rafael J. Wysocki
Quoted 31 lines · Dave Chinner
Quoted 30 lines · Rafael J. Wysocki
Quoted 28 lines · Dave Chinner
Quoted 16 lines · Rafael J. Wysocki
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Jeremy Fitzhardinge
Quoted 5 lines · Dave Chinner
Quoted 3 lines · Rafael J. Wysocki

Well, it seems we can handle this on the block layer level, by temporarily replacing the elevator with something that will selectively prevent fs I/O from reaching the layers below it.

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

Why? What part of freeze_bdev() doesn't work for you?

Well, my original problem - which is still an issue - is that a process writing to a frozen XFS filesystem is stuck in D state, and therefore cannot be frozen as part of suspend.

I thought we were talking about the post-freezer situation.

Quoted 8 lines
Quoted 5 lines

Silly me - how could I forget the three headed monkey getting in the way of our happy trip to beer island?

Seriously, though, how is stopping I/O in the elevator is going to change that?

We can do that after creating the image and before we let devices run again. This way we won't need to worry about the freezer.

You're suggesting that you let processes trying to do I/O continue until *after* the memory image is taken?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I'm not going to let the data get to the disk.

Yes, but you still haven't answered the original question - What are you going to do with sync I/O that leaves a process in D state because you've prevented the I/O from being completed?

I don't want to intercept those processes, just allow them to block on that I/O.

Quoted 7 lines

So you're going to allow them to go to D state somewhere. Ok, so what's the problem with blocking them in vfs_check_frozen(), then?

Quoted 1 line

Do all of the filesystems implement the freezing?

Most of the major ones - those that implement ->write_super_lockfs() should work just fine.

Okay, so we can do that.

I'm surely not against freezing of the filesystems before hibernation at least. In fact we tried that in the past, but there were some locking problems I was unable to resolve at that time.

Unfortunately I'm not very familiar with the VFS and filesystems code, so some experts' help would be very much appreciated.

Thanks,
Rafael

#46 lore raw
In reply to Dave Chinner
Quoted 5 lines · Dave Chinner
Quoted 4 lines · Elias Oltmanns
Quoted 1 line · Rafael J. Wysocki

Do all of the filesystems implement the freezing?

Quoted 7 lines
Quoted 1 line

There is some work in progress [1].

[....]

Quoted 1 line

[1] http://permalink.gmane.org/gmane.linux.file-systems/24716

No, that's the userspace ioctl interface to enable freezing from something other than dm-snapshot. The filesystems that is aimed at already support freezing via freeze_bdev().

Yes, Christoph mentioned that too and I should have realised it had I looked at those patches properly.

Regards,

Elias

#34 lore raw
In reply to Rafael J. Wysocki
Quoted 10 lines · Rafael J. Wysocki
Quoted 6 lines
Quoted 2 lines

I talked with Jens about it on a very general level, but it seems doable at first sight.

Why would you hack the blok layer when we already have a perfectly fine facility to archive what you want? freeze_bdev is there exactly for the purpose to make the filesystem consistant on disk and then freeze all I/O.

We tried that in the past and it didn't work very well due to some bad interactions with the md layer that we wanted to stay functional while we were saving the image.

Hm, details or a link?

Quoted 1 line

Also, do all of the supported filesystems implement this feature?

ext3, ext4, gfs2, jfs, reiserfs, xfs, all provide a write_super_lockfs op, which is what freeze_bdev uses. I think that the rest is generic, for simpler filesystems.

-Eric