Skip to contentSkip to conversation

ext3fs: umount+sync not enough to guarantee metadata-on-disk

DISCUSSION Mark Lord 7 participants 20 messages Jun 7, 2007 – Jun 14, 2007

lore
Share & tools
lore

Mark Lord

author
#0 lore raw

Andrew / Stephen / Ted,

I have a MythTV PVR box here, which has had intermittent shutdown issues over the past while.

The main storage for recordings is a 2-drive RAID0 array, formatted with ext3fs. The system runs a 2.6.17 Ubuntu kernel that I've tailored/rebuild for this specific machine.

My observation is, if I delete a couple of 25+ GByte files, and then immediately shutdown the system, the disks are still being written to at the point when the power goes off (halt -f -p).

The overall sequence is something like this:

1. Delete the files in Myth, which uses a "delete slowly" function to avoid locking up the machine for the 30-60 seconds that this would otherwise require. Myth appears to open the file, unlink it, and then sit in a loop doing small ftruncate's until nothing is left.

2. When I trigger the shutdown whilst this is happening, Myth gets killed off, and so the unlinked file is automatically closed. and the kernel (filesystem) code begins finishing the delete operation.

3. The shutdown scripts do their thing quickly, so the delete is *still* underway when the umount commands are issued. On this system, I use this sequence:

## /var/lib/mythtv is the recording's ext3fs, on /dev/md0 (RAID0):
mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv
sync
mount / -oremount,ro
sync
sleep 1
hdparm -W0 /dev/sda /dev/sdb
sync
sleep 2
halt -f -p

4. The hard drive light is on solid throughout, including at the point when the power goes out.

5. On the next reboot, there is a LONG pause (20-30 seconds) at the point where /var/lib/mythtv is remounted --> indicating unfinished business from the journal file that needs to be replayed (eg. the file deletion).

So.. how can I guarantee a quiescent filesystem before doing "halt -f -p" ?? This looks pretty dangerous as-is.

Thanks

#1 lore raw
Quoted 13 lines · On Thu, 07 Jun 2007 09:44:24 -0400 Mark Lord <[email protected]> wrote:

Andrew / Stephen / Ted,

I have a MythTV PVR box here, which has had intermittent shutdown issues over the past while.

The main storage for recordings is a 2-drive RAID0 array, formatted with ext3fs. The system runs a 2.6.17 Ubuntu kernel that I've tailored/rebuild for this specific machine.

My observation is, if I delete a couple of 25+ GByte files, and then immediately shutdown the system, the disks are still being written to at the point when the power goes off (halt -f -p).

Quoted 6 lines

The overall sequence is something like this:

1. Delete the files in Myth, which uses a "delete slowly" function to avoid locking up the machine for the 30-60 seconds that this would otherwise require. Myth appears to open the file, unlink it, and then sit in a loop doing small ftruncate's until nothing is left.

sigh. yup.

Quoted 9 lines

2. When I trigger the shutdown whilst this is happening, Myth gets killed off, and so the unlinked file is automatically closed. and the kernel (filesystem) code begins finishing the delete operation.

3. The shutdown scripts do their thing quickly, so the delete is *still* underway when the umount commands are issued. On this system, I use this sequence:

## /var/lib/mythtv is the recording's ext3fs, on /dev/md0 (RAID0):

I assume the applikcaton has already been killed at this stage, and it is blocked in the kernel running the truncate?

Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Quoted 9 lines

sync
mount / -oremount,ro
sync
sleep 1
hdparm -W0 /dev/sda /dev/sdb
sync
sleep 2
halt -f -p

Quoted 6 lines

4. The hard drive light is on solid throughout, including at the point when the power goes out.

5. On the next reboot, there is a LONG pause (20-30 seconds) at the point where /var/lib/mythtv is remounted --> indicating unfinished business from the journal file that needs to be replayed (eg. the file deletion).

That opened-but-deleted file's inode is on the orphan list.

See, the unlink-then-slowly-truncate trick is done in this fashion so that if the box crashes during the slow unlink, the orhpan list handling on the reboot will finish off the truncate for us.

Quoted 2 lines

So.. how can I guarantee a quiescent filesystem before doing "halt -f -p" ?? This looks pretty dangerous as-is.

Wait for the killed-off applicaiton to actually exit, perhaps? But that unmount should have failed.

author
#2 lore raw
Quoted 1 line · Andrew Morton

On Thu, 07 Jun 2007 09:44:24 -0400 Mark Lord <[email protected]> wrote:

..

Quoted 4 lines
Quoted 4 lines

2. When I trigger the shutdown whilst this is happening, Myth gets killed off, and so the unlinked file is automatically closed. and the kernel (filesystem) code begins finishing the delete operation.

Quoted 8 lines
Quoted 5 lines

3. The shutdown scripts do their thing quickly, so the delete is *still* underway when the umount commands are issued. On this system, I use this sequence:

## /var/lib/mythtv is the recording's ext3fs, on /dev/md0 (RAID0):

I assume the applikcaton has already been killed at this stage, and it is blocked in the kernel running the truncate?

Yes, I believe I saw that once.

Quoted 6 lines
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Actually, what I expect to happen is for the remount,ro to block until the file deletion completes. But it doesn't.

Once a f/s is read-only, there should be NO writing to it. Right?

I don't know if the umount worked or not, but the f/s ought to be read-only at this point, so why is it still writing to the device?

I'll instrument the shutdown more for next time, to see if the remount and umount really do succeed or not. Mmm.. do they log anything on failure?

Quoted 16 lines
Quoted 9 lines

sync
mount / -oremount,ro
sync
sleep 1
hdparm -W0 /dev/sda /dev/sdb
sync
sleep 2
halt -f -p

Quoted 6 lines

4. The hard drive light is on solid throughout, including at the point when the power goes out.

5. On the next reboot, there is a LONG pause (20-30 seconds) at the point where /var/lib/mythtv is remounted --> indicating unfinished business from the journal file that needs to be replayed (eg. the file deletion).

Quoted 5 lines

That opened-but-deleted file's inode is on the orphan list.

See, the unlink-then-slowly-truncate trick is done in this fashion so that if the box crashes during the slow unlink, the orhpan list handling on the reboot will finish off the truncate for us.

Yes, absolutely.

Quoted 5 lines
Quoted 2 lines

So.. how can I guarantee a quiescent filesystem before doing "halt -f -p" ?? This looks pretty dangerous as-is.

Wait for the killed-off applicaiton to actually exit, perhaps? But that unmount should have failed.

But some applications just "hang" regardless, and so this cannot wait forever. There must be *some* way to know when a filesystem is really quiescent and therefore safe to power off?

Cheers

#7 lore raw
In reply to Andrew Morton
Quoted 6 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

#3 lore raw
In reply to Mark Lord

Hi,

Quoted 9 lines · Mark Lord
Quoted 6 lines
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Actually, what I expect to happen is for the remount,ro to block until the file deletion completes. But it doesn't.

No -- all that the remount,ro sees is that there's a fd still open for write. It has no idea if or when that fd is going to get closed, so it should fail. Returning -EBUSY is the only thing it _can_ do: waiting for the fs to be remountable might wait forever.

So the fs is still writable after that point.

--Stephen

#8 lore raw
In reply to Chuck Ebbert
Quoted 9 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

author
#9 lore raw
Quoted 17 lines · Andrew Morton
Quoted 7 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 5 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Yeah, I really don't know what to do with this either. We have to have a bounds on how long we wait at shutdown, but there doesn't seem to be an easy way to get notified once a filesystem becomes idle (?).

I suppose I could have the script loop on /proc/interrupts until it sees the disk activity has tapered off..

Cheers

author
#16 lore raw
In reply to Chuck Ebbert
Quoted 8 lines · Chuck Ebbert
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

The part that gets me here, and that others might be missing, is that we are not waiting for ftruncate at this point.

We're waiting for unlink. The application that was doing ftruncate in tiny little doses has been sent a kill-9 signal, so what should be happening now (confirmed by disk activity LEDs) is the file should just be getting deleted the same as if we did "rm bigfile" on it.

And I kind of expected sync or remount,ro to complete *after* the unlink finishes..

Oh well.

#10 lore raw
In reply to Mark Lord
Quoted 24 lines · On Thu, 07 Jun 2007 17:38:54 -0400 Mark Lord <[email protected]> wrote:
Quoted 17 lines · Andrew Morton
Quoted 7 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 5 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Quoted 8 lines

Yeah, I really don't know what to do with this either. We have to have a bounds on how long we wait at shutdown, but there doesn't seem to be an easy way to get notified once a filesystem becomes idle (?).

I suppose I could have the script loop on /proc/interrupts until it sees the disk activity has tapered off..

I don't recall clarity on this question: did the umount fail?

Because it should have, in which case your script can poll that.

author
#11 lore raw
Quoted 30 lines · Andrew Morton
Quoted 27 lines · On Thu, 07 Jun 2007 17:38:54 -0400 Mark Lord <[email protected]> wrote:
Quoted 16 lines · Andrew Morton
Quoted 7 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 5 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Yeah, I really don't know what to do with this either. We have to have a bounds on how long we wait at shutdown, but there doesn't seem to be an easy way to get notified once a filesystem becomes idle (?).

Quoted 7 lines
Quoted 3 lines

I suppose I could have the script loop on /proc/interrupts until it sees the disk activity has tapered off..

I don't recall clarity on this question: did the umount fail?

Because it should have, in which case your script can poll that.

I haven't had the opportunity to instrument/retest that part, but would it really make any difference?

The process is already a Zombie at this point, existing only because it was killed during a syscall and seems to have gotten stuck there. So it won't be closing anything unless that has already happened during the conversion to Zombie (?).

But yeah, I'll get back here again once I see if the remount and umount work or not.

Cheers

author
#12 lore raw
In reply to Andrew Morton
Quoted 17 lines · Andrew Morton
Quoted 7 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 5 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Okay, I added more instrumentation and retested today.

Good and Bad.
The umount does indeed fail while the massive unlink is happening, so I can just loop on that a few times before giving up.

But.. the earlier "remount,ro".. well.. I don't know what it does. I did get it to lock up solid, though.. hung on the "remount,ro" when issued during an unlink of a 15GB file. The disk I/O eventually completes, and drives go idle, but the system remains hung inside the remount,ro call.

Alt-sysrq-T was functioning, so I have some screen shots (.jpg) here:

http://rtr.ca/remount_ro/

That's definitely a bug.
For now, I'll just not attempt the remount,ro on this system, and have it loop for a minute attempting umount instead.

Cheers

#17 lore raw
In reply to Mark Lord
Quoted 10 lines
Quoted 8 lines · Chuck Ebbert
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

Quoted 7 lines

The part that gets me here, and that others might be missing, is that we are not waiting for ftruncate at this point.

We're waiting for unlink. The application that was doing ftruncate in tiny little doses has been sent a kill-9 signal, so what should be happening now (confirmed by disk activity LEDs) is the file should just be getting deleted the same as if we did "rm bigfile" on it.

But if that app has been waiting in D state, kill -9 does nothing to it
until it wakes up, doesn't it? So fd's are still open and umount fails.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#13 lore raw
In reply to Mark Lord
Quoted 36 lines
Quoted 17 lines · Andrew Morton
Quoted 7 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 5 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Okay, I added more instrumentation and retested today.

Good and Bad.
The umount does indeed fail while the massive unlink is happening, so I can just loop on that a few times before giving up.

But.. the earlier "remount,ro".. well.. I don't know what it does. I did get it to lock up solid, though.. hung on the "remount,ro" when issued during an unlink of a 15GB file. The disk I/O eventually completes, and drives go idle, but the system remains hung inside the remount,ro call.

Quoted 3 lines

Alt-sysrq-T was functioning, so I have some screen shots (.jpg) here:

http://rtr.ca/remount_ro/

Thanks for the traces.

Quoted 1 line

That's definitely a bug.

Yes. We have a nice lock inversion there. ext3_remount() is called
with sb->s_lock held and waits for transaction to finish in journal_lock_updates(). On the other hand ext3_orphan_del() is called inside a transaction and tries to do lock_super()... Bad luck.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
author
#18 lore raw
In reply to Jan Kara
Quoted 11 lines · Jan Kara
Quoted 11 lines
Quoted 7 lines · Chuck Ebbert
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

The part that gets me here, and that others might be missing, is that we are not waiting for ftruncate at this point.

Quoted 6 lines
Quoted 4 lines

We're waiting for unlink. The application that was doing ftruncate in tiny little doses has been sent a kill-9 signal, so what should be happening now (confirmed by disk activity LEDs) is the file should just be getting deleted the same as if we did "rm bigfile" on it.

But if that app has been waiting in D state, kill -9 does nothing to it
until it wakes up, doesn't it? So fd's are still open and umount fails.

Yeah, except the task is in Zombie state at this point.
But the umount does fail properly, so I no longer complain about that.

Cheers

author
#14 lore raw
In reply to Jan Kara
Quoted 36 lines · Jan Kara
Quoted 36 lines
Quoted 16 lines · Andrew Morton
Quoted 7 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 5 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Okay, I added more instrumentation and retested today.

Good and Bad.
The umount does indeed fail while the massive unlink is happening, so I can just loop on that a few times before giving up.

But.. the earlier "remount,ro".. well.. I don't know what it does. I did get it to lock up solid, though.. hung on the "remount,ro" when issued during an unlink of a 15GB file. The disk I/O eventually completes, and drives go idle, but the system remains hung inside the remount,ro call.

Alt-sysrq-T was functioning, so I have some screen shots (.jpg) here:

Quoted 9 lines
Quoted 1 line

http://rtr.ca/remount_ro/

Thanks for the traces.

Quoted 1 line

That's definitely a bug.

Yes. We have a nice lock inversion there. ext3_remount() is called
with sb->s_lock held and waits for transaction to finish in journal_lock_updates(). On the other hand ext3_orphan_del() is called inside a transaction and tries to do lock_super()... Bad luck.

Peachy. Do you have enough knowledge here to generate a fix for this?
Maybe just have the remount break out, releasing all locks, and then loop and retry (or return -EBUSY?) when this happens?

Cheers

#15 lore raw
Quoted 44 lines · Mark Lord
Quoted 43 lines · Jan Kara
Quoted 37 lines
Quoted 18 lines · Andrew Morton
Quoted 8 lines · On Thu, 07 Jun 2007 12:11:58 -0400 Chuck Ebbert <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 3 lines

mount /var/lib/mythtv -oremount,ro
sync
umount /var/lib/mythtv

Did this succeed? If the application is still truncating that file, the
umount should have failed.

Shouldn't sync should wait for truncate to finish?

I can't think of anything in there at present which would cause that to happen, and it's not immediately obvious how we _could_ make it happen - we
have an inode which potentially has no dirty pages and which is itself clean. The truncate can span multiple journal commits, so forcing a journal commit in sync() won't necessarily block behind the truncate.

Quoted 6 lines

I guess we could ask sync to speculatively take and release every inode's i_mutex or something. But even that would involve quite some hoop-jumping
due to those infuriating spinlock-protected list_heads on the superblock.

hmm.

Okay, I added more instrumentation and retested today.

Good and Bad.
The umount does indeed fail while the massive unlink is happening, so I can just loop on that a few times before giving up.

But.. the earlier "remount,ro".. well.. I don't know what it does. I did get it to lock up solid, though.. hung on the "remount,ro" when issued during an unlink of a 15GB file. The disk I/O eventually completes, and drives go idle, but the system remains hung inside the remount,ro call.

Quoted 3 lines

Alt-sysrq-T was functioning, so I have some screen shots (.jpg) here:

http://rtr.ca/remount_ro/

Thanks for the traces.

Quoted 1 line

That's definitely a bug.

Quoted 8 lines
Quoted 4 lines

Yes. We have a nice lock inversion there. ext3_remount() is called
with sb->s_lock held and waits for transaction to finish in journal_lock_updates(). On the other hand ext3_orphan_del() is called inside a transaction and tries to do lock_super()... Bad luck.

Peachy. Do you have enough knowledge here to generate a fix for this? Maybe just have the remount break out, releasing all locks, and then loop and retry (or return -EBUSY?) when this happens?

Yes, I'll try to cook up some patch. As I'm looking through the code,
ext3_remount seems to be the only place where we need to start a transaction under s_lock. So probably we could release sb->s_lock for the time we have to wait for a transaction...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs
#4 lore raw
In reply to Mark Lord

Hi!

Quoted 10 lines
Quoted 3 lines

Did this succeed? If the application is still
truncating that file, the
umount should have failed.

Actually, what I expect to happen is for the remount,ro to block until the file deletion completes. But it doesn't.

Once a f/s is read-only, there should be NO writing to it. Right?

Linux happily writes to filesystems mounted read-only. It will replay journal on them.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#19 lore raw
In reply to Mark Lord

Hi!

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

Did this succeed? If the application is still
truncating that file, the
umount should have failed.

Shouldn't sync should wait for truncate to finish?

The part that gets me here, and that others might be missing,
is that we are not waiting for ftruncate at this point.

Quoted 8 lines

We're waiting for unlink. The application that was doing ftruncate
in tiny little doses has been sent a kill-9 signal, so what should
be happening now (confirmed by disk activity LEDs) is the file should
just be getting deleted the same as if we did "rm bigfile" on it.

Well, AFAICT kill-9 signal delivery can take time. It still might be doing ftruncate.

It would be interesting to know when in that sequence mythtv dies... or perhaps put loop 'while killall -9 mythtv returns success, sleep 1' into shutdown scripts?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw
In reply to Pavel Machek

Hi,

Quoted 5 lines · Pavel Machek
Quoted 2 lines

Once a f/s is read-only, there should be NO writing to it. Right?

Linux happily writes to filesystems mounted read-only. It will replay journal on them.

Only at mount time, not on unmount; and it does check whether the underlying device is truly readonly or not first (assuming bdev_read_only() is working on the device in question.)

--Stephen

#6 lore raw
In reply to Pavel Machek
Quoted 11 lines · Pavel Machek

Hi!

Quoted 6 lines
Quoted 2 lines

Did this succeed? If the application is still truncating that file, the umount should have failed.

Actually, what I expect to happen is for the remount,ro to block until the file deletion completes. But it doesn't.

Once a f/s is read-only, there should be NO writing to it. Right?

Linux happily writes to filesystems mounted read-only. It will replay journal on them.

That's a bug and needs fixed. Read only means read _only_.

And the question still remains; why is sync() not blocking until the file has been completely unlinked and the disk is consistent?