Add timeout feature
Subsystems (4)
PATCH 3/3 Add timeout feature
The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.
o Freeze the filesystem
int ioctl(int fd, int FIFREEZE, long *timeout_sec)
fd: The file descriptor of the mountpoint
FIFREEZE: request code for the freeze
timeout_sec: the timeout period in seconds
If it's 0 or 1, the timeout isn't set.
This special case of "1" is implemented to keep the compatibility with XFS applications.
Return value: 0 if the operation succeeds. Otherwise, -1
o Reset the timeout period
int ioctl(int fd, int FIFREEZE_RESET_TIMEOUT, long *timeout_sec)
fd:file descriptor of mountpoint
FIFREEZE_RESET_TIMEOUT: request code for reset of timeout period
timeout_sec: new timeout period in seconds
Return value: 0 if the operation succeeds. Otherwise, -1 Error number: If the filesystem has already been unfrozen,
errno is set to EINVAL.
--- drivers/md/dm.c | 2 - fs/block_dev.c | 2 + fs/buffer.c | 44 ++++++++++++++++++++++++--- fs/ioctl.c | 71 ++++++++++++++++++++++++++++++++++++++++++-- fs/super.c | 37 ++++++++++++++++++++++ fs/xfs/xfs_fsops.c | 2 - include/linux/buffer_head.h | 4 +- include/linux/fs.h | 8 ++++ 8 files changed, 159 insertions(+), 11 deletions(-)
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/drivers/md/dm.c linux-2.6.27-rc5-timeout/ drivers/md/dm.c
drivers/md/dm.c +1 -1
@@ -1451,7 +1451,7 @@ static int lock_fs(struct mapped_deviceWARN_ON (md->frozen_sb); - md->frozen_sb =freeze_bdev (md->suspended_bdev); + md->frozen_sb =freeze_bdev (md->suspended_bdev,0 );if (IS_ERR (md->frozen_sb)) { r =PTR_ERR (md->frozen_sb); md->frozen_sb =NULL ;
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/block_dev.c linux-2.6.27-rc5-timeout/f s/block_dev.c
fs/block_dev.c +2 -0
@@ -287,6 +287,8 @@ static void init_once(void *foo)inode_init_once (&ei->vfs_inode);/* Initialize mutex for freeze. */ mutex_init (&bdev->bd_fsfreeze_mutex); +/* Setup freeze timeout function. */ +INIT_DELAYED_WORK (&bdev->bd_fsfreeze_timeout, fsfreeze_timeout); }static inline void __bd_forget (struct inode *inode)
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/buffer.c linux-2.6.27-rc5-timeout/fs/b uffer.c
fs/buffer.c +39 -5
90 lines not on this page · raw message
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/ioctl.c linux-2.6.27-rc5-timeout/fs/io ctl.c
fs/ioctl.c +68 -3
104 lines not on this page · raw message
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/super.c linux-2.6.27-rc5-timeout/fs/su per.c
fs/super.c +37 -0
@@ -981,3 +981,40 @@ struct vfsmount *kern_mount_data(struct }EXPORT_SYMBOL_GPL (kern_mount_data); + +/* +* fsfreeze_timeout - Thaw the filesystem. +* +* @work: work queue (delayed_work.work) +* +* Called by the delayed work when elapsing the timeout period. +* Thaw the filesystem. +*/ +void fsfreeze_timeout (struct work_struct *work) +{ +struct block_device *bd =container_of (work, + struct block_device, bd_fsfreeze_timeout.work); +struct super_block *sb =get_super (bd); + +thaw_bdev_core (bd, sb,0 ); + +if (sb) +drop_super (sb); +} + +/* +* add_fsfreeze_timeout - Add timeout for freeze. +* +* @bdev: block device struct +* @timeout_msec: timeout period +* +* Add the delayed work for freeze timeout to the delayed work queue. +*/ +void add_fsfreeze_timeout (struct block_device *bdev,unsigned int timeout_msec) +{ +s64 timeout_jiffies =msecs_to_jiffies (timeout_msec); + +/* Set delayed work queue */ +cancel_delayed_work_sync (&bdev->bd_fsfreeze_timeout); +schedule_delayed_work (&bdev->bd_fsfreeze_timeout, timeout_jiffies); +}
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/xfs/xfs_fsops.c linux-2.6.27-rc5-timeo ut/fs/xfs/xfs_fsops.c
fs/xfs/xfs_fsops.c +1 -1
@@ -621,7 +621,7 @@ xfs_fs_goingdown( {switch (inflags) {case XFS_FSOP_GOING_FLAGS_DEFAULT : { -struct super_block *sb =freeze_bdev (mp->m_super->s_bdev); +struct super_block *sb =freeze_bdev (mp->m_super->s_bdev,0 );if (sb && !IS_ERR (sb)) {xfs_force_shutdown (mp,SHUTDOWN_FORCE_UMOUNT );
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/include/linux/buffer_head.h linux-2.6.27- rc5-timeout/include/linux/buffer_head.h
include/linux/buffer_head.h +3 -1
@@ -169,8 +169,10 @@ int sync_blockdev(struct block_device *bvoid __wait_on_buffer (struct buffer_head *);wait_queue_head_t *bh_waitq_head (struct buffer_head *bh);int fsync_bdev (struct block_device *); -struct super_block *freeze_bdev (struct block_device *); +struct super_block *freeze_bdev (struct block_device *, +unsigned int timeout_msec);int thaw_bdev (struct block_device *,struct super_block *); +int thaw_bdev_core (struct block_device *,struct super_block *,int );int fsync_super (struct super_block *);int fsync_no_super (struct block_device *); struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,
diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/include/linux/fs.h linux-2.6.27-rc5-timeo ut/include/linux/fs.h
include/linux/fs.h +8 -0
@@ -8,6 +8,7 @@#include <linux/limits.h> #include <linux/ioctl.h> +#include <linux/workqueue.h> /* * It's silly to have NR_OPEN bigger than NR_FILE, but you can change
@@ -228,6 +229,7 @@ extern int dir_notify_enable;#define FIGETBSZ _IO(0x00,2)/* get the block size used for bmap */ #define FIFREEZE _IOWR('X', 119, int)/* Freeze */ #define FITHAW _IOWR('X', 120, int)/* Thaw */ +#define FIFREEZE_RESET_TIMEOUT _IO(0x00, 3)/* Reset freeze timeout */ #define FS_IOC_GETFLAGS _IOR('f', 1, long)#define FS_IOC_SETFLAGS _IOW('f', 2, long)
@@ -581,6 +583,8 @@ struct block_device {int bd_fsfreeze_count;/* Mutex for freeze */ struct mutex bd_fsfreeze_mutex; +/* Delayed work for freeze */ +struct delayed_work bd_fsfreeze_timeout; };/*
@@ -2160,5 +2164,9 @@ int proc_nr_files(struct ctl_table *tablint get_filesystem_list (char * buf); +extern void add_fsfreeze_timeout (struct block_device *bdev, +unsigned int timeout_msec); +extern void fsfreeze_timeout (struct work_struct *work); +#endif /* __KERNEL__ */ #endif /* _LINUX_FS_H */
Quoted 8 lines · Takashi Sato
The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.
And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Takashi Sato
The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.
Quoted 4 lines
And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.
I agree with Christoph here, I think that the timeout is unneeded.
Regards,
Ric
Hi,
Quoted 13 lines · Ric Wheeler
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Takashi Sato
The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.
Quoted 4 lines
Quoted 2 lines
And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.
I agree with Christoph here, I think that the timeout is unneeded.
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern be solved?
If so, I will implement it.
Cheers, Takashi
Quoted 21 lines · Takashi Sato
Hi,
Quoted 13 lines · Ric Wheeler
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Takashi Sato
The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.
Quoted 4 lines
Quoted 2 lines
And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.
I agree with Christoph here, I think that the timeout is unneeded.
Quoted 9 lines
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern be solved?
If so, I will implement it.Cheers, Takashi
I think that is certainly part a big part of my concern.
Also note that the timeout seems to be quite low relative to say the standard timeout for a SCSI device (30 seconds worst case).
In general, I am quite supportive of the patch series and think that this is a great addition.
Thanks!
Ric
Hi Ric and Christoph,
Quoted 12 lines · Ric Wheeler
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines
And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.
I agree with Christoph here, I think that the timeout is unneeded.
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern be solved?
If so, I will implement it.
Quoted 7 lines
Quoted 2 lines
Cheers, Takashi
I think that is certainly part a big part of my concern.
Also note that the timeout seems to be quite low relative to say the standard timeout for a SCSI device (30 seconds worst case).
In general, I am quite supportive of the patch series and think that this is a great addition.
Thank you for your comments.
Christoph, do you have any comments about this solution?
If it's OK, I will change the freeze patch so that the unfreeze ioctl sets ETIMEDOUT to errno when the timeout occurs.
Cheers, Takashi
On Fri, 26 Sep 2008 17:52:35 +0900, Takashi Sato said:
(Sorry, am reading the threads out of temporal sequence....)
Quoted 6 lines
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern
be solved?
If so, I will implement it.
That would also address my concerns about merging patches 8 and 10 of the other patch series (because patch 10 wouldn't be needed then)...
Quoted 3 lines · Takashi Sato
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.
The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.
I'm still not really sold on the timeout, but I think the above was the intent.
-Eric
Quoted 6 lines · Eric Sandeen
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
Quoted 6 lines
Quoted 2 lines
What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.
The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.
But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Eric Sandeen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.
The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.
But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.
I agree. Was just trying to clarify the above point.
But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)
Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.
And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.
-Eric
Quoted 9 lines · Eric Sandeen
Quoted 2 lines · Christoph Hellwig
But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.
I agree. Was just trying to clarify the above point.
But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)
Quoted 7 lines
Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.
And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.
Since this patch hit fsdev, there have been an equal number of supporters and opponents of the timeout.
I'm not opposed to the timeout on the API, but I don't think it is needed if we have a system configurable timeout (default is no timeout) that can be changed by an admin.
My experience is that a timeout is not needed protect against a stupid admin or against software bugs.
The justification for a timeout as far as I am concerned is so the admin can log in and reset hung hardware. If we think there is no chance of forcing the external device that went to sleep to respond so the system can continue to be used, then I don't think a timeout has any valid use.
My timeout desire is based on some past SAN behavior and I'm OK if people argue those devices should just be fixed. But we argued the same thing and were ignored because bad device behavior did not stop people from buying them.
jim
Quoted 21 lines · Eric Sandeen
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Eric Sandeen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.
The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.
But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.
I agree. Was just trying to clarify the above point.
But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)
Quoted 7 lines
Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.
And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.
Can you docuument what administrator must not do for freezing to be safe?
What if so much dirty data accumulates on frozen filesystem that there's not enough memory for the unfreeze tool?
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Hi,
Quoted 21 lines · Eric Sandeen
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Eric Sandeen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato
I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.
The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.
But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.
I agree. Was just trying to clarify the above point.
But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)
Quoted 7 lines
Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.
And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.
I think we need the timeout for the case someone dirties so much data with mmap, hence the freeze process is swapped out and cannot unfreeze.
Cheers, Takashi
Quoted 2 lines · Takashi Sato
I think we need the timeout for the case someone dirties so much data with mmap, hence the freeze process is swapped out and cannot unfreeze.
That is not supposed to happen. That's why write blocks early on a frozen filesystem (the shared mmap write path is currently missing the check, but that's a rather small patch)