Skip to contentSkip to conversation

Add timeout feature

PATCH In review Takashi Sato 7 participants 15 messages Sep 8, 2008 – Oct 9, 2008

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 3/3 Add timeout feature

The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.
o Freeze the filesystem
int ioctl(int fd, int FIFREEZE, long *timeout_sec)
fd: The file descriptor of the mountpoint
FIFREEZE: request code for the freeze
timeout_sec: the timeout period in seconds
If it's 0 or 1, the timeout isn't set.
This special case of "1" is implemented to keep the compatibility with XFS applications.
Return value: 0 if the operation succeeds. Otherwise, -1

o Reset the timeout period
int ioctl(int fd, int FIFREEZE_RESET_TIMEOUT, long *timeout_sec)
fd:file descriptor of mountpoint
FIFREEZE_RESET_TIMEOUT: request code for reset of timeout period
timeout_sec: new timeout period in seconds
Return value: 0 if the operation succeeds. Otherwise, -1 Error number: If the filesystem has already been unfrozen,
errno is set to EINVAL.

---
 drivers/md/dm.c             |    2 -
 fs/block_dev.c              |    2 +
 fs/buffer.c                 |   44 ++++++++++++++++++++++++---
 fs/ioctl.c                  |   71 ++++++++++++++++++++++++++++++++++++++++++--
 fs/super.c                  |   37 ++++++++++++++++++++++
 fs/xfs/xfs_fsops.c          |    2 -
 include/linux/buffer_head.h |    4 +-
 include/linux/fs.h          |    8 ++++
 8 files changed, 159 insertions(+), 11 deletions(-)

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/drivers/md/dm.c linux-2.6.27-rc5-timeout/ drivers/md/dm.c

drivers/md/dm.c +1 -1
@@ -1451,7 +1451,7 @@ static int lock_fs(struct mapped_device 
 
 	WARN_ON(md->frozen_sb);
 
-	md->frozen_sb = freeze_bdev(md->suspended_bdev);
+	md->frozen_sb = freeze_bdev(md->suspended_bdev, 0);
 	if (IS_ERR(md->frozen_sb)) {
 		r = PTR_ERR(md->frozen_sb);
 		md->frozen_sb = NULL;

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/block_dev.c linux-2.6.27-rc5-timeout/f s/block_dev.c

fs/block_dev.c +2 -0
@@ -287,6 +287,8 @@ static void init_once(void *foo)
 	inode_init_once(&ei->vfs_inode);
 	/* Initialize mutex for freeze. */
 	mutex_init(&bdev->bd_fsfreeze_mutex);
+	/* Setup freeze timeout function. */
+	INIT_DELAYED_WORK(&bdev->bd_fsfreeze_timeout, fsfreeze_timeout);
 }
 
 static inline void __bd_forget(struct inode *inode)

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/buffer.c linux-2.6.27-rc5-timeout/fs/b uffer.c

fs/buffer.c +39 -5

90 lines not on this page · raw message

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/ioctl.c linux-2.6.27-rc5-timeout/fs/io ctl.c

fs/ioctl.c +68 -3

104 lines not on this page · raw message

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/super.c linux-2.6.27-rc5-timeout/fs/su per.c

fs/super.c +37 -0
@@ -981,3 +981,40 @@ struct vfsmount *kern_mount_data(struct 
 }
 
 EXPORT_SYMBOL_GPL(kern_mount_data);
+
+/*
+ * fsfreeze_timeout - Thaw the filesystem.
+ *
+ * @work:	work queue (delayed_work.work)
+ *
+ * Called by the delayed work when elapsing the timeout period.
+ * Thaw the filesystem.
+ */
+void fsfreeze_timeout(struct work_struct *work)
+{
+	struct block_device *bd = container_of(work,
+			struct block_device, bd_fsfreeze_timeout.work);
+	struct super_block *sb = get_super(bd);
+
+	thaw_bdev_core(bd, sb, 0);
+
+	if (sb)
+		drop_super(sb);
+}
+
+/*
+ * add_fsfreeze_timeout - Add timeout for freeze.
+ *
+ * @bdev:		block device struct
+ * @timeout_msec:	timeout period
+ *
+ * Add the delayed work for freeze timeout to the delayed work queue.
+ */
+void add_fsfreeze_timeout(struct block_device *bdev, unsigned int timeout_msec)
+{
+	s64 timeout_jiffies = msecs_to_jiffies(timeout_msec);
+
+	/* Set delayed work queue */
+	cancel_delayed_work_sync(&bdev->bd_fsfreeze_timeout);
+	schedule_delayed_work(&bdev->bd_fsfreeze_timeout, timeout_jiffies);
+}

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/fs/xfs/xfs_fsops.c linux-2.6.27-rc5-timeo ut/fs/xfs/xfs_fsops.c

fs/xfs/xfs_fsops.c +1 -1
@@ -621,7 +621,7 @@ xfs_fs_goingdown(
 {
 	switch (inflags) {
 	case XFS_FSOP_GOING_FLAGS_DEFAULT: {
-		struct super_block *sb = freeze_bdev(mp->m_super->s_bdev);
+		struct super_block *sb = freeze_bdev(mp->m_super->s_bdev, 0);
 
 		if (sb && !IS_ERR(sb)) {
 			xfs_force_shutdown(mp, SHUTDOWN_FORCE_UMOUNT);

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/include/linux/buffer_head.h linux-2.6.27- rc5-timeout/include/linux/buffer_head.h

include/linux/buffer_head.h +3 -1
@@ -169,8 +169,10 @@ int sync_blockdev(struct block_device *b
 void __wait_on_buffer(struct buffer_head *);
 wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);
 int fsync_bdev(struct block_device *);
-struct super_block *freeze_bdev(struct block_device *);
+struct super_block *freeze_bdev(struct block_device *,
+				unsigned int timeout_msec);
 int thaw_bdev(struct block_device *, struct super_block *);
+int thaw_bdev_core(struct block_device *, struct super_block *, int);
 int fsync_super(struct super_block *);
 int fsync_no_super(struct block_device *);
 struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,

diff -uprN -X linux-2.6.27-rc5.org/Documentation/dontdiff linux-2.6.27-rc5-xfs/include/linux/fs.h linux-2.6.27-rc5-timeo ut/include/linux/fs.h

include/linux/fs.h +8 -0
@@ -8,6 +8,7 @@
 
 #include <linux/limits.h>
 #include <linux/ioctl.h>
+#include <linux/workqueue.h>
 
 /*
  * It's silly to have NR_OPEN bigger than NR_FILE, but you can change
@@ -228,6 +229,7 @@ extern int dir_notify_enable;
 #define FIGETBSZ   _IO(0x00,2)	/* get the block size used for bmap */
 #define FIFREEZE	_IOWR('X', 119, int)	/* Freeze */
 #define FITHAW		_IOWR('X', 120, int)	/* Thaw */
+#define	FIFREEZE_RESET_TIMEOUT	_IO(0x00, 3)	/* Reset freeze timeout */
 
 #define	FS_IOC_GETFLAGS			_IOR('f', 1, long)
 #define	FS_IOC_SETFLAGS			_IOW('f', 2, long)
@@ -581,6 +583,8 @@ struct block_device {
 	int			bd_fsfreeze_count;
 	/* Mutex for freeze */
 	struct mutex		bd_fsfreeze_mutex;
+	/* Delayed work for freeze */
+	struct delayed_work	bd_fsfreeze_timeout;
 };
 
 /*
@@ -2160,5 +2164,9 @@ int proc_nr_files(struct ctl_table *tabl
 
 int get_filesystem_list(char * buf);
 
+extern void add_fsfreeze_timeout(struct block_device *bdev,
+				unsigned int timeout_msec);
+extern void fsfreeze_timeout(struct work_struct *work);
+
 #endif /* __KERNEL__ */
 #endif /* _LINUX_FS_H */
#1 lore raw
Quoted 8 lines · Takashi Sato

The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.

And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.

#2 lore raw
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Takashi Sato

The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.

Quoted 4 lines

And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.

I agree with Christoph here, I think that the timeout is unneeded.

Regards,

Ric

#3 lore raw

Hi,

Quoted 13 lines · Ric Wheeler
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Takashi Sato

The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.

Quoted 4 lines
Quoted 2 lines

And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.

I agree with Christoph here, I think that the timeout is unneeded.

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern be solved?
If so, I will implement it.

Cheers, Takashi

#4 lore raw
Quoted 21 lines · Takashi Sato

Hi,

Quoted 13 lines · Ric Wheeler
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Takashi Sato

The timeout feature is added to "freeze ioctl" to solve a deadlock when the freezer accesses a frozen filesystem. And new ioctl to reset the timeout period is added to extend the timeout period. For example, the freezer resets the timeout period to 10 seconds every 5 seconds. In this approach, even if the freezer causes a deadlock by accessing the frozen filesystem, it will be solved by the timeout in 10 seconds and the freezer will be able to recognize that at the next reset of timeout period.

Quoted 4 lines
Quoted 2 lines

And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.

I agree with Christoph here, I think that the timeout is unneeded.

Quoted 9 lines

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern be solved?
If so, I will implement it.

Cheers, Takashi

I think that is certainly part a big part of my concern.

Also note that the timeout seems to be quite low relative to say the standard timeout for a SCSI device (30 seconds worst case).

In general, I am quite supportive of the patch series and think that this is a great addition.

Thanks!

Ric

#5 lore raw

Hi Ric and Christoph,

Quoted 12 lines · Ric Wheeler
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines

And as with all previous posting I still fundamentally disagree about the need of this functionality. We don't need a timeout for freezing.

I agree with Christoph here, I think that the timeout is unneeded.

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern be solved?
If so, I will implement it.

Quoted 7 lines
Quoted 2 lines

Cheers, Takashi

I think that is certainly part a big part of my concern.

Also note that the timeout seems to be quite low relative to say the standard timeout for a SCSI device (30 seconds worst case).

In general, I am quite supportive of the patch series and think that this is a great addition.

Thank you for your comments.
Christoph, do you have any comments about this solution?

If it's OK, I will change the freeze patch so that the unfreeze ioctl sets ETIMEDOUT to errno when the timeout occurs.

Cheers, Takashi

#6 lore raw
In reply to Takashi Sato

On Fri, 26 Sep 2008 17:52:35 +0900, Takashi Sato said:

(Sorry, am reading the threads out of temporal sequence....)

Quoted 6 lines

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.
If the freezer can recognize it by the unfreeze ioctl's errono, will your concern
be solved?
If so, I will implement it.

That would also address my concerns about merging patches 8 and 10 of the other patch series (because patch 10 wouldn't be needed then)...

#7 lore raw
In reply to Takashi Sato
Quoted 3 lines · Takashi Sato

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.

What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.

#8 lore raw
Quoted 7 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.

What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.

The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.

I'm still not really sold on the timeout, but I think the above was the intent.

-Eric

#9 lore raw
Quoted 6 lines · Eric Sandeen
Quoted 5 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.

Quoted 6 lines
Quoted 2 lines

What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.

The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.

But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.

#10 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Eric Sandeen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.

The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.

But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.

I agree. Was just trying to clarify the above point.

But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)

Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.

And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.

-Eric

#11 lore raw
Quoted 9 lines · Eric Sandeen
Quoted 2 lines · Christoph Hellwig

But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.

I agree. Was just trying to clarify the above point.

But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)

Quoted 7 lines

Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.

And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.

Since this patch hit fsdev, there have been an equal number of supporters and opponents of the timeout.

I'm not opposed to the timeout on the API, but I don't think it is needed if we have a system configurable timeout (default is no timeout) that can be changed by an admin.

My experience is that a timeout is not needed protect against a stupid admin or against software bugs.

The justification for a timeout as far as I am concerned is so the admin can log in and reset hung hardware. If we think there is no chance of forcing the external device that went to sleep to respond so the system can continue to be used, then I don't think a timeout has any valid use.

My timeout desire is based on some past SAN behavior and I'm OK if people argue those devices should just be fixed. But we argued the same thing and were ignored because bad device behavior did not stop people from buying them.

jim

#12 lore raw
In reply to Eric Sandeen
Quoted 21 lines · Eric Sandeen
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Eric Sandeen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.

The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.

But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.

I agree. Was just trying to clarify the above point.

But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)

Quoted 7 lines

Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.

And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.

Can you docuument what administrator must not do for freezing to be safe?

What if so much dirty data accumulates on frozen filesystem that there's not enough memory for the unfreeze tool?

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#13 lore raw
In reply to Eric Sandeen

Hi,

Quoted 21 lines · Eric Sandeen
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Eric Sandeen
Quoted 4 lines · Christoph Hellwig
Quoted 3 lines · Takashi Sato

I think that your concern is that the freezer cannot recognize the occurrence of a timeout and it continues the backup process and the backup data is corrupted finally.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

What timeout should happen? the freeze ioctl must not return until the filesystem is a clean state and all writes are blocked.

The suggestion was that *UN*freeze would return ETIMEDOUT if the filesystem had already unfrozen itself, I think. That way you know that the snapshot you just took is worthless, at least.

But why would the filesystem every unfreeze itself? That defeats the whole point of freezing it.

I agree. Was just trying to clarify the above point.

But there have been what, 12 submissions now, with the unfreeze timeout in place so it's a persistent theme ;)

Quoted 7 lines

Perhaps a demonstration of just how easy (or not easy) it is to deadlock a filesystem by freezing the root might be in order, at least.

And even if it is relatively easy, I still maintain that it is the administrator's role to not inflict damage on the machine being administered. There are a lot of potentially dangerous tools at root's disposal; why this particular one needs a nanny I'm still not quite sure.

I think we need the timeout for the case someone dirties so much data with mmap, hence the freeze process is swapped out and cannot unfreeze.

Cheers, Takashi

#14 lore raw
Quoted 2 lines · Takashi Sato

I think we need the timeout for the case someone dirties so much data with mmap, hence the freeze process is swapped out and cannot unfreeze.

That is not supposed to happen. That's why write blocks early on a frozen filesystem (the shared mmap write path is currently missing the check, but that's a rather small patch)