Skip to contentSkip to conversation

hang when copying large file to disc

BUG, bisect Jeremiah Mahler 2 participants 3 messages Jan 31, 2015 – Feb 1, 2015 VFS

lore
Share & tools
lore
#0 lore raw

If a large file (> 1 GB) is transferred from a USB stick or a mmc to an local disc (using ext4), it will proceed normally until it has transferred approximately 600 MB. Then the speed will drop off to zero and the terminal performing the operation will usually hang. And sometimes other terminals will hang as well. A reboot is required to get the system working again.

mount /dev/sdb1 /mnt
pv /mnt/large_file > large_file_out
(... ~600 MB ... hang)

I have performed a bisect and found that commit ef39794651347 introduced the bug. This commit is present in the latest -next 20150130.

From ef397946513470381c0e9ed5b355cd1c9855a364 Mon Sep 17 00:00:00 2001
From: Theodore Ts'o <[email protected]>
Date: Sat, 24 Jan 2015 20:51:05 -0500
Subject: [PATCH] vfs: add support for a lazytime mount option

Add a new mount option which enables a new "lazytime" mode. This mode causes atime, mtime, and ctime updates to only be made to the in-memory version of the inode. The on-disk times will only get updated when (a) if the inode needs to be updated for some non-time related change, (b) if userspace calls fsync(), syncfs() or sync(), or
(c) just before an undeleted inode is evicted from memory.

This is OK according to POSIX because there are no guarantees after a crash unless userspace explicitly requests via a fsync(2) call.

For workloads which feature a large number of random write to a preallocated file, the lazytime mount option significantly reduces writes to the inode table. The repeated 4k writes to a single block will result in undesirable stress on flash devices and SMR disk drives. Even on conventional HDD's, the repeated writes to the inode table block will trigger Adjacent Track Interference (ATI) remediation latencies, which very negatively impact long tail latencies --- which is a very big deal for web serving tiers (for example).

Google-Bug-Id: 18297052

---

   fs/ext4/inode.c                  |  6 ++++
   fs/fs-writeback.c                | 62 +++++++++++++++++++++++++++++++++-------
   fs/gfs2/file.c                   |  4 +--
   fs/inode.c                       | 56 +++++++++++++++++++++++++-----------
   fs/jfs/file.c                    |  2 +-
   fs/libfs.c                       |  2 +-
   fs/proc_namespace.c              |  1 +
   fs/sync.c                        |  8 ++++++
   include/linux/backing-dev.h      |  1 +
   include/linux/fs.h               |  5 ++++
   include/trace/events/writeback.h | 60 +++++++++++++++++++++++++++++++++++++-
   include/uapi/linux/fs.h          |  4 ++-
   mm/backing-dev.c                 | 10 +++++--
   13 files changed, 186 insertions(+), 35 deletions(-)

If I can do anything else to help, let me know.

Signature · 1 line
- Jeremiah Mahler
#1 lore raw
Quoted 8 lines · Jeremiah Mahler

If a large file (> 1 GB) is transferred from a USB stick or a mmc to an local disc (using ext4), it will proceed normally until it has transferred approximately 600 MB. Then the speed will drop off to zero and the terminal performing the operation will usually hang. And sometimes other terminals will hang as well. A reboot is required to get the system working again.

Quoted 6 lines

mount /dev/sdb1 /mnt
pv /mnt/large_file > large_file_out
(... ~600 MB ... hang)

I have performed a bisect and found that commit ef39794651347 introduced the bug. This commit is present in the latest -next 20150130.

Thanks for the bug report. I was able to reproduce the problem, and found the problem. Unfortunately I accidentally introduce this in my most recent patchset. I'll fix it up, but here is the patch until I get the a new version pushed out.

fs/fs-writeback.c +1 -1
index 82c2984..0046861 100644
@@ -1192,9 +1192,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 
 		trace_writeback_dirty_inode(inode, flags);
 	}
-	dirtytime = flags & I_DIRTY_TIME;
 	if (flags & I_DIRTY_INODE)
 		flags &= ~I_DIRTY_TIME;
+	dirtytime = flags & I_DIRTY_TIME;
 
 	/*
 	 * Paired with smp_mb() in __writeback_single_inode() for the

- Ted

#2 lore raw

Ted,

Quoted 2 lines · Theodore Ts'o
Quoted 1 line · Jeremiah Mahler

[...]

Quoted 22 lines

Thanks for the bug report. I was able to reproduce the problem, and found the problem. Unfortunately I accidentally introduce this in my most recent patchset. I'll fix it up, but here is the patch until I get the a new version pushed out.

fs/fs-writeback.c +1 -1
index 82c2984..0046861 100644
@@ -1192,9 +1192,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 
 		trace_writeback_dirty_inode(inode, flags);
 	}
-	dirtytime = flags & I_DIRTY_TIME;
 	if (flags & I_DIRTY_INODE)
 		flags &= ~I_DIRTY_TIME;
+	dirtytime = flags & I_DIRTY_TIME;
 
 	/*
 	 * Paired with smp_mb() in __writeback_single_inode() for the
Quoted 1 line

- Ted

That solved the problem. Thanks for the quick fix.

Signature · 1 line
- Jeremiah Mahler