create_inode: fix copying large files
PATCH v2 1/1 create_inode: fix copying large files
When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:
copy_file_chunk() passes an offset (off_t, 64-bit typically) to ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset instead.
try_lseek_copy() rounds the data and hole offsets as found by lseek() to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize to off_t to ensure this doesn't happen.
--- misc/create_inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
misc/create_inode.c +3 -3
@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file, ptr += blen;continue ; } - err =ext2fs_file_lseek (e2_file, off + bpos, + err =ext2fs_file_llseek (e2_file, off + bpos,EXT2_SEEK_SET ,NULL );if (err) goto fail;
@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,if (hole <0 )return EXT2_ET_UNIMPLEMENTED ; - data_blk = data & ~(fs->blocksize -1 ); - hole_blk = (hole + (fs->blocksize -1 )) & ~(fs->blocksize -1 ); + data_blk = data & ~(off_t)(fs->blocksize -1 ); + hole_blk = (hole + (off_t)(fs->blocksize -1 )) & ~(off_t)(fs->blocksize -1 ); err =copy_file_chunk (fs, fd, e2_file, data_blk, hole_blk, buf, zerobuf);if (err)
Signature · 1 line
2.11.0
Quoted 8 lines · Ross Burton
When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:
copy_file_chunk()passes an offset (off_t, 64-bit typically) toext2fs_file_lseek()which expects aext2_off_t(typedef to__u32) so the value is truncated. Solve by callingext2fs_file_llseek()which takes a u64 offset instead.
Quoted 5 lines
try_lseek_copy()rounds the data and hole offsets as found bylseek()to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize tooff_tto ensure this doesn't happen.Signed-off-by Ross Burton <[email protected]>
Looks ok,
--D
Quoted 31 lines
--- misc/create_inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)misc/create_inode.c +3
-3@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file, ptr += blen;continue ; } - err =ext2fs_file_lseek (e2_file, off + bpos, + err =ext2fs_file_llseek (e2_file, off + bpos,EXT2_SEEK_SET ,NULL );if (err) goto fail;@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,if (hole <0 )return EXT2_ET_UNIMPLEMENTED ; - data_blk = data & ~(fs->blocksize -1 ); - hole_blk = (hole + (fs->blocksize -1 )) & ~(fs->blocksize -1 ); + data_blk = data & ~(off_t)(fs->blocksize -1 ); + hole_blk = (hole + (off_t)(fs->blocksize -1 )) & ~(off_t)(fs->blocksize -1 ); err =copy_file_chunk (fs, fd, e2_file, data_blk, hole_blk, buf, zerobuf);if (err)Signature · 1 line
2.11.0
Obviously this was a problem that the test suite should have spotted. Would it be simple enough to write a test suite to make a 4.1GB file, put in in a ext with -d, and the md5 the file in the file system? I was using sudo and loopback mounts which isn't really ideal for the test suite.
Ross
Quoted 52 lines · Darrick J. Wong
Quoted 8 lines · Ross Burton
When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:
copy_file_chunk()passes an offset (off_t, 64-bit typically) toext2fs_file_lseek()which expects aext2_off_t(typedef to__u32) so the value is truncated. Solve by callingext2fs_file_llseek()which takes a u64 offset instead.Quoted 5 lines
try_lseek_copy()rounds the data and hole offsets as found bylseek()to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize tooff_tto ensure this doesn't happen.Signed-off-by Ross Burton <[email protected]>Looks ok,
Reviewed-by Darrick J. Wong <[email protected]>--D
Quoted 31 lines
--- misc/create_inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)misc/create_inode.c +3
-4@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file, ptr += blen;continue ; } - err =ext2fs_file_lseek (e2_file, off + bpos, + err =ext2fs_file_llseek (e2_file, off + bpos,EXT2_SEEK_SET ,NULL );if (err) goto fail;@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,if (hole <0 )return EXT2_ET_UNIMPLEMENTED ; - data_blk = data & ~(fs->blocksize -1 ); - hole_blk = (hole + (fs->blocksize -1 )) & ~(fs->blocksize -1 ); + data_blk = data & ~(off_t)(fs->blocksize -1 ); + hole_blk = (hole + (off_t)(fs->blocksize -1 )) & ~(off_t)(fs->blocksize -1 ); err =copy_file_chunk (fs, fd, e2_file, data_blk, hole_blk, buf, zerobuf);if (err) --2.11.0
Gentle reminder to the maintainers that this patch hasn't been merged yet.
Ross
Quoted 52 lines · Darrick J. Wong
Quoted 8 lines · Ross Burton
When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:
copy_file_chunk()passes an offset (off_t, 64-bit typically) toext2fs_file_lseek()which expects aext2_off_t(typedef to__u32) so the value is truncated. Solve by callingext2fs_file_llseek()which takes a u64 offset instead.Quoted 5 lines
try_lseek_copy()rounds the data and hole offsets as found bylseek()to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize tooff_tto ensure this doesn't happen.Signed-off-by Ross Burton <[email protected]>Looks ok,
Reviewed-by Darrick J. Wong <[email protected]>--D
Quoted 31 lines
--- misc/create_inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)misc/create_inode.c +3
-4@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file, ptr += blen;continue ; } - err =ext2fs_file_lseek (e2_file, off + bpos, + err =ext2fs_file_llseek (e2_file, off + bpos,EXT2_SEEK_SET ,NULL );if (err) goto fail;@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,if (hole <0 )return EXT2_ET_UNIMPLEMENTED ; - data_blk = data & ~(fs->blocksize -1 ); - hole_blk = (hole + (fs->blocksize -1 )) & ~(fs->blocksize -1 ); + data_blk = data & ~(off_t)(fs->blocksize -1 ); + hole_blk = (hole + (off_t)(fs->blocksize -1 )) & ~(off_t)(fs->blocksize -1 ); err =copy_file_chunk (fs, fd, e2_file, data_blk, hole_blk, buf, zerobuf);if (err) --2.11.0
Theodore Y. Ts'o
AppliedQuoted 9 lines · Darrick J. Wong
Quoted 8 lines · Ross Burton
When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:
copy_file_chunk()passes an offset (off_t, 64-bit typically) toext2fs_file_lseek()which expects aext2_off_t(typedef to__u32) so the value is truncated. Solve by callingext2fs_file_llseek()which takes a u64 offset instead.
Quoted 8 lines
Quoted 5 lines
try_lseek_copy()rounds the data and hole offsets as found bylseek()to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize tooff_tto ensure this doesn't happen.Signed-off-by Ross Burton <[email protected]>Looks ok,
Reviewed-by Darrick J. Wong <[email protected]>
Thanks, applied.
- Ted