Skip to contentSkip to conversation

create_inode: fix copying large files

PATCH Applied Ross Burton 3 participants 5 messages Feb 1, 2019 – Feb 7, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 create_inode: fix copying large files

When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:

copy_file_chunk() passes an offset (off_t, 64-bit typically) to ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset instead.

try_lseek_copy() rounds the data and hole offsets as found by lseek() to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize to off_t to ensure this doesn't happen.

---
 misc/create_inode.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
misc/create_inode.c +3 -3
index 05aa6363..1b35c769 100644
@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
 				ptr += blen;
 				continue;
 			}
-			err = ext2fs_file_lseek(e2_file, off + bpos,
+			err = ext2fs_file_llseek(e2_file, off + bpos,
 						EXT2_SEEK_SET, NULL);
 			if (err)
 				goto fail;
@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
 		if (hole < 0)
 			return EXT2_ET_UNIMPLEMENTED;
 
-		data_blk = data & ~(fs->blocksize - 1);
-		hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
+		data_blk = data & ~(off_t)(fs->blocksize - 1);
+		hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
 		err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
 				      zerobuf);
 		if (err)
Signature · 1 line
2.11.0
Reviewed-by on v2 1/1
#1 lore raw
Quoted 8 lines · Ross Burton

When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:

copy_file_chunk() passes an offset (off_t, 64-bit typically) to ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset instead.

Quoted 5 lines

try_lseek_copy() rounds the data and hole offsets as found by lseek() to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize to off_t to ensure this doesn't happen.

Looks ok,

--D

Quoted 31 lines
---
 misc/create_inode.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
misc/create_inode.c +3 -3
index 05aa6363..1b35c769 100644
@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
 				ptr += blen;
 				continue;
 			}
-			err = ext2fs_file_lseek(e2_file, off + bpos,
+			err = ext2fs_file_llseek(e2_file, off + bpos,
 						EXT2_SEEK_SET, NULL);
 			if (err)
 				goto fail;
@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
 		if (hole < 0)
 			return EXT2_ET_UNIMPLEMENTED;
 
-		data_blk = data & ~(fs->blocksize - 1);
-		hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
+		data_blk = data & ~(off_t)(fs->blocksize - 1);
+		hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
 		err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
 				      zerobuf);
 		if (err)
Signature · 1 line
2.11.0
#2 lore raw

Obviously this was a problem that the test suite should have spotted. Would it be simple enough to write a test suite to make a 4.1GB file, put in in a ext with -d, and the md5 the file in the file system? I was using sudo and loopback mounts which isn't really ideal for the test suite.

Ross

Quoted 52 lines · Darrick J. Wong
Quoted 8 lines · Ross Burton

When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:

copy_file_chunk() passes an offset (off_t, 64-bit typically) to ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset instead.

Quoted 5 lines

try_lseek_copy() rounds the data and hole offsets as found by lseek() to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize to off_t to ensure this doesn't happen.

Looks ok,

--D

Quoted 31 lines
---
 misc/create_inode.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
misc/create_inode.c +3 -4
index 05aa6363..1b35c769 100644
@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
                              ptr += blen;
                              continue;
                      }
-                     err = ext2fs_file_lseek(e2_file, off + bpos,
+                     err = ext2fs_file_llseek(e2_file, off + bpos,
                                              EXT2_SEEK_SET, NULL);
                      if (err)
                              goto fail;
@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
              if (hole < 0)
                      return EXT2_ET_UNIMPLEMENTED;
 
-             data_blk = data & ~(fs->blocksize - 1);
-             hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
+             data_blk = data & ~(off_t)(fs->blocksize - 1);
+             hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
              err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
                                    zerobuf);
              if (err)
--

2.11.0

#3 lore raw
In reply to Darrick J. Wong

Gentle reminder to the maintainers that this patch hasn't been merged yet.

Ross

Quoted 52 lines · Darrick J. Wong
Quoted 8 lines · Ross Burton

When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:

copy_file_chunk() passes an offset (off_t, 64-bit typically) to ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset instead.

Quoted 5 lines

try_lseek_copy() rounds the data and hole offsets as found by lseek() to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize to off_t to ensure this doesn't happen.

Looks ok,

--D

Quoted 31 lines
---
 misc/create_inode.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
misc/create_inode.c +3 -4
index 05aa6363..1b35c769 100644
@@ -438,7 +438,7 @@ static errcode_t copy_file_chunk(ext2_filsys fs, int fd, ext2_file_t e2_file,
                              ptr += blen;
                              continue;
                      }
-                     err = ext2fs_file_lseek(e2_file, off + bpos,
+                     err = ext2fs_file_llseek(e2_file, off + bpos,
                                              EXT2_SEEK_SET, NULL);
                      if (err)
                              goto fail;
@@ -480,8 +480,8 @@ static errcode_t try_lseek_copy(ext2_filsys fs, int fd, struct stat *statbuf,
              if (hole < 0)
                      return EXT2_ET_UNIMPLEMENTED;
 
-             data_blk = data & ~(fs->blocksize - 1);
-             hole_blk = (hole + (fs->blocksize - 1)) & ~(fs->blocksize - 1);
+             data_blk = data & ~(off_t)(fs->blocksize - 1);
+             hole_blk = (hole + (off_t)(fs->blocksize - 1)) & ~(off_t)(fs->blocksize - 1);
              err = copy_file_chunk(fs, fd, e2_file, data_blk, hole_blk, buf,
                                    zerobuf);
              if (err)
--

2.11.0

#4 lore raw
In reply to Darrick J. Wong
Quoted 9 lines · Darrick J. Wong
Quoted 8 lines · Ross Burton

When copying large files into a ext filesystem at mkfs time the copy fails at 2^31 bytes in. There are two problems:

copy_file_chunk() passes an offset (off_t, 64-bit typically) to ext2fs_file_lseek() which expects a ext2_off_t (typedef to __u32) so the value is truncated. Solve by calling ext2fs_file_llseek() which takes a u64 offset instead.

Quoted 8 lines
Quoted 5 lines

try_lseek_copy() rounds the data and hole offsets as found by lseek() to block boundaries, but the calculation gets truncated to 32-bits. Solve by casting the 32-bit blocksize to off_t to ensure this doesn't happen.

Looks ok,

Thanks, applied.

- Ted