DD-ing from device to device.
Hi,
I should NOT get a "file too large" error when copying from a device to a device, right?
I should NOT get a "file too large" if the files are openeed using the "O_LARGEFILE" option, right?
Well:
execve("/bin/dd", ["dd", "if=/dev/hda", "of=/dev/hdc", "bs=1024k", "count=10"], [/* 46 vars */]) = 0 [... libs and stuff ... ]open("/dev/hda", O_RDONLY|O_LARGEFILE) = 4open("/dev/hdc", O_RDWR|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 5
[....signals and stuff. ]read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576
munmap(0x400fb000, 1052672) = 0 write(2, "10+0 records in\n", 16) = 16 write(2, "10+0 records out\n", 17) = 17 close(4) = 0 close(5) = 0 _exit(0) = ?
But without the "count=10" I get:
read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048576read(4, ""..., 1048576) = 1048576write(5, ""..., 1048576) = 1048575write(5, ".", 1) = -1 EFBIG (File too large)
This is on 2.2.14. I Could swear we made a working copy of a disk 30 minutes earlier....
Roger.
Signature · 4 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * There are old pilots, and there are bold pilots. * There are also old, bald pilots.
Quoted 14 lines · Rogier Wolff
I should NOT get a "file too large" error when copying from a device to a device, right?
I should NOT get a "file too large" if the files are openeed using the "
O_LARGEFILE" option, right?
read(4, ""..., 1048576)= 1048576write(5, ""..., 1048576)= 1048576read(4, ""..., 1048576)= 1048576write(5, ""..., 1048576)= 1048575write(5, ".", 1)= -1 EFBIG (File too large)
Quoted 2 lines
This is on 2.2.14. I Could swear we made a working copy of a disk 30 minutes earlier....
Hmm, you mean 2.4.14 I take it? There is another report saying 2.4.14 also "Creating partitions under 2.4.14", and I have read several more recently but am unsure of the exact kernel version. What fs are you using, just in case it matters?
I know for sure that 2.4.13+ext3 is working mostly OK, as I have been playing with multi-TB file sizes (sparse of course) although there is a minor bug in the case where you hit the fs size maximum. I'm glad my patch isn't in yet, or I would be getting flak over this I'm sure.
The only problem is that I can't see anything in the 2.4.14 patch which would cause this problem. All the previous reports had to do with ulimit, caused by su'ing to root instead of logging into root, but I'm not sure exactly where the problem lies.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 15 lines · Andreas Dilger
Quoted 14 lines · Rogier Wolff
I should NOT get a "file too large" error when copying from a device to a device, right?
I should NOT get a "file too large" if the files are openeed using the "
O_LARGEFILE" option, right?
read(4, ""..., 1048576)= 1048576write(5, ""..., 1048576)= 1048576read(4, ""..., 1048576)= 1048576write(5, ""..., 1048576)= 1048575write(5, ".", 1)= -1 EFBIG (File too large)
Quoted 4 lines
Quoted 2 lines
This is on 2.2.14. I Could swear we made a working copy of a disk 30 minutes earlier....
Hmm, you mean 2.4.14 I take it?
Typo. Yes.
Quoted 4 lines
There is another report saying 2.4.14
also "Creating partitions under 2.4.14", and I have read several more recently but am unsure of the exact kernel version. What fs are you using, just in case it matters?
ext2.
It first worked on one machine, then we moved the harddisk to another machine and it suddenly stopped working as described above.
We since then moved back to the first machine, and it worked again. Then we moved to the second machine, which now works great too.
In short: Cannot reproduce anymore....
Quoted 9 lines
I know for sure that 2.4.13+ext3 is working mostly OK, as I have been playing with multi-TB file sizes (sparse of course) although there is a minor bug in the case where you hit the fs size maximum. I'm glad my patch isn't in yet, or I would be getting flak over this I'm sure.
The only problem is that I can't see anything in the 2.4.14 patch which would cause this problem. All the previous reports had to do with ulimit, caused by su'ing to root instead of logging into root, but I'm not sure exactly where the problem lies.
Gotcha!!!!
The "wouldn't work" case was tested by me, logged in as wolff, su-ing to root, and the "works just fine" cases were tested by a guy who logs in to the machine on the console (as root).
Now, can someone tell me why "unlimited" is interpreted somehow as 2G or something thereabouts? :
/home/wolff> limit
cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize unlimited memoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 4095 openfiles 1024 /home/wolff> su Password: /home/wolff# limit cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize unlimited memoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 4095 openfiles 1024
/home/wolff# cat /proc/version
Linux version 2.4.9 (wolff@machine) (gcc version 2.95.2 19991024 (release)) #3 SMP
Mon Sep 10 09:17:17 BST 2001
/home/wolff#
(The machine was downgraded due to other problems. )
Roger.
Signature · 4 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * There are old pilots, and there are bold pilots. * There are also old, bald pilots.
Quoted 6 lines · Rogier Wolff
Now, can someone tell me why "unlimited" is interpreted somehow as 2G or something thereabouts? :
/home/wolff> limit
cputime unlimited
filesize unlimited
I seem to recall seeing something about this on lkml: the kernel considers 0xffffffff to be unlimited internally, but the setrlimit syscall converts user-supplied 0xffffffff into 0x7fffffff.
Here's the relevant thread:
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0111.1/0086.html
Marius Gedminas
Signature · 2 lines
"If it ain't broke, don't fix it." - Bert Lantz
Quoted 6 lines · Rogier Wolff
Quoted 4 lines
There is another report saying 2.4.14
also "Creating partitions under 2.4.14", and I have read several more recently but am unsure of the exact kernel version. What fs are you using, just in case it matters?ext2.
Well, I just tried this on ext2 instead of ext3 (on my 2.4.13 system) and it worked fine as a logged-in non-root user (creates a 16GB sparse file):
dd if=/dev/zero of=tt bs=1k count=1 seek=16M
Quoted 36 lines
Quoted 9 lines
I know for sure that 2.4.13+ext3 is working mostly OK, as I have been playing with multi-TB file sizes (sparse of course) although there is a minor bug in the case where you hit the fs size maximum. I'm glad my patch isn't in yet, or I would be getting flak over this I'm sure.
The only problem is that I can't see anything in the 2.4.14 patch which would cause this problem. All the previous reports had to do with ulimit, caused by su'ing to root instead of logging into root, but I'm not sure exactly where the problem lies.
Gotcha!!!!
The "wouldn't work" case was tested by me, logged in as wolff, su-ing to root, and the "works just fine" cases were tested by a guy who logs in to the machine on the console (as root).
Now, can someone tell me why "unlimited" is interpreted somehow as 2G or something thereabouts? :
/home/wolff> limitcputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize unlimited memoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 4095 openfiles 1024 /home/wolff> su Password: /home/wolff# limit cputime unlimited filesize unlimited
Quoted 8 lines
datasize unlimited stacksize unlimited coredumpsize unlimited memoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 4095 openfiles 1024
Well, because of 32-bit API issues "unlimited" actually IS the same as 2G for 32-bit systems, but the code internally checks if the limit is equal to RLIM_INFINITY (mm/filemap.c:generic_file_write()) and (should) ignore it if so. Thus it is impossible to set a ulimit exactly 2GB, but that isn't really a problem.
Hmm, looking at the user-space header <customs/customs.h>, it has RLIM_INFINITY as 0x7fffffff, the <bits/resource.h> has:
#ifndef __USE_FILE_OFFSET64
# define RLIM_INFINITY ((long int) (~0UL >> 1)) #else # define RLIM_INFINITY 0x7fffffffffffffffLL #endif
but the kernel code has RLIM_INFINITY as ~0UL for most arches.
Quoted 6 lines
/home/wolff# cat
/proc/version
Linux version 2.4.9 (wolff@machine) (gcc version 2.95.2 19991024 (release)) #3 SMP
Mon Sep 10 09:17:17 BST 2001
/home/wolff#(The machine was downgraded due to other problems. )
Can you test the "dd" above to ensure it works with your tools and the old kernel? For your next 2.4.14 kernel build, it may be instructive to put a printk() inside the 3 checks in generic_file_write() before it outputs SIGXFSZ, which tells us limit and RLIM_INIFINITY, pos and count, and pos and s_maxbytes are, respectively. This will also tell us what limit is being hit (although it is most likely a ulimit issue).
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 6 lines · Andreas Dilger
Quoted 5 lines · Rogier Wolff
Quoted 4 lines
There is another report saying 2.4.14
also "Creating partitions under 2.4.14", and I have read several more recently but am unsure of the exact kernel version. What fs are you using, just in case it matters?
Quoted 6 lines
Quoted 1 line
ext2.
Well, I just tried this on ext2 instead of ext3 (on my 2.4.13 system) and it worked fine as a logged-in non-root user (creates a 16GB sparse file):
dd if=/dev/zero of=tt bs=1k count=1 seek=16M
/tmp> dd if=/dev/zero of=tt bs=1k count=1 seek=16M
dd: tt: Invalid argument
1+0 records in
1+0 records out
/tmp> dd if=/dev/zero of=tt bs=1k seek=2047k
19913+0 records in
19912+0 records out
^C
/tmp> ls -al tt
ls: tt: Value too large for defined data type
/tmp> su
Password:
/tmp# rm tt
rm: cannot remove `tt': Value too large for defined data type
/tmp# mv tt xx
mv: tt: Value too large for defined data type
/tmp# rm -f tt
rm: cannot remove `tt': Value too large for defined data type
/tmp# dd if=/dev/zero of=uu bs=1k count=2050 seek=2047k
2050+0 records in
2050+0 records out
/tmp# l uu
ls: uu: Value too large for defined data type
/tmp#
Quoted 6 lines
Can you test the "dd" above to ensure it works with your tools and the old kernel? For your next 2.4.14 kernel build, it may be instructive to put a
printk()inside the 3 checks ingeneric_file_write()before it outputs SIGXFSZ, which tells us limit andRLIM_INIFINITY, pos and count, and pos ands_maxbytesare, respectively. This will also tell us what limit is being hit (although it is most likely a ulimit issue).
Grmbl... I'll see what I can do.
Rogier.
Signature · 4 lines
** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 ** *-- BitWizard writes Linux device drivers for any device you may have! --* * There are old pilots, and there are bold pilots. * There are also old, bald pilots.
Quoted 8 lines · Rogier Wolff
Quoted 1 line · Andreas Dilger
dd if=/dev/zero of=tt bs=1k count=1 seek=16M
/tmp> dd if=/dev/zero of=tt bs=1k count=1 seek=16M
dd: tt: Invalid argument
1+0 records in
1+0 records out
Invalid argument is probably from ftruncate.
Quoted 12 lines
/tmp> dd if=/dev/zero of=tt bs=1k seek=2047k
19913+0 records in
19912+0 records out
^C
/tmp> ls -al tt
ls: tt: Value too large for defined data type
/tmp> su
Password:
/tmp# rm tt
rm: cannot remove `tt': Value too large for defined data type
/tmp# mv tt xx
mv: tt: Value too large for defined data type
Quoted 8 lines
/tmp# rm -f tt
rm: cannot remove `tt': Value too large for defined data type
/tmp# dd if=/dev/zero of=uu bs=1k count=2050 seek=2047k
2050+0 records in
2050+0 records out
/tmp# l uu
ls: uu: Value too large for defined data type
/tmp#
Looks like your fileutils and/or shell and/or glibc are conspiring against you.
Quoted 8 lines
Quoted 6 lines
Can you test the "dd" above to ensure it works with your tools and the old kernel? For your next 2.4.14 kernel build, it may be instructive to put a
printk()inside the 3 checks ingeneric_file_write()before it outputs SIGXFSZ, which tells us limit andRLIM_INIFINITY, pos and count, and pos ands_maxbytesare, respectively. This will also tell us what limit is being hit (although it is most likely a ulimit issue).Grmbl... I'll see what I can do.
Start by upgrading your tools to largefile aware ones.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/