how can I copy files bigger than ~32 GB when using compress-force?
Quoted 3 lines
I'm assuming this works without compress-force? I can make a guess at what is happening, the compression forces a relatively small extent size, and this is making our worst case metadata reservations get upset.
Yes, it works without compress-force.
Interesting is that cp or rsync sometimes just exit quite fast with "no space left".
Sometimes, they just "hang" (waited up to about an hour) - file size does not grow anymore, last modified time is not updated, iostat does not show any bytes read/written, there are no btrfs or any other processes taking too much CPU, cp/rsync is not in "D" state (although it gets to "D" state and uses 100% CPU as I try to kill it).
Could it be we're hitting two different bugs here?
Quoted 1 line
Does it happen with any 32gb file that doesn't compress well?
The 220 GB qcow2 file was basically uncompressible (backuppc archive full of bzip2-compressed files).
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 11 lines · Tomasz Chmielewski
Quoted 3 lines
I'm assuming this works without compress-force? I can make a guess at what is happening, the compression forces a relatively small extent size, and this is making our worst case metadata reservations get upset.
Yes, it works without compress-force.
Interesting is that cp or rsync sometimes just exit quite fast with "no space left".
Sometimes, they just "hang" (waited up to about an hour) - file size does not grow anymore, last modified time is not updated, iostat
Sorry is this hang/fast exit with or without compress-force.
Quoted 5 lines
does not show any bytes read/written, there are no btrfs or any other processes taking too much CPU, cp/rsync is not in "D" state (although it gets to "D" state and uses 100% CPU as I try to kill it).
Quoted 7 lines
Could it be we're hitting two different bugs here?
Quoted 1 line
Does it happen with any 32gb file that doesn't compress well?
The 220 GB qcow2 file was basically uncompressible (backuppc archive full of bzip2-compressed files).
Ok, I think I know what is happening here, they all lead to the same chunk of code. I'll be able to reproduce this locally.
-chris
I once tried to copy a chroot onto a USB mount, mounted with the 'compress,noatime' option, and the same thing kept happening to me. I started using slower methods to transfer the files over, and I no longer encountered the hanging problem. I was not copying any big files, but it was a chroot, so there were lots of small files.
On Mon, Oct 4, 2010 at 5:28 PM, Chris Mason <[email protected]> wr= ote:
Quoted 3 lines · Tomasz Chmielewski
Quoted 1 line
Quoted 1 line
I'm assuming this works without compress-force? =A0I can make a gu=
ess at
Quoted 1 line
Quoted 1 line
Quoted 1 line
what is happening, the compression forces a relatively small exten=
t
Quoted 1 line
Quoted 1 line
Quoted 1 line
size, and this is making our worst case metadata reservations get =
upset.
Quoted 7 lines
Quoted 7 lines
Yes, it works without compress-force.
Interesting is that cp or rsync sometimes just exit quite fast with "no space left".
Sometimes, they just "hang" (waited up to about an hour) - file siz=
e
Quoted 15 lines
Quoted 1 line
does not grow anymore, last modified time is not updated, iostat
Sorry is this hang/fast exit with or without compress-force.
Quoted 11 lines
does not show any bytes read/written, there are no btrfs or any other processes taking too much CPU, cp/rsync is not in "D" state (although it gets to "D" state and uses 100% CPU as I try to kill it).
Could it be we're hitting two different bugs here?
Quoted 1 line
Does it happen with any 32gb file that doesn't compress well?
The 220 GB qcow2 file was basically uncompressible (backuppc archiv=
e
Quoted 9 lines
Quoted 1 line
full of bzip2-compressed files).
Ok, I think I know what is happening here, they all lead to the same chunk of code. =A0I'll be able to reproduce this locally.
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
Quoted 2 lines
the body of a message to [email protected]
More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Chris Mason
Quoted 6 lines · Tomasz Chmielewski
Quoted 3 lines
I'm assuming this works without compress-force? I can make a guess at what is happening, the compression forces a relatively small extent size, and this is making our worst case metadata reservations get upset.
Yes, it works without compress-force.
Quoted 7 lines
Quoted 5 lines
Interesting is that cp or rsync sometimes just exit quite fast with "no space left".
Sometimes, they just "hang" (waited up to about an hour) - file size does not grow anymore, last modified time is not updated, iostat
Sorry is this hang/fast exit with or without compress-force.
It only seems to happen with compress-force.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 7 lines · Chris Mason
Quoted 4 lines
Quoted 1 line
Does it happen with any 32gb file that doesn't compress well?
The 220 GB qcow2 file was basically uncompressible (backuppc archive full of bzip2-compressed files).
Ok, I think I know what is happening here, they all lead to the same chunk of code. I'll be able to reproduce this locally.
FYI, qemu/kvm doesn't seem to like its files located on btrfs mounted with compress-force.
I have a filesystem mounted with noatime,compress-force, where I created a 100GB sparse file.
There, I wanted to install a Linux distribution - however, the whole qemu-kvm process hanged with these entries being repeated over and over. It's not possible to kill the qemu-kvm process (even with kill -9) etc.
[103678.068429] INFO: task qemu-kvm:18722 blocked for more than 120 seconds. [103678.068434] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [103678.068439] qemu-kvm D 00000001062a02ba 0 18722 18607 0x00000000 [103678.068446] ffff88008d80bb38 0000000000000082 ffff880000000000 00000000000148c0 [103678.068453] ffff88008725db00 00000000000148c0 ffff88008d80bfd8 00000000000148c0 [103678.068459] ffff88008d80a000 00000000000148c0 ffff88008725dea8 00000000000148c0 [103678.068466] Call Trace: [103678.068490] [<ffffffffa06a3ee5>] btrfs_start_ordered_extent+0x75/0xc0 [btrfs] [103678.068499] [<ffffffff8107e5e0>] ? autoremove_wake_function+0x0/0x40 [103678.068517] [<ffffffffa06a44a1>] btrfs_wait_ordered_range+0xc1/0x160 [btrfs] [103678.068535] [<ffffffffa0695922>] btrfs_file_aio_write+0x542/0x9a0 [btrfs] [103678.068542] [<ffffffff8108f562>] ? futex_wait_queue_me+0xc2/0x100 [103678.068549] [<ffffffff81082462>] ? hrtimer_cancel+0x22/0x30 [103678.068568] [<ffffffffa06953e0>] ? btrfs_file_aio_write+0x0/0x9a0 [btrfs] [103678.068576] [<ffffffff81144533>] do_sync_readv_writev+0xd3/0x110 [103678.068583] [<ffffffff81091db9>] ? do_futex+0x109/0xb00 [103678.068590] [<ffffffff811d9789>] ? security_file_permission+0x29/0xa0 [103678.068597] [<ffffffff811447f4>] do_readv_writev+0xd4/0x1f0 [103678.068603] [<ffffffff810714ef>] ? kill_pid_info+0x3f/0x60 [103678.068610] [<ffffffff81144958>] vfs_writev+0x48/0x60 [103678.068615] [<ffffffff81144c92>] sys_pwritev+0xa2/0xc0 [103678.068623] [<ffffffff8100af82>] system_call_fastpath+0x16/0x1b
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 10 lines · Tomasz Chmielewski
Quoted 7 lines · Chris Mason
Quoted 4 lines
Quoted 1 line
Does it happen with any 32gb file that doesn't compress well?
The 220 GB qcow2 file was basically uncompressible (backuppc archive full of bzip2-compressed files).
Ok, I think I know what is happening here, they all lead to the same chunk of code. I'll be able to reproduce this locally.
Quoted 8 lines
FYI, qemu/kvm doesn't seem to like its files located on btrfs mounted with compress-force.
I have a filesystem mounted with noatime,compress-force, where I created a 100GB sparse file.
There, I wanted to install a Linux distribution - however, the whole qemu-kvm process hanged with these entries being repeated over and over. It's not possible to kill the qemu-kvm process (even with kill -9) etc.[103678.068429] INFO: task qemu-kvm:18722 blocked for more than 120 seconds.
Hmm, I see it blocks infinitely like this whether qemu-kvm tries to access a sparse file mounted with compress, compress-force, or no compression at all.
Also hangs with non-sparse files mounted without compression (didn't try with compression).
So must be some other bug.
And, I see occasional:
bad ordered accounting left 4096 size 12288
when it happens.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org