No space left on device (28)
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00
# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt
# btrfs fi show
Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
Total devices 1 FS bytes used 15.07TB
devid 1 size 18.19TB used 18.19TB path /dev/dm-3# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunks
What wrong here? What is about the last 3TB?
Greets.
Stefan
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 23 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3
Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunksWhat wrong here? What is about the last 3TB?
Which kernel are you running?
-chris
Hi Chris,
Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:
Quoted 24 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 23 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3
Quoted 8 lines
Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunksWhat wrong here? What is about the last 3TB?
Which kernel are you running?
-chris
vanilla 3.8.3.
Stefan
Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
Quoted 31 lines
Hi Chris,
Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:
Quoted 27 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 26 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunks
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
What wrong here? What is about the last 3TB?
Which kernel are you running?
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
-chris
Yes, i've now upgraded to 3.9-rc3 same result.
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": No space left on device (28)
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.98GB Metadata: total=8.00MB, used=0.00
Stefan
Am 21.03.2013 20:02, schrieb Chris Mason:
Quoted 45 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
Quoted 31 lines
Hi Chris,
Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:
Quoted 27 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 26 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunksQuoted 7 lines
Quoted 5 lines
Quoted 1 line
What wrong here? What is about the last 3TB?
Which kernel are you running?
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
-chris
Should i try to downgrade?
Greets,
Stefan
Am 21.03.2013 um 20:42 schrieb Stefan Priebe <[email protected]>:
Quoted 59 lines
Yes, i've now upgraded to 3.9-rc3 same result.
rsync: rename "
/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": No space left on device (28)# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.98GB Metadata: total=8.00MB, used=0.00Stefan
Am 21.03.2013 20:02, schrieb Chris Mason:
Quoted 45 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
Quoted 31 lines
Hi Chris,
Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:
Quoted 27 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 26 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunksQuoted 7 lines
Quoted 5 lines
Quoted 1 line
What wrong here? What is about the last 3TB?
Which kernel are you running?
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
-chris
On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:
I might be wrong here, but doesn't this
Quoted 4 lines
rsync: rename "
/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
...try to move a file from
"/mnt/.software/"
to
".software/"
(relative to current dir)??
Maybe you end up trying to rsync your array to a small root partition or something?
Quoted 61 lines
No space left on device (28)
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.98GB Metadata: total=8.00MB, used=0.00Stefan
Am 21.03.2013 20:02, schrieb Chris Mason:
Quoted 36 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
Quoted 35 lines
Hi Chris,
Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:
Quoted 31 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 23 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunksWhat wrong here? What is about the last 3TB?
Which kernel are you running?
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
With respect, Roman
Quoted 12 lines · Roman Mamedov
On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:I might be wrong here, but doesn't this
Quoted 4 lines
rsync: rename "
/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/"" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":...try to move a file from
Quoted 7 lines
"
/mnt/.software/"to
".software/"
(relative to current dir)??
No; that's rsync giving the full path, and then the target path relative to the command it was given. The filename itself (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to temporarily, so it can mv it over the original in an atomic fashion...
Stefan: ...which means that the actual copy succeeded, which suggests that this is more of a metadata enospc thing.
You might try btrfs balance start -musage=5 (instead of -dusage), and if that doesn't report any chunks balanced, try a high number until it does.
No this is fine it's rsync style same happens with cp.
Am 22.03.2013 um 07:13 schrieb Roman Mamedov <[email protected]>:
Quoted 88 lines
On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:I might be wrong here, but doesn't this
Quoted 4 lines
rsync: rename "
/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":...try to move a file from
"
/mnt/.software/"to
".software/"
(relative to current dir)??
Maybe you end up trying to rsync your array to a small root partition or something?
Quoted 60 lines
No space left on device (28)
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.98GB Metadata: total=8.00MB, used=0.00Stefan
Am 21.03.2013 20:02, schrieb Chris Mason:
Quoted 36 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
Quoted 35 lines
Hi Chris,
Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:
Quoted 31 lines
Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
Quoted 23 lines
Hi,
while copying a 15GB file to my btrfs volume i'm getting "No space left on device".
Some information:
# btrfs filesystem df /mnt/ Data: total=18.14TB, used=15.05TB System, DUP: total=8.00MB, used=1.94MB System: total=4.00MB, used=0.00 Metadata, DUP: total=23.98GB, used=17.97GB Metadata: total=8.00MB, used=0.00# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/raid54tb1 19T 16T 3,1T 84% /mnt# btrfs fi show Label: none uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a Total devices 1 FS bytes used 15.07TB devid 1 size 18.19TB used 18.19TB path /dev/dm-3Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/ Done, had to relocate 0 out of 18604 chunksWhat wrong here? What is about the last 3TB?
Which kernel are you running?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 2 lines
With respect, Roman
Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%
Am 22.03.2013 um 07:24 schrieb cwillu <[email protected]>:
Quoted 32 lines · Roman Mamedov
Quoted 12 lines
On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:I might be wrong here, but doesn't this
Quoted 4 lines
rsync: rename "
/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/"" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":...try to move a file from
Quoted 7 lines
"
/mnt/.software/"to
".software/"
(relative to current dir)??
No; that's rsync giving the full path, and then the target path relative to the command it was given. The filename itself (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to temporarily, so it can mv it over the original in an atomic fashion...
Stefan: ...which means that the actual copy succeeded, which suggests that this is more of a metadata enospc thing.
You might try btrfs balance start -musage=5 (instead of -dusage), and if that doesn't report any chunks balanced, try a high number until it does.
Quoted 1 line · Stefan Priebe - Profihost AG
Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%
You tried -musage=5? Your original email said -dusage=5.
Hi,
Am 22.03.2013 07:41, schrieb cwillu:> On Fri, Mar 22, 2013 at 12:39 AM, Stefan Priebe - Profihost AG
Quoted 5 lines
Quoted 2 lines · <[email protected]> wrote:
Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%
You tried -musage=5? Your original email said -dusage=5.
sorry i missed the difference - but it does not work:
~# btrfs fi balance start -musage=5 /mnt/
ERROR: error during balancing '/mnt/' - No space left on device There may be more info in syslog - try dmesg | tail
~# dmesg|tail
[ 1183.019367] device fsid fc23c4a8-a351-4c91-96a2-ee6abeffe59a devid 1 transid 6224 /dev/mapper/raid54tb1 [ 1183.019860] btrfs: disk space caching is enabled [42719.915781] btrfs: relocating block group 4194304 flags 4 [42727.916141] btrfs: 5 enospc errors during balance
Stefan
Hi Chris,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Which kernel are you running?
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Greets,
Stefan
Quoted 4 lines · Stefan Priebe - Profihost AG
Hi Chris,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Which kernel are you running?
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,
Josef
Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:
Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 8 lines
Hi Chris,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Which kernel are you running?
-chris
vanilla 3.8.3.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,
Great!
Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.
Stefan
Quoted 17 lines · Stefan Priebe - Profihost AG
Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:Quoted 15 lines · Stefan Priebe - Profihost AG
Quoted 13 lines
Hi Chris,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Which kernel are you running?
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Quoted 8 lines
Quoted 2 lines
Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,
Great!
Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.
Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,
Josef
fs/btrfs/extent-tree.c +5 -0
@@ -4161,6 +4161,11 @@ out: ret =0 ; }if (flushing) { +if (ret == -ENOSPC ) { +printk (KERN_ERR "returning enospc, dumping space info\n" ); +dump_space_info (space_info,0 ,0 ); + } +spin_lock (&space_info->lock); space_info->flush =0 ;wake_up_all (&space_info->wait);
fs/btrfs/inode.c +2 -4
@@ -3679,11 +3679,9 @@ static struct btrfs_trans_handle *__unlink_start_trans(struct inode *dir,* 1 for the dir item * 1 for the dir index * 1 for the inode ref -* 1 for the inode ref in the tree log -* 2 for the dir entries in the log * 1 for the inode */ - trans =btrfs_start_transaction (root,8 ); + trans =btrfs_start_transaction (root,5 );if (!IS_ERR (trans) ||PTR_ERR (trans) != -ENOSPC )return trans;
@@ -8127,7 +8125,7 @@ static int btrfs_rename(struct inode *old_dir, struct dentry *old_dentry,* inodes. So 5 * 2 is 10, plus 1 for the new link, so 11 total items * should cover the worst case number of items we'll modify. */ - trans =btrfs_start_transaction (root,20 ); + trans =btrfs_start_transaction (root,11 );if (IS_ERR (trans)) { ret =PTR_ERR (trans); goto out_notrans;
Signature · 1 line
1.7.7.6
Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:
Quoted 21 lines · Stefan Priebe - Profihost AG
Quoted 20 lines
Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:Quoted 17 lines · Stefan Priebe - Profihost AG
Quoted 4 lines
Hi Chris,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Which kernel are you running?
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
-chris
vanilla 3.8.3.
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,
Quoted 8 lines
Quoted 5 lines
Great!
Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.
Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,
Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some.
Rsync output:.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-wm8350.c
12156 100% 11.17kB/s 0:00:01 (xfer#21395, to-check=1008/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-x1205.c
16460 100% 15.05kB/s 0:00:01 (xfer#21396, to-check=1007/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/systohc.c
1223 100% 1.12kB/s 0:00:01 (xfer#21397, to-check=1006/52625)
.software/kernel/linux-3.9-rc3/drivers/s390/
.software/kernel/linux-3.9-rc3/drivers/s390/Makefile
144 100% 0.13kB/s 0:00:01 (xfer#21398, to-check=1052/52672).software/kernel/linux-3.9-rc3/drivers/s390/block/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/block" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/char/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/char" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/cio/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/cio" failed: No space left on device (28)
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.88pm8607.c.YBi0Rz" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/crypto/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/crypto" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/kvm/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/kvm" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/net/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/net" failed: No space left on device (28)
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Kconfig.0lsfuE" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Makefile.C9wI7I" failed: No space left on device (28) .software/kernel/linux-3.9-rc3/drivers/s390/scsi/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/scsi" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/sbus/
.software/kernel/linux-3.9-rc3/drivers/sbus/Makefile
70 100% 0.06kB/s 0:00:01 (xfer#21399, to-check=1003/52824)
.software/kernel/linux-3.9-rc3/drivers/sbus/char/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/sbus/char" failed: No space left on device (28)
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.aat2870-regulator.c.BGDwPN" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/scsi/
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.ko.cmd
208 100% 0.16kB/s 0:00:01 (xfer#21400, to-check=1407/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.mod.o.cmd
27987 100% 21.54kB/s 0:00:01 (xfer#21401, to-check=1406/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.o.cmd
43340 100% 32.63kB/s 0:00:01 (xfer#21402, to-check=1405/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.ko.cmd
204 100% 15.32kB/s 0:00:00 (xfer#21403, to-check=1404/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.mod.o.cmd
27975 100% 1.91MB/s 0:00:00 (xfer#21404, to-check=1403/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.o.cmd
43327 100% 983.99kB/s 0:00:00 (xfer#21405, to-check=1402/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.ko.cmd
208 100% 4.72kB/s 0:00:00 (xfer#21406, to-check=1401/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.mod.o.cmd
27987 100% 621.16kB/s 0:00:00 (xfer#21407, to-check=1400/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.o.cmd
43340 100% 742.53kB/s 0:00:00 (xfer#21408, to-check=1399/53242)dmesg Output was pretty long so i posted to pastebin:
http://pastebin.com/raw.php?i=5kUS5MGA
Greets,
Stefan
Quoted 28 lines · Stefan Priebe
Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:Quoted 26 lines · Stefan Priebe - Profihost AG
Quoted 25 lines
Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 8 lines
Hi Chris,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Which kernel are you running?
-chris
vanilla 3.8.3.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,
Great!
Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.
Quoted 7 lines
Quoted 3 lines
Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,
Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some.
Ok so I've been working on another ENOSPC related problem that I think is affecting you as well. So I'm going to nail that down and see if that helps you too, if not we'll poke around your problem some more and try and work out what's happening. I'll get back to this fresh Monday morning. Thanks,
Josef
Hi Jsoef,
thanks!
Am 22.03.2013 21:49, schrieb Josef Bacik:
Quoted 44 lines · Stefan Priebe
Quoted 28 lines
Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:Quoted 26 lines · Stefan Priebe - Profihost AG
Quoted 25 lines
Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 8 lines
Hi Chris,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Which kernel are you running?
-chris
vanilla 3.8.3.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?
any ideas what i can check? 3.9-rc3 gives me same results.
Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,
Great!
Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.
Quoted 7 lines
Quoted 3 lines
Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,
Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some.
Ok so I've been working on another ENOSPC related problem that I think is affecting you as well. So I'm going to nail that down and see if that helps you too, if not we'll poke around your problem some more and try and work out what's happening. I'll get back to this fresh Monday morning. Thanks,
Josef
Quoted 3 lines · Stefan Priebe
Hi Jsoef,
thanks!
Ok I don't think the thing I just fixed will make any difference for you so here's another debug patch, just apply it on top of what I've already sent you and re-run and give me the dmesg again. Thanks,
Josef
fs/btrfs/extent-tree.c +5 -1
@@ -4162,7 +4162,11 @@ out: }if (flushing) {if (ret == -ENOSPC ) { -printk (KERN_ERR "returning enospc, dumping space info\n" ); +spin_lock (&block_rsv->lock); +printk (KERN_ERR "returning enospc, space_info %u, size %Lu reserved %Lu, " +"flush %d, flush_state %d dumping space info\n" , block_rsv->type, + block_rsv->size, block_rsv->reserved, flush, flush_state); +spin_unlock (&block_rsv->lock);dump_space_info (space_info,0 ,0 ); }
Signature · 1 line
1.7.7.6
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0, reserved=73728, may_use=6438297600, readonly=65536
Am 25.03.2013 21:14, schrieb Josef Bacik:
Quoted 30 lines · Stefan Priebe
Quoted 3 lines
Hi Jsoef,
thanks!
Ok I don't think the thing I just fixed will make any difference for you so here's another debug patch, just apply it on top of what I've already sent you and re-run and give me the dmesg again. Thanks,
Josef
fs/btrfs/extent-tree.c +5
-1@@ -4162,7 +4162,11 @@ out: }if (flushing) {if (ret == -ENOSPC ) { -printk (KERN_ERR "returning enospc, dumping space info\n" ); +spin_lock (&block_rsv->lock); +printk (KERN_ERR "returning enospc, space_info %u, size %Lu reserved %Lu, " +"flush %d, flush_state %d dumping space info\n" , block_rsv->type, + block_rsv->size, block_rsv->reserved, flush, flush_state); +spin_unlock (&block_rsv->lock);dump_space_info (space_info,0 ,0 ); }
Quoted 27 lines · Stefan Priebe
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,
Quoted 2 lines
reserved=73728, may_use=6438297600, readonly=65536
Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,
Josef
Hi Josef,
Am 26.03.2013 13:53, schrieb Josef Bacik:
Quoted 31 lines · Stefan Priebe
Quoted 27 lines
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,Quoted 2 lines
reserved=73728, may_use=6438297600, readonly=65536
Quoted 2 lines
Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,
no - it's just mounted with mount -o noatime
:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
Greets,
Stefan
Quoted 34 lines · Stefan Priebe - Profihost AG
Hi Josef,
Am 26.03.2013 13:53, schrieb Josef Bacik:
Quoted 31 lines · Stefan Priebe
Quoted 29 lines
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,reserved=73728, may_use=6438297600, readonly=65536
Quoted 8 lines
Quoted 2 lines
Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,
no - it's just mounted with mount -o noatime
:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
Josef
fs/btrfs/extent-tree.c +14 -2
@@ -3803,6 +3803,19 @@ static int can_overcommit(struct btrfs_root *root,return 0 ; } +static int btrfs_try_writeback (struct super_block *sb,unsigned long nr_pages, +enum wb_reason reason) +{ +if (!writeback_in_progress (sb->s_bdi) && +down_read_trylock (&sb->s_umount)) { +writeback_inodes_sb_nr (sb, nr_pages, reason); +up_read (&sb->s_umount); +return 1 ; + } + +return 0 ; +} +void btrfs_writeback_inodes_sb_nr (struct btrfs_root *root,unsigned long nr_pages) {
@@ -3810,8 +3823,7 @@ void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,int started;/* If we can not start writeback, just sync all the delalloc file. */ - started =try_to_writeback_inodes_sb_nr (sb, nr_pages, -WB_REASON_FS_FREE_SPACE ); + started =btrfs_try_writeback (sb, nr_pages,WB_REASON_FS_FREE_SPACE );if (!started) {/* * We needn't worry the filesystem going from r/w to r/o though
Signature · 1 line
1.7.7.6
Hi Josef,
Am 26.03.2013 14:30, schrieb Josef Bacik:
Quoted 38 lines · Stefan Priebe - Profihost AG
Quoted 37 lines
Hi Josef,
Am 26.03.2013 13:53, schrieb Josef Bacik:
Quoted 33 lines · Stefan Priebe
Quoted 27 lines
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,Quoted 2 lines
reserved=73728, may_use=6438297600, readonly=65536
Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,
Quoted 8 lines
Quoted 5 lines
no - it's just mounted with mount -o noatime
:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/ /mnt/
sending incremental file list.etc_openvpn/ipp.txt
229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log
104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
19431424 61% 7.35MB/s 0:00:01the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,
reserved=49152, may_use=6438453248, readonly=65536
Stefan
Quoted 72 lines · Stefan Priebe - Profihost AG
Hi Josef,
Am 26.03.2013 14:30, schrieb Josef Bacik:
Quoted 38 lines · Stefan Priebe - Profihost AG
Quoted 37 lines
Hi Josef,
Am 26.03.2013 13:53, schrieb Josef Bacik:
Quoted 33 lines · Stefan Priebe
Quoted 27 lines
Hi,
output here:
[ 590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.548280] space_info 4 has 6439292928 free, is full [ 590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536 [ 590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.552264] space_info 4 has 6439284736 free, is full [ 590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 590.556258] space_info 4 has 6439284736 free, is full [ 590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536 [ 591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 591.147373] space_info 4 has 6439235584 free, is full [ 591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536 [ 595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 595.562390] space_info 4 has 6439120896 free, is full [ 595.562393] space_info total=25748307968, used=19309047808, pinned=0,Quoted 2 lines
reserved=73728, may_use=6438297600, readonly=65536
Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,
Quoted 8 lines
Quoted 5 lines
no - it's just mounted with mount -o noatime
:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01
Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Ok so then this is probably it, let me know if it helps. Thanks,
Josef
fs/btrfs/ordered-data.c +2 -0
@@ -557,6 +557,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)INIT_LIST_HEAD (&splice);INIT_LIST_HEAD (&works); +mutex_lock (&root->fs_info->ordered_operations_mutex);spin_lock (&root->fs_info->ordered_extent_lock);list_splice_init (&root->fs_info->ordered_extents, &splice);while (!list_empty (&splice)) {
@@ -600,6 +601,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)cond_resched (); } +mutex_unlock (&root->fs_info->ordered_operations_mutex); }/*
Signature · 1 line
1.7.7.6
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:
Quoted 39 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Quoted 1 line
Ok so then this is probably it, let me know if it helps. Thanks,
OK it now has copied a lot of files (170) without an error all were very small.
Then this again:
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.vhost_net.mod.GrhWet" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/vhost_net.mod": No space left on device (28)
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rhine.mod.X2Ofhz" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rhine.mod": No space left on device (28)
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rng.mod.RAyxkF" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rng.mod": No space left on device (28)
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.virtio.mod.w7INoL" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/virtio.mod": No space left on device (28)^C
[ 5012.468988] space_info total=25748307968, used=19308773376, pinned=0, reserved=69632, may_use=6438354944, readonly=65536 [ 5012.472981] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 5012.472982] space_info 4 has 6439399424 free, is full [ 5012.472982] space_info total=25748307968, used=19308773376, pinned=0, reserved=69632, may_use=6438354944, readonly=65536 [ 5012.476974] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 5012.476975] space_info 4 has 6439399424 free, is full [ 5012.476976] space_info total=25748307968, used=19308773376, pinned=0, reserved=69632, may_use=6438354944, readonly=65536 [ 5012.480968] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 5012.480969] space_info 4 has 6439399424 free, is full
Stefan
Quoted 41 lines · Stefan Priebe - Profihost AG
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:Quoted 39 lines
Quoted 38 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Quoted 5 lines
Quoted 1 line
Ok so then this is probably it, let me know if it helps. Thanks,
OK it now has copied a lot of files (170) without an error all were very small.
Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,
Josef
fs/btrfs/extent-tree.c +12 -0
@@ -4026,6 +4026,15 @@ static int flush_space(struct btrfs_root *root,return ret; } + +static void dump_block_rsv (struct btrfs_block_rsv *block_rsv) +{ +spin_lock (&block_rsv->lock); +printk (KERN_ERR "dumping block rsv %u, size %Lu reserved %Lu\n" , + block_rsv->type, block_rsv->size, block_rsv->reserved); +spin_unlock (&block_rsv->lock); +} +/** * reserve_metadata_bytes - try to reserve bytes from the block_rsv's space * @root - the root we're allocating for
@@ -4179,6 +4188,9 @@ out:"flush %d, flush_state %d dumping space info\n" , block_rsv->type, block_rsv->size, block_rsv->reserved, flush, flush_state);spin_unlock (&block_rsv->lock); +dump_block_rsv (&root->fs_info->delalloc_block_rsv); +dump_block_rsv (&root->fs_info->delayed_block_rsv); +dump_block_rsv (&root->fs_info->global_block_rsv);dump_space_info (space_info,0 ,0 ); }
Signature · 1 line
1.7.7.6
HI,
Am 26.03.2013 16:25, schrieb Josef Bacik:
Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Ok so then this is probably it, let me know if it helps. Thanks,
Quoted 7 lines
Quoted 3 lines
OK it now has copied a lot of files (170) without an error all were very small.
Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,
Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438354944, readonly=65536
Stefan
Quoted 72 lines · Stefan Priebe
HI,
Am 26.03.2013 16:25, schrieb Josef Bacik:
Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Ok so then this is probably it, let me know if it helps. Thanks,
Quoted 7 lines
Quoted 3 lines
OK it now has copied a lot of files (170) without an error all were very small.
Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,
Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,
Quoted 2 lines
reserved=45056, may_use=6438354944, readonly=65536
Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,
Josef
fs/btrfs/extent-tree.c +6 -1
@@ -4471,7 +4471,12 @@ static void update_global_block_rsv(struct btrfs_fs_info *fs_info)spin_lock (&sinfo->lock);spin_lock (&block_rsv->lock); - block_rsv->size = num_bytes; +/* +* Limit the global block rsv to 512mb, we have infrastructure in place +* to throttle reservations if we start getting low on global block rsv +* space. +*/ + block_rsv->size =min_t (u64, num_bytes,512 *1024 *1024 ); num_bytes = sinfo->bytes_used + sinfo->bytes_pinned + sinfo->bytes_reserved + sinfo->bytes_readonly +
Signature · 1 line
1.7.7.6
Hi Josef,
Am 26.03.2013 18:45, schrieb Josef Bacik:
Quoted 72 lines
Quoted 69 lines
Am 26.03.2013 16:25, schrieb Josef Bacik:
Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Ok so then this is probably it, let me know if it helps. Thanks,
Quoted 7 lines
Quoted 3 lines
OK it now has copied a lot of files (170) without an error all were very small.
Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,
Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,Quoted 2 lines
reserved=45056, may_use=6438354944, readonly=65536
Quoted 2 lines
Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,
Yes - this works now. Which of the patches can i drop? Do i just need the last one? Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5?
Thanks and greets,
Stefan
Quoted 75 lines · Stefan Priebe
Hi Josef,
Am 26.03.2013 18:45, schrieb Josef Bacik:
Quoted 72 lines
Quoted 71 lines
Am 26.03.2013 16:25, schrieb Josef Bacik:
Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Ok so then this is probably it, let me know if it helps. Thanks,
Quoted 7 lines
Quoted 3 lines
OK it now has copied a lot of files (170) without an error all were very small.
Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,
Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,reserved=45056, may_use=6438354944, readonly=65536
Quoted 8 lines
Quoted 2 lines
Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,
Yes - this works now. Which of the patches can i drop? Do i just need the last one?
Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5?
So I'll put together all of the patches that actually need to go up for this and post them, but basically its the mutex patch, the last patch I sent you and the one that adjusts the reservations for rename and delete. Thanks,
Josef
Hi,
but when i transfer big files i see now this one:
[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds. [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20368.895140] rsync D ffffffff8160f580 0 14911 1 0x00000000 [20368.895148] ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 ffff88022394f800 [20368.895158] ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 0000000000012c40 [20368.895163] ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 ffff8801cefce130 [20368.895169] Call Trace: [20368.895180] [<ffffffff8151a774>] schedule+0x24/0x70 [20368.895207] [<ffffffffa0158c75>] wait_current_trans.isra.32+0x95/0x100 [btrfs] [20368.895214] [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60 [20368.895236] [<ffffffffa015a45d>] start_transaction.part.33+0x13d/0x4d0 [btrfs] [20368.895252] [<ffffffff811420f3>] ? inode_permission+0x13/0x50 [20368.895271] [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs] [20368.895287] [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 [btrfs] [20368.895302] [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs] [20368.895307] [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122 [20368.895312] [<ffffffff81055ff0>] ? ns_capable+0x30/0x60 [20368.895317] [<ffffffff811413bd>] ? generic_permission+0xbd/0x110 [20368.895336] [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs] [20368.895341] [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0 [20368.895345] [<ffffffff81143ad7>] vfs_unlink+0x37/0x50 [20368.895349] [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240 [20368.895354] [<ffffffff81146171>] sys_unlink+0x11/0x20 [20368.895359] [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
Speed is just 100kb/s instead of 100MB/s.
Stefan
Am 26.03.2013 20:16, schrieb Josef Bacik:
Quoted 91 lines · Stefan Priebe
Quoted 75 lines
Hi Josef,
Am 26.03.2013 18:45, schrieb Josef Bacik:
Quoted 72 lines
Quoted 71 lines
Am 26.03.2013 16:25, schrieb Josef Bacik:
Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines
Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines
Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime:~# cat /proc/mounts | grep btrfs /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,
It still does not fix the problem.
The rsync output looks like this so it does not work for file a but then continues on c d e, ...
sync -av --progress /backup/
/mnt/
sending incremental file list.etc_openvpn/ipp.txt229 100% 3.99kB/s 0:00:00 (xfer#2, to-check=1009/1196) .etc_openvpn/openvpn-status.log 360 100% 6.28kB/s 0:00:00 (xfer#3, to-check=1007/1196)rsync: rename "
/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log104188 100% 147.67kB/s 0:00:00 (xfer#4, to-check=1131/2700) .log/auth.log 15211522 100% 2.97MB/s 0:00:04 (xfer#5, to-check=1105/2700) .log/auth.log.1 19431424 61% 7.35MB/s 0:00:01Quoted 7 lines
the dmesg output looks like this:
[ 551.321576] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 551.323694] space_info 4 has 6439526400 free, is full [ 551.323696] space_info total=25748307968, used=19308666880, pinned=0,reserved=49152, may_use=6438453248, readonly=65536
Ok so then this is probably it, let me know if it helps. Thanks,
Quoted 7 lines
Quoted 3 lines
OK it now has copied a lot of files (170) without an error all were very small.
Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,
Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.527392] dumping block rsv 2, size 0 reserved 0 [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608 [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.646958] space_info 4 has 6439428096 free, is full [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536 [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7 dumping space info [ 9587.727000] dumping block rsv 2, size 0 reserved 0 [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304 [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640 [ 9587.839935] space_info 4 has 6439428096 free, is full [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,reserved=45056, may_use=6438354944, readonly=65536
Quoted 8 lines
Quoted 2 lines
Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,
Yes - this works now. Which of the patches can i drop? Do i just need the last one?
Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5?So I'll put together all of the patches that actually need to go up for this and post them, but basically its the mutex patch, the last patch I sent you and the one that adjusts the reservations for rename and delete. Thanks,
Josef
Quoted 36 lines · Stefan Priebe
Hi,
but when i transfer big files i see now this one:
[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds. [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20368.895140] rsync D ffffffff8160f580 0 14911 1 0x00000000 [20368.895148] ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 ffff88022394f800 [20368.895158] ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 0000000000012c40 [20368.895163] ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 ffff8801cefce130 [20368.895169] Call Trace: [20368.895180] [<ffffffff8151a774>] schedule+0x24/0x70 [20368.895207] [<ffffffffa0158c75>] wait_current_trans.isra.32+0x95/0x100 [btrfs] [20368.895214] [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60 [20368.895236] [<ffffffffa015a45d>] start_transaction.part.33+0x13d/0x4d0 [btrfs] [20368.895252] [<ffffffff811420f3>] ? inode_permission+0x13/0x50 [20368.895271] [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs] [20368.895287] [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 [btrfs] [20368.895302] [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs] [20368.895307] [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122 [20368.895312] [<ffffffff81055ff0>] ? ns_capable+0x30/0x60 [20368.895317] [<ffffffff811413bd>] ? generic_permission+0xbd/0x110 [20368.895336] [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs] [20368.895341] [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0 [20368.895345] [<ffffffff81143ad7>] vfs_unlink+0x37/0x50 [20368.895349] [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240 [20368.895354] [<ffffffff81146171>] sys_unlink+0x11/0x20 [20368.895359] [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
Quoted 2 lines
Speed is just 100kb/s instead of 100MB/s.
Hrm I wonder if 512 is too small for your case, can you add this patch to the pile and see what dmesg says when you are having these problems? Thanks,
Josef
fs/btrfs/transaction.c +13 -3
@@ -31,6 +31,7 @@#include "inode-map.h" #include "volumes.h" #include "dev-replace.h" +#include "math.h" #define BTRFS_ROOT_TRANS_TAG 0
@@ -576,10 +577,19 @@ void btrfs_throttle(struct btrfs_root *root)static int should_end_transaction (struct btrfs_trans_handle *trans,struct btrfs_root *root) { -int ret; +struct btrfs_block_rsv *block_rsv = &root->fs_info->global_block_rsv; +u64 num_bytes =0 ; +int ret =1 ; - ret =btrfs_block_rsv_check (root, &root->fs_info->global_block_rsv,5 ); -return ret ?1 :0 ; +spin_lock (&block_rsv->lock); + num_bytes =div_factor (block_rsv->size,5 ); +if (block_rsv->reserved >= num_bytes) + ret =0 ; +else +printk (KERN_ERR "we're pretty low, setting blocked, reserved %Lu, size %Lu, num %Lu\n" , + block_rsv->reserved, block_rsv->size, num_bytes); +spin_unlock (&block_rsv->lock); +return ret; } int btrfs_should_end_transaction(struct btrfs_trans_handle *trans,
Signature · 1 line
1.7.7.6
HI,
Am 26.03.2013 20:38, schrieb Josef Bacik:
Quoted 37 lines · Stefan Priebe
Quoted 36 lines
Hi,
but when i transfer big files i see now this one:
[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds. [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20368.895140] rsync D ffffffff8160f580 0 14911 1 0x00000000 [20368.895148] ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 ffff88022394f800 [20368.895158] ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 0000000000012c40 [20368.895163] ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 ffff8801cefce130 [20368.895169] Call Trace: [20368.895180] [<ffffffff8151a774>] schedule+0x24/0x70 [20368.895207] [<ffffffffa0158c75>] wait_current_trans.isra.32+0x95/0x100 [btrfs] [20368.895214] [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60 [20368.895236] [<ffffffffa015a45d>] start_transaction.part.33+0x13d/0x4d0 [btrfs] [20368.895252] [<ffffffff811420f3>] ? inode_permission+0x13/0x50 [20368.895271] [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs] [20368.895287] [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 [btrfs] [20368.895302] [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs] [20368.895307] [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122 [20368.895312] [<ffffffff81055ff0>] ? ns_capable+0x30/0x60 [20368.895317] [<ffffffff811413bd>] ? generic_permission+0xbd/0x110 [20368.895336] [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs] [20368.895341] [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0 [20368.895345] [<ffffffff81143ad7>] vfs_unlink+0x37/0x50 [20368.895349] [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240 [20368.895354] [<ffffffff81146171>] sys_unlink+0x11/0x20 [20368.895359] [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
Quoted 5 lines
Quoted 2 lines
Speed is just 100kb/s instead of 100MB/s.
Hrm I wonder if 512 is too small for your case, can you add this patch to the pile and see what dmesg says when you are having these problems? Thanks,
No idea why but i can't reproduce... ;-(
Stefan