Skip to contentSkip to conversation

No space left on device (28)

DISCUSSION Stefan Priebe - Profihost AG 6 participants 35 messages Mar 21, 2013 – Mar 26, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
        Total devices 1 FS bytes used 15.07TB
        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Greets.
Stefan

#1 lore raw

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 23 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
        Total devices 1 FS bytes used 15.07TB
        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Which kernel are you running?

-chris

#2 lore raw

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:

Quoted 24 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 23 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
       Total devices 1 FS bytes used 15.07TB
       devid    1 size 18.19TB used 18.19TB path /dev/dm-3
Quoted 8 lines
Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Which kernel are you running?

-chris

vanilla 3.8.3.

Stefan

#3 lore raw

Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)

Quoted 31 lines

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:

Quoted 27 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 26 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
       Total devices 1 FS bytes used 15.07TB
       devid    1 size 18.19TB used 18.19TB path /dev/dm-3
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

What wrong here? What is about the last 3TB?

Which kernel are you running?

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

-chris

#4 lore raw

Yes, i've now upgraded to 3.9-rc3 same result.

rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": No space left on device (28)

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.98GB
Metadata: total=8.00MB, used=0.00

Stefan

Am 21.03.2013 20:02, schrieb Chris Mason:

Quoted 45 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)

Quoted 31 lines

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:

Quoted 27 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 26 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
        Total devices 1 FS bytes used 15.07TB
        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

What wrong here? What is about the last 3TB?

Which kernel are you running?

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

-chris

#5 lore raw

Should i try to downgrade?

Greets,
Stefan

Am 21.03.2013 um 20:42 schrieb Stefan Priebe <[email protected]>:

Quoted 59 lines

Yes, i've now upgraded to 3.9-rc3 same result.

rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": No space left on device (28)

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.98GB
Metadata: total=8.00MB, used=0.00

Stefan

Am 21.03.2013 20:02, schrieb Chris Mason:

Quoted 45 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)

Quoted 31 lines

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:

Quoted 27 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 26 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
       Total devices 1 FS bytes used 15.07TB
       devid    1 size 18.19TB used 18.19TB path /dev/dm-3
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

What wrong here? What is about the last 3TB?

Which kernel are you running?

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

-chris

#6 lore raw

Hi Chris,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Which kernel are you running?

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Greets,
Stefan

#7 lore raw
Quoted 4 lines · Stefan Priebe - Profihost AG

Hi Chris,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Which kernel are you running?

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,

Josef

#8 lore raw

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:

Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 8 lines

Hi Chris,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Which kernel are you running?

-chris

vanilla 3.8.3.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,

Great!

Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.

Stefan

#9 lore raw
Quoted 17 lines · Stefan Priebe - Profihost AG

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:

Quoted 15 lines · Stefan Priebe - Profihost AG
Quoted 13 lines

Hi Chris,

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Which kernel are you running?

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Quoted 8 lines
Quoted 2 lines

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,

Great!

Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.

Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,

Josef

fs/btrfs/extent-tree.c +5 -0
index 9ac2eca..aabaea6 100644
@@ -4161,6 +4161,11 @@ out:
 			ret = 0;
 	}
 	if (flushing) {
+		if (ret == -ENOSPC) {
+			printk(KERN_ERR "returning enospc, dumping space info\n");
+			dump_space_info(space_info, 0, 0);
+		}
+
 		spin_lock(&space_info->lock);
 		space_info->flush = 0;
 		wake_up_all(&space_info->wait);
fs/btrfs/inode.c +2 -4
index ca1b767..3980ae7 100644
@@ -3679,11 +3679,9 @@ static struct btrfs_trans_handle *__unlink_start_trans(struct inode *dir,
 	 * 1 for the dir item
 	 * 1 for the dir index
 	 * 1 for the inode ref
-	 * 1 for the inode ref in the tree log
-	 * 2 for the dir entries in the log
 	 * 1 for the inode
 	 */
-	trans = btrfs_start_transaction(root, 8);
+	trans = btrfs_start_transaction(root, 5);
 	if (!IS_ERR(trans) || PTR_ERR(trans) != -ENOSPC)
 		return trans;
 
@@ -8127,7 +8125,7 @@ static int btrfs_rename(struct inode *old_dir, struct dentry *old_dentry,
 	 * inodes.  So 5 * 2 is 10, plus 1 for the new link, so 11 total items
 	 * should cover the worst case number of items we'll modify.
 	 */
-	trans = btrfs_start_transaction(root, 20);
+	trans = btrfs_start_transaction(root, 11);
 	if (IS_ERR(trans)) {
                 ret = PTR_ERR(trans);
                 goto out_notrans;
Signature · 1 line
1.7.7.6
#10 lore raw

Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:

Quoted 21 lines · Stefan Priebe - Profihost AG
Quoted 20 lines

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:

Quoted 17 lines · Stefan Priebe - Profihost AG
Quoted 4 lines

Hi Chris,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Which kernel are you running?

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,

Quoted 8 lines
Quoted 5 lines

Great!

Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.

Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,

Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some.

Rsync output:
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-wm8350.c

       12156 100%   11.17kB/s    0:00:01 (xfer#21395, to-check=1008/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-x1205.c
       16460 100%   15.05kB/s    0:00:01 (xfer#21396, to-check=1007/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/systohc.c
        1223 100%    1.12kB/s    0:00:01 (xfer#21397, to-check=1006/52625)
.software/kernel/linux-3.9-rc3/drivers/s390/
.software/kernel/linux-3.9-rc3/drivers/s390/Makefile
         144 100%    0.13kB/s    0:00:01 (xfer#21398, to-check=1052/52672)

.software/kernel/linux-3.9-rc3/drivers/s390/block/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/block" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/char/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/char" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/cio/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/cio" failed: No space left on device (28)
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.88pm8607.c.YBi0Rz" failed: No space left on device (28) *** Skipping any contents from this failed directory *** .software/kernel/linux-3.9-rc3/drivers/s390/crypto/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/crypto" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/kvm/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/kvm" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/net/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/net" failed: No space left on device (28)
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Kconfig.0lsfuE" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Makefile.C9wI7I" failed: No space left on device (28) .software/kernel/linux-3.9-rc3/drivers/s390/scsi/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/scsi" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/sbus/
.software/kernel/linux-3.9-rc3/drivers/sbus/Makefile

          70 100%    0.06kB/s    0:00:01 (xfer#21399, to-check=1003/52824)

.software/kernel/linux-3.9-rc3/drivers/sbus/char/
rsync: recv_generator: mkdir "/mnt/.software/kernel/linux-3.9-rc3/drivers/sbus/char" failed: No space left on device (28)
rsync: mkstemp "/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.aat2870-regulator.c.BGDwPN" failed: No space left on device (28) *** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/scsi/
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.ko.cmd

         208 100%    0.16kB/s    0:00:01 (xfer#21400, to-check=1407/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.mod.o.cmd
       27987 100%   21.54kB/s    0:00:01 (xfer#21401, to-check=1406/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.o.cmd
       43340 100%   32.63kB/s    0:00:01 (xfer#21402, to-check=1405/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.ko.cmd
         204 100%   15.32kB/s    0:00:00 (xfer#21403, to-check=1404/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.mod.o.cmd
       27975 100%    1.91MB/s    0:00:00 (xfer#21404, to-check=1403/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.o.cmd
       43327 100%  983.99kB/s    0:00:00 (xfer#21405, to-check=1402/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.ko.cmd
         208 100%    4.72kB/s    0:00:00 (xfer#21406, to-check=1401/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.mod.o.cmd
       27987 100%  621.16kB/s    0:00:00 (xfer#21407, to-check=1400/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.o.cmd
       43340 100%  742.53kB/s    0:00:00 (xfer#21408, to-check=1399/53242)

dmesg Output was pretty long so i posted to pastebin:
http://pastebin.com/raw.php?i=5kUS5MGA

Greets,
Stefan

#11 lore raw
Quoted 28 lines · Stefan Priebe

Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:

Quoted 26 lines · Stefan Priebe - Profihost AG
Quoted 25 lines

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:

Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 8 lines

Hi Chris,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Which kernel are you running?

-chris

vanilla 3.8.3.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,

Great!

Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.

Quoted 7 lines
Quoted 3 lines

Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,

Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some.

Ok so I've been working on another ENOSPC related problem that I think is affecting you as well. So I'm going to nail that down and see if that helps you too, if not we'll poke around your problem some more and try and work out what's happening. I'll get back to this fresh Monday morning. Thanks,

Josef

#12 lore raw

Hi Jsoef,

thanks!

Am 22.03.2013 21:49, schrieb Josef Bacik:

Quoted 44 lines · Stefan Priebe
Quoted 28 lines

Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:

Quoted 26 lines · Stefan Priebe - Profihost AG
Quoted 25 lines

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:

Quoted 9 lines · Stefan Priebe - Profihost AG
Quoted 8 lines

Hi Chris,

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Which kernel are you running?

-chris

vanilla 3.8.3.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a patch for you to run so I can narrow down what's going on. Thanks,

Great!

Thanks - just wanted to know that it's not my fault. I'm happy to test the patch and provide feedback.

Quoted 7 lines
Quoted 3 lines

Ok I think we are way over-reserving for rename, can you give this patch a whirl and see what happens? If it still fails can you capture dmesg and reply with that so I can see what's going on. Thanks,

Thanks for the patch. I was able to copy some files and i'm still able put it copy some then fails on some then copies some then fails on some.

Ok so I've been working on another ENOSPC related problem that I think is affecting you as well. So I'm going to nail that down and see if that helps you too, if not we'll poke around your problem some more and try and work out what's happening. I'll get back to this fresh Monday morning. Thanks,

Josef

#13 lore raw
Quoted 3 lines · Stefan Priebe

Hi Jsoef,

thanks!

Ok I don't think the thing I just fixed will make any difference for you so here's another debug patch, just apply it on top of what I've already sent you and re-run and give me the dmesg again. Thanks,

Josef

fs/btrfs/extent-tree.c +5 -1
index aabaea6..bf6433f 100644
@@ -4162,7 +4162,11 @@ out:
 	}
 	if (flushing) {
 		if (ret == -ENOSPC) {
-			printk(KERN_ERR "returning enospc, dumping space info\n");
+			spin_lock(&block_rsv->lock);
+			printk(KERN_ERR "returning enospc, space_info %u, size %Lu reserved %Lu, "
+			       "flush %d, flush_state %d  dumping space info\n", block_rsv->type,
+			       block_rsv->size, block_rsv->reserved, flush, flush_state);
+			spin_unlock(&block_rsv->lock);
 			dump_space_info(space_info, 0, 0);
 		}
 
Signature · 1 line
1.7.7.6
#14 lore raw

Hi,

output here:

[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, reserved=73728, may_use=6438297600, readonly=65536

Am 25.03.2013 21:14, schrieb Josef Bacik:

Quoted 30 lines · Stefan Priebe
Quoted 3 lines

Hi Jsoef,

thanks!

Ok I don't think the thing I just fixed will make any difference for you so here's another debug patch, just apply it on top of what I've already sent you and re-run and give me the dmesg again. Thanks,

Josef

fs/btrfs/extent-tree.c +5 -1
index aabaea6..bf6433f 100644
@@ -4162,7 +4162,11 @@ out:
  	}
  	if (flushing) {
  		if (ret == -ENOSPC) {
-			printk(KERN_ERR "returning enospc, dumping space info\n");
+			spin_lock(&block_rsv->lock);
+			printk(KERN_ERR "returning enospc, space_info %u, size %Lu reserved %Lu, "
+			       "flush %d, flush_state %d  dumping space info\n", block_rsv->type,
+			       block_rsv->size, block_rsv->reserved, flush, flush_state);
+			spin_unlock(&block_rsv->lock);
  			dump_space_info(space_info, 0, 0);
  		}
 
#15 lore raw
Quoted 27 lines · Stefan Priebe

Hi,

output here:

[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
Quoted 2 lines

reserved=73728, may_use=6438297600, readonly=65536

Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,

Josef

#16 lore raw

Hi Josef,

Am 26.03.2013 13:53, schrieb Josef Bacik:

Quoted 31 lines · Stefan Priebe
Quoted 27 lines

Hi,

output here:

[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
Quoted 2 lines

reserved=73728, may_use=6438297600, readonly=65536

Quoted 2 lines

Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,

no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Greets,
Stefan

#17 lore raw
Quoted 34 lines · Stefan Priebe - Profihost AG

Hi Josef,

Am 26.03.2013 13:53, schrieb Josef Bacik:

Quoted 31 lines · Stefan Priebe
Quoted 29 lines

Hi,

output here:

[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 

reserved=73728, may_use=6438297600, readonly=65536

Quoted 8 lines
Quoted 2 lines

Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,

no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

Josef

fs/btrfs/extent-tree.c +14 -2
index bf6433f..84e8909 100644
@@ -3803,6 +3803,19 @@ static int can_overcommit(struct btrfs_root *root,
 	return 0;
 }
 
+static int btrfs_try_writeback(struct super_block *sb, unsigned long nr_pages,
+			       enum wb_reason reason)
+{
+	if (!writeback_in_progress(sb->s_bdi) &&
+	    down_read_trylock(&sb->s_umount)) {
+		writeback_inodes_sb_nr(sb, nr_pages, reason);
+		up_read(&sb->s_umount);
+		return 1;
+	}
+
+	return 0;
+}
+
 void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
 				  unsigned long nr_pages)
 {
@@ -3810,8 +3823,7 @@ void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
 	int started;
 
 	/* If we can not start writeback, just sync all the delalloc file. */
-	started = try_to_writeback_inodes_sb_nr(sb, nr_pages,
-						      WB_REASON_FS_FREE_SPACE);
+	started = btrfs_try_writeback(sb, nr_pages, WB_REASON_FS_FREE_SPACE);
 	if (!started) {
 		/*
 		 * We needn't worry the filesystem going from r/w to r/o though
Signature · 1 line
1.7.7.6
#18 lore raw

Hi Josef,

Am 26.03.2013 14:30, schrieb Josef Bacik:

Quoted 38 lines · Stefan Priebe - Profihost AG
Quoted 37 lines

Hi Josef,

Am 26.03.2013 13:53, schrieb Josef Bacik:

Quoted 33 lines · Stefan Priebe
Quoted 27 lines

Hi,

output here:

[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
Quoted 2 lines

reserved=73728, may_use=6438297600, readonly=65536

Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,

Quoted 8 lines
Quoted 5 lines

no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

         229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
         360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

      104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
    15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
    19431424  61%    7.35MB/s    0:00:01

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Stefan

#19 lore raw
Quoted 72 lines · Stefan Priebe - Profihost AG

Hi Josef,

Am 26.03.2013 14:30, schrieb Josef Bacik:

Quoted 38 lines · Stefan Priebe - Profihost AG
Quoted 37 lines

Hi Josef,

Am 26.03.2013 13:53, schrieb Josef Bacik:

Quoted 33 lines · Stefan Priebe
Quoted 27 lines

Hi,

output here:

[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
Quoted 2 lines

reserved=73728, may_use=6438297600, readonly=65536

Weird, we have all the flushing stuff set and yet there is still a whole lot of outstanding reservations. Do you have compression enabled? Thanks,

Quoted 8 lines
Quoted 5 lines

no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

         229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
         360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

      104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
    15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
    19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Ok so then this is probably it, let me know if it helps. Thanks,

Josef

fs/btrfs/ordered-data.c +2 -0
index dc08d77..005c45d 100644
@@ -557,6 +557,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)
 	INIT_LIST_HEAD(&splice);
 	INIT_LIST_HEAD(&works);
 
+	mutex_lock(&root->fs_info->ordered_operations_mutex);
 	spin_lock(&root->fs_info->ordered_extent_lock);
 	list_splice_init(&root->fs_info->ordered_extents, &splice);
 	while (!list_empty(&splice)) {
@@ -600,6 +601,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)
 
 		cond_resched();
 	}
+	mutex_unlock(&root->fs_info->ordered_operations_mutex);
 }
 
 /*
Signature · 1 line
1.7.7.6
#20 lore raw

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 39 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

         229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
         360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

      104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
    15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
    19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Quoted 1 line

Ok so then this is probably it, let me know if it helps. Thanks,

OK it now has copied a lot of files (170) without an error all were very small.

Then this again:
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.vhost_net.mod.GrhWet" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/vhost_net.mod": No space left on device (28)
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rhine.mod.X2Ofhz" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rhine.mod": No space left on device (28)
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rng.mod.RAyxkF" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rng.mod": No space left on device (28)
rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.virtio.mod.w7INoL" -> ".software/kernel/linux-3.9-rc3/.tmp_versions/virtio.mod": No space left on device (28)^C

[ 5012.468988] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.472981] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.472982] space_info 4 has 6439399424 free, is full
[ 5012.472982] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.476974] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.476975] space_info 4 has 6439399424 free, is full
[ 5012.476976] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.480968] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.480969] space_info 4 has 6439399424 free, is full

Stefan

#21 lore raw
Quoted 41 lines · Stefan Priebe - Profihost AG

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 39 lines
Quoted 38 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

         229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
         360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

      104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
    15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
    19431424  61%    7.35MB/s    0:00:01

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Quoted 5 lines
Quoted 1 line

Ok so then this is probably it, let me know if it helps. Thanks,

OK it now has copied a lot of files (170) without an error all were very small.

Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,

Josef

fs/btrfs/extent-tree.c +12 -0
index 84e8909..1cf810a 100644
@@ -4026,6 +4026,15 @@ static int flush_space(struct btrfs_root *root,
 
 	return ret;
 }
+
+static void dump_block_rsv(struct btrfs_block_rsv *block_rsv)
+{
+	spin_lock(&block_rsv->lock);
+	printk(KERN_ERR "dumping block rsv %u, size %Lu reserved %Lu\n",
+	       block_rsv->type, block_rsv->size, block_rsv->reserved);
+	spin_unlock(&block_rsv->lock);
+}
+
 /**
  * reserve_metadata_bytes - try to reserve bytes from the block_rsv's space
  * @root - the root we're allocating for
@@ -4179,6 +4188,9 @@ out:
 			       "flush %d, flush_state %d  dumping space info\n", block_rsv->type,
 			       block_rsv->size, block_rsv->reserved, flush, flush_state);
 			spin_unlock(&block_rsv->lock);
+			dump_block_rsv(&root->fs_info->delalloc_block_rsv);
+			dump_block_rsv(&root->fs_info->delayed_block_rsv);
+			dump_block_rsv(&root->fs_info->global_block_rsv);
 			dump_space_info(space_info, 0, 0);
 		}
 
Signature · 1 line
1.7.7.6
#22 lore raw

HI,

Am 26.03.2013 16:25, schrieb Josef Bacik:

Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
     19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Ok so then this is probably it, let me know if it helps. Thanks,

Quoted 7 lines
Quoted 3 lines

OK it now has copied a lot of files (170) without an error all were very small.

Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,

Output is now:

[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, reserved=45056, may_use=6438354944, readonly=65536

Stefan

#23 lore raw
Quoted 72 lines · Stefan Priebe

HI,

Am 26.03.2013 16:25, schrieb Josef Bacik:

Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
     19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Ok so then this is probably it, let me know if it helps. Thanks,

Quoted 7 lines
Quoted 3 lines

OK it now has copied a lot of files (170) without an error all were very small.

Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,

Output is now:

[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, 
reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, 
Quoted 2 lines

reserved=45056, may_use=6438354944, readonly=65536

Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,

Josef

fs/btrfs/extent-tree.c +6 -1
index 1cf810a..ac415cf7 100644
@@ -4471,7 +4471,12 @@ static void update_global_block_rsv(struct btrfs_fs_info *fs_info)
 	spin_lock(&sinfo->lock);
 	spin_lock(&block_rsv->lock);
 
-	block_rsv->size = num_bytes;
+	/*
+	 * Limit the global block rsv to 512mb, we have infrastructure in place
+	 * to throttle reservations if we start getting low on global block rsv
+	 * space.
+	 */
+	block_rsv->size = min_t(u64, num_bytes, 512 * 1024 * 1024);
 
 	num_bytes = sinfo->bytes_used + sinfo->bytes_pinned +
 		    sinfo->bytes_reserved + sinfo->bytes_readonly +
Signature · 1 line
1.7.7.6
#24 lore raw

Hi Josef,

Am 26.03.2013 18:45, schrieb Josef Bacik:

Quoted 72 lines
Quoted 69 lines

Am 26.03.2013 16:25, schrieb Josef Bacik:

Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

           229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
           360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

        104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
      15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
      19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Ok so then this is probably it, let me know if it helps. Thanks,

Quoted 7 lines
Quoted 3 lines

OK it now has copied a lot of files (170) without an error all were very small.

Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,

Output is now:

[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,
Quoted 2 lines

reserved=45056, may_use=6438354944, readonly=65536

Quoted 2 lines

Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,

Yes - this works now. Which of the patches can i drop? Do i just need the last one? Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5?

Thanks and greets,
Stefan

#25 lore raw
Quoted 75 lines · Stefan Priebe

Hi Josef,

Am 26.03.2013 18:45, schrieb Josef Bacik:

Quoted 72 lines
Quoted 71 lines

Am 26.03.2013 16:25, schrieb Josef Bacik:

Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

           229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
           360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

        104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
      15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
      19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Ok so then this is probably it, let me know if it helps. Thanks,

Quoted 7 lines
Quoted 3 lines

OK it now has copied a lot of files (170) without an error all were very small.

Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,

Output is now:

[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,

reserved=45056, may_use=6438354944, readonly=65536

Quoted 8 lines
Quoted 2 lines

Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,

Yes - this works now. Which of the patches can i drop? Do i just need the last one?
Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5?

So I'll put together all of the patches that actually need to go up for this and post them, but basically its the mutex patch, the last patch I sent you and the one that adjusts the reservations for rename and delete. Thanks,

Josef

#26 lore raw

Hi,

but when i transfer big files i see now this one:

[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
[20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[20368.895140] rsync           D ffffffff8160f580     0 14911      1 0x00000000
[20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 ffff88022394f800
[20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 0000000000012c40
[20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 ffff8801cefce130
[20368.895169] Call Trace:
[20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
[20368.895207]  [<ffffffffa0158c75>] wait_current_trans.isra.32+0x95/0x100 [btrfs]
[20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
[20368.895236]  [<ffffffffa015a45d>] start_transaction.part.33+0x13d/0x4d0 [btrfs]
[20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
[20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
[20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 [btrfs]
[20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs]
[20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
[20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
[20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
[20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
[20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
[20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
[20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
[20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
[20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b

Speed is just 100kb/s instead of 100MB/s.

Stefan

Am 26.03.2013 20:16, schrieb Josef Bacik:

Quoted 91 lines · Stefan Priebe
Quoted 75 lines

Hi Josef,

Am 26.03.2013 18:45, schrieb Josef Bacik:

Quoted 72 lines
Quoted 71 lines

Am 26.03.2013 16:25, schrieb Josef Bacik:

Quoted 44 lines · Stefan Priebe - Profihost AG
Quoted 43 lines

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:

Quoted 40 lines
Quoted 31 lines
Quoted 9 lines
Quoted 6 lines

Am 26.03.2013 13:53, schrieb Josef Bacik:
no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Ok I think I see what's going on. Can you try this patch and see if it fixes it? Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt

            229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
            360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)

rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" -> ".etc_openvpn/ipp.txt": No space left on device (28) .log/
.log/UcliEvt.log

         104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
       15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
       19431424  61%    7.35MB/s    0:00:01
Quoted 7 lines

the dmesg output looks like this:

[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,

reserved=49152, may_use=6438453248, readonly=65536

Ok so then this is probably it, let me know if it helps. Thanks,

Quoted 7 lines
Quoted 3 lines

OK it now has copied a lot of files (170) without an error all were very small.

Welp progress is good. Throw this into the mix and go again, it's just adding some more debugging so I can make sure I'm going down the right rabbit hole. Thanks,

Output is now:

[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,

reserved=45056, may_use=6438354944, readonly=65536

Quoted 8 lines
Quoted 2 lines

Well then that looks like I was going down the wrong rabbit hole. This should fix you up, for real this time ;). Thanks,

Yes - this works now. Which of the patches can i drop? Do i just need the last one?
Is it safe to add another 18TB raid via converting it to btrfs raid0? Will the fix be part of 3.9-rc5?

So I'll put together all of the patches that actually need to go up for this and post them, but basically its the mutex patch, the last patch I sent you and the one that adjusts the reservations for rename and delete. Thanks,

Josef

#27 lore raw
Quoted 36 lines · Stefan Priebe

Hi,

but when i transfer big files i see now this one:

[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
[20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[20368.895140] rsync           D ffffffff8160f580     0 14911      1 
0x00000000
[20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 
ffff88022394f800
[20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 
0000000000012c40
[20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 
ffff8801cefce130
[20368.895169] Call Trace:
[20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
[20368.895207]  [<ffffffffa0158c75>] 
wait_current_trans.isra.32+0x95/0x100 [btrfs]
[20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
[20368.895236]  [<ffffffffa015a45d>] 
start_transaction.part.33+0x13d/0x4d0 [btrfs]
[20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
[20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
[20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 
[btrfs]
[20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs]
[20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
[20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
[20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
[20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
[20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
[20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
[20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
[20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
[20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
Quoted 2 lines

Speed is just 100kb/s instead of 100MB/s.

Hrm I wonder if 512 is too small for your case, can you add this patch to the pile and see what dmesg says when you are having these problems? Thanks,

Josef

fs/btrfs/transaction.c +13 -3
index 50767bb..d19c9f6 100644
@@ -31,6 +31,7 @@
 #include "inode-map.h"
 #include "volumes.h"
 #include "dev-replace.h"
+#include "math.h"
 
 #define BTRFS_ROOT_TRANS_TAG 0
 
@@ -576,10 +577,19 @@ void btrfs_throttle(struct btrfs_root *root)
 static int should_end_transaction(struct btrfs_trans_handle *trans,
 				  struct btrfs_root *root)
 {
-	int ret;
+	struct btrfs_block_rsv *block_rsv = &root->fs_info->global_block_rsv;
+	u64 num_bytes = 0;
+	int ret = 1;
 
-	ret = btrfs_block_rsv_check(root, &root->fs_info->global_block_rsv, 5);
-	return ret ? 1 : 0;
+	spin_lock(&block_rsv->lock);
+	num_bytes = div_factor(block_rsv->size, 5);
+	if (block_rsv->reserved >= num_bytes)
+		ret = 0;
+	else
+		printk(KERN_ERR "we're pretty low, setting blocked, reserved %Lu, size %Lu, num %Lu\n",
+		       block_rsv->reserved, block_rsv->size, num_bytes);
+	spin_unlock(&block_rsv->lock);
+	return ret;
 }
 
 int btrfs_should_end_transaction(struct btrfs_trans_handle *trans,
Signature · 1 line
1.7.7.6
#28 lore raw

HI,
Am 26.03.2013 20:38, schrieb Josef Bacik:

Quoted 37 lines · Stefan Priebe
Quoted 36 lines

Hi,

but when i transfer big files i see now this one:

[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
[20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[20368.895140] rsync           D ffffffff8160f580     0 14911      1
0x00000000
[20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198
ffff88022394f800
[20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8
0000000000012c40
[20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88
ffff8801cefce130
[20368.895169] Call Trace:
[20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
[20368.895207]  [<ffffffffa0158c75>]
wait_current_trans.isra.32+0x95/0x100 [btrfs]
[20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
[20368.895236]  [<ffffffffa015a45d>]
start_transaction.part.33+0x13d/0x4d0 [btrfs]
[20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
[20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
[20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20
[btrfs]
[20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs]
[20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
[20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
[20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
[20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
[20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
[20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
[20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
[20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
[20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
Quoted 5 lines
Quoted 2 lines

Speed is just 100kb/s instead of 100MB/s.

Hrm I wonder if 512 is too small for your case, can you add this patch to the pile and see what dmesg says when you are having these problems? Thanks,

No idea why but i can't reproduce... ;-(

Stefan

#29 lore raw
In reply to Stefan Priebe

On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:

I might be wrong here, but doesn't this

Quoted 4 lines

rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":

...try to move a file from

"/mnt/.software/"

to

".software/"

(relative to current dir)??

Maybe you end up trying to rsync your array to a small root partition or something?

Quoted 61 lines

No space left on device (28)

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.98GB
Metadata: total=8.00MB, used=0.00

Stefan

Am 21.03.2013 20:02, schrieb Chris Mason:

Quoted 36 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)

Quoted 35 lines

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:

Quoted 31 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 23 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
        Total devices 1 FS bytes used 15.07TB
        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Which kernel are you running?

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
With respect,
Roman
#30 lore raw
Quoted 12 lines · Roman Mamedov

On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:

I might be wrong here, but doesn't this

Quoted 4 lines

rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/"" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":

...try to move a file from

Quoted 7 lines

"/mnt/.software/"

to

".software/"

(relative to current dir)??

No; that's rsync giving the full path, and then the target path relative to the command it was given. The filename itself (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to temporarily, so it can mv it over the original in an atomic fashion...

Stefan: ...which means that the actual copy succeeded, which suggests that this is more of a metadata enospc thing.

You might try btrfs balance start -musage=5 (instead of -dusage), and if that doesn't report any chunks balanced, try a high number until it does.

#31 lore raw

Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%

Am 22.03.2013 um 07:24 schrieb cwillu <[email protected]>:

Quoted 32 lines · Roman Mamedov
Quoted 12 lines

On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:

I might be wrong here, but doesn't this

Quoted 4 lines

rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/"" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":

...try to move a file from

Quoted 7 lines

"/mnt/.software/"

to

".software/"

(relative to current dir)??

No; that's rsync giving the full path, and then the target path relative to the command it was given. The filename itself (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to temporarily, so it can mv it over the original in an atomic fashion...

Stefan: ...which means that the actual copy succeeded, which suggests that this is more of a metadata enospc thing.

You might try btrfs balance start -musage=5 (instead of -dusage), and if that doesn't report any chunks balanced, try a high number until it does.

#32 lore raw
Quoted 1 line · Stefan Priebe - Profihost AG

Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%

You tried -musage=5? Your original email said -dusage=5.

#33 lore raw

Hi,

Am 22.03.2013 07:41, schrieb cwillu:> On Fri, Mar 22, 2013 at 12:39 AM, Stefan Priebe - Profihost AG

Quoted 5 lines
Quoted 2 lines · <[email protected]> wrote:

Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%

You tried -musage=5? Your original email said -dusage=5.

sorry i missed the difference - but it does not work:

~# btrfs fi balance start -musage=5 /mnt/
ERROR: error during balancing '/mnt/' - No space left on device There may be more info in syslog - try dmesg | tail

~# dmesg|tail

[ 1183.019367] device fsid fc23c4a8-a351-4c91-96a2-ee6abeffe59a devid 1
transid 6224 /dev/mapper/raid54tb1
[ 1183.019860] btrfs: disk space caching is enabled
[42719.915781] btrfs: relocating block group 4194304 flags 4
[42727.916141] btrfs: 5 enospc errors during balance

Stefan

#34 lore raw
In reply to Roman Mamedov

No this is fine it's rsync style same happens with cp.

Am 22.03.2013 um 07:13 schrieb Roman Mamedov <[email protected]>:

Quoted 88 lines

On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <[email protected]> wrote:

I might be wrong here, but doesn't this

Quoted 4 lines

rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":

...try to move a file from

"/mnt/.software/"

to

".software/"

(relative to current dir)??

Maybe you end up trying to rsync your array to a small root partition or something?

Quoted 60 lines

No space left on device (28)

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.98GB
Metadata: total=8.00MB, used=0.00

Stefan

Am 21.03.2013 20:02, schrieb Chris Mason:

Quoted 36 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)

Quoted 35 lines

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <[email protected]>:

Quoted 31 lines

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)

Quoted 23 lines

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00
# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
       Total devices 1 FS bytes used 15.07TB
       devid    1 size 18.19TB used 18.19TB path /dev/dm-3
Quoted 4 lines
# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Which kernel are you running?

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

-chris

vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations. Are you able to try a slightly more experimental kernel?

-chris

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
With respect,
Roman