Crash in __btrfs_reserve_extent
Hi guys,
I just had my Fedora 11 workstation (kernel version 2.6.29.4-167.fc11.i686.PAE) crash hard. The following was printed to a logged-in SSH session:
------------[ cut here ]------------ invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.1/1-2.1.1/devnum Process pdflush (pid: 11675, ti=d4944000 task=ee516500 task.ti=d4944000) Stack: c2920960 00000024 00000000 00000000 00001000 00000000 00000000 f6d72660 f5c8a150 00001000 00000000 a55cf000 d4945ccb d4945c50 d4945c48 f8a0d70e Call Trace: [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs] [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs] [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs] [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs] [<c0565323>] ? __lookup_tag+0x89/0xe3 [<c05653ec>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs] [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs] [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs] [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs] [<c04852bc>] ? do_writepages+0x25/0x39 [<c04bea15>] ? __writeback_single_inode+0x15c/0x27a [<c0667d88>] ? dm_any_congested+0x32/0x3d [<f8a15c3b>] ? btrfs_congested_fn+0x38/0x66 [btrfs] [<c04bee90>] ? generic_sync_sb_inodes+0x1d9/0x2f6 [<c04bf156>] ? writeback_inodes+0x82/0xca [<c048598c>] ? background_writeout+0x7b/0xa7 [<c04860c2>] ? pdflush+0x130/0x1dc [<c0485911>] ? background_writeout+0x0/0xa7 [<c0485f92>] ? pdflush+0x0/0x1dc [<c0446fc8>] ? kthread+0x41/0x65 [<c0446f87>] ? kthread+0x0/0x65 [<c0409dbf>] ? kernel_thread_helper+0x7/0x10 Code: e8 17 46 a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 18 d0 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:d4945ba4 ------------[ cut here ]------------ invalid opcode: 0000 [#2] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.1/1-2.1.1/devnum Process qemu-kvm (pid: 11670, ti=f0c6a000 task=d9c89940 task.ti=f0c6a000) Stack: 00000000 00000024 00000000 00000000 00001000 00000000 00000000 f4d04630 f5c8a150 00001000 00000000 05f82000 f0c6bc67 f0c6bbec f0c6bbe4 f8a0d70e Call Trace: [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs] [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs] [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs] [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs] [<c0565323>] ? __lookup_tag+0x89/0xe3 [<c05653ec>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs] [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs] [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs] [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs] [<f8a2d0be>] ? btrfs_fdatawrite_range+0x67/0x6f [btrfs] [<f8a23245>] ? btrfs_file_write+0x440/0x627 [btrfs] [<c05365ef>] ? security_file_permission+0x14/0x16 [<c04a8b59>] ? rw_verify_area+0x9a/0xbb [<f8a22e05>] ? btrfs_file_write+0x0/0x627 [btrfs] [<c04a9229>] ? vfs_write+0x95/0xf4 [<c04a92e0>] ? sys_pwrite64+0x58/0x70 [<c040945f>] ? sysenter_do_call+0x12/0x34 Code: e8 17 46 a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 18 d0 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:f0c6bb40 ------------[ cut here ]------------ invalid opcode: 0000 [#3] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.1/1-2.1.1/devnum Process qemu-kvm (pid: 11674, ti=c583c000 task=d9c8f1a0 task.ti=c583c000) Stack: 00000000 00000024 00000000 00000000 00001000 00000000 00000000 f4d04ed0 f5c8a150 00001000 00000000 03fbd000 c583dc67 c583dbec c583dbe4 f8a0d70e Call Trace: [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs] [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs] [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs] [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs] [<c0565323>] ? __lookup_tag+0x89/0xe3 [<c05653ec>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs] [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs] [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs] [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs] [<f8a2d0be>] ? btrfs_fdatawrite_range+0x67/0x6f [btrfs] [<f8a23245>] ? btrfs_file_write+0x440/0x627 [btrfs] [<c05365ef>] ? security_file_permission+0x14/0x16 [<c04a8b59>] ? rw_verify_area+0x9a/0xbb [<f8a22e05>] ? btrfs_file_write+0x0/0x627 [btrfs] [<c04a9229>] ? vfs_write+0x95/0xf4 [<c04a92e0>] ? sys_pwrite64+0x58/0x70 [<c040945f>] ? sysenter_do_call+0x12/0x34 Code: e8 17 46 a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 18 d0 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:c583db40
I have btrfs on my root filesystem and at the time of the crash I taring together some files from a NFS filesystem onto it. There was plenty of free space on the btrfs filesystem.
I wasn't able to Google up another report with a similar-looking crash, so I thought you might be interested.
Best regards,
Signature · 3 lines
Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com/ Tel: +47 21 54 41 27
* Tore Anderson
Quoted 3 lines
I have btrfs on my root filesystem and at the time of the crash I taring together some files from a NFS filesystem onto it. There was plenty of free space on the btrfs filesystem.
Some more info - the file system is acting really strange after a reboot. A lot of messages like these are printed to /var/log/messages:
no space left, need 4096, 0 delalloc bytes, 16577376256 bytes_used, 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 69816320 may use16647192576 total
Many commands gives the error message "no space left on device". Curiously enough /var/log/messages is stored on the very same file system, and new lines appears in it, so it appears those writes are avoiding the ENOSPC somehow.
Anyway, there's plenty of free space on the file system:
Filesystem Size Used Avail Use% Mounted on/dev/mapper/vg_echo-lv_root 228G 16G 212G 7% /
It has never been resized, by the way. I found that this issue had been mentioned on the mailing list by Hugo Mills (Cc-ed) before after all - apologies for posting a dupe:
http://thread.gmane.org/gmane.comp.file-systems.btrfs/2694
Is this bug fixed in later versions of btrfs or is it still unresolved, I wonder? If the latter, I'll be glad to help out with any debugging info you might need before I try to fix it (or re-install).
Best regards,
Signature · 3 lines
Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com/ Tel: +47 21 54 41 27
Quoted 27 lines · Tore Anderson
* Tore Anderson
Quoted 3 lines
I have btrfs on my root filesystem and at the time of the crash I taring together some files from a NFS filesystem onto it. There was plenty of free space on the btrfs filesystem.
Some more info - the file system is acting really strange after a reboot. A lot of messages like these are printed to
/var/log/messages:no space left, need 4096, 0 delalloc bytes, 16577376256
bytes_used, 0bytes_reserved, 0bytes_pinned, 0bytes_readonly, 69816320 may use16647192576 totalMany commands gives the error message "no space left on device". Curiously enough
/var/log/messagesis stored on the very same file system, and new lines appears in it, so it appears those writes are avoiding the ENOSPC somehow.Anyway, there's plenty of free space on the file system:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_echo-lv_root228G 16G 212G 7% /It has never been resized, by the way. I found that this issue had been mentioned on the mailing list by Hugo Mills (Cc-ed) before after all - apologies for posting a dupe:
Quoted 6 lines
http://thread.gmane.org/gmane.comp.file-systems.btrfs/2694
Is this bug fixed in later versions of btrfs or is it still unresolved, I wonder? If the latter, I'll be glad to help out with any debugging info you might need before I try to fix it (or re-install).
This is one of the gotchas of btrfs, there is not proper ENOSPC handling, just a few things in place that are a bit conservative to make sure you don't panic the box. Btrfs has seperate zones used for data and metadata, and these chunks are allocated in 1gb chunks, so you have 212gb of space thats allowed for data use, and 16gb thats allowed for metadata use. By default every 12 (or it may be 8, i forget) chunks we allocate for data, we allocate 1 for metadata, which ends up with like 8% of the disk being used for metadata.
Now in your case you can run btrfsctl -b and it will re-balance the space on the drive, and it may give you more space back. It could also possible panic the box, so make sure you are all backed up :). Thanks,
Josef
Hello Josef,
* Josef Bacik
Quoted 8 lines
This is one of the gotchas of btrfs, there is not proper ENOSPC handling, just a few things in place that are a bit conservative to make sure you don't panic the box. Btrfs has seperate zones used for data and metadata, and these chunks are allocated in 1gb chunks, so you have 212gb of space thats allowed for data use, and 16gb thats allowed for metadata use. By default every 12 (or it may be 8, i forget) chunks we allocate for data, we allocate 1 for metadata, which ends up with like 8% of the disk being used for metadata.
Hmm, okay. I was aware of the fact that it didn't handle the disk filling up too well, but I didn't know that would cause an issue so long before the disk has gone full? I mean, according to df my there's 16 GB of files on my disk (something I verified with du), while according to you there should be 212 GB available for that in total.
Has metadata and data _both_ been stored in the 16 GB zone reserved for metadata, for some reason? It would make sense that I ran into ENOSPC in that case, since the metadata-reserved zone now is indeed completely full. If I understand you correctly, though, the data is stored in the 212 GB large zone, not the 16 GB large metadata zone - but if that's the case I don't understand how I could have hit ENOSPC?
Quoted 3 lines
Now in your case you can run btrfsctl -b and it will re-balance the space on the drive, and it may give you more space back. It could also possible panic the box, so make sure you are all backed up :).
Thanks for the tipe, but the "-b" option doesn't seem to be present in btrfsctl (built from a day-fresh git checkout)...?
Best regards,
Signature · 3 lines
Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com/ Tel: +47 21 54 41 27
Quoted 19 lines · Tore Anderson
Hello Josef,
* Josef Bacik
Quoted 8 lines
This is one of the gotchas of btrfs, there is not proper ENOSPC handling, just a few things in place that are a bit conservative to make sure you don't panic the box. Btrfs has seperate zones used for data and metadata, and these chunks are allocated in 1gb chunks, so you have 212gb of space thats allowed for data use, and 16gb thats allowed for metadata use. By default every 12 (or it may be 8, i forget) chunks we allocate for data, we allocate 1 for metadata, which ends up with like 8% of the disk being used for metadata.
Hmm, okay. I was aware of the fact that it didn't handle the disk filling up too well, but I didn't know that would cause an issue so long before the disk has gone full? I mean, according to df my there's 16 GB of files on my disk (something I verified with du), while according to you there should be 212 GB available for that in total.
Quoted 7 lines
Has metadata and data _both_ been stored in the 16 GB zone reserved for metadata, for some reason? It would make sense that I ran into ENOSPC in that case, since the metadata-reserved zone now is indeed completely full. If I understand you correctly, though, the data is stored in the 212 GB large zone, not the 16 GB large metadata zone - but if that's the case I don't understand how I could have hit ENOSPC?
Ooookay, now that I'm awake lets try this again :). You are right, I thought it was 212GB used, not 16GB used. Something has gone horribly wrong, btrfs seems to think thats all the space it can use. The command you need to use is btrfs-vol -b. Please try that and see if it behaves better. Before you run that please do a btrfs-show and post the output, I'd like to see how big the fs thinks its supposed to be. Thanks,
Josef
* Josef Bacik
Quoted 7 lines
Ooookay, now that I'm awake lets try this again :). You are right, I thought it was 212GB used, not 16GB used. Something has gone horribly wrong, btrfs seems to think thats all the space it can use. The command you need to use is btrfs-vol -b. Please try that and see if it behaves better. Before you run that please do a btrfs-show and post the output, I'd like to see how big the fs thinks its supposed to be.
It doesn't print much of anything, I'm afraid:
root@echo:~/misc/btrfs-progs-unstable# ./btrfs-show /dev/mapper/vg_echo-lv_root failed to read /dev/sr0 failed to read /dev/sdf failed to read /dev/sde failed to read /dev/sdd failed to read /dev/sdc failed to read /dev/sdb Btrfs Btrfs v0.19
Same output with btrfs-progs-0.19. The devices it's complaining about is a USB memory card reader and my optical drive, if I disconnect those from the SCSI layer the output is simply "Btrfs Btrfs v0.19".
I tried to run btrfsck earlier today, but it only gave me an error message. Perhaps that just made things worse?
I've got a btrfs-image dump that I took right after the problem showed for the first time (before the fsck), by the way. It's available at
<http://greed.fud.no/hosed_rootfs.btrfsimage.Z> in case you have any use for it.
Best regards,
Signature · 3 lines
Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com/ Tel: +47 21 54 41 27
Quoted 25 lines · Tore Anderson
* Josef Bacik
Quoted 7 lines
Ooookay, now that I'm awake lets try this again :). You are right, I thought it was 212GB used, not 16GB used. Something has gone horribly wrong, btrfs seems to think thats all the space it can use. The command you need to use is btrfs-vol -b. Please try that and see if it behaves better. Before you run that please do a btrfs-show and post the output, I'd like to see how big the fs thinks its supposed to be.
It doesn't print much of anything, I'm afraid:
root@echo:~/misc/btrfs-progs-unstable# ./btrfs-show /dev/mapper/vg_echo-lv_root failed to read /dev/sr0 failed to read /dev/sdf failed to read /dev/sde failed to read /dev/sdd failed to read /dev/sdc failed to read /dev/sdb Btrfs Btrfs v0.19Same output with btrfs-progs-0.19. The devices it's complaining about is a USB memory card reader and my optical drive, if I disconnect those from the SCSI layer the output is simply "Btrfs Btrfs v0.19".
Quoted 8 lines
I tried to run btrfsck earlier today, but it only gave me an error message. Perhaps that just made things worse?
I've got a btrfs-image dump that I took right after the problem showed for the first time (before the fsck), by the way. It's available at
<http://greed.fud.no/hosed_rootfs.btrfsimage.Z> in case you have any use for it.
What kernel are you running? I need to get a look at what code you are on. Thanks for the image, I'm having some trouble getting it to work, but I will let you know what I come up with.
Josef
* Josef Bacik
Quoted 3 lines
What kernel are you running? I need to get a look at what code you are on. Thanks for the image, I'm having some trouble getting it to work, but I will let you know what I come up with.
I was running 2.6.29.4-167.fc11.i686.PAE at the time of the crash, and when I've been having problems. I see that there's a new F11 kernel out - I'll try to upgrade to that one and will let you know if there's any change (I doubt it - there's no mention of btrfs-related changes).
Here's the output from btrfs-show, just figured out that I shouldn't have supplied any arguments (thanks to Robert F=F6rster):
Label: none uuid: 94762921-dbc0-4faf-a4e5-f5acdaf305a8
Total devices 1 FS bytes used 15.82GB devid 1 size 227.53GB used 227.53GB path /dev/root
I just tried to make a new image with btrfs-image from btrfs-progs-unstable (using "btrfs-image -c 9 /dev/vg_echo/lv_root image2.Z").
Unlike when making the first image, the file system was mounted. The image is at <http://greed.fud.no/image2.Z> - is that any better?
Best regards,
--=20
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Tore Anderson
* Josef Bacik
=20Quoted 3 lines
What kernel are you running? I need to get a look at what code you are on. Thanks for the image, I'm having some trouble getting it to work, but I will let you know what I come up with.
=20
I was running 2.6.29.4-167.fc11.i686.PAE at the time of the crash, an=
d
Quoted 2 lines
when I've been having problems. I see that there's a new F11 kernel out - I'll try to upgrade to that one and will let you know if there'=
s
Quoted 1 line
any change (I doubt it - there's no mention of btrfs-related changes)=
=2E
Quoted 7 lines
=20
Here's the output from btrfs-show, just figured out that I shouldn't have supplied any arguments (thanks to Robert F=F6rster):
=20
Label: none uuid: 94762921-dbc0-4faf-a4e5-f5acdaf305a8Total devices 1 FS bytes used 15.82GB devid 1 size 227.53GB used 227.53GB path /dev/root
Quoted 7 lines
=20
I just tried to make a new image with btrfs-image from btrfs-progs-unstable (using "btrfs-image -c 9/dev/vg_echo/lv_rootimage2.Z").
Unlike when making the first image, the file system was mounted. The image is at <http://greed.fud.no/image2.Z> - is that any better? =20
Ok good news is, btrfs-vol -b will fix your problem. Somehow most of y= our disk has
been allocated to use metadata. So did you have a whole bunch of stuff=
on this
disk and then delete it all? Because that would put you in that situat= ion. If
you have not then there is likely a bug in the metadata ratio stuff tha= t needs
to be fixed. Thankfully btrfs-vol -b will move everything around so yo= u are
good to go, so if it is a bug that will keep you going until I can find=
and fix
the problem. btrfs-vol -b will take a while to run, so don't worry if = it sits
there forever, you can run dmesg to watch its progress. Let me know if=
you had
a bunch of stuff on this disk that you deleted, cause if there was good=
reason
for it to allocate all this metadata space then thats fine, but if not = I need to
start looking at a root cause. Thanks,
Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
* Josef Bacik
Quoted 5 lines
Ok good news is, btrfs-vol -b will fix your problem. Somehow most of your disk has been allocated to use metadata. So did you have a whole bunch of stuff on this disk and then delete it all? Because that would put you in that situation. If you have not then there is likely a bug in the metadata ratio stuff that needs to be fixed.
As far as I know there hasn't been a lot of stuff on the file system, I'm afraid. The file system was created by the Fedora 11 installer, and it has just been used as the system drive (I've got /home on NFS).
btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn. I've was able to get output from dmesg before my SSH sessions started hanging - maybe you can make anything out of it? Anyway, right now I have no more remote access to the box so any further debugging will have to wait until tomorrow morning.
btrfs relocating chunk 129952120832
btrfs relocating block group 129952120832 flags 1
btrfs allocation failed flags 1, wanted 4096space_info has 4096 free, is fullspace_info total=16647192576, pinned=0, delalloc=57856000, may_use=1691648, used=16645496832 block group 12582912 has 8388608 bytes, 8384512 used 0 pinned 4096 reserved
0 blocks of free space at or bigger than bytes is
block group 1103101952 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 3250585600 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 4324327424 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 5398069248 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 6471811072 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 7545552896 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 30094131200 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 31167873024 has 1073741824 bytes, 1072078848 used 0 pinned 1662976 reserved 0 blocks of free space at or bigger than bytes is
block group 32241614848 has 1073741824 bytes, 1073721344 used 0 pinned 20480 reserved 0 blocks of free space at or bigger than bytes is
block group 40831549440 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 46200258560 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 47274000384 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserve 0 blocks of free space at or bigger than bytes is
block group 49421484032 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 58011418624 has 1073741824 bytes, 1073737728 used 0 pinned 4096 reserved 0 blocks of free space at or bigger than bytes is
block group 128878379008 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 129952120832 has 532676608 bytes, 532672512 used 0 pinned 0 reserved entry offset 130484793344, bytes 4096
1 blocks of free space at or bigger than bytes is
------------[ cut here ]------------ kernel BUG at fs/btrfs/extent-tree.c:2905! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings Modules linked in: nls_utf8 cifs nfs lockd nfs_acl auth_rpcgss ipt_MASQUERADE iptable_nat nf_nat rfcomm sco bridge stp llc bnep l2cap sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 cpufreq_ondemand acpi_cpufreq dm_multipath kvm_intel kvm uinput snd_hda_codec_realtek snd_hda_intel firewire_ohci ppdev snd_hda_codec firewire_core i2c_i801 snd_hwdep snd_pcm snd_timer crc_itu_t btusb snd soundcore snd_page_alloc usb_storage parport_pc floppy parport bluetooth sky2 asus_atk0110 hwmon iTCO_wdt iTCO_vendor_support serio_raw pata_jmicron ata_generic pata_acpi btrfs zlib_deflate libcrc32c nouveau drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
Pid: 30, comm: pdflush Not tainted (2.6.29.6-213.fc11.i686.PAE #1) P5K-VM
EIP: 0060:[<f8a0d543>] EFLAGS: 00010257 CPU: 1
EIP is at __btrfs_reserve_extent+0x339/0x347 [btrfs]
EAX: f5c8aadc EBX: f5c8aa50 ECX: 00000000 EDX: 00000001
ESI: f5c8aadc EDI: f58c786c EBP: f5c03bfc ESP: f5c03ba4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process pdflush (pid: 30, ti=f5c02000 task=f5c08000 task.ti=f5c02000)
Stack: fffff000 00000000 00000000 f5c03ccb ffffffff ffffffff c1c00000 0000000d c3296380 00000024 00000000 00000000 00001000 00000000 00000000 f4d09690 f5c8aad0 00001000 00000000 0019b000 f5c03ccb f5c03c50 f5c03c48 f8a0d70e Call Trace: [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs] [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs] [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs] [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs] [<c0564dfb>] ? __lookup_tag+0x89/0xe3 [<c0564ec4>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs] [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs] [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs] [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs] [<c0484bb8>] ? do_writepages+0x25/0x39 [<c04be369>] ? __writeback_single_inode+0x140/0x241 [<c0667944>] ? dm_any_congested+0x32/0x3d [<f8a15c3b>] ? btrfs_congested_fn+0x38/0x66 [btrfs] [<c04be7c7>] ? generic_sync_sb_inodes+0x1d9/0x2f6 [<c04bea8d>] ? writeback_inodes+0x82/0xca [<c0485288>] ? background_writeout+0x7b/0xa7 [<c04859be>] ? pdflush+0x130/0x1dc [<c048520d>] ? background_writeout+0x0/0xa7 [<c048588e>] ? pdflush+0x0/0x1dc [<c04468bc>] ? kthread+0x41/0x65 [<c044687b>] ? kthread+0x0/0x65 [<c0409dbf>] ? kernel_thread_helper+0x7/0x10 Code: e8 ef 3e a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 0c c9 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:f5c03ba4 ---[ end trace 209348013c0e69ac ]---
btrfs allocation failed flags 1, wanted 4096space_info has 4096 free, is fullspace_info total=16647192576, pinned=0, delalloc=132501504, may_use=1691648, used=16645500928 block group 12582912 has 8388608 bytes, 8388608 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 1103101952 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 3250585600 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 4324327424 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 5398069248 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 6471811072 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 7545552896 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 30094131200 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 31167873024 has 1073741824 bytes, 1072078848 used 0 pinned 1662976 reserved 0 blocks of free space at or bigger than bytes is
block group 32241614848 has 1073741824 bytes, 1073721344 used 0 pinned 20480 reserved 0 blocks of free space at or bigger than bytes is
block group 40831549440 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 46200258560 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 47274000384 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 49421484032 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 58011418624 has 1073741824 bytes, 1073737728 used 0 pinned 4096 reserved 0 blocks of free space at or bigger than bytes is
block group 128878379008 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved 0 blocks of free space at or bigger than bytes is
block group 129952120832 has 532676608 bytes, 532672512 used 0 pinned 0 reserved entry offset 130484793344, bytes 4096
1 blocks of free space at or bigger than bytes is
------------[ cut here ]------------ kernel BUG at fs/btrfs/extent-tree.c:2905! invalid opcode: 0000 [#2] SMP last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings Modules linked in: nls_utf8 cifs nfs lockd nfs_acl auth_rpcgss ipt_MASQUERADE iptable_nat nf_nat rfcomm sco bridge stp llc bnep l2cap sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 cpufreq_ondemand acpi_cpufreq dm_multipath kvm_intel kvm uinput snd_hda_codec_realtek snd_hda_intel firewire_ohci ppdev snd_hda_codec firewire_core i2c_i801 snd_hwdep snd_pcm snd_timer crc_itu_t btusb snd soundcore snd_page_alloc usb_storage parport_pc floppy parport bluetooth sky2 asus_atk0110 hwmon iTCO_wdt iTCO_vendor_support serio_raw pata_jmicron ata_generic pata_acpi btrfs zlib_deflate libcrc32c nouveau drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
Pid: 2656, comm: btrfs-vol Tainted: G D (2.6.29.6-213.fc11.i686.PAE #1) P5K-VM
EIP: 0060:[<f8a0d543>] EFLAGS: 00210257 CPU: 1
EIP is at __btrfs_reserve_extent+0x339/0x347 [btrfs]
EAX: f5c8aadc EBX: f5c8aa50 ECX: 00000000 EDX: 00000001
ESI: f5c8aadc EDI: f58c786c EBP: f09cb7e4 ESP: f09cb78c
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process btrfs-vol (pid: 2656, ti=f09ca000 task=f0995860 task.ti=f09ca000)
Stack: fffff000 00000000 00000000 f09cb8b3 ffffffff ffffffff 00000000 00000000 00000000 00000024 00000000 00000000 00001000 00000000 00000000 f4d096f0 f5c8aad0 00001000 00000000 000aa000 f09cb8b3 f09cb838 f09cb830 f8a0d70e Call Trace: [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs] [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs] [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs] [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs] [<c0564dc7>] ? __lookup_tag+0x55/0xe3 [<c0564ec4>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs] [<c0716989>] ? _spin_lock_irq+0x21/0x25 [<c043dac2>] ? run_timer_softirq+0x1ae/0x1c0 [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs] [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs] [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs] [<c0484bb8>] ? do_writepages+0x25/0x39 [<c04be369>] ? __writeback_single_inode+0x140/0x241 [<c056459d>] ? prop_fraction_single+0x35/0x55 [<c04be7c7>] ? generic_sync_sb_inodes+0x1d9/0x2f6 [<c04bea8d>] ? writeback_inodes+0x82/0xca [<c0485464>] ? balance_dirty_pages_ratelimited_nr+0x137/0x23a [<f8a0ad09>] ? relocate_inode_pages+0x2fa/0x305 [btrfs] [<f8a0ae69>] ? relocate_data_extent+0x155/0x173 [btrfs] [<f8a165cf>] ? btrfs_read_fs_root_no_name+0x77/0x100 [btrfs] [<f8a0fb0e>] ? relocate_one_extent+0x181/0x371 [btrfs] [<f8a18e57>] ? btrfs_end_transaction+0xf/0x11 [btrfs] [<f8a0a26f>] ? __alloc_chunk_for_shrink+0x10d/0x125 [btrfs] [<f8a0ff52>] ? btrfs_relocate_block_group+0x254/0x3dc [btrfs] [<f8a36105>] ? btrfs_relocate_chunk+0x53/0x419 [btrfs] [<f8a2e89e>] ? map_private_extent_buffer+0x96/0xb8 [btrfs] [<f8a2e90f>] ? map_extent_buffer+0x4f/0x7f [btrfs] [<c0425a4b>] ? kunmap_atomic+0x6e/0x7c [<f8a2e32f>] ? unmap_extent_buffer+0x11/0x13 [btrfs] [<f8a274c1>] ? btrfs_dev_extent_chunk_offset+0xa2/0xae [btrfs] [<f8a366b3>] ? btrfs_shrink_device+0x1e8/0x29c [btrfs] [<f8a3692a>] ? btrfs_balance+0xec/0x243 [btrfs] [<c053947f>] ? avc_has_perm+0x41/0x4e [<f8a3a8e1>] ? btrfs_ioctl+0x72c/0x837 [btrfs] [<f8a3a1b5>] ? btrfs_ioctl+0x0/0x837 [btrfs] [<c04b2623>] ? vfs_ioctl+0x1d/0x76 [<c04b2f16>] ? do_vfs_ioctl+0x480/0x4ba [<c053b04c>] ? selinux_file_ioctl+0x43/0x46 [<c04b2f96>] ? sys_ioctl+0x46/0x66 [<c040945f>] ? sysenter_do_call+0x12/0x34 Code: e8 ef 3e a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 0c c9 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:f09cb78c ---[ end trace 209348013c0e69ad ]---
Signature · 3 lines
Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com/ Tel: +47 21 54 41 27
Quoted 12 lines · Tore Anderson
* Josef Bacik
Quoted 5 lines
Ok good news is, btrfs-vol -b will fix your problem. Somehow most of your disk has been allocated to use metadata. So did you have a whole bunch of stuff on this disk and then delete it all? Because that would put you in that situation. If you have not then there is likely a bug in the metadata ratio stuff that needs to be fixed.
As far as I know there hasn't been a lot of stuff on the file system, I'm afraid. The file system was created by the Fedora 11 installer, and it has just been used as the system drive (I've got /home on NFS).
Quoted 8 lines
btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn. I've was able to get output from dmesg before my SSH sessions started hanging - maybe you can make anything out of it? Anyway, right now I have no more remote access to the box so any further debugging will have to wait until tomorrow morning.
btrfs relocating chunk 129952120832
btrfs relocating block group 129952120832 flags 1
Ugh, it hit the data group first, which it can't relocate because there is nowhere to put the blocks.
-chris
Quoted 12 lines · Tore Anderson
* Josef Bacik
Quoted 5 lines
Ok good news is, btrfs-vol -b will fix your problem. Somehow most of your disk has been allocated to use metadata. So did you have a whole bunch of stuff on this disk and then delete it all? Because that would put you in that situation. If you have not then there is likely a bug in the metadata ratio stuff that needs to be fixed.
As far as I know there hasn't been a lot of stuff on the file system, I'm afraid. The file system was created by the Fedora 11 installer, and it has just been used as the system drive (I've got /home on NFS).
Quoted 6 lines
btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn. I've was able to get output from dmesg before my SSH sessions started hanging - maybe you can make anything out of it? Anyway, right now I have no more remote access to the box so any further debugging will have to wait until tomorrow morning.
Hrm yeah I was afraid of that. So you will have to try and free up some data so the balancer as enough room to move things around. Also one thing that _may_ help is mounting with nocow for a little bit, so any writing (like logging and such) just over-writes stuff instead of needing a new extent, until btrfs-vol -b can finish, and then you can go back to the normal mount options. I still can't figure out how this happened, but one thing to do for now would be to mount with metadata_ratio=100. This will make you more likely to panic in the event that you run out of metadata, but its not that likely to happen, especially with as much disk space as you have. Sorry about all of this, hopefully a better solution will be coming down in a few months. Thanks,
Josef
Quoted 20 lines · Josef Bacik
Quoted 12 lines · Tore Anderson
* Josef Bacik
Quoted 5 lines
Ok good news is, btrfs-vol -b will fix your problem. Somehow most of your disk has been allocated to use metadata. So did you have a whole bunch of stuff on this disk and then delete it all? Because that would put you in that situation. If you have not then there is likely a bug in the metadata ratio stuff that needs to be fixed.
As far as I know there hasn't been a lot of stuff on the file system, I'm afraid. The file system was created by the Fedora 11 installer, and it has just been used as the system drive (I've got /home on NFS).
Quoted 6 lines
btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn. I've was able to get output from dmesg before my SSH sessions started hanging - maybe you can make anything out of it? Anyway, right now I have no more remote access to the box so any further debugging will have to wait until tomorrow morning.
Quoted 10 lines
Hrm yeah I was afraid of that. So you will have to try and free up some data so the balancer as enough room to move things around. Also one thing that _may_ help is mounting with nocow for a little bit, so any writing (like logging and such) just over-writes stuff instead of needing a new extent, until btrfs-vol -b can finish, and then you can go back to the normal mount options. I still can't figure out how this happened, but one thing to do for now would be to mount with metadata_ratio=100. This will make you more likely to panic in the event that you run out of metadata, but its not that likely to happen, especially with as much disk space as you have. Sorry about all of this, hopefully a better solution will be coming down in a few months. Thanks,
We can code a fix for the balancer problem quickly, but you might have a hard time recompiling it and getting it on that box.
If this data is critical we'll write something up.
-chris
Hi,
* Josef Bacik
Quoted 2 lines
Quoted 2 lines
Sorry about all of this, hopefully a better solution will be coming down in a few months.
Don't be - the only reason why I'm running btrfs at this point is to test and hopefully help out improving it by reporting the problems I'm running into. :-)
Thanks for the suggestions!
* Chris Mason
Quoted 4 lines
We can code a fix for the balancer problem quickly, but you might have a hard time recompiling it and getting it on that box.
If this data is critical we'll write something up.
There's no critical data on this filesystem, it's only used for my Fedora installation. Won't take me long to re-install from scratch if that's what it takes. All og my important data (home directory) is on NFS. Using NFS I can easily get new versions of btrfs-progs onto the box with the hosed btrfs, too.
So if you meant to code up the fix as a band-aid only for me, don't bother (but thanks for the offer)! On the other hand, if you wanted to get such a fix into btrfs-progs anyway I'll be happy to test it out for you.
Best regards,
Signature · 3 lines
Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com/ Tel: +47 21 54 41 27