can not allocate space for caching data
Hi, Chris
There is something wrong with this patch:
commit 83a50de97fe96aca82389e061862ed760ece2283
Author: Chris Mason <[email protected]> Date: Mon Dec 13 15:06:46 2010 -0500
Btrfs: prevent RAID level downgrades when space is low
The extent allocator has code that allows us to fill allocations from any available block group, even if it doesn't match the raid level we've requested.
This was put in because adding a new drive to a filesystem made with the default mkfs options actually upgrades the metadata from single spindle dup to full RAID1.
But, the code also allows us to allocate from a raid0 chunk when we really want a raid1 or raid10 chunk. This can cause big trouble because mkfs creates a small (4MB) raid0 chunk for data and metadata which then goes unused for raid1/raid10 installs.
The allocator will happily wander in and allocate from that chunk when things get tight, which is not correct.
The fix here is to make sure that we provide duplication when the caller has asked for it. It does all the dups to be any raid level, which preserves the dup->raid1 upgrade abilities.
Btrfs has added the space of single chunks and raid0 chunks into the space information, so when we use btrfs_check_data_free_space() to check if there is some space for storing file data, this function may return true. So we write the data into the cache successfully. But, the extent allocator can not allocate any space to store that cached data, and then the file system panic.
I think we subtract that space from the space information, or split the space information into two types, one is used to manage the chunks with duplication, the other manages the other chunks.
Excerpts from Miao Xie's message of 2010-12-20 07:25:10 -0500:
Quoted 39 lines
Hi, Chris
There is something wrong with this patch:
commit
83a50de97fe96aca82389e061862ed760ece2283Author: Chris Mason <[email protected]> Date: Mon Dec 13 15:06:46 2010 -0500Btrfs: prevent RAID level downgrades when space is low
The extent allocator has code that allows us to fill allocations from any available block group, even if it doesn't match the raid level we've requested.
This was put in because adding a new drive to a filesystem made with the default mkfs options actually upgrades the metadata from single spindle dup to full RAID1.
But, the code also allows us to allocate from a raid0 chunk when we really want a raid1 or raid10 chunk. This can cause big trouble because mkfs creates a small (4MB) raid0 chunk for data and metadata which then goes unused for raid1/raid10 installs.
The allocator will happily wander in and allocate from that chunk when things get tight, which is not correct.
The fix here is to make sure that we provide duplication when the caller has asked for it. It does all the dups to be any raid level, which preserves the
dup->raid1upgrade abilities.Signed-off-by Chris Mason <[email protected]>Btrfs has added the space of single chunks and raid0 chunks into the space information, so when we use
btrfs_check_data_free_space()to check if there is some space for storing file data, this function may return true. So we write the data into the cache successfully. But, the extent allocator can not allocate any space to store that cached data, and then the file system panic.
Quoted 3 lines
I think we subtract that space from the space information, or split the space information into two types, one is used to manage the chunks with duplication, the other manages the other chunks.
Ok, do you have a test case that triggers this? I'll work out a patch. Yan Zheng's original idea of 'the chunks should be readonly' should help us deduct them from the total.
-chris
Quoted 40 lines · Chris Mason
Excerpts from Miao Xie's message of 2010-12-20 07:25:10 -0500:
Quoted 39 lines
Hi, Chris
There is something wrong with this patch:
commit
83a50de97fe96aca82389e061862ed760ece2283Author: Chris Mason<[email protected]> Date: Mon Dec 13 15:06:46 2010 -0500Btrfs: prevent RAID level downgrades when space is low
The extent allocator has code that allows us to fill allocations from any available block group, even if it doesn't match the raid level we've requested.
This was put in because adding a new drive to a filesystem made with the default mkfs options actually upgrades the metadata from single spindle dup to full RAID1.
But, the code also allows us to allocate from a raid0 chunk when we really want a raid1 or raid10 chunk. This can cause big trouble because mkfs creates a small (4MB) raid0 chunk for data and metadata which then goes unused for raid1/raid10 installs.
The allocator will happily wander in and allocate from that chunk when things get tight, which is not correct.
The fix here is to make sure that we provide duplication when the caller has asked for it. It does all the dups to be any raid level, which preserves the
dup->raid1upgrade abilities.Signed-off-by Chris Mason<[email protected]>Btrfs has added the space of single chunks and raid0 chunks into the space information, so when we use
btrfs_check_data_free_space()to check if there is some space for storing file data, this function may return true. So we write the data into the cache successfully. But, the extent allocator can not allocate any space to store that cached data, and then the file system panic.
Quoted 7 lines
Quoted 3 lines
I think we subtract that space from the space information, or split the space information into two types, one is used to manage the chunks with duplication, the other manages the other chunks.
Ok, do you have a test case that triggers this? I'll work out a patch. Yan Zheng's original idea of 'the chunks should be readonly' should help us deduct them from the total.
# mkfs.btrfs -d raid1 /dev/sda9 /dev/sda10 # mount /dev/sda9 /mnt # dd if=/dev/zero of=/mnt/tmpfile0 bs=4K count=999999999999999999 (fill the file system) # umount /mnt # mount /dev/sda9 /mnt # dd if=/dev/zero of=/mnt/tmpfile1 bs=4K count=1000 # sync
Thanks
Miao
Excerpts from Miao Xie's message of 2010-12-20 08:13:14 -0500:
Quoted 31 lines · Chris Mason
Quoted 22 lines
Excerpts from Miao Xie's message of 2010-12-20 07:25:10 -0500:
Quoted 21 lines
Hi, Chris
There is something wrong with this patch:
commit
83a50de97fe96aca82389e061862ed760ece2283Author: Chris Mason<[email protected]> Date: Mon Dec 13 15:06:46 2010 -0500Btrfs: prevent RAID level downgrades when space is low
The extent allocator has code that allows us to fill allocations from any available block group, even if it doesn't match the raid level we've requested.
Btrfs has added the space of single chunks and raid0 chunks into the space information, so when we use
btrfs_check_data_free_space()to check if there is some space for storing file data, this function may return true. So we write the data into the cache successfully. But, the extent allocator can not allocate any space to store that cached data, and then the file system panic.Quoted 7 lines
Quoted 3 lines
I think we subtract that space from the space information, or split the space information into two types, one is used to manage the chunks with duplication, the other manages the other chunks.
Ok, do you have a test case that triggers this? I'll work out a patch. Yan Zheng's original idea of 'the chunks should be readonly' should help us deduct them from the total.
Quoted 8 lines
# mkfs.btrfs -d raid1 /dev/sda9 /dev/sda10 # mount /dev/sda9 /mnt # dd if=/dev/zero of=/mnt/tmpfile0 bs=4K count=999999999999999999 (fill the file system) # umount /mnt # mount /dev/sda9 /mnt # dd if=/dev/zero of=/mnt/tmpfile1 bs=4K count=1000 # sync
Looks like we've got an off by one bug in set_block_group_ro, which is why our block group isn't getting set to ro. With this patch, we're properly setting the block group ro, and the enospc accounting is done correctly.
It should also be able to replace my commit above. Please take a look, Zheng does this look correct to you?
fs/btrfs/extent-tree.c +2 -1
@@ -7970,13 +7970,14 @@ static int set_block_group_ro(struct btrfs_block_group_cache *cache)if (sinfo->bytes_used + sinfo->bytes_reserved + sinfo->bytes_pinned + sinfo->bytes_may_use + sinfo->bytes_readonly + - cache->reserved_pinned + num_bytes < sinfo->total_bytes) { + cache->reserved_pinned + num_bytes <= sinfo->total_bytes) { sinfo->bytes_readonly += num_bytes; sinfo->bytes_reserved += cache->reserved_pinned; cache->reserved_pinned =0 ; cache->ro =1 ; ret =0 ; } +spin_unlock (&cache->lock);spin_unlock (&sinfo->lock);return ret;
Quoted 15 lines · On Mon, Dec 20, 2010 at 11:41 PM, Chris Mason <[email protected]> = wrote:
Excerpts from Miao Xie's message of 2010-12-20 08:13:14 -0500:
Quoted 14 lines · Chris Mason
Quoted 13 lines
Excerpts from Miao Xie's message of 2010-12-20 07:25:10 -0500:
Quoted 12 lines
Hi, Chris
There is something wrong with this patch:
commit
83a50de97fe96aca82389e061862ed760ece2283
Author: Chris Mason<[email protected]>
Date: =A0 Mon Dec 13 15:06:46 2010 -0500=A0 =A0 =A0Btrfs: prevent RAID level downgrades when space is low
=A0 =A0 =A0The extent allocator has code that allows us to fill =A0 =A0 =A0allocations from any available block group, even if it=
doesn't
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
=A0 =A0 =A0match the raid level we've requested.
Btrfs has added the space of single chunks and raid0 chunks into =
the space
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
information, so when we use
btrfs_check_data_free_space()to chec=
k if there
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
is some space for storing file data, this function may return tru=
e. So we
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
write the data into the cache successfully. But, the extent alloc=
ator can
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
not allocate any space to store that cached data, and then the fi=
le system
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
panic.
I think we subtract that space from the space information, or spl=
it the space
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
information into two types, one is used to manage the chunks with=
duplication,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
the other manages the other chunks.
Ok, do you have a test case that triggers this? =A0I'll work out a=
patch.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Yan Zheng's original idea of 'the chunks should be readonly' shoul=
d help
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
us deduct them from the total.
# mkfs.btrfs -d raid1 /dev/sda9 /dev/sda10 # mount /dev/sda9 /mnt # dd if=3D/dev/zero of=3D/mnt/tmpfile0 bs=3D4K count=3D9999999999999=
99999
Quoted 7 lines
Quoted 5 lines
=A0 =A0(fill the file system)
# umount /mnt # mount /dev/sda9 /mnt # dd if=3D/dev/zero of=3D/mnt/tmpfile1 bs=3D4K count=3D1000 # syncLooks like we've got an off by one bug in
set_block_group_ro, which i=
s
Quoted 1 line
why our block group isn't getting set to ro. =A0With this patch, we'r=
e
Quoted 1 line
properly setting the block group ro, and the enospc accounting is don=
e
Quoted 3 lines
correctly.
It should also be able to replace my commit above. =A0Please take a l=
ook,
fs/btrfs/extent-tree.c static int set_block_group_ro(struct btrfs_bl=
Quoted 7 lines
Zheng does this look correct to you?
fs/btrfs/extent-tree.c +0
-0@@ -7970,13 +7970,14 @@ static int set_block_group_ro(struct btrfs_bl=
ock_group_cache *cache)
Quoted 5 lines
=A0 =A0 =A0 =A0if (
sinfo->bytes_used+sinfo->bytes_reserved+ sinfo-=bytes_pinned+
=A0 =A0 =A0 =A0 =A0 =A0sinfo->bytes_may_use +sinfo->bytes_readonly+
- =A0 =A0 =A0 =A0 =A0cache->reserved_pinned+num_bytes< sinfo->tot=
al_bytes) {
Quoted 1 line
+ =A0 =A0 =A0 =A0 =A0
cache->reserved_pinned+num_bytes<=3D sinfo->=
total_bytes) {
Quoted 2 lines
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sinfo->bytes_readonly +=3D
num_bytes; =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0sinfo->bytes_reserved +=3D cache->rese=
rved_pinned;
Quoted 9 lines
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0cache->reserved_pinned =3D 0; =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0cache->ro =3D 1; =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ret =3D 0;
=A0 =A0 =A0 =A0}
+
=A0 =A0 =A0 =A0spin_unlock(&cache->lock);
=A0 =A0 =A0 =A0spin_unlock(&sinfo->lock);
=A0 =A0 =A0 =A0return ret;
Looks good for me,
Yan, Zheng
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html