Skip to contentSkip to conversation

Non-existent qgroup in parent-child relation prevents makes qgroup commands fail

DISCUSSION Andrei Borzenkov 2 participants 6 messages Aug 2, 2019 – Aug 3, 2019 Btrfs file system

lore
Share & tools
lore
#0 lore raw

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically destroys snapshots and apparently either kernel or snapper now also remove corresponding qgroup. I played with snapshots and created several top level qgroups that included snapshot qgroups existing at this time. Now these snapshots are gone, their qgroups are gone ... and what can I do? I have no way to even know what is wrong because the very command that shows it fails immediately.

bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789 (0/789 QGROUP_RELATION 2/792)
(0/789 QGROUP_RELATION 2/793)
(0/789 QGROUP_RELATION 2/795)
(0/789 QGROUP_RELATION 2/799)
(0/789 QGROUP_RELATION 2/800)
(0/789 QGROUP_RELATION 2/803)
(0/789 QGROUP_RELATION 2/804)
(0/789 QGROUP_RELATION 2/805)
(0/789 QGROUP_RELATION 2/806)
(0/789 QGROUP_RELATION 2/807)
(0/789 QGROUP_RELATION 2/808)
(0/789 QGROUP_RELATION 2/809)
(0/789 QGROUP_RELATION 2/814)
(0/789 QGROUP_RELATION 2/818)
(0/789 QGROUP_RELATION 2/819)
(2/792 QGROUP_RELATION 0/789)
(2/793 QGROUP_RELATION 0/789)
(2/795 QGROUP_RELATION 0/789)
(2/799 QGROUP_RELATION 0/789)
(2/800 QGROUP_RELATION 0/789)
(2/803 QGROUP_RELATION 0/789)
(2/804 QGROUP_RELATION 0/789)
(2/805 QGROUP_RELATION 0/789)
(2/806 QGROUP_RELATION 0/789)
(2/807 QGROUP_RELATION 0/789)
(2/808 QGROUP_RELATION 0/789)
(2/809 QGROUP_RELATION 0/789)
(2/814 QGROUP_RELATION 0/789)
(2/818 QGROUP_RELATION 0/789)
(2/819 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

And even if I find it out, I cannot fix it anyway

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
ERROR: unable to assign quota group: Invalid argument bor@tw:~/python-btrfs/examples>

I can remove parent qgroup, but it does not clean up parent-child relationship

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 . bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792 (0/789 QGROUP_RELATION 2/792)
(2/792 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

#1 lore raw
Quoted 9 lines · Andrei Borzenkov

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically destroys snapshots and apparently either kernel or snapper now also remove corresponding qgroup. I played with snapshots and created several top level qgroups that included snapshot qgroups existing at this time. Now these snapshots are gone, their qgroups are gone ...

Kernel version please.

IIRC latest upstream kernel doesn't remove the level 0 qgroup. It may be the userspace doing it improperly.

Quoted 37 lines

and what can I
do? I have no way to even know what is wrong because the very command that shows it fails immediately.

bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789 (0/789 QGROUP_RELATION 2/792)
(0/789 QGROUP_RELATION 2/793)
(0/789 QGROUP_RELATION 2/795)
(0/789 QGROUP_RELATION 2/799)
(0/789 QGROUP_RELATION 2/800)
(0/789 QGROUP_RELATION 2/803)
(0/789 QGROUP_RELATION 2/804)
(0/789 QGROUP_RELATION 2/805)
(0/789 QGROUP_RELATION 2/806)
(0/789 QGROUP_RELATION 2/807)
(0/789 QGROUP_RELATION 2/808)
(0/789 QGROUP_RELATION 2/809)
(0/789 QGROUP_RELATION 2/814)
(0/789 QGROUP_RELATION 2/818)
(0/789 QGROUP_RELATION 2/819)
(2/792 QGROUP_RELATION 0/789)
(2/793 QGROUP_RELATION 0/789)
(2/795 QGROUP_RELATION 0/789)
(2/799 QGROUP_RELATION 0/789)
(2/800 QGROUP_RELATION 0/789)
(2/803 QGROUP_RELATION 0/789)
(2/804 QGROUP_RELATION 0/789)
(2/805 QGROUP_RELATION 0/789)
(2/806 QGROUP_RELATION 0/789)
(2/807 QGROUP_RELATION 0/789)
(2/808 QGROUP_RELATION 0/789)
(2/809 QGROUP_RELATION 0/789)
(2/814 QGROUP_RELATION 0/789)
(2/818 QGROUP_RELATION 0/789)
(2/819 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

Quoted 1 line

And even if I find it out, I cannot fix it anyway

Furthermore, latest kernel should automatically remove the relation when deleting the qgroup.

Would you please provide the (minimal) script/reproducer causing the situation and kernel version?

Thanks,
Qu

Quoted 14 lines

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
ERROR: unable to assign quota group: Invalid argument bor@tw:~/python-btrfs/examples>

I can remove parent qgroup, but it does not clean up parent-child relationship

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 . bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792 (0/789 QGROUP_RELATION 2/792)
(2/792 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

#2 lore raw

03.08.2019 2:09, Qu Wenruo пишет:

Quoted 13 lines · Andrei Borzenkov
Quoted 9 lines

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically destroys snapshots and apparently either kernel or snapper now also remove corresponding qgroup. I played with snapshots and created several top level qgroups that included snapshot qgroups existing at this time. Now these snapshots are gone, their qgroups are gone ...

Quoted 3 lines

Kernel version please.

IIRC latest upstream kernel doesn't remove the level 0 qgroup.

Yes?

Quoted 2 lines

It may be the userspace doing it improperly.

Not sure what "improperly" means here. snapper removes qgroup after deleting snapshot. What is "improper" here?

Snapper obviously does not track every parent-child relationship beyond what it itself cares about (single summary qrgoup that includes all snapshots).

Quoted 37 lines
Quoted 37 lines

and what can I
do? I have no way to even know what is wrong because the very command that shows it fails immediately.

bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789 (0/789 QGROUP_RELATION 2/792)
(0/789 QGROUP_RELATION 2/793)
(0/789 QGROUP_RELATION 2/795)
(0/789 QGROUP_RELATION 2/799)
(0/789 QGROUP_RELATION 2/800)
(0/789 QGROUP_RELATION 2/803)
(0/789 QGROUP_RELATION 2/804)
(0/789 QGROUP_RELATION 2/805)
(0/789 QGROUP_RELATION 2/806)
(0/789 QGROUP_RELATION 2/807)
(0/789 QGROUP_RELATION 2/808)
(0/789 QGROUP_RELATION 2/809)
(0/789 QGROUP_RELATION 2/814)
(0/789 QGROUP_RELATION 2/818)
(0/789 QGROUP_RELATION 2/819)
(2/792 QGROUP_RELATION 0/789)
(2/793 QGROUP_RELATION 0/789)
(2/795 QGROUP_RELATION 0/789)
(2/799 QGROUP_RELATION 0/789)
(2/800 QGROUP_RELATION 0/789)
(2/803 QGROUP_RELATION 0/789)
(2/804 QGROUP_RELATION 0/789)
(2/805 QGROUP_RELATION 0/789)
(2/806 QGROUP_RELATION 0/789)
(2/807 QGROUP_RELATION 0/789)
(2/808 QGROUP_RELATION 0/789)
(2/809 QGROUP_RELATION 0/789)
(2/814 QGROUP_RELATION 0/789)
(2/818 QGROUP_RELATION 0/789)
(2/819 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

Quoted 5 lines
Quoted 1 line

And even if I find it out, I cannot fix it anyway

Furthermore, latest kernel should automatically remove the relation when deleting the qgroup.

Yes, seems to be the case now with 5.2.3.

Quoted 3 lines

Would you please provide the (minimal) script/reproducer causing the situation and kernel version?

I cannot reproduce it on purpose with kernel 5.2.3 and btrfsprogs 5.1. I do not remember when I created the configuration in question, it probably was late 4.x or early 5.x. System was periodically updated since then and I noticed it only now.

Actually kernel should be dropping parent-child relation when removing child since kernel 4.1 (commit f5a6b1c53bdd44f79e3904c0f5e59f956b49b2c8). May be there is (was) some other code path that skipped it.

OTOH this is not atomic. First qgroup item itself is removed, then relation. If removing relation fails for whatever reason item itself is already gone and we have situation above.

Quoted 18 lines

Thanks,
Qu

Quoted 14 lines

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
ERROR: unable to assign quota group: Invalid argument bor@tw:~/python-btrfs/examples>

I can remove parent qgroup, but it does not clean up parent-child relationship

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 . bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792 (0/789 QGROUP_RELATION 2/792)
(2/792 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

#3 lore raw
Quoted 18 lines · Andrei Borzenkov

03.08.2019 2:09, Qu Wenruo пишет:

Quoted 13 lines · Andrei Borzenkov
Quoted 9 lines

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically destroys snapshots and apparently either kernel or snapper now also remove corresponding qgroup. I played with snapshots and created several top level qgroups that included snapshot qgroups existing at this time. Now these snapshots are gone, their qgroups are gone ...

Quoted 3 lines

Kernel version please.

IIRC latest upstream kernel doesn't remove the level 0 qgroup.

Quoted 7 lines

Yes?

Quoted 2 lines

It may be the userspace doing it improperly.

Not sure what "improperly" means here. snapper removes qgroup after deleting snapshot. What is "improper" here?

Doing without using the qgroup ioctl, but some extra flag in snapshot creation/deletion, which can also add relation at subv/snapshot creation time.

Quoted 59 lines

Snapper obviously does not track every parent-child relationship beyond what it itself cares about (single summary qrgoup that includes all snapshots).

Quoted 37 lines
Quoted 37 lines

and what can I
do? I have no way to even know what is wrong because the very command that shows it fails immediately.

bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 0/789 (0/789 QGROUP_RELATION 2/792)
(0/789 QGROUP_RELATION 2/793)
(0/789 QGROUP_RELATION 2/795)
(0/789 QGROUP_RELATION 2/799)
(0/789 QGROUP_RELATION 2/800)
(0/789 QGROUP_RELATION 2/803)
(0/789 QGROUP_RELATION 2/804)
(0/789 QGROUP_RELATION 2/805)
(0/789 QGROUP_RELATION 2/806)
(0/789 QGROUP_RELATION 2/807)
(0/789 QGROUP_RELATION 2/808)
(0/789 QGROUP_RELATION 2/809)
(0/789 QGROUP_RELATION 2/814)
(0/789 QGROUP_RELATION 2/818)
(0/789 QGROUP_RELATION 2/819)
(2/792 QGROUP_RELATION 0/789)
(2/793 QGROUP_RELATION 0/789)
(2/795 QGROUP_RELATION 0/789)
(2/799 QGROUP_RELATION 0/789)
(2/800 QGROUP_RELATION 0/789)
(2/803 QGROUP_RELATION 0/789)
(2/804 QGROUP_RELATION 0/789)
(2/805 QGROUP_RELATION 0/789)
(2/806 QGROUP_RELATION 0/789)
(2/807 QGROUP_RELATION 0/789)
(2/808 QGROUP_RELATION 0/789)
(2/809 QGROUP_RELATION 0/789)
(2/814 QGROUP_RELATION 0/789)
(2/818 QGROUP_RELATION 0/789)
(2/819 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

Quoted 5 lines
Quoted 1 line

And even if I find it out, I cannot fix it anyway

Furthermore, latest kernel should automatically remove the relation when deleting the qgroup.

Yes, seems to be the case now with 5.2.3.

Quoted 3 lines

Would you please provide the (minimal) script/reproducer causing the situation and kernel version?

I cannot reproduce it on purpose with kernel 5.2.3 and btrfsprogs 5.1. I do not remember when I created the configuration in question, it probably was late 4.x or early 5.x. System was periodically updated since then and I noticed it only now.

Quoted 8 lines

Actually kernel should be dropping parent-child relation when removing child since kernel 4.1 (commit f5a6b1c53bdd44f79e3904c0f5e59f956b49b2c8). May be there is (was) some other code path that skipped it.

OTOH this is not atomic. First qgroup item itself is removed, then relation. If removing relation fails for whatever reason item itself is already gone and we have situation above.

You're right about the removal, although the term is not "atomic".

For qgroup item deletion, we can return -ENOENT or other serious tree corruption error from btrfs_search_slot()/btrfs_del_item().

But even for -ENOENT case, we still tries to delete the relation item. So that part is OK.

The problem is the del_qgroup_relation part, where if we hit even -ENOENT we abort deleting even if we may have extra items needs to delete.

We need a little more polish on the error handler.

So is the btrfs_del_qgroup_relation() where it doesn't delete all relations, but only one.

I'll soon send out patches to address your bugs.

Thanks,
Qu

Quoted 21 lines
Quoted 19 lines

Thanks,
Qu

Quoted 14 lines

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup remove 0/789 2/792 .
ERROR: unable to assign quota group: Invalid argument bor@tw:~/python-btrfs/examples>

I can remove parent qgroup, but it does not clean up parent-child relationship

bor@tw:~/python-btrfs/examples> sudo btrfs qgroup destroy 2/792 . bor@tw:~/python-btrfs/examples> sudo ./show_tree_keys.py 8 . | grep 2/792 (0/789 QGROUP_RELATION 2/792)
(2/792 QGROUP_RELATION 0/789)
bor@tw:~/python-btrfs/examples>

#4 lore raw

03.08.2019 9:17, Qu Wenruo пишет:

Quoted 26 lines · Andrei Borzenkov
Quoted 23 lines

03.08.2019 2:09, Qu Wenruo пишет:

Quoted 13 lines · Andrei Borzenkov
Quoted 9 lines

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically destroys snapshots and apparently either kernel or snapper now also remove corresponding qgroup. I played with snapshots and created several top level qgroups that included snapshot qgroups existing at this time. Now these snapshots are gone, their qgroups are gone ...

Quoted 3 lines

Kernel version please.

IIRC latest upstream kernel doesn't remove the level 0 qgroup.

Yes?

Quoted 2 lines

It may be the userspace doing it improperly.

Quoted 7 lines
Quoted 2 lines

Not sure what "improperly" means here. snapper removes qgroup after deleting snapshot. What is "improper" here?

Doing without using the qgroup ioctl, but some extra flag in snapshot creation/deletion, which can also add relation at subv/snapshot creation time.

As far as I can tell, this is exactly what snapper does:

            if (qgroup != no_qgroup)
            {
                size_t size = sizeof(btrfs_qgroup_inherit) +

sizeof(((btrfs_qgroup_inherit*) 0)->qgroups[0]);
vector<char> buffer(size, 0); struct btrfs_qgroup_inherit* inherit =
(btrfs_qgroup_inherit*) &buffer[0];

                inherit->num_qgroups = 1;
                inherit->num_ref_copies = 0;
                inherit->num_excl_copies = 0;
                inherit->qgroups[0] = qgroup;

                args_v2.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
                args_v2.size = size;
                args_v2.qgroup_inherit = inherit;
            }

Do you say it should not be doing it?

#5 lore raw
Quoted 55 lines · Andrei Borzenkov

03.08.2019 9:17, Qu Wenruo пишет:

Quoted 26 lines · Andrei Borzenkov
Quoted 23 lines

03.08.2019 2:09, Qu Wenruo пишет:

Quoted 13 lines · Andrei Borzenkov
Quoted 9 lines

bor@tw:~> sudo btrfs qgroup show .
ERROR: cannot find the qgroup 0/789
bor@tw:~>

Fine. This openSUSE with snapper which creates and automatically destroys snapshots and apparently either kernel or snapper now also remove corresponding qgroup. I played with snapshots and created several top level qgroups that included snapshot qgroups existing at this time. Now these snapshots are gone, their qgroups are gone ...

Quoted 3 lines

Kernel version please.

IIRC latest upstream kernel doesn't remove the level 0 qgroup.

Yes?

Quoted 2 lines

It may be the userspace doing it improperly.

Quoted 7 lines
Quoted 2 lines

Not sure what "improperly" means here. snapper removes qgroup after deleting snapshot. What is "improper" here?

Doing without using the qgroup ioctl, but some extra flag in snapshot creation/deletion, which can also add relation at subv/snapshot creation time.

As far as I can tell, this is exactly what snapper does:

            if (qgroup != no_qgroup)
            {
                size_t size = sizeof(btrfs_qgroup_inherit) +

sizeof(((btrfs_qgroup_inherit*) 0)->qgroups[0]);
vector<char> buffer(size, 0); struct btrfs_qgroup_inherit* inherit =
(btrfs_qgroup_inherit*) &buffer[0];

                inherit->num_qgroups = 1;
                inherit->num_ref_copies = 0;
                inherit->num_excl_copies = 0;
                inherit->qgroups[0] = qgroup;

                args_v2.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
                args_v2.size = size;
                args_v2.qgroup_inherit = inherit;
            }
Quoted 2 lines

Do you say it should not be doing it?

For creation, that's OK, as long as it's not using the deprecated copy version..

Thanks,
Qu