problem with ceph and btrfs patch: set journal_info in async trans commit worker

DISCUSSION Stefan Priebe - Profihost AG 2 participants 3 messages Nov 14, 2012 – Nov 15, 2012 Btrfs file system

Share & tools

Stefan Priebe - Profihost AG

author

2012-11-14 13:42 UTC #0 lore raw

Hello list,

i wanted to try out ceph with latest vanilla kernel 3.7-rc5. I was seeing a massive performance degration. I see around 22x btrfs-endio-write processes every 10-20 seconds and they run a long time while consuming a massive amount of CPU.

So my performance of 23.000 iops drops to an up and down of 23.000 iops to 0 - avg is now 2500 iops instead of 23.000.

Git bisect shows me commit: e209db7ace281ca347b1ac699bf1fb222eac03fe "Btrfs: set journal_info in async trans commit worker" as the problematic patch.

When i revert this one everything is fine again.

Is this known?

Greets,
Stefan

Miao Xie

2012-11-15 05:18 UTC #1 lore raw

Hi, Stefan

Quoted 11 lines · Stefan Priebe - Profihost AG

Hello list,
i wanted to try out ceph with latest vanilla kernel 3.7-rc5. I was seeing a massive performance degration. I see around 22x btrfs-endio-write processes every 10-20 seconds and they run a long time while consuming a massive amount of CPU.
So my performance of 23.000 iops drops to an up and down of 23.000 iops to 0 - avg is now 2500 iops instead of 23.000.
Git bisect shows me commit: e209db7ace281ca347b1ac699bf1fb222eac03fe "Btrfs: set journal_info in async trans commit worker" as the problematic patch.
When i revert this one everything is fine again.
Is this known?

Could you try the following patch?

http://marc.info/?l=linux-btrfs&m=135175512030453&w=2

I think the patch

Btrfs: set journal_info in async trans commit worker

is not the real reason that caused the regression.

I guess it is caused by the bug of the reservation. When we join the same transaction handle more than 2 times, the pointer of the reservation in the transaction handle would be lost, and the statistical data in the reservation would be corrupted. And then we would trigger the space flush, which may block your tasks.

Thanks
Miao

Quoted 8 lines

Greets,
Stefan

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe - Profihost AG

author

2012-11-15 08:50 UTC #2 lore raw

Hi Miao,

Am 15.11.2012 06:18, schrieb Miao Xie:

Quoted 23 lines

Hi, Stefan
Quoted 11 lines · Stefan Priebe - Profihost AG
Hello list,
i wanted to try out ceph with latest vanilla kernel 3.7-rc5. I was seeing a massive performance degration. I see around 22x btrfs-endio-write processes every 10-20 seconds and they run a long time while consuming a massive amount of CPU.
So my performance of 23.000 iops drops to an up and down of 23.000 iops to 0 - avg is now 2500 iops instead of 23.000.
Git bisect shows me commit: e209db7ace281ca347b1ac699bf1fb222eac03fe "Btrfs: set journal_info in async trans commit worker" as the problematic patch.
When i revert this one everything is fine again.
Is this known?
Could you try the following patch?
http://marc.info/?l=linux-btrfs&m=135175512030453&w=2
I think the patch
Btrfs: set journal_info in async trans commit worker

Quoted 7 lines

is not the real reason that caused the regression.
I guess it is caused by the bug of the reservation. When we join the same transaction handle more than 2 times, the pointer of the reservation in the transaction handle would be lost, and the statistical data in the reservation would be corrupted. And then we would trigger the space flush, which may block your tasks.

i applied your whole patchset. It looks a lot better now but avg iops is now 5000 iops and not 23.000 like when removing the mentioned commit (e209db7ace281ca347b1ac699bf1fb222eac03fe).

Stefan