btrfs-convert processing time

DISCUSSION Olivier Bonvalet 3 participants 12 messages Feb 20, 2012 – Mar 31, 2012 Btrfs file system

lore

Share & tools

lore

Olivier Bonvalet

author

2012-02-20 12:51 UTC #0 lore raw

Hi,

I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the time needed to do that conversion.

The first FS is on a 500GiB block device, and btrfs-convert is running since more than 48h :
root 1978 25.6 47.7 748308 732556 ? D Feb18 944:44 btrfs-convert /dev/vg-backup/backup

The second is on a 340GiB block device, and the processing time is similar :
root 30192 5.2 3.0 73180 61520 pts/0 R Feb18 121:04 btrfs-convert /dev/vg-balor/backup

Is it "normal", or is there a problem during the processing ?

Thanks,
Olivier

Hubert Kario

2012-02-20 13:20 UTC #1 lore raw

Quoted 15 lines · On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:

Hi,
I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the time needed to do that conversion.
The first FS is on a 500GiB block device, and btrfs-convert is running since more than 48h :
root 1978 25.6 47.7 748308 732556 ? D Feb18 944:44
btrfs-convert /dev/vg-backup/backup
The second is on a 340GiB block device, and the processing time is similar :
root 30192 5.2 3.0 73180 61520 pts/0 R Feb18 121:04
btrfs-convert /dev/vg-balor/backup

Quoted 4 lines

Is it "normal", or is there a problem during the processing ?
Thanks,
Olivier

How much free space did those FS have? How many files were on them (was they dominated by small or large files)?

Regards,

Signature · 5 lines

Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawerów 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl

Olivier Bonvalet

author

2012-02-20 13:41 UTC #2 lore raw

Quoted 18 lines · Hubert Kario

Quoted 17 lines · On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:
Hi,
I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the time needed to do that conversion.
The first FS is on a 500GiB block device, and btrfs-convert is running since more than 48h :
root 1978 25.6 47.7 748308 732556 ? D Feb18 944:44
btrfs-convert /dev/vg-backup/backup
The second is on a 340GiB block device, and the processing time is similar :
root 30192 5.2 3.0 73180 61520 pts/0 R Feb18 121:04
btrfs-convert /dev/vg-balor/backup
Is it "normal", or is there a problem during the processing ?

Quoted 7 lines

Quoted 2 lines
Thanks,
Olivier
How much free space did those FS have? How many files were on them (was they dominated by small or large files)?
Regards,

Lot of small files (like compressed email from Maildir), and lot of hardlinks, and probably low free space (near 15% I suppose).

So I think I have my answer :)

Thanks

Hubert Kario

2012-02-20 13:50 UTC #3 lore raw

Quoted 5 lines · On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:

Quoted 4 lines · Hubert Kario
Quoted 3 lines · On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:
Hi,
=20
I'm trying to convert two ext4 FS to btrfs, but I'm surprised by t=

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
time needed to do that conversion.
=20
The first FS is on a 500GiB block device, and btrfs-convert is run=

ning

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
since more than 48h :
root 1978 25.6 47.7 748308 732556 ? D Feb18 944:44
btrfs-convert /dev/vg-backup/backup
=20
The second is on a 340GiB block device, and the processing time is
similar : root 30192 5.2 3.0 73180 61520 pts/0 R Feb1=

Quoted 9 lines

Quoted 9 lines
Quoted 7 lines
121:04 btrfs-convert /dev/vg-balor/backup
=20
=20
Is it "normal", or is there a problem during the processing ? =20
Thanks,
Olivier
=20
How much free space did those FS have? How many files were on them =

(was

Quoted 10 lines

Quoted 3 lines
they dominated by small or large files)?
=20
Regards,
=20
Lot of small files (like compressed email from Maildir), and lot of hardlinks, and probably low free space (near 15% I suppose). =20
=20
So I think I have my answer :)
=20

Yes, this is probably the worst possible combination.

Plese keep us updated. Just to have exact numbers for new users.

Regards,
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=F3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Fajar A. Nugraha

2012-02-20 14:00 UTC #4 lore raw

Quoted 11 lines · Hubert Kario

Quoted 6 lines · On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
Lot of small files (like compressed email from Maildir), and lot of hardlinks, and probably low free space (near 15% I suppose).
So I think I have my answer :)
Yes, this is probably the worst possible combination.
Plese keep us updated. Just to have exact numbers for new users.

... although it would probably fail anyway due to btrfs hardlink limit in the same directory.

Signature · 1 line

Fajar

Hubert Kario

2012-02-20 14:12 UTC #5 lore raw

Chris: What will btrfs-convert do when it encounters a directory with m= ore=20
hardlinks than the btrfs limit?

Quoted 3 lines · On Monday 20 of February 2012 21:00:34 Fajar A. Nugraha wrote:

Quoted 2 lines · Hubert Kario
Quoted 1 line · On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
Lot of small files (like compressed email from Maildir), and lot o=

Quoted 10 lines

Quoted 8 lines
Quoted 4 lines
hardlinks, and probably low free space (near 15% I suppose). =20
=20
So I think I have my answer :)
=20
Yes, this is probably the worst possible combination. =20
Plese keep us updated. Just to have exact numbers for new users.
=20
... although it would probably fail anyway due to btrfs hardlink limi=

Quoted 1 line

in the same directory.

This problem also went through my mind.

Olivier Bonvalet

author

2012-02-20 14:29 UTC #6 lore raw

In reply to Fajar A. Nugraha

Quoted 9 lines · Fajar A. Nugraha

Quoted 8 lines · Hubert Kario
Quoted 6 lines · On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
Lot of small files (like compressed email from Maildir), and lot of hardlinks, and probably low free space (near 15% I suppose).
So I think I have my answer :)

Quoted 8 lines

Quoted 3 lines
Yes, this is probably the worst possible combination.
Plese keep us updated. Just to have exact numbers for new users.
... although it would probably fail anyway due to btrfs hardlink limit in the same directory.

And in that case, btrfs-convert will abort, or ignore the error, or just hang ?

Fajar A. Nugraha

2012-02-21 05:51 UTC #7 lore raw

On Mon, Feb 20, 2012 at 9:29 PM, Olivier Bonvalet <[email protected]=

Quoted 4 lines

wrote:
Quoted 2 lines · Fajar A. Nugraha
On Mon, Feb 20, 2012 at 8:50 PM, Hubert Kario<[email protected]> =A0wro=

te:

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines · On Monday 20 of February 2012 14:41:33 Olivier Bonvalet wrote:
Quoted 2 lines
Lot of small files (like compressed email from Maildir), and lot o=

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
hardlinks, and probably low free space (near 15% I suppose).
So I think I have my answer :)

Quoted 7 lines

Quoted 7 lines
Quoted 3 lines
Yes, this is probably the worst possible combination.
Plese keep us updated. Just to have exact numbers for new users.
... although it would probably fail anyway due to btrfs hardlink lim=

Quoted 4 lines

Quoted 2 lines
in the same directory.
And in that case, btrfs-convert will abort, or ignore the error, or j=

ust

Quoted 1 line

hang ?

On my simple test with ubuntu precise, loop-mounted ext4, 8k hardlinks:

$ sudo btrfs-convert /dev/loop0
creating btrfs metadata.
$ echo $?
139

so no useful error message, but it doesn't crash. And when mounted the device still shows ext4.

A successful conversionn would look like this:

$ sudo btrfs-convert /dev/loop1
creating btrfs metadata.
creating ext2fs image file.
cleaning up system chunk.
conversion complete.

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Olivier Bonvalet

author

2012-02-22 16:55 UTC #8 lore raw

So, the btrfs-convert for the smaller drive is done... after near 5 days. Which stats can I give you ?

It's a 340GB LVM block device, and "btrfs filesystem df /backup/" say that :

Data: total=225.97GB, used=181.94GB
System: total=32.00MB, used=24.00KB
Metadata: total=111.00GB, used=91.56GB

If I mount the ext2_saved/image copy, I can see 257GB of data (78% of the block device used), with 17M of inodes.

The other btrfs-convert stay running.

Olivier Bonvalet

author

2012-03-30 19:23 UTC #9 lore raw

Le 22/02/2012 17:55, Olivier Bonvalet a =E9crit :

Quoted 1 line

So, the btrfs-convert for the smaller drive is done... after near 5 d=

ays. Which stats can I give you ?

Quoted 2 lines

It's a 340GB LVM block device, and "btrfs filesystem df /backup/" say=

that :

Quoted 7 lines

Data: total=3D225.97GB, used=3D181.94GB
System: total=3D32.00MB, used=3D24.00KB
Metadata: total=3D111.00GB, used=3D91.56GB
If I mount the ext2_saved/image copy, I can see 257GB of data (78% of=

the block device used), with 17M of inodes.

Quoted 5 lines

The other btrfs-convert stay running.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs=

" in

Quoted 3 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Well, the second one is still running :

root! backup:~# uptime
21:17:43 up 41 days, 19:59, 1 user, load average: 2.06, 1.90, 1.88

root! backup:~# ps auxw | grep btrfs
root 1978 25.6 74.3 1269072 1141212 ? D Feb18 15421:50 btrf=
s-convert /dev/vg-backup/backup

root! backup:~# iostat -k
Linux 2.6.42.3-dae-xen (backup) 30/03/2012 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle

          10,54    0,06    2,36   54,21    0,09   32,74

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvda 724,70 175,50 3181,13 634342597 11498370400

xvdy              0,00         0,00         0,00       1381          0
xvdz              0,00         0,00         0,00       4065          0
dm-0              0,49         3,22         1,78   11627497    6435028

dm-1 837,57 171,69 3178,57 620593936 11489105416
dm-2 0,34 0,59 0,78 2120800 2830584

11498370400 kB written, so, 11TB written to convert a 518GB partition ?

but more important : is it safe to abort the process ?

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hubert Kario

2012-03-31 20:50 UTC #10 lore raw

Quoted 5 lines · On Friday 30 of March 2012 21:23:52 Olivier Bonvalet wrote:

Le 22/02/2012 17:55, Olivier Bonvalet a =E9crit :
Quoted 4 lines
So, the btrfs-convert for the smaller drive is done... after near 5 days. Which stats can I give you ?
=20
It's a 340GB LVM block device, and "btrfs filesystem df /backup/" s=

Quoted 8 lines

Quoted 8 lines
that :
=20
Data: total=3D225.97GB, used=3D181.94GB
System: total=3D32.00MB, used=3D24.00KB
Metadata: total=3D111.00GB, used=3D91.56GB
=20
=20
If I mount the ext2_saved/image copy, I can see 257GB of data (78% =

Quoted 9 lines

Quoted 4 lines
the block device used), with 17M of inodes.
=20
=20
The other btrfs-convert stay running.
=20
Well, the second one is still running :
=20
root! backup:~# uptime
21:17:43 up 41 days, 19:59, 1 user, load average: 2.06, 1.90, 1.8=

Quoted 5 lines

=20
=20
root! backup:~# ps auxw | grep btrfs
root 1978 25.6 74.3 1269072 1141212 ? D Feb18 15421:50
btrfs-convert /dev/vg-backup/backup

Wow, that's /long/...

Quoted 7 lines

root! backup:~# iostat -k
Linux 2.6.42.3-dae-xen (backup) 30/03/2012 _x86_64_ (2 CPU)
=20
avg-cpu: %user %nice %system %iowait %steal %idle
           10,54    0,06    2,36   54,21    0,09   32,74
=20
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrt=

Quoted 1 line

xvda 724,70 175,50 3181,13 634342597 114983704=

Quoted 1 line

xvdy 0,00 0,00 0,00 1381 =

Quoted 1 line

xvdz 0,00 0,00 0,00 4065 =

Quoted 1 line

dm-0 0,49 3,22 1,78 11627497 643502=

Quoted 1 line

dm-1 837,57 171,69 3178,57 620593936 114891054=

Quoted 1 line

dm-2 0,34 0,59 0,78 2120800 283058=

Quoted 2 lines

=20
11498370400 kB written, so, 11TB written to convert a 518GB partition=

Quoted 3 lines

=20
=20
but more important : is it safe to abort the process ?

I'm quite sure you won't get a mountable filesystem, either ext or btrf= s.=20
btrfs-convert has to move at least some ext3 metadata blocks, so ext FS=
will=20
be broken. I don't know if btrfs-convert writes whole btrfs tree and th= en=20
updates it as it finds new inodes in ext3 or just goes over it sequenti= ally.=20
But either way, you'll get at best btrfs with lots of files missing.

So the short answer is: no.

Hubert Kario

2012-02-20 13:21 UTC #11 lore raw

In reply to Olivier Bonvalet

(sorry for the duplicate, previous one has broken signature)

Quoted 6 lines · On Monday 20 of February 2012 13:51:29 Olivier Bonvalet wrote:

Hi,
=20
I'm trying to convert two ext4 FS to btrfs, but I'm surprised by the time needed to do that conversion.
=20
The first FS is on a 500GiB block device, and btrfs-convert is runnin=

Quoted 5 lines

since more than 48h :
root 1978 25.6 47.7 748308 732556 ? D Feb18 944:44
btrfs-convert /dev/vg-backup/backup
=20
The second is on a 340GiB block device, and the processing time is si=

milar :

Quoted 8 lines

root 30192 5.2 3.0 73180 61520 pts/0 R Feb18 121:04
btrfs-convert /dev/vg-balor/backup
=20
=20
Is it "normal", or is there a problem during the processing ? =20
Thanks,
Olivier

How much free space did those FS have? How many files were on them (was=
they=20
dominated by small or large files)?