Skip to contentSkip to conversation

XFS & LVM: unexpected cp when issuing mv

DISCUSSION Michael Monnerie 4 participants 8 messages Nov 29, 2009 – Dec 1, 2009

lore
Share & tools
lore
#0 lore raw

I have an unexpected behaviour and I hope someone can explain me the reasons:

This is an openSUSE 11.2 virtual machine within XENserver. XENserver can only create 2TB disks, but I needed more. So I create 2x 2TB disks for that VM. These disks have no partitions, but are straight LVM:

# pvscan
  PV /dev/xvdb   VG sharestore   lvm2 [1,95 TB / 0    free]
  PV /dev/xvdc   VG sharestore   lvm2 [1,95 TB / 0    free]
  Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0   ]

I created one VG, and then one LV:

# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "sharestore" using metadata type lvm2
# lvscan
  ACTIVE            '/dev/sharestore/public' [3,91 TB] inherit

On that LV, I created an XFS filesystem, mounted from /etc/fstab:
/dev/sharestore/public /disks/sharestore xfs noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota

Now when I move from one dir to another, example mv /disks/sharestore/upload/* /disks/sharestore/download/

within some dirs it's a simple mv where only metadata is moved, but with some dirs it's a physical cp+rm of the files. You can easily see that by the speed of the mv, plus with iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
xvdb 0,00 0,00 0,00 647,31 0,00 28424,75
87,82 18,46 29,71 0,24 15,65
xvdc 0,00 0,40 631,14 2,40 26928,54 76,65
85,25 5,56 8,69 1,56 98,84

Until now I believed that a mv within one filesystem is always just a metadata mv. But it seems I found a case now where even within the same filesystem a physical cp+rm is done. Can someone explain me

1) why this happens
2) how I can prevent this?

We have files >5G there, often 20G or more, so a mv should just be a metadata mv, everything else is inacceptable.
Could it be the way I created the VG + LV, that there's a cp instead mv? How could I create all that to get a normal behaviour?

Maybe like this?:
1) create VG only on one disk
2) create LV on that disk
3) create XFS
4) extend VG to 2nd disk
5) extend LV to 2nd disk
6) xfs_growfs to 2nd disk

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

Please paste the output of the following 2 commands:

df /disks/sharestore/upload/

df /disks/sharestore/download/

Quoted 44 lines · Michael Monnerie

I have an unexpected behaviour and I hope someone can explain me the reasons:

This is an openSUSE 11.2 virtual machine within XENserver. XENserver can only create 2TB disks, but I needed more. So I create 2x 2TB disks for that VM. These disks have no partitions, but are straight LVM:

# pvscan
  PV /dev/xvdb   VG sharestore   lvm2 [1,95 TB / 0    free]
  PV /dev/xvdc   VG sharestore   lvm2 [1,95 TB / 0    free]
  Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0   ]

I created one VG, and then one LV:

# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "sharestore" using metadata type lvm2
# lvscan
  ACTIVE            '/dev/sharestore/public' [3,91 TB] inherit

On that LV, I created an XFS filesystem, mounted from /etc/fstab:
/dev/sharestore/public /disks/sharestore xfs noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota

Now when I move from one dir to another, example mv /disks/sharestore/upload/* /disks/sharestore/download/

within some dirs it's a simple mv where only metadata is moved, but with some dirs it's a physical cp+rm of the files. You can easily see that by the speed of the mv, plus with iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util

xvdb              0,00     0,00    0,00  647,31     0,00 28424,75    87,82    18,46   29,71   0,24  15,65
xvdc              0,00     0,40  631,14    2,40 26928,54    76,65    85,25     5,56    8,69   1,56  98,84

Until now I believed that a mv within one filesystem is always just a metadata mv. But it seems I found a case now where even within the same filesystem a physical cp+rm is done. Can someone explain me

1) why this happens
2) how I can prevent this?

We have files >5G there, often 20G or more, so a mv should just be a metadata mv, everything else is inacceptable. Could it be the way I created the VG + LV, that there's a cp instead mv?
How could I create all that to get a normal behaviour?

Maybe like this?:
1) create VG only on one disk
2) create LV on that disk
3) create XFS
4) extend VG to 2nd disk
5) extend LV to 2nd disk
6) xfs_growfs to 2nd disk

Quoted 9 lines

mfg zmi

------------------------------------------------------------------------

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 1 line · On Sonntag, 29. November 2009 Asdo wrote:

Please paste the output of the following 2 commands:

# df  /disks/sharestore/upload/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/sharestore-public
                     4192254976 1956672968 2235582008  47% 
/disks/sharestore
# df  /disks/sharestore/download/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/sharestore-public
                     4192254976 1956675156 2235579820  47% 
/disks/sharestore

Yes, it really *is* the same filesystem.

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
In reply to Michael Monnerie
Quoted 18 lines · Michael Monnerie

I have an unexpected behaviour and I hope someone can explain me the reasons:

This is an openSUSE 11.2 virtual machine within XENserver. XENserver can only create 2TB disks, but I needed more. So I create 2x 2TB disks for that VM. These disks have no partitions, but are straight LVM:

# pvscan
  PV /dev/xvdb   VG sharestore   lvm2 [1,95 TB / 0    free]
  PV /dev/xvdc   VG sharestore   lvm2 [1,95 TB / 0    free]
  Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0   ]

I created one VG, and then one LV:

# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "sharestore" using metadata type lvm2
# lvscan
  ACTIVE            '/dev/sharestore/public' [3,91 TB] inherit
Quoted 3 lines

On that LV, I created an XFS filesystem, mounted from /etc/fstab:
/dev/sharestore/public /disks/sharestore xfs noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota

You are using project quotas.

Quoted 13 lines

Now when I move from one dir to another, example mv /disks/sharestore/upload/* /disks/sharestore/download/

within some dirs it's a simple mv where only metadata is moved, but with some dirs it's a physical cp+rm of the files. You can easily see that by the speed of the mv, plus with iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
xvdb 0,00 0,00 0,00 647,31 0,00 28424,75
87,82 18,46 29,71 0,24 15,65
xvdc 0,00 0,40 631,14 2,40 26928,54 76,65
85,25 5,56 8,69 1,56 98,84

Quoted 5 lines

Until now I believed that a mv within one filesystem is always just a metadata mv. But it seems I found a case now where even within the same filesystem a physical cp+rm is done. Can someone explain me

1) why this happens

It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.

Quoted 1 line

2) how I can prevent this?

You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#4 lore raw
Quoted 8 lines · On Montag, 30. November 2009 Dave Chinner wrote:
Quoted 1 line

1) why this happens

It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.

Quoted 5 lines
Quoted 1 line

2) how I can prevent this?

You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.

Shit. So I have to turn project quotas off. I can't accept the extra load for a simple move, as there are tons of data. Maybe the projet code could be redesigned to allow a simple move? Is it that complicated?

If I change from project to user quota - I guess the same would still happen? If so, I have to drop quota support, and build a script to check quotas manually. It's a pity. :-(

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
Quoted 14 lines · Michael Monnerie
Quoted 7 lines · On Montag, 30. November 2009 Dave Chinner wrote:
Quoted 1 line

1) why this happens

It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.

Quoted 5 lines
Quoted 1 line

2) how I can prevent this?

You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.

Quoted 7 lines

Shit. So I have to turn project quotas off. I can't accept the extra load for a simple move, as there are tons of data. Maybe the projet code could be redesigned to allow a simple move? Is it that complicated?

If I change from project to user quota - I guess the same would still happen? If so, I have to drop quota support, and build a script to check quotas manually. It's a pity. :-(

The quota inheritance property Dave described only applies to project quotas, not group or user quotas.

-Alex

Quoted 1 line

mfg zmi

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
Quoted 3 lines · On Montag, 30. November 2009 Alex Elder wrote:

The quota inheritance property Dave described only applies to project quotas, not group or user quotas.

Thanks. I thought about it but can't use them anyway, as I would have to change the owner of files on upload. But I want to keep that information. So turning off quota is our only chance by now :-(

mfg zmi

Signature · 5 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
In reply to Michael Monnerie
Quoted 11 lines · Michael Monnerie
Quoted 10 lines · On Montag, 30. November 2009 Dave Chinner wrote:
Quoted 1 line

1) why this happens

It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.

Quoted 1 line

2) how I can prevent this?

Quoted 7 lines
Quoted 3 lines

You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.

Shit. So I have to turn project quotas off. I can't accept the extra load for a simple move, as there are tons of data. Maybe the projet code could be redesigned to allow a simple move? Is it that complicated?

IIRC, it was a design decision to make project quotas behave this way. Returning EXDEV made the code to handle this very simple - userspace already handles this error correctly and all the kernel needs is 3 lines of code to detect differing project IDs on the inodes and return EXDEV.

To do it without copying is definitely possible - it requires extending the rename transaction to directly modify the project quota id of the target inodes and the relevant dquots as well. It becomes quite complicated because it also has to handle the cases of source or destination not having quotas at all, the destination not having enough space (i.e. EDQUOT), etc.

All the nasty corner cases are handled by userspace by returning EXDEV to force it to copy rather than renaming, so the decision was made to do the simple (if slow) thing for relatively uncommon operation of moving data between projects.

Quoted 2 lines

If I change from project to user quota - I guess the same would still happen?

No - the EXDEV error (and therefore the copy) is a feature of project quotas.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs