XFS & LVM: unexpected cp when issuing mv
I have an unexpected behaviour and I hope someone can explain me the reasons:
This is an openSUSE 11.2 virtual machine within XENserver. XENserver can only create 2TB disks, but I needed more. So I create 2x 2TB disks for that VM. These disks have no partitions, but are straight LVM:
# pvscan PV /dev/xvdb VG sharestore lvm2 [1,95 TB / 0 free] PV /dev/xvdc VG sharestore lvm2 [1,95 TB / 0 free] Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0 ]
I created one VG, and then one LV:
# vgscan Reading all physical volumes. This may take a while... Found volume group "sharestore" using metadata type lvm2 # lvscan ACTIVE '/dev/sharestore/public' [3,91 TB] inherit
On that LV, I created an XFS filesystem, mounted from /etc/fstab:/dev/sharestore/public /disks/sharestore xfs noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota
Now when I move from one dir to another, example mv /disks/sharestore/upload/* /disks/sharestore/download/
within some dirs it's a simple mv where only metadata is moved, but with some dirs it's a physical cp+rm of the files. You can easily see that by the speed of the mv, plus with iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
xvdb 0,00 0,00 0,00 647,31 0,00 28424,75
87,82 18,46 29,71 0,24 15,65
xvdc 0,00 0,40 631,14 2,40 26928,54 76,65
85,25 5,56 8,69 1,56 98,84
Until now I believed that a mv within one filesystem is always just a metadata mv. But it seems I found a case now where even within the same filesystem a physical cp+rm is done. Can someone explain me
1) why this happens
2) how I can prevent this?
We have files >5G there, often 20G or more, so a mv should just be a metadata mv, everything else is inacceptable.
Could it be the way I created the VG + LV, that there's a cp instead mv? How could I create all that to get a normal behaviour?
Maybe like this?:
1) create VG only on one disk
2) create LV on that disk
3) create XFS
4) extend VG to 2nd disk
5) extend LV to 2nd disk
6) xfs_growfs to 2nd disk
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Please paste the output of the following 2 commands:
df /disks/sharestore/upload/
df /disks/sharestore/download/
Quoted 44 lines · Michael Monnerie
I have an unexpected behaviour and I hope someone can explain me the reasons:
This is an openSUSE 11.2 virtual machine within XENserver. XENserver can only create 2TB disks, but I needed more. So I create 2x 2TB disks for that VM. These disks have no partitions, but are straight LVM:
# pvscan PV /dev/xvdb VG sharestore lvm2 [1,95 TB / 0 free] PV /dev/xvdc VG sharestore lvm2 [1,95 TB / 0 free] Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0 ]I created one VG, and then one LV:
# vgscan Reading all physical volumes. This may take a while... Found volume group "sharestore" using metadata type lvm2 # lvscan ACTIVE '/dev/sharestore/public' [3,91 TB] inheritOn that LV, I created an XFS filesystem, mounted from
/etc/fstab:/dev/sharestore/public/disks/sharestore xfs noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquotaNow when I move from one dir to another, example mv /disks/sharestore/upload/* /disks/sharestore/download/
within some dirs it's a simple mv where only metadata is moved, but with some dirs it's a physical cp+rm of the files. You can easily see that by the speed of the mv, plus with iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %utilxvdb 0,00 0,00 0,00 647,31 0,00 28424,75 87,82 18,46 29,71 0,24 15,65 xvdc 0,00 0,40 631,14 2,40 26928,54 76,65 85,25 5,56 8,69 1,56 98,84Until now I believed that a mv within one filesystem is always just a metadata mv. But it seems I found a case now where even within the same filesystem a physical cp+rm is done. Can someone explain me
1) why this happens
2) how I can prevent this?We have files >5G there, often 20G or more, so a mv should just be a metadata mv, everything else is inacceptable. Could it be the way I created the VG + LV, that there's a cp instead mv?
How could I create all that to get a normal behaviour?Maybe like this?:
1) create VG only on one disk
2) create LV on that disk
3) create XFS
4) extend VG to 2nd disk
5) extend LV to 2nd disk
6)xfs_growfsto 2nd disk
Quoted 9 lines
mfg zmi
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 1 line · On Sonntag, 29. November 2009 Asdo wrote:
Please paste the output of the following 2 commands:
# df /disks/sharestore/upload/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/sharestore-public
4192254976 1956672968 2235582008 47%
/disks/sharestore# df /disks/sharestore/download/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/sharestore-public
4192254976 1956675156 2235579820 47%
/disks/sharestoreYes, it really *is* the same filesystem.
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 18 lines · Michael Monnerie
I have an unexpected behaviour and I hope someone can explain me the reasons:
This is an openSUSE 11.2 virtual machine within XENserver. XENserver can only create 2TB disks, but I needed more. So I create 2x 2TB disks for that VM. These disks have no partitions, but are straight LVM:
# pvscan PV /dev/xvdb VG sharestore lvm2 [1,95 TB / 0 free] PV /dev/xvdc VG sharestore lvm2 [1,95 TB / 0 free] Total: 2 [3,91 TB] / in use: 2 [3,91 TB] / in no VG: 0 [0 ]I created one VG, and then one LV:
# vgscan Reading all physical volumes. This may take a while... Found volume group "sharestore" using metadata type lvm2 # lvscan ACTIVE '/dev/sharestore/public' [3,91 TB] inherit
Quoted 3 lines
On that LV, I created an XFS filesystem, mounted from
/etc/fstab:/dev/sharestore/public/disks/sharestore xfs noatime,nodiratime,logbufs=8,logbsize=256k,attr2,nobarrier,largeio,swalloc,inode64,prjquota
You are using project quotas.
Quoted 13 lines
Now when I move from one dir to another, example mv /disks/sharestore/upload/* /disks/sharestore/download/
within some dirs it's a simple mv where only metadata is moved, but with some dirs it's a physical cp+rm of the files. You can easily see that by the speed of the mv, plus with iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-
sz avgqu-sz await svctm %util
xvdb 0,00 0,00 0,00 647,31 0,00 28424,75
87,82 18,46 29,71 0,24 15,65
xvdc 0,00 0,40 631,14 2,40 26928,54 76,65
85,25 5,56 8,69 1,56 98,84
Quoted 5 lines
Until now I believed that a mv within one filesystem is always just a metadata mv. But it seems I found a case now where even within the same filesystem a physical cp+rm is done. Can someone explain me
1) why this happens
It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.
Quoted 1 line
2) how I can prevent this?
You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 8 lines · On Montag, 30. November 2009 Dave Chinner wrote:
Quoted 1 line
1) why this happens
It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.
Quoted 5 lines
Quoted 1 line
2) how I can prevent this?
You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.
Shit. So I have to turn project quotas off. I can't accept the extra load for a simple move, as there are tons of data. Maybe the projet code could be redesigned to allow a simple move? Is it that complicated?
If I change from project to user quota - I guess the same would still happen? If so, I have to drop quota support, and build a script to check quotas manually. It's a pity. :-(
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 14 lines · Michael Monnerie
Quoted 7 lines · On Montag, 30. November 2009 Dave Chinner wrote:
Quoted 1 line
1) why this happens
It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.
Quoted 5 lines
Quoted 1 line
2) how I can prevent this?
You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.
Quoted 7 lines
Shit. So I have to turn project quotas off. I can't accept the extra load for a simple move, as there are tons of data. Maybe the projet code could be redesigned to allow a simple move? Is it that complicated?
If I change from project to user quota - I guess the same would still happen? If so, I have to drop quota support, and build a script to check quotas manually. It's a pity. :-(
The quota inheritance property Dave described only applies to project quotas, not group or user quotas.
-Alex
Quoted 1 line
mfg zmi
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 3 lines · On Montag, 30. November 2009 Alex Elder wrote:
The quota inheritance property Dave described only applies to project quotas, not group or user quotas.
Thanks. I thought about it but can't use them anyway, as I would have to change the owner of files on upload. But I want to keep that information. So turning off quota is our only chance by now :-(
mfg zmi
Signature · 5 lines
// Michael Monnerie, Ing.BSc ----- http://it-management.at // Tel: 0660 / 415 65 31 .network.your.ideas. // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import" // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4 // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 11 lines · Michael Monnerie
Quoted 10 lines · On Montag, 30. November 2009 Dave Chinner wrote:
Quoted 1 line
1) why this happens
It happens if you move from one project directory heirarchy to another - rename is not allowed across project quota boundaries as the moved data has to be correctly attributed to the new project. Hence it causes a mv to do a copy/unlink by returning a EXDEV error to the rename.
Quoted 1 line
2) how I can prevent this?
Quoted 7 lines
Quoted 3 lines
You can't if you are moving from one project to another. If you move within the project heirarchy, then it will be a rename as per normal.
Shit. So I have to turn project quotas off. I can't accept the extra load for a simple move, as there are tons of data. Maybe the projet code could be redesigned to allow a simple move? Is it that complicated?
IIRC, it was a design decision to make project quotas behave this way. Returning EXDEV made the code to handle this very simple - userspace already handles this error correctly and all the kernel needs is 3 lines of code to detect differing project IDs on the inodes and return EXDEV.
To do it without copying is definitely possible - it requires extending the rename transaction to directly modify the project quota id of the target inodes and the relevant dquots as well. It becomes quite complicated because it also has to handle the cases of source or destination not having quotas at all, the destination not having enough space (i.e. EDQUOT), etc.
All the nasty corner cases are handled by userspace by returning EXDEV to force it to copy rather than renaming, so the decision was made to do the simple (if slow) thing for relatively uncommon operation of moving data between projects.
Quoted 2 lines
If I change from project to user quota - I guess the same would still happen?
No - the EXDEV error (and therefore the copy) is a feature of project quotas.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs