Skip to contentSkip to conversation

no space left on device with 662G free

DISCUSSION Dave Howorth 4 participants 6 messages Sep 15, 2010

lore
Share & tools
lore
#0 lore raw

Hello,

I recently started using XFS when I set up a new server and I'm having a problem when copying data on to it from an old server. cp reports errors like:

cp: cannot create symbolic link `/nfs/scop4/data/trembl/xml/releases/2006-01-10/Q6RXU6.xml': No space left on device

and later

cp: cannot create directory `/nfs/scop4/data/trembl/xml/releases/2006-01-24': No space left on device

The filesystem is in a 2 TB LVM on an md RAID and according to df there's 662 GB space free:

/dev/mapper/vg--storage-lv--data

                      2.0T  1.4T  662G  68% /nfs/scop4/data

There are a quite a few directories in the filesystem. Some of the directories contain many millions of files and some directories consist entirely of symlinks, if any of that's relevant.

The filesystem is on a newish machine running openSUSE 11.2 (Linux scop4 2.6.31.12-0.2-desktop #1 SMP PREEMPT 2010-03-16 21:25:39 +0100 x86_64 x86_64 x86_64 GNU/Linux) and the data is being mirrored from an older machine that uses a reiser filesystem and which stores the data without problem.

Some folks on the suse mailing list said it was probably an inode problem and suggested I run df -i:

# df -i /nfs/scop4/data
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg--storage-lv--data
                     429496704 -18446744073280007488
18446744073709504192    - /nfs/scop4/data

I don't understand that output, especially the negative number! I then read about the inode64 mount option in the XFS FAQ and I believe I've now enabled that and remounted the filesystem. mtab shows

/dev/mapper/vg--storage-lv--data /nfs/scop4/data xfs rw,noatime,inode64 0 0

But I'm still seeing the same errors as before. Is there something else I need to do to enable inode64, or am I looking in the wrong direction?

Thanks, Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 9 lines · Dave Howorth

Hello,

I recently started using XFS when I set up a new server and I'm having a problem when copying data on to it from an old server. cp reports errors like:

cp: cannot create symbolic link `/nfs/scop4/data/trembl/xml/releases/2006-01-10/Q6RXU6.xml': No space left on device

Please send the output of this command:

xfs_info /dev/mapper/vg--storage-lv--data

I'm pretty sure you've run out of inodes.

-Alex

Quoted 39 lines

and later

cp: cannot create directory `/nfs/scop4/data/trembl/xml/releases/2006-01-24': No space left on device

The filesystem is in a 2 TB LVM on an md RAID and according to df there's 662 GB space free:

/dev/mapper/vg--storage-lv--data

                      2.0T  1.4T  662G  68% /nfs/scop4/data

There are a quite a few directories in the filesystem. Some of the directories contain many millions of files and some directories consist entirely of symlinks, if any of that's relevant.

The filesystem is on a newish machine running openSUSE 11.2 (Linux scop4 2.6.31.12-0.2-desktop #1 SMP PREEMPT 2010-03-16 21:25:39 +0100 x86_64 x86_64 x86_64 GNU/Linux) and the data is being mirrored from an older machine that uses a reiser filesystem and which stores the data without problem.

Some folks on the suse mailing list said it was probably an inode problem and suggested I run df -i:

# df -i /nfs/scop4/data
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg--storage-lv--data
                     429496704 -18446744073280007488
18446744073709504192    - /nfs/scop4/data

I don't understand that output, especially the negative number! I then read about the inode64 mount option in the XFS FAQ and I believe I've now enabled that and remounted the filesystem. mtab shows

/dev/mapper/vg--storage-lv--data /nfs/scop4/data xfs rw,noatime,inode64 0 0

But I'm still seeing the same errors as before. Is there something else I need to do to enable inode64, or am I looking in the wrong direction?

Quoted 6 lines

Thanks, Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 5 lines · Alex Elder

Please send the output of this command:

xfs_info /dev/mapper/vg--storage-lv--data

I'm pretty sure you've run out of inodes.

# xfs_info /dev/mapper/vg--storage-lv--data
meta-data=/dev/mapper/vg--storage-lv--data isize=256    agcount=4,
agsize=134217728 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=536870912, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

Cheers, Dave

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw

Hi Dave

Quoted 13 lines
Quoted 1 line

I'm pretty sure you've run out of inodes.

# xfs_info /dev/mapper/vg--storage-lv--data
meta-data=/dev/mapper/vg--storage-lv--data isize=256    agcount=4,
agsize=134217728 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=536870912, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0

Try:

# xfs_growfs -m 10 /dev/mapper/vg--storage-lv--data

It will increase your inode max percentage from 5 to 10.

cheers,
Michael

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw
In reply to Dave Howorth

Hi,

Quoted 38 lines

I recently started using XFS when I set up a new server and I'm having a problem when copying data on to it from an old server. cp reports errors like:

cp: cannot create symbolic link `/nfs/scop4/data/trembl/xml/releases/2006-01-10/Q6RXU6.xml': No space left on device

and later

cp: cannot create directory `/nfs/scop4/data/trembl/xml/releases/2006-01-24': No space left on device

The filesystem is in a 2 TB LVM on an md RAID and according to df there's 662 GB space free:

/dev/mapper/vg--storage-lv--data

                      2.0T  1.4T  662G  68% /nfs/scop4/data

There are a quite a few directories in the filesystem. Some of the directories contain many millions of files and some directories consist entirely of symlinks, if any of that's relevant.

The filesystem is on a newish machine running openSUSE 11.2 (Linux scop4 2.6.31.12-0.2-desktop #1 SMP PREEMPT 2010-03-16 21:25:39 +0100 x86_64 x86_64 x86_64 GNU/Linux) and the data is being mirrored from an older machine that uses a reiser filesystem and which stores the data without problem.

Some folks on the suse mailing list said it was probably an inode problem and suggested I run df -i:

# df -i /nfs/scop4/data
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg--storage-lv--data
                     429496704 -18446744073280007488
18446744073709504192    - /nfs/scop4/data
Quoted 8 lines

I don't understand that output, especially the negative number! I then read about the inode64 mount option in the XFS FAQ and I believe I've now enabled that and remounted the filesystem. mtab shows

/dev/mapper/vg--storage-lv--data /nfs/scop4/data xfs rw,noatime,inode64 0 0

But I'm still seeing the same errors as before. Is there something else I need to do to enable inode64, or am I looking in the wrong direction?

Seems like you have a cross between

http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F

and

http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_receive_No_space_left_on_device_after_xfs_growfs.3F

Perhaps not using inode64 from the beginning looks similar to a xfs_growfs?

Regards,
@ndy

Signature · 8 lines
[email protected]
http://www.ashurst.eu.org/
0x7EBA75FF

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs