Skip to contentSkip to conversation

ARC-1260: No space left on device, when there is (or should be) free space left

DISCUSSION Magnus Naeslund 4 participants 7 messages Jun 18, 2007

lore
Share & tools
lore
#0 lore raw

It seems I have a problem accessing the whole device, it filled up too quickly. I thought I was avoiding the 2tb limit when not using any partition tables, shame on me :)

# df -B 1 /dev/sdb
Filesystem           1B-blocks      Used Available Use% Mounted on
/dev/sdb             5999417057280 1908287229952 4091129827328  32% /storage/disk1

(Ok here I have 68% free space on our 5.5TB device)

# echo zzz > /storage/disk1/dummy
/storage/disk1/dummy: No space left on device

(But here it seems to think the device is full)

It's an amd64 based system running 2.6.21.5.
Please tell me if you need more information!

Regards,
Magnus

#1 lore raw
Quoted 8 lines · Magnus Naeslund

It seems I have a problem accessing the whole device, it filled up too quickly. I thought I was avoiding the 2tb limit when not using any partition tables, shame on me :)

# df -B 1 /dev/sdb
Filesystem           1B-blocks      Used Available Use% Mounted on
/dev/sdb             5999417057280 1908287229952 4091129827328  32% /storage/disk1

(Ok here I have 68% free space on our 5.5TB device)

$ df -i

rday

Signature · 7 lines
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry
Waterloo, Ontario, CANADA

http://fsdev.net/wiki/index.php?title=Main_Page
========================================================================
#2 lore raw
Quoted 5 lines · Robert P. J. Day
Quoted 2 lines

(Ok here I have 68% free space on our 5.5TB device)

$ df -i

Duh, this was it!
What is the recommended method on how to increase the inode count on the filesystem (there is no need for online resize in this case)? I have the resize_inode option set when looking at the tune2fs -l output.

Magnus

#3 lore raw
Quoted 3 lines · Magnus Naeslund

Duh, this was it!
What is the recommended method on how to increase the inode count on the filesystem (there is no need for online resize in this case)? I have the resize_inode option set when looking at the tune2fs -l output.

Unfortunately, at the moment there isn't a way to increase the inode count without resizing the filesystem (which will add more blocks and inodes as you add more blockgroups). That's because the number of inodes per block group is fixed at mke2fs time.

In theory it should be possible to whip up a program which would implement this, but it would be a non-trivial exercise. So at the moment, the only solution is to recreate it with a larger number of inodes, or resize the filesystem.

How are you using the filesystem? This wouldn't happen to be one of the backup schemes that use hard links and huge numbers of directories, would it? And how did you create the filesystem originally? Normally mke2fs is quite generous with the number of inodes it creates to avoid this problem. Did you use -T largefile or -T largefile4 by any chance? Or did you manually specify a non-standard inode_ratio size?

- Ted

#4 lore raw
Quoted 9 lines · Theodore Tso

How are you using the filesystem? This wouldn't happen to be one of the backup schemes that use hard links and huge numbers of directories, would it? And how did you create the filesystem originally? Normally mke2fs is quite generous with the number of inodes it creates to avoid this problem. Did you use -T largefile or -T largefile4 by any chance? Or did you manually specify a non-standard inode_ratio size?

It's a filesystem containing images varying from 300kb to 2mb of size. It currently contains 1426394 files in 4125 directories.

Magnus

I believe this has the appropriate information:
tune2fs 1.40-WIP (14-Nov-2006)

Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed directory hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1430528
Block count:              1464843264
Reserved block count:     0
Free blocks:              998810993
Free inodes:              0
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      674
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32
Inode blocks per group:   1
Filesystem created:       Sun Apr  1 12:46:39 2007
Last mount time:          Mon Jun 18 14:23:26 2007
Last write time:          Mon Jun 18 14:23:26 2007
Mount count:              8
Maximum mount count:      20
Last checked:             Sun Apr  1 12:46:39 2007
Check interval:           15552000 (6 months)
Next check after:         Fri Sep 28 12:46:39 2007
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      f252b473-6db0-499b-8de1-c788e84563dc
Journal backup:           inode blocks
#5 lore raw
Quoted 11 lines · Magnus Naeslund

It's a filesystem containing images varying from 300kb to 2mb of size. It currently contains 1426394 files in 4125 directories.

Magnus

Inode count:              1430528
Block count:              1464843264
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32

Well, *thar's* your problem! :-)

I'm guessing you created the filesystem with mke2fs -T largefile4. This basically told mke2fs that you anticipated the filesystem to have an average inode size of 4 megabytes. Given that your filesystem has files ranging from 300k and 2mb, that's why you're running into problems.

- Ted

#6 lore raw
In reply to Magnus Naeslund
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"MN" == Magnus Naeslund <[email protected]> writes:

MN> Inode count: 1430528

That's stunningly small; I created an 8TB partition here and made an ext3 filesystem in it with the default parameters; I got nearly 1000 times as many inodes as you have:

Inode count:              1072414720
Block count:              2144799744

- J<