Printk for ENOSPC due to lack of inodes

Hi,

While diagnosing a MySQL crash (on a Centos 5.7 box), I noticed that it had failed with ENOSPC earlier; it was rebooted after that; now after reboot, even though space was there, ENOSPC was showing up, I also did df -i and it showed inodes available. At this point, mounting with inode64 option was tried, which fixed it.
http://oss.sgi.com/archives/xfs/2011-03/msg00299.html helped me here.

So, I have attached a patch here.

=====================================================================

When a ENOSPC is encountered and it is due to lack of inodes (particularly without inode64), it is not possible to detect this (df -i doesn't help here), so adding a printk which can aid in detecting this.

Signed-off-by Raghavendra D Prabhu <[email protected]>

---
 fs/xfs/xfs_qm.c       |    3 +++
 fs/xfs/xfs_vnodeops.c |   10 ++++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

fs/xfs/xfs_qm.c +3 -0

index c872fea..fbefa87 100644
@@ -987,6 +987,9 @@ xfs_qm_qino_alloc(
 
     error = xfs_dir_ialloc(&tp, NULL, S_IFREG, 1, 0, 0, 1, ip, &committed);
     if (error) {
+        if (error == ENOSPC)
+            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
         xfs_trans_cancel(tp, XFS_TRANS_RELEASE_LOG_RES |
                  XFS_TRANS_ABORT);
         return error;

fs/xfs/xfs_vnodeops.c +8 -2

index ebdb888..7542c36 100644
@@ -946,8 +946,11 @@ xfs_create(
     error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev,
                    prid, resblks > 0, &ip, &committed);
     if (error) {
-        if (error == ENOSPC)
+        if (error == ENOSPC) {
+            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
             goto out_trans_cancel;
+        }
         goto out_trans_abort;
     }
 

@@ -1610,8 +1613,11 @@ xfs_symlink(
     error = xfs_dir_ialloc(&tp, dp, S_IFLNK | (mode & ~S_IFMT), 1, 0,
                    prid, resblks > 0, &ip, NULL);
     if (error) {
-        if (error == ENOSPC)
+        if (error == ENOSPC) {
+            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
             goto error_return;
+        }
         goto error1;
     }
 

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 9 lines · Raghavendra D Prabhu

Hi,

While diagnosing a MySQL crash (on a Centos 5.7 box), I noticed that it had failed with ENOSPC earlier; it was rebooted after that; now after reboot, even though space was there, ENOSPC was showing up, I also did df -i and it showed inodes available. At this point, mounting with inode64 option was tried, which fixed it.
http://oss.sgi.com/archives/xfs/2011-03/msg00299.html helped me here.

Yeah, that's kind of a bummer. And given the semi-ugly situation we're in with inode32, maybe a syslog message would be good.

But a few things; I don't think we want to warn on every inode-allocation ENOSPC. Ideally I'd probably do a WARN_ON_ONCE or a ratelimited printk. xfs_warn_once_per_fs()? :)

I'd probably also want to only do it in the case where the ENOSPC was due to either inode32, or due to fragmented freespace.

Without looking very hard yet; could this be done in xfs_dialloc() so that the exact reason for the ENOSPC can be issued? (maxicount, inode32, or no free contiguous space...)

-Eric

Quoted 62 lines

So, I have attached a patch here.

=====================================================================

When a ENOSPC is encountered and it is due to lack of inodes (particularly without inode64), it is not possible to detect this (df -i doesn't help here), so adding a printk which can aid in detecting this.

Signed-off-by Raghavendra D Prabhu <[email protected]>

---
 fs/xfs/xfs_qm.c       |    3 +++
 fs/xfs/xfs_vnodeops.c |   10 ++++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

fs/xfs/xfs_qm.c +3 -0

index c872fea..fbefa87 100644
@@ -987,6 +987,9 @@ xfs_qm_qino_alloc(
 
     error = xfs_dir_ialloc(&tp, NULL, S_IFREG, 1, 0, 0, 1, ip, &committed);
     if (error) {
+        if (error == ENOSPC)
+            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
         xfs_trans_cancel(tp, XFS_TRANS_RELEASE_LOG_RES |
                  XFS_TRANS_ABORT);
         return error;

fs/xfs/xfs_vnodeops.c +8 -2

index ebdb888..7542c36 100644
@@ -946,8 +946,11 @@ xfs_create(
     error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev,
                    prid, resblks > 0, &ip, &committed);
     if (error) {
-        if (error == ENOSPC)
+        if (error == ENOSPC) {
+            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
             goto out_trans_cancel;
+        }
         goto out_trans_abort;
     }
 

@@ -1610,8 +1613,11 @@ xfs_symlink(
     error = xfs_dir_ialloc(&tp, dp, S_IFLNK | (mode & ~S_IFMT), 1, 0,
                    prid, resblks > 0, &ip, NULL);
     if (error) {
-        if (error == ENOSPC)
+        if (error == ENOSPC) {
+            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
             goto error_return;
+        }
         goto error1;
     }
 

Quoted 4 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Hi,

Thanks for the review. I have now moved it inside xfs_dialloc. Along with adding the message, I noticed that the loop

    while (!agi->agi_freecount) {
    }

is redundant when noroom=1 and okalloc=0.

Also, xfs_ialloc_ag_alloc function in the loop calls
============

    if (mp->m_maxicount &&
        mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {

===============

condition again.

So I have moved xfs_tran_brelse etc. into the condition along with message.

Is this logic valid? If it is, then I will look into rate-limiting the message etc.

Quoted 16 lines · Raghavendra D Prabhu

Hi,

Thanks for the review. I have now moved it inside xfs_dialloc. Along with adding the message, I noticed that the loop

    while (!agi->agi_freecount) {
    }

is redundant when noroom=1 and okalloc=0.

Also, xfs_ialloc_ag_alloc function in the loop calls
============

    if (mp->m_maxicount &&
        mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {

===============

Quoted 5 lines

condition again.

So I have moved xfs_tran_brelse etc. into the condition along with message.

Is this logic valid? If it is, then I will look into rate-limiting the message etc.

It'd be easiest to understand this new change as a patch rather than as a description.

If you are changing logic or flow in addition to adding the messages, it should almost certainly be sent as more than one patch.

Thanks,

-eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 14 lines · Raghavendra D Prabhu

Hi,

Thanks for the review. I have now moved it inside xfs_dialloc. Along with adding the message, I noticed that the loop

    while (!agi->agi_freecount) {
    }

is redundant when noroom=1 and okalloc=0.

Also, xfs_ialloc_ag_alloc function in the loop calls
============

    if (mp->m_maxicount &&
        mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {

Quoted 7 lines

===============

condition again.

So I have moved xfs_tran_brelse etc. into the condition along with message.

Is this logic valid? If it is, then I will look into rate-limiting the message etc.

It'd be easiest to understand this new change as a patch rather than as a description.

If you are changing logic or flow in addition to adding the messages, it should almost certainly be sent as more than one patch.

Thanks,

-eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs