Skip to contentSkip to conversation

2nd Attempt - FSCK Errors

DISCUSSION Stephen Elliott 2 participants 3 messages May 8, 2013 VFS

lore
Share & tools
lore
#0 lore raw

Hi,

Well... I have some new and interesting information.

After deleting the file, and re-creating from backup, the file system got mounted read only and the following errors were logged:

May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904376:freeing already freed block (bit 1144) May  8 14:58:15 despair kernel: Aborting journal on device dm-0-8.
May  8 14:58:15 despair kernel: EXT4-fs (dm-0): Remounting filesystem read-only May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904377:freeing already freed block (bit 1145) May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904378:freeing already freed block (bit 1146) May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904379:freeing already freed block (bit 1147) May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904380:freeing already freed block (bit 1148) May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904381:freeing already freed block (bit 1149) May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904382:freeing already freed block (bit 1150) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904383:freeing already freed block (bit 1151) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904384:freeing already freed block (bit 1152) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904385:freeing already freed block (bit 1153) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904386:freeing already freed block (bit 1154) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949296:freeing already freed block (bit 13296) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949297:freeing already freed block (bit 13297) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949298:freeing already freed block (bit 13298) May  8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949299:freeing already freed block (bit 13299) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949300:freeing already freed block (bit 13300) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949301:freeing already freed block (bit 13301) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949302:freeing already freed block (bit 13302) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949303:freeing already freed block (bit 13303) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949304:freeing already freed block (bit 13304) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949305:freeing already freed block (bit 13305) May  8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949306:freeing already freed block (bit 13306)

These are the same blocks slated as multiply claimed :)

And then running an FSCK, we got the following:

***** File system check forced at Wed May 8 15:16:50 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012)
/dev/c/c: recovering journal
Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong for group #5124 (28170, counted=28159). Fix? yes

Free blocks count wrong for group #5125 (25861, counted=25850). Fix? yes

Free blocks count wrong (420683133, counted=420644972). Fix? yes

Free inodes count wrong (29595347, counted=29595271). Fix? yes

/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
/dev/c/c: 616825/30212096 files (13.6% non-contiguous), 62748564/483393536 blocks

Volume seems clean now after reboot and keeps coming back clean.

Any ideas based on this? The old Database was over 1.6 GB in size. The re-created one from backup is much smaller 33MB (as it was compacted before restoring). Once these DB files grow they don't get smaller until compacted. Although I'm sure file size should not have a bearing on this.

Many Thanks
Stephen Elliott

From: Stephen Elliott [mailto:[email protected]]
Sent: 26 April 2013 22:37
To: '[email protected]'
Subject: FSCK Errors

Hi all,

Just rebooted my box today after 200 days uptime and thought I'd request a volume scan and it found errors! I've never had a power outage etc so am keen to know what could have caused this file system corruption? Anyu ideas???

I'm running 4.2.21 on a ReadyNAS Pro6, but ultimately it is a Linux (Debian) 2.6.37.6. based system underneath.

***** File system check forced at Fri Apr 26 20:08:38 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) Pass 1: Checking inodes, blocks, and sizes Inode 4195619, i_blocks is 3135728, should be 3135904. Fix? yes

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 4195619: 167904376 167904377 167904378 167904379 167904380 167904381 167904382 167904383 167904384 167904385 167904386 167949296 167949297 167949298 167949299 167949300 167949301 167949302 167949303 167949304 167949305 167949306 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.)

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
/dev/c/c: 615898/30212096 files (13.6% non-contiguous), 62353456/483393536 blocks

I'm curious to know what this is all about, shared with 0 files!!!!:

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Every follow on boot with FSCK yields the following:

***** File system check forced at Fri Apr 26 20:21:42 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) Pass 1: Checking inodes, blocks, and sizes

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 4195619: 167904376 167904377 167904378 167904379 167904380 167904381 167904382 167904383 167904384 167904385 167904386 167949296 167949297 167949298 167949299 167949300 167949301 167949302 167949303 167949304 167949305 167949306 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.)

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013)
has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/c/c: 615898/30212096 files (13.6% non-contiguous), 62353456/483393536 blocks

Still same thing with multiply claimed blocks :)☹

Any ideas?

Many Thanks
Stephen Elliott

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#1 lore raw
Quoted 7 lines · Stephen Elliott

Well... I have some new and interesting information.

After deleting the file, and re-creating from backup, the file system got mounted read only and the following errors were logged:

May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904376:freeing already freed block (bit 1144) May  8 14:58:15 despair kernel: Aborting journal on device dm-0-8.
May  8 14:58:15 despair kernel: EXT4-fs (dm-0): Remounting filesystem read-only
May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904377:freeing already freed block (bit 1145)

These errors are consistent with duplicate blocks being inside the same file. It deleted the first set of blocks properly, then when it got to the duplicate referenced ones it spit the errors about the bits already being freed.

Quoted 1 line

Volume seems clean now after reboot and keeps coming back clean.

Right, since the corruption was only in that one file.

Quoted 1 line

Any ideas based on this?

Did you ever run the "debugfs -c -R 'stat <4195619> ' /dev/c/c" command? (Sorry, I don't recall, lots of other things on the go.) That would at least allow us to see how the corruption was present in the file and create a test case.

Without that, we can only speculate on what was wrong, and we might not be able to find the problem.

Cheers, Andreas

Quoted 56 lines

From: Stephen Elliott [mailto:[email protected]]
Sent: 26 April 2013 22:37
To: '[email protected]'
Subject: FSCK Errors

Hi all,

Just rebooted my box today after 200 days uptime and thought I'd request a volume scan and it found errors! I've never had a power outage etc so am keen to know what could have caused this file system corruption? Anyu ideas???

I'm running 4.2.21 on a ReadyNAS Pro6, but ultimately it is a Linux (Debian) 2.6.37.6. based system underneath.

***** File system check forced at Fri Apr 26 20:08:38 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) Pass 1: Checking inodes, blocks, and sizes Inode 4195619, i_blocks is 3135728, should be 3135904. Fix? yes

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 4195619: 167904376 167904377 167904378 167904379 167904380 167904381 167904382 167904383 167904384 167904385 167904386 167949296 167949297 167949298 167949299 167949300 167949301 167949302 167949303 167949304 167949305 167949306 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.)

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/c/c: ***** FILE SYSTEM WAS MODIFIED *****
/dev/c/c: 615898/30212096 files (13.6% non-contiguous), 62353456/483393536 blocks

I'm curious to know what this is all about, shared with 0 files!!!!:

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Every follow on boot with FSCK yields the following:

***** File system check forced at Fri Apr 26 20:21:42 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) Pass 1: Checking inodes, blocks, and sizes

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 4195619: 167904376 167904377 167904378 167904379 167904380 167904381 167904382 167904383 167904384 167904385 167904386 167949296 167949297 167949298 167949299 167949300 167949301 167949302 167949303 167949304 167949305 167949306 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.)

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013)
has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/c/c: 615898/30212096 files (13.6% non-contiguous), 62353456/483393536 blocks

Still same thing with multiply claimed blocks :)☹

Any ideas?

Quoted 7 lines

Many Thanks
Stephen Elliott

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

Thanks Andreas. Unfortunately I was unable to gather this.

Even your speculation would be appreciated :) I am watching this box like a hawk now. Will perform weekly scans for a while.

-----Original Message-----
From: Andreas Dilger [mailto:[email protected]]
Sent: 08 May 2013 17:22
To: <[email protected]>
Cc: <[email protected]>
Subject: Re: 2nd Attempt - FSCK Errors

Quoted 9 lines · Stephen Elliott

Well... I have some new and interesting information.

After deleting the file, and re-creating from backup, the file system got mounted read only and the following errors were logged:

May  8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904376:freeing already freed block (bit 1144) May  8 14:58:15 despair kernel: Aborting journal on device dm-0-8.
May  8 14:58:15 despair kernel: EXT4-fs (dm-0): Remounting filesystem 

read-only May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0):
mb_free_blocks:1411: group 5124block 167904377:freeing already freed block (bit 1145)

These errors are consistent with duplicate blocks being inside the same file. It deleted the first set of blocks properly, then when it got to the duplicate referenced ones it spit the errors about the bits already being freed.

Quoted 1 line

Volume seems clean now after reboot and keeps coming back clean.

Right, since the corruption was only in that one file.

Quoted 1 line

Any ideas based on this?

Did you ever run the "debugfs -c -R 'stat <4195619> ' /dev/c/c" command? (Sorry, I don't recall, lots of other things on the go.) That would at least allow us to see how the corruption was present in the file and create a test case.

Without that, we can only speculate on what was wrong, and we might not be able to find the problem.

Cheers, Andreas

Quoted 82 lines

From: Stephen Elliott [mailto:[email protected]]
Sent: 26 April 2013 22:37
To: '[email protected]'
Subject: FSCK Errors

Hi all,

Just rebooted my box today after 200 days uptime and thought I'd request a volume scan and it found errors! I've never had a power outage etc so am keen to know what could have caused this file system corruption? Anyu ideas???

I'm running 4.2.21 on a ReadyNAS Pro6, but ultimately it is a Linux (Debian) 2.6.37.6. based system underneath.

***** File system check forced at Fri Apr 26 20:08:38 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) Pass 1:
Checking inodes, blocks, and sizes Inode 4195619, i_blocks is 3135728, should be 3135904. Fix? yes

Running additional passes to resolve blocks claimed by more than one inode... Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 4195619: 167904376 167904377 167904378 167904379 167904380 167904381 167904382 167904383 167904384 167904385 167904386 167949296 167949297 167949298 167949299 167949300 167949301 167949302 167949303 167949304 167949305 167949306 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.)

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information

/dev/c/c: ***** FILE SYSTEM WAS MODIFIED ***** /dev/c/c: 615898/30212096 files (13.6% non-contiguous), 62353456/483393536 blocks

I'm curious to know what this is all about, shared with 0 files!!!!:

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode #4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Every follow on boot with FSCK yields the following:

***** File system check forced at Fri Apr 26 20:21:42 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) Pass 1:
Checking inodes, blocks, and sizes

Running additional passes to resolve blocks claimed by more than one inode... Pass 1B: Rescanning for multiply-claimed blocks Multiply-claimed block(s) in inode 4195619: 167904376 167904377 167904378 167904379 167904380 167904381 167904382 167904383 167904384 167904385 167904386 167949296 167949297 167949298 167949299 167949300 167949301 167949302 167949303 167949304 167949305 167949306 Pass 1C: Scanning directories for inodes with multiply-claimed blocks Pass 1D: Reconciling multiply-claimed blocks (There are 1 inodes containing multiply-claimed blocks.)

File /PREMIER/Premier Automation Purchase OrdersApp V18.5.mdb (inode
#4195619, mod time Fri Apr 26 20:07:42 2013) has 22 multiply-claimed block(s), shared with 0 file(s):
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/c/c: 615898/30212096 files (13.6% non-contiguous), 62353456/483393536 blocks

Still same thing with multiply claimed blocks :)?

Any ideas?

Many Thanks
Stephen Elliott

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html