nilfs_cpfile_delete_checkpoints: cannot delete block

DISCUSSION David Arendt 2 participants 7 messages May 2, 2009 – May 5, 2009 5 messages not in the archive

Subsystems (2)

VFS NILFS2 filesystem

lore

Share & tools

lore

David Arendt

author

2009-05-02 22:55 UTC #0 lore raw

Hi,

Until now nilfs-2.0.12 has run very stable without data corruption. However on one partition (600G) I have got the following errors while running the cleaner:

nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2

This is a partition mainly holding large temporary render files (can be up to 25gb/file). There are currently 132702 snapshots.

As this partition is not used during the next few days, I will leave it with the error so if you would like me to test further things, please let me know.

Bye,
David Arendt

A message that is not in the archive (sent off-list or not yet received): 49FCCF6F.3040101-/[email protected]

Ryusuke Konishi

2009-05-03 08:08 UTC #2 lore raw

Hi David,

Quoted 12 lines · David Arendt

Hi,
Until now nilfs-2.0.12 has run very stable without data corruption. However on one partition (600G) I have got the following errors while running the cleaner:
nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
This is a partition mainly holding large temporary render files (can be up to 25gb/file). There are currently 132702 snapshots.

Quoted 6 lines

As this partition is not used during the next few days, I will leave it with the error so if you would like me to test further things, please let me know.
Bye,
David Arendt

I have reviewed the function in question, but could not find any likely problems.

Could you try the following patch?

It's applicable to v2.0.12.

I have some pending patches later than 2.0.12, but they seem to be independent with your problem.

Thanks,
Ryusuke Konishi
--

fs/cpfile.c +6 -2

index 038d660..9a6a6ae 100644
@@ -342,8 +342,12 @@ int nilfs_cpfile_delete_checkpoints(struct inode *cpfile,
 					cpfile, cno);
 				if (ret == 0)
 					continue;
-				printk(KERN_ERR "%s: cannot delete block\n",
-				       __func__);
+				printk(KERN_ERR "%s: cannot delete block: "
+				       "cno=%llu, range = [%llu, %llu)\n",
+				       __func__,
+				       (unsigned long long)cno,
+				       (unsigned long long)start,
+				       (unsigned long long)end);
 				goto out_sem;
 			}
 		}

fs/dat.c +1 -3

index 523eee7..7463301 100644
@@ -381,11 +381,9 @@ int nilfs_dat_translate(struct inode *dat, __u64 vblocknr, sector_t *blocknrp)
 	entry = nilfs_palloc_block_get_entry(dat, vblocknr, entry_bh, kaddr);
 	blocknr = le64_to_cpu(entry->de_blocknr);
 	if (blocknr == 0) {
-#ifdef CONFIG_NILFS_DEBUG
 		printk(KERN_DEBUG "%s: invalid virtual block number: %llu\n",
 		       __func__, (unsigned long long)vblocknr);
-		BUG();
-#endif
+		WARN_ON(1);
 		ret = -ENOENT;
 		goto out;
 	}

Signature · 1 line

1.6.2

A message that is not in the archive (sent off-list or not yet received): [email protected]

David Arendt

author

2009-05-03 09:26 UTC #4 lore raw

Hi,

I have tried your patch.

The more verbose error message is:

nilfs_cpfile_delete_checkpoints: cannot delete block: cno=1407, range = [11, 75990)
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2

Bye,
David Arendt

Quoted 73 lines · Ryusuke Konishi

Hi David,
Quoted 12 lines · David Arendt
Hi,
Until now nilfs-2.0.12 has run very stable without data corruption. However on one partition (600G) I have got the following errors while running the cleaner:
nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
This is a partition mainly holding large temporary render files (can be up to 25gb/file). There are currently 132702 snapshots.
Quoted 7 lines
As this partition is not used during the next few days, I will leave it with the error so if you would like me to test further things, please let me know.
Bye,
David Arendt
I have reviewed the function in question, but could not find any likely problems.
Could you try the following patch?
It's applicable to v2.0.12.
I have some pending patches later than 2.0.12, but they seem to be independent with your problem.
Thanks,
Ryusuke Konishi
--
fs/cpfile.c +6 -2
index 038d660..9a6a6ae 100644
@@ -342,8 +342,12 @@ int nilfs_cpfile_delete_checkpoints(struct inode *cpfile,
 					cpfile, cno);
 				if (ret == 0)
 					continue;
-				printk(KERN_ERR "%s: cannot delete block\n",
-				       __func__);
+				printk(KERN_ERR "%s: cannot delete block: "
+				       "cno=%llu, range = [%llu, %llu)\n",
+				       __func__,
+				       (unsigned long long)cno,
+				       (unsigned long long)start,
+				       (unsigned long long)end);
 				goto out_sem;
 			}
 		}
fs/dat.c +1 -3
index 523eee7..7463301 100644
@@ -381,11 +381,9 @@ int nilfs_dat_translate(struct inode *dat, __u64 vblocknr, sector_t *blocknrp)
 	entry = nilfs_palloc_block_get_entry(dat, vblocknr, entry_bh, kaddr);
 	blocknr = le64_to_cpu(entry->de_blocknr);
 	if (blocknr == 0) {
-#ifdef CONFIG_NILFS_DEBUG
 		printk(KERN_DEBUG "%s: invalid virtual block number: %llu\n",
 		       __func__, (unsigned long long)vblocknr);
-		BUG();
-#endif
+		WARN_ON(1);
 		ret = -ENOENT;
 		goto out;
 	}

A message that is not in the archive (sent off-list or not yet received): 49FD6359.1020405-/[email protected]

Ryusuke Konishi

2009-05-03 09:44 UTC #6 lore raw

Hi!

Quoted 9 lines · David Arendt

Hi,
I have tried your patch.
The more verbose error message is:
nilfs_cpfile_delete_checkpoints: cannot delete block: cno=1407, range = [11, 75990)
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2

You didn't see any DAT warnings?

If so, do you think the range of deleting checkpoints (i.e. 11 ~ 75990 - 1) is proper?

How is the output of lscp?

Ryusuke Konishi

Quoted 77 lines

Bye,
David Arendt
Quoted 73 lines · Ryusuke Konishi
Hi David,
Quoted 12 lines · David Arendt
Hi,
Until now nilfs-2.0.12 has run very stable without data corruption. However on one partition (600G) I have got the following errors while running the cleaner:
nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
This is a partition mainly holding large temporary render files (can be up to 25gb/file). There are currently 132702 snapshots.
Quoted 7 lines
As this partition is not used during the next few days, I will leave it with the error so if you would like me to test further things, please let me know.
Bye,
David Arendt
I have reviewed the function in question, but could not find any likely problems.
Could you try the following patch?
It's applicable to v2.0.12.
I have some pending patches later than 2.0.12, but they seem to be independent with your problem.
Thanks,
Ryusuke Konishi
--
fs/cpfile.c +6 -2
index 038d660..9a6a6ae 100644
@@ -342,8 +342,12 @@ int nilfs_cpfile_delete_checkpoints(struct inode *cpfile,
 					cpfile, cno);
 				if (ret == 0)
 					continue;
-				printk(KERN_ERR "%s: cannot delete block\n",
-				       __func__);
+				printk(KERN_ERR "%s: cannot delete block: "
+				       "cno=%llu, range = [%llu, %llu)\n",
+				       __func__,
+				       (unsigned long long)cno,
+				       (unsigned long long)start,
+				       (unsigned long long)end);
 				goto out_sem;
 			}
 		}
fs/dat.c +1 -3
index 523eee7..7463301 100644
@@ -381,11 +381,9 @@ int nilfs_dat_translate(struct inode *dat, __u64 vblocknr, sector_t *blocknrp)
 	entry = nilfs_palloc_block_get_entry(dat, vblocknr, entry_bh, kaddr);
 	blocknr = le64_to_cpu(entry->de_blocknr);
 	if (blocknr == 0) {
-#ifdef CONFIG_NILFS_DEBUG
 		printk(KERN_DEBUG "%s: invalid virtual block number: %llu\n",
 		       __func__, (unsigned long long)vblocknr);
-		BUG();
-#endif
+		WARN_ON(1);
 		ret = -ENOENT;
 		goto out;
 	}

A message that is not in the archive (sent off-list or not yet received): [email protected]

David Arendt

author

2009-05-03 10:06 UTC #8 lore raw

Hi,

I didn't see any DAT warnings.

Using lscp I see that the first entry is

1428 2009-03-30 02:13:06 cp - 259 74436

The last entry is

134128 2009-05-03 00:04:28 cp i 81813 876

If you want to have the full output of lscp please tell me, then I will send it to you without sending to the mailinglist as the file has 10mb.

Bye,
David Arendt

Quoted 104 lines · Ryusuke Konishi

Hi!
Quoted 10 lines · David Arendt
Hi,
I have tried your patch.
The more verbose error message is:
nilfs_cpfile_delete_checkpoints: cannot delete block: cno=1407, range = [11, 75990)
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
You didn't see any DAT warnings?
If so, do you think the range of deleting checkpoints (i.e. 11 ~ 75990 - 1) is proper?
How is the output of lscp?
Ryusuke Konishi
Quoted 80 lines
Bye,
David Arendt
Quoted 75 lines · Ryusuke Konishi
Hi David,
Quoted 12 lines · David Arendt
Hi,
Until now nilfs-2.0.12 has run very stable without data corruption. However on one partition (600G) I have got the following errors while running the cleaner:
nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
This is a partition mainly holding large temporary render files (can be up to 25gb/file). There are currently 132702 snapshots.
Quoted 8 lines
As this partition is not used during the next few days, I will leave it with the error so if you would like me to test further things, please let me know.
Bye,
David Arendt
I have reviewed the function in question, but could not find any likely problems.
Could you try the following patch?
It's applicable to v2.0.12.
I have some pending patches later than 2.0.12, but they seem to be independent with your problem.
Thanks,
Ryusuke Konishi
--
fs/cpfile.c +6 -2
index 038d660..9a6a6ae 100644
@@ -342,8 +342,12 @@ int nilfs_cpfile_delete_checkpoints(struct inode *cpfile,
 					cpfile, cno);
 				if (ret == 0)
 					continue;
-				printk(KERN_ERR "%s: cannot delete block\n",
-				       __func__);
+				printk(KERN_ERR "%s: cannot delete block: "
+				       "cno=%llu, range = [%llu, %llu)\n",
+				       __func__,
+				       (unsigned long long)cno,
+				       (unsigned long long)start,
+				       (unsigned long long)end);
 				goto out_sem;
 			}
 		}
fs/dat.c +1 -3
index 523eee7..7463301 100644
@@ -381,11 +381,9 @@ int nilfs_dat_translate(struct inode *dat, __u64 vblocknr, sector_t *blocknrp)
 	entry = nilfs_palloc_block_get_entry(dat, vblocknr, entry_bh, kaddr);
 	blocknr = le64_to_cpu(entry->de_blocknr);
 	if (blocknr == 0) {
-#ifdef CONFIG_NILFS_DEBUG
 		printk(KERN_DEBUG "%s: invalid virtual block number: %llu\n",
 		       __func__, (unsigned long long)vblocknr);
-		BUG();
-#endif
+		WARN_ON(1);
 		ret = -ENOENT;
 		goto out;
 	}

David Arendt

author

2009-05-04 04:16 UTC #9 lore raw

In reply to a message not in the archive

Hi,

This night. I had lots of:

nilfs_btree_propagate: key = 67, level == 0

On the parition where cleanerd has failed.

A try to umount it resulted in a hang with the following message:

NILFS warning (device sda10): nilfs_segctor_destroy: dirty file(s) after the final construction

Bye,
David Arendt

Quoted 73 lines · Ryusuke Konishi

Hi David,
Quoted 12 lines · David Arendt
Hi,
Until now nilfs-2.0.12 has run very stable without data corruption. However on one partition (600G) I have got the following errors while running the cleaner:
nilfs_cpfile_delete_checkpoints: cannot delete block
NILFS: GC failed during preparation: cannot delete checkpoints: err=-2
This is a partition mainly holding large temporary render files (can be up to 25gb/file). There are currently 132702 snapshots.
Quoted 7 lines
As this partition is not used during the next few days, I will leave it with the error so if you would like me to test further things, please let me know.
Bye,
David Arendt
I have reviewed the function in question, but could not find any likely problems.
Could you try the following patch?
It's applicable to v2.0.12.
I have some pending patches later than 2.0.12, but they seem to be independent with your problem.
Thanks,
Ryusuke Konishi
--
fs/cpfile.c +6 -2
index 038d660..9a6a6ae 100644
@@ -342,8 +342,12 @@ int nilfs_cpfile_delete_checkpoints(struct inode *cpfile,
 					cpfile, cno);
 				if (ret == 0)
 					continue;
-				printk(KERN_ERR "%s: cannot delete block\n",
-				       __func__);
+				printk(KERN_ERR "%s: cannot delete block: "
+				       "cno=%llu, range = [%llu, %llu)\n",
+				       __func__,
+				       (unsigned long long)cno,
+				       (unsigned long long)start,
+				       (unsigned long long)end);
 				goto out_sem;
 			}
 		}
fs/dat.c +1 -3
index 523eee7..7463301 100644
@@ -381,11 +381,9 @@ int nilfs_dat_translate(struct inode *dat, __u64 vblocknr, sector_t *blocknrp)
 	entry = nilfs_palloc_block_get_entry(dat, vblocknr, entry_bh, kaddr);
 	blocknr = le64_to_cpu(entry->de_blocknr);
 	if (blocknr == 0) {
-#ifdef CONFIG_NILFS_DEBUG
 		printk(KERN_DEBUG "%s: invalid virtual block number: %llu\n",
 		       __func__, (unsigned long long)vblocknr);
-		BUG();
-#endif
+		WARN_ON(1);
 		ret = -ENOENT;
 		goto out;
 	}

A message that is not in the archive (sent off-list or not yet received): 49FE6C18.3050707-/[email protected]

Ryusuke Konishi

2009-05-05 11:23 UTC #11 lore raw

Hi David,

Quoted 7 lines · David Arendt

Hi,
This night. I had lots of:
nilfs_btree_propagate: key = 67, level == 0
On the parition where cleanerd has failed.

This error is related to the GC failure.

Both logs indicate that btree look-up of the 67th block on the checkpoint file failed.

I suspect inconsistency between the block on page cache and btree; the block was removed from the btree but were remaining on the page cache.

Could you try the following bugfix patch?

The patch ensures to clear dirty state of page and buffer after removal of block, and would prevent the inconsistency.

Thanks in advance,
Ryusuke Konishi
--

fs/btnode.c +1 -3

index 5e83c60..11a7305 100644
@@ -176,7 +176,6 @@ void nilfs_btnode_delete(struct buffer_head *bh)
 	struct address_space *mapping;
 	struct page *page = bh->b_page;
 	pgoff_t index = page_index(page);
-	int still_dirty;
 
 	page_cache_get(page);
 	lock_page(page);

@@ -186,12 +185,11 @@ void nilfs_btnode_delete(struct buffer_head *bh)
 		BH_DEBUG(bh, "deleting unused btnode buffer");
 
 	nilfs_forget_buffer(bh);
-	still_dirty = PageDirty(page);
 	mapping = page->mapping;
 	unlock_page(page);
 	page_cache_release(page);
 
-	if (!still_dirty && mapping)
+	if (mapping)
 		invalidate_inode_pages2_range(mapping, index, index);
 }

fs/mdt.c +4 -5

index 2792e76..4c9fb00 100644
@@ -327,7 +327,7 @@ int nilfs_mdt_delete_block(struct inode *inode, unsigned long block)
 
 	mdt_debug(3, "called (ino=%lu, blkoff=%lu)\n", inode->i_ino, block);
 	err = nilfs_bmap_delete(ii->i_bmap, block);
-	if (likely(!err)) {
+	if (!err || err == -ENOENT) {
 		nilfs_mdt_mark_dirty(inode);
 		nilfs_mdt_forget_block(inode, block);
 	}

@@ -357,7 +357,6 @@ int nilfs_mdt_forget_block(struct inode *inode, unsigned long block)
 	struct page *page;
 	unsigned long first_block;
 	int ret = 0;
-	int still_dirty;
 
 	mdt_debug(3, "called (ino=%lu, blkoff=%lu)\n", inode->i_ino, block);
 	page = find_lock_page(inode->i_mapping, index);

@@ -373,13 +372,13 @@ int nilfs_mdt_forget_block(struct inode *inode, unsigned long block)
 
 		bh = nilfs_page_get_nth_block(page, block - first_block);
 		nilfs_forget_buffer(bh);
+	} else {
+		__nilfs_clear_page_dirty(page);
 	}
-	still_dirty = PageDirty(page);
 	unlock_page(page);
 	page_cache_release(page);
 
-	if (still_dirty ||
-	    invalidate_inode_pages2_range(inode->i_mapping, index, index) != 0)
+	if (invalidate_inode_pages2_range(inode->i_mapping, index, index) != 0)
 		ret = -EBUSY;
 	mdt_debug(3, "done (err=%d)\n", ret);
 	return ret;

fs/page.c +2 -1

index 9cf93c3..d333fef 100644
@@ -129,7 +129,8 @@ void nilfs_forget_buffer(struct buffer_head *bh)
 
 	lock_buffer(bh);
 	clear_buffer_nilfs_volatile(bh);
-	if (test_clear_buffer_dirty(bh) && nilfs_page_buffers_clean(page))
+	clear_buffer_dirty(bh);
+	if (nilfs_page_buffers_clean(page))
 		__nilfs_clear_page_dirty(page);
 
 	clear_buffer_uptodate(bh);