Skip to contentSkip to conversation

2.6.19 file content corruption on ext3

PATCH NAK Andrei Popa 43 participants 311 messages Dec 17, 2006 – Jan 7, 2007 3 messages not in the archive Page 2 of 2

Subsystems (15)
lore
Share & tools
lore
#173 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to [email protected]
Quoted 3 lines · On 12/27/06, [email protected] <[email protected]> wrote:

I do get this error on reiserfs ( old one, didn't try on reiser4 ). Stock 2.6.19 plus reiser4 patch. Previously reported by me only in the debian bts.

I've had reports of corrupted data on earlier kernel releases with reiserfs3, which were fixed by upgrading to reiserfs4.

Jari Sundell

#174 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds

What would also actually be interesting is whether somebody can reproduce this on Reiserfs, for example. I _think_ all the reports I've seen are on ext2 or ext3, and if this is somehow writeback-related, it could be some bug that is just shared between the two by virtue of them still having a lot of stuff in common.

Linus

I've been following this thread for a while now as I started experiencing file corruption in rtorrent when I upgraded to 2.6.19. I am using reiserfs.

Signature · 1 line
Tom Lanyon
#175 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 3 lines · Tom Lanyon

I've been following this thread for a while now as I started experiencing file corruption in rtorrent when I upgraded to 2.6.19. I am using reiserfs.

However, moving to 2.6.20-rc3 does indeed seem to fix the issue thus far...

Signature · 1 line
Tom Lanyon
#176 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Tom Lanyon
Quoted 7 lines · On Sun, 7 Jan 2007 12:36:18 +1030 "Tom Lanyon" <[email protected]> wrote:
Quoted 6 lines · Linus Torvalds

What would also actually be interesting is whether somebody can reproduce this on Reiserfs, for example. I _think_ all the reports I've seen are on ext2 or ext3, and if this is somehow writeback-related, it could be some bug that is just shared between the two by virtue of them still having a lot of stuff in common.

Quoted 5 lines
Quoted 1 line

Linus

I've been following this thread for a while now as I started experiencing file corruption in rtorrent when I upgraded to 2.6.19. I am using reiserfs.

reiserfs defaults to data=ordered, so it's quite possibly the same bug.

#177 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 6 lines

Quoting Linus Torvalds <[email protected]>:
Subject: Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Peter, tell me I'm crazy, but with the new rules, the following condition is a bug:

Quoted 7 lines

- shared mapping
- writable
- not already marked dirty in the PTE

because that combination means that the hardware can mark the PTE dirty without us even realizing (and thus not marking the "struct page *" dirty).

Er.
Sorry about bumping in, and I'm not sure I understand all of the discussion, but this reminded me of an old issue with COW that created what looks like a vaguely similiar data corruption on infiniband. We solved this for infiniband with MADV_DONTFORK, but I always wondered why does it not affect other parts of kernel. Small reminder from that discussion:

down mmap sem
get user pages
up mmap sem
page becomes shared, and COW (e.g. fork)
process writes to first byte of page <----- gets a copy Now we had a problem: struct page that we got from get user pages does not point to a correct page in our process.
For example: if at some point we map this page for DMA, and hardware writes to last byte of page -----> process does not see this data.

So for infiniband, what we do is a combination of
- prevent page from becoming COW while hardware might DMA to this page, and
- ask users not to write to page if hardware might DMA to same page (even if its using different bytes).

I just wandered - is there some chance something like this could be happening in the fs code?

HTH,

Signature · 1 line
MST
#178 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

How about this particularly stupid diff? (please test with something that _would_ cause corruption normally).

It is _entirely_ untested, but what it tries to do is to simply serialize any writeback in progress with any process that tries to re-map a shared page into its address space and dirty it. I haven't tested it, and maybe it misses some case, but it looks likea good way to try to avoid races with marking pages dirty and the writeback phase ..

The apt cache files (/var/cache/apt/*.bin) still get corrupted with this patch and 2.6.19.

Gordon

fs/buffer.c 2006-12-21 01:16:31.000000000 -0700 renamed from fs/buffer.c 2006-11-29 14:57:37.000000000 -0700 +1 -12
diff -Naupr linux-2.6.19.orig/fs/buffer.c linux-2.6.19/fs/buffer.c
@@ -2832,7 +2832,7 @@ int try_to_free_buffers(struct page *pag
       int ret = 0;
 
       BUG_ON(!PageLocked(page));
-       if (PageWriteback(page))
+       if (PageDirty(page) || PageWriteback(page))
               return 0;
 
       if (mapping == NULL) {          /* can this still happen? */
@@ -2843,17 +2843,6 @@ int try_to_free_buffers(struct page *pag
       spin_lock(&mapping->private_lock);
       ret = drop_buffers(page, &buffers_to_free);
       spin_unlock(&mapping->private_lock);
-       if (ret) {
-               /*
-                * If the filesystem writes its buffers by hand (eg ext3)
-                * then we can have clean buffers against a dirty page.  We
-                * clean the page here; otherwise later reattachment of buffers
-                * could encounter a non-uptodate page, which is unresolvable.
-                * This only applies in the rare case where try_to_free_buffers
-                * succeeds but the page is not freed.
-                */
-               clear_page_dirty(page);
-       }

out:

       if (buffers_to_free) {
               struct buffer_head *bh = buffers_to_free;

diff -Naupr linux-2.6.19.orig/fs/hugetlbfs/inode.c linux-2.6.19/fs/hugetlbfs/inode.c
--- linux-2.6.19.orig/fs/hugetlbfs/inode.c 2006-11-29
14:57:37.000000000 -0700
+++ linux-2.6.19/fs/hugetlbfs/inode.c 2006-12-21 01:15:21.000000000 -0700
@@ -176,7 +176,7 @@ static int hugetlbfs_commit_write(struct

static void truncate_huge_page(struct page *page)
{
- clear_page_dirty(page);
+ cancel_dirty_page(page, /* No IO accounting for huge pages? */0);

       ClearPageUptodate(page);
       remove_from_page_cache(page);
       put_page(page);

diff -Naupr linux-2.6.19.orig/include/linux/page-flags.h linux-2.6.19/include/linux/page-flags.h
--- linux-2.6.19.orig/include/linux/page-flags.h 2006-11-29
14:57:37.000000000 -0700
+++ linux-2.6.19/include/linux/page-flags.h 2006-12-21
01:15:21.000000000 -0700
@@ -253,15 +253,11 @@ static inline void SetPageUptodate(struc

struct page; /* forward declaration */

-int test_clear_page_dirty(struct page *page);
+extern void cancel_dirty_page(struct page *page, unsigned int account_size); +
int test_clear_page_writeback(struct page *page);
int test_set_page_writeback(struct page *page);

-static inline void clear_page_dirty(struct page *page) -{
- test_clear_page_dirty(page);
-}
-
static inline void set_page_writeback(struct page *page)

{
       test_set_page_writeback(page);
mm/memory.c 2006-12-24 11:04:03.000000000 -0700 renamed from mm/memory.c 2006-11-29 14:57:37.000000000 -0700 +1 -0
diff -Naupr linux-2.6.19.orig/mm/memory.c linux-2.6.19/mm/memory.c
@@ -1534,6 +1534,7 @@ static int do_wp_page(struct mm_struct *
                       if (!pte_same(*page_table, orig_pte))
                               goto unlock;
               }
+               wait_on_page_writeback(old_page);
               dirty_page = old_page;
               get_page(dirty_page);
               reuse = 1;
@@ -1832,6 +1833,33 @@ void unmap_mapping_range(struct address_

}
EXPORT_SYMBOL(unmap_mapping_range);

+static void check_last_page(struct address_space *mapping, loff_t size) +{

+       pgoff_t index;
+       unsigned int offset;
+       struct page *page;

+
+ if (!mapping)
+ return;
+ offset = size & ~PAGE_MASK;
+ if (!offset)
+ return;

+       index = size >> PAGE_SHIFT;
+       page = find_lock_page(mapping, index);
+       if (page) {
+               unsigned int check = 0;
+               unsigned char *kaddr = kmap_atomic(page, KM_USER0);
+               do {
+                       check += kaddr[offset++];
+               } while (offset < PAGE_SIZE);
+               kunmap_atomic(kaddr,KM_USER0);
+               unlock_page(page);
+               page_cache_release(page);
+               if (check)

+ printk("%s: BADNESS: truncate check %u\n",
current->comm, check);
+ }
+}
+
/**
* vmtruncate - unmap mappings "freed" by truncate() syscall
* @inode: inode of the file used
@@ -1865,6 +1893,7 @@ do_expand:

               goto out_sig;
       if (offset > inode->i_sb->s_maxbytes)
               goto out_big;
+       check_last_page(mapping, inode->i_size);
       i_size_write(inode, offset);

out_truncate:
@@ -2206,6 +2235,7 @@ retry:

                               page_cache_release(new_page);
                               return VM_FAULT_SIGBUS;
                       }
+                       wait_on_page_writeback(new_page);
               }
       }

diff -Naupr linux-2.6.19.orig/mm/page-writeback.c linux-2.6.19/mm/page-writeback.c
--- linux-2.6.19.orig/mm/page-writeback.c 2006-11-29
14:57:37.000000000 -0700
+++ linux-2.6.19/mm/page-writeback.c 2006-12-21 01:26:53.000000000 -0700
@@ -843,39 +843,6 @@ int set_page_dirty_lock(struct page *pag EXPORT_SYMBOL(set_page_dirty_lock);

/*
- * Clear a page's dirty flag, while caring for dirty memory accounting.
- * Returns true if the page was previously dirty.
- */
-int test_clear_page_dirty(struct page *page)
-{
- struct address_space *mapping = page_mapping(page);
- unsigned long flags;
-
- if (mapping) {
- write_lock_irqsave(&mapping->tree_lock, flags);
- if (TestClearPageDirty(page)) {
- radix_tree_tag_clear(&mapping->page_tree,
- page_index(page),
- PAGECACHE_TAG_DIRTY);
- write_unlock_irqrestore(&mapping->tree_lock, flags);
- /*

-                        * We can continue to use `mapping' here because the
-                        * page is locked, which pins the address_space
-                        */

- if (mapping_cap_account_dirty(mapping)) {
- page_mkclean(page);
- dec_zone_page_state(page, NR_FILE_DIRTY);
- }
- return 1;

-               }
-               write_unlock_irqrestore(&mapping->tree_lock, flags);
-               return 0;

- }
- return TestClearPageDirty(page);
-}
-EXPORT_SYMBOL(test_clear_page_dirty);
-
-/*
* Clear a page's dirty flag, while caring for dirty memory accounting.
* Returns true if the page was previously dirty.
*

mm/rmap.c 2006-12-22 23:25:09.000000000 -0700 renamed from mm/rmap.c 2006-11-29 14:57:37.000000000 -0700 +0 -0
diff -Naupr linux-2.6.19.orig/mm/rmap.c linux-2.6.19/mm/rmap.c
@@ -432,7 +432,7 @@ static int page_mkclean_one(struct page
{
       struct mm_struct *mm = vma->vm_mm;
       unsigned long address;
-       pte_t *pte, entry;
+       pte_t *pte;
       spinlock_t *ptl;
       int ret = 0;

@@ -444,17 +444,18 @@ static int page_mkclean_one(struct page

       if (!pte)
               goto out;

- if (!pte_dirty(*pte) && !pte_write(*pte))
- goto unlock;
+ if (pte_dirty(*pte) || pte_write(*pte)) {
+ pte_t entry;

-       entry = ptep_get_and_clear(mm, address, pte);
-       entry = pte_mkclean(entry);
-       entry = pte_wrprotect(entry);
-       ptep_establish(vma, address, pte, entry);
-       lazy_mmu_prot_update(entry);
-       ret = 1;
+               flush_cache_page(vma, address, pte_pfn(*pte));
+               entry = ptep_clear_flush(vma, address, pte);
+               entry = pte_wrprotect(entry);
+               entry = pte_mkclean(entry);
+               set_pte_at(vma, address, pte, entry);
+               lazy_mmu_prot_update(entry);
+               ret = 1;

+ }

-unlock:

       pte_unmap_unlock(pte, ptl);
out:
       return ret;
@@ -489,6 +490,8 @@ int page_mkclean(struct page *page)
               if (mapping)
                       ret = page_mkclean_file(mapping, page);
       }

+ if (page_test_and_clear_dirty(page))
+ ret = 1;

       return ret;
}

@@ -587,8 +590,6 @@ void page_remove_rmap(struct page *page)
* Leaving it set also helps swapoff to reinstate ptes
* faster for those pages still in swapcache. */
- if (page_test_and_clear_dirty(page))
- set_page_dirty(page);
__dec_zone_page_state(page,
PageAnon(page) ? NR_ANON_PAGES :

NR_FILE_MAPPED);
       }
@@ -607,6 +608,7 @@ static int try_to_unmap_one(struct page
       pte_t pteval;
       spinlock_t *ptl;
       int ret = SWAP_AGAIN;

+ struct page *dirty_page = NULL;

       address = vma_address(page, vma);
       if (address == -EFAULT)

@@ -633,7 +635,7 @@ static int try_to_unmap_one(struct page

       /* Move the dirty bit to the physical page now the pte is gone. */
       if (pte_dirty(pteval))

- set_page_dirty(page);
+ dirty_page = page;

       /* Update high watermark before we lower rss */
       update_hiwater_rss(mm);

@@ -684,6 +686,8 @@ static int try_to_unmap_one(struct page

out_unmap:

       pte_unmap_unlock(pte, ptl);
+       if (dirty_page)
+               set_page_dirty(dirty_page);
out:
       return ret;
}
@@ -915,6 +919,9 @@ int try_to_unmap(struct page *page, int
       else
               ret = try_to_unmap_file(page, migration);

+ if (page_test_and_clear_dirty(page))
+ set_page_dirty(page);
+

       if (!page_mapped(page))
               ret = SWAP_SUCCESS;
       return ret;
mm/truncate.c 2006-12-23 13:21:42.000000000 -0700 renamed from mm/truncate.c 2006-11-29 14:57:37.000000000 -0700 +0 -0
diff -Naupr linux-2.6.19.orig/mm/truncate.c linux-2.6.19/mm/truncate.c
@@ -50,6 +50,21 @@ static inline void truncate_partial_page
               do_invalidatepage(page, partial);

}

+void cancel_dirty_page(struct page *page, unsigned int account_size) +{
+ /* If we're cancelling the page, it had better not be mapped
any more */+ if (page_mapped(page)) {
+ static unsigned int warncount;
+
+ WARN_ON(++warncount < 5);
+ }
+
+ if (TestClearPageDirty(page) && account_size &&
+ mapping_cap_account_dirty(page->mapping))
+ dec_zone_page_state(page, NR_FILE_DIRTY);
+}
+
+
/*
* If truncate cannot remove the fs-private metadata from the page, the page
* becomes anonymous. It will be left on the LRU and may even be mapped into
@@ -66,10 +81,11 @@ truncate_complete_page(struct address_sp

       if (page->mapping != mapping)
               return;

+ cancel_dirty_page(page, PAGE_CACHE_SIZE);
+

       if (PagePrivate(page))
               do_invalidatepage(page, 0);
-       clear_page_dirty(page);
       ClearPageUptodate(page);
       ClearPageMappedToDisk(page);
       remove_from_page_cache(page);
@@ -348,7 +364,6 @@ int invalidate_inode_pages2_range(struct
               for (i = 0; !ret && i < pagevec_count(&pvec); i++) {
                       struct page *page = pvec.pages[i];
                       pgoff_t page_index;

- int was_dirty;

                       lock_page(page);
                       if (page->mapping != mapping) {
@@ -384,12 +399,8 @@ int invalidate_inode_pages2_range(struct
                                         PAGE_CACHE_SIZE, 0);
                               }
                       }

- was_dirty = test_clear_page_dirty(page);
- if (!invalidate_complete_page2(mapping, page)) {
- if (was_dirty)
- set_page_dirty(page);
+ if (!invalidate_complete_page2(mapping, page))

                               ret = -EIO;
-                       }
                       unlock_page(page);
               }
               pagevec_release(&pvec);
Signature · 1 line
Gordon Farquharson
#179 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 3 lines · Gordon Farquharson

The apt cache files (/var/cache/apt/*.bin) still get corrupted with this patch and 2.6.19.

Yeah, if my guess about do_no_page() is right, _none_ of the previous patches should have ANY effect what-so-ever. In fact, I'd say that even the "ext3 works in writeback mode" thing that Andrei reports is probably a total fluke brought on by timing changes rather than anything else.

So please try the latest patch instead (on top of anything that shows corruption reliably - the patch should be _totally_ independent of all the other issues, and I think it will apply cleanly on top of 2.6.18.3 and 2.6.19 too, so anything that shows corruption is a fine target - but try to choose something that has been the "best" at corrupting things for you, to make the testing as good as possible).

Patch included here again (although I think you were cc'd on my previous email too, so you should already have it, and our emails just crossed)

And if this doesn't fix it, I don't know what will..

Linus

---

mm/memory.c +5 -3
index 563792f..cf429c4 100644
@@ -2247,21 +2249,23 @@ retry:
 	if (pte_none(*page_table)) {
 		flush_icache_page(vma, new_page);
 		entry = mk_pte(new_page, vma->vm_page_prot);
-		if (write_access)
-			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
-		set_pte_at(mm, address, page_table, entry);
 		if (anon) {
 			inc_mm_counter(mm, anon_rss);
 			lru_cache_add_active(new_page);
 			page_add_new_anon_rmap(new_page, vma, address);
+			if (write_access)
+				entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 		} else {
 			inc_mm_counter(mm, file_rss);
 			page_add_file_rmap(new_page);
+			entry = pte_wrprotect(entry);
 			if (write_access) {
 				dirty_page = new_page;
 				get_page(dirty_page);
+				entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 			}
 		}
+		set_pte_at(mm, address, page_table, entry);
 	} else {
 		/* One of our sibling threads was faster, back out. */
 		page_cache_release(new_page);
#180 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 18 lines · Linus Torvalds
Quoted 3 lines · Gordon Farquharson

The apt cache files (/var/cache/apt/*.bin) still get corrupted with this patch and 2.6.19.

Yeah, if my guess about do_no_page() is right, _none_ of the previous patches should have ANY effect what-so-ever. In fact, I'd say that even the "ext3 works in writeback mode" thing that Andrei reports is probably a total fluke brought on by timing changes rather than anything else.

So please try the latest patch instead (on top of anything that shows corruption reliably - the patch should be _totally_ independent of all the other issues, and I think it will apply cleanly on top of 2.6.18.3 and 2.6.19 too, so anything that shows corruption is a fine target - but try to choose something that has been the "best" at corrupting things for you, to make the testing as good as possible).

Quoted 4 lines

Patch included here again (although I think you were cc'd on my previous email too, so you should already have it, and our emails just crossed)

And if this doesn't fix it, I don't know what will..

With latest git and patches:
http://lkml.org/lkml/diff/2006/12/24/56/1
http://lkml.org/lkml/diff/2006/12/24/61/1

Hash check on download completion found bad chunks, consider using "safe_sync".

Quoted 35 lines

Linus

---

mm/memory.c +5 -3
index 563792f..cf429c4 100644
@@ -2247,21 +2249,23 @@ retry:
 	if (pte_none(*page_table)) {
 		flush_icache_page(vma, new_page);
 		entry = mk_pte(new_page, vma->vm_page_prot);
-		if (write_access)
-			entry = maybe_mkwrite(pte_mkdirty(entry), vma);
-		set_pte_at(mm, address, page_table, entry);
 		if (anon) {
 			inc_mm_counter(mm, anon_rss);
 			lru_cache_add_active(new_page);
 			page_add_new_anon_rmap(new_page, vma, address);
+			if (write_access)
+				entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 		} else {
 			inc_mm_counter(mm, file_rss);
 			page_add_file_rmap(new_page);
+			entry = pte_wrprotect(entry);
 			if (write_access) {
 				dirty_page = new_page;
 				get_page(dirty_page);
+				entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 			}
 		}
+		set_pte_at(mm, address, page_table, entry);
 	} else {
 		/* One of our sibling threads was faster, back out. */
 		page_cache_release(new_page);
#181 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 3 lines · Andrei Popa

Hash check on download completion found bad chunks, consider using "safe_sync".

Dang. Did you get any warning messages from the kernel?

Linus

#182 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Andrei Popa

Hash check on download completion found bad chunks, consider using "safe_sync".

Dang. Did you get any warning messages from the kernel?

only these:
ACPI: EC: evaluating _Q80
ACPI: EC: evaluating _Q80
ACPI: EC: evaluating _Q80

but I don't think has anything to do with...

Quoted 1 line

Linus

#183 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · Andrei Popa

Hash check on download completion found bad chunks, consider using "safe_sync".

Dang. Did you get any warning messages from the kernel?

Linus

BTW, rmap.c patch is broken - needs at least

---

mm/rmap.c +1 -1
index 57306fa..669acb2 100644
@@ -452,7 +452,7 @@ static int page_mkclean_one(struct page 
 		entry = ptep_clear_flush(vma, address, pte);
 		entry = pte_wrprotect(entry);
 		entry = pte_mkclean(entry);
-		set_pte_at(vma, address, pte, entry);
+		set_pte_at(mm, address, pte, entry);
 		lazy_mmu_prot_update(entry);
 		ret = 1;
 	}
#184 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 5 lines · Al Viro
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Andrei Popa
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Hash check on download completion found bad chunks, consider using "safe_sync".

Dang. Did you get any warning messages from the kernel?

Linus

BTW, rmap.c patch is broken - needs at least

... but that doesn't affect most of the architectures - only sparc64 and some of powerpc. So it's definitely not enough.

#185 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 1 line · Linus Torvalds

And if this doesn't fix it, I don't know what will..

Sorry, but it still fails (on top of plain 2.6.19).

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#186 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

/etc/fstab: ext2 nobh
/etc/fstab: ext3 data=writeback,nobh

It seems that busybox mount ignores the nobh option but both ext2 and ext3 data=writeback work for me. This is with plain 2.6.19 which normally always fails.

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#187 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

I don't think it's a page table issue any more, it just doesn't look likely with the ARM UP corruption. It's also not apparently even on a cacheline boundary, so it probably is really a dirty bit that got cleared wrogn due to some race with IO.

So, until now it's only been reported for SMP on i386? I'm seeing the issue on my Pentium-M Notebook (Thinkpad R52) over here, UP kernel, no preempt.

I've first seen it with 2.6.20-rc1, but am running 2.6.20-rc2 now. The corruption pattern looks like the one already reported, rtorrent hash check fails (for some files it succeeds at first, but fails after "echo 1 > /proc/sys/vm/drop_caches"), the corruption is zeroes at the end of page instead of data.

ii  rtorrent       0.6.4-1        ncurses BitTorrent client based on LibTorren
ii  libtorrent9    0.10.4-1       a C++ BitTorrent library

.config:
# Automatically generated make config: don't edit # Linux kernel version: 2.6.20-rc2
# Mon Dec 25 14:00:03 2006
#
CONFIG_X86_32=y
CONFIG_GENERIC_TIME=y
CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
# CONFIG_IPC_NS is not set
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_UTS_NS is not set
# CONFIG_AUDIT is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_SLAB=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
# CONFIG_SLOB is not set

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y

#
# Block layer
#
CONFIG_BLOCK=y
CONFIG_LBD=y
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_LSF is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_AS=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="anticipatory"

#
# Processor type and features
#
# CONFIG_SMP is not set
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_PARAVIRT is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
CONFIG_MPENTIUMM=y
# CONFIG_MCORE2 is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_DELL_RBU is not set
CONFIG_DCDBAS=m
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
CONFIG_HZ_300=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=300
# CONFIG_KEXEC is not set
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_COMPAT_VDSO=y

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_LEGACY is not set
# CONFIG_PM_DEBUG is not set
# CONFIG_PM_SYSFS_DEPRECATED is not set
CONFIG_SOFTWARE_SUSPEND=y
CONFIG_PM_STD_PARTITION=""

#
# ACPI (Advanced Configuration and Power Interface) Support #
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SLEEP_PROC_FS=y
# CONFIG_ACPI_SLEEP_PROC_SLEEP is not set CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=y
CONFIG_ACPI_HOTKEY=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
CONFIG_ACPI_IBM=m
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_ACPI_CUSTOM_DSDT is not set CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_STAT_DETAILS is not set CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=y CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

#
# CPUFreq processor drivers
#
# CONFIG_X86_ACPI_CPUFREQ is not set
# CONFIG_X86_POWERNOW_K6 is not set
# CONFIG_X86_POWERNOW_K7 is not set
# CONFIG_X86_POWERNOW_K8 is not set
# CONFIG_X86_GX_SUSPMOD is not set
CONFIG_X86_SPEEDSTEP_CENTRINO=y
CONFIG_X86_SPEEDSTEP_CENTRINO_ACPI=y
CONFIG_X86_SPEEDSTEP_CENTRINO_TABLE=y
CONFIG_X86_SPEEDSTEP_ICH=y
CONFIG_X86_SPEEDSTEP_SMI=y
# CONFIG_X86_P4_CLOCKMOD is not set
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
# CONFIG_X86_LONGRUN is not set
# CONFIG_X86_LONGHAUL is not set

#

# shared options
#
# CONFIG_X86_ACPI_CPUFREQ_PROC_INTF is not set
CONFIG_X86_SPEEDSTEP_LIB=y
# CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCIEPORTBUS=y
# CONFIG_HOTPLUG_PCI_PCIE is not set
CONFIG_PCIEAER=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_MULTITHREAD_PROBE is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
CONFIG_PCCARD=y
# CONFIG_PCMCIA_DEBUG is not set
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_PCMCIA_IOCTL=y
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
# CONFIG_PD6729 is not set
# CONFIG_I82092 is not set
# CONFIG_I82365 is not set
# CONFIG_TCIC is not set
CONFIG_PCMCIA_PROBE=y
CONFIG_PCCARD_NONSTATIC=y

#
# PCI Hotplug Support
#
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_FAKE is not set
# CONFIG_HOTPLUG_PCI_COMPAQ is not set
CONFIG_HOTPLUG_PCI_IBM=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=y
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_MISC=y

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
# CONFIG_NETDEBUG is not set
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=y
# CONFIG_TCP_CONG_HTCP is not set
CONFIG_TCP_CONG_HSTCP=y
# CONFIG_TCP_CONG_HYBLA is not set
CONFIG_TCP_CONG_VEGAS=y
# CONFIG_TCP_CONG_SCALABLE is not set
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
# CONFIG_DEFAULT_BIC is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_VEGAS is not set
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set

#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
CONFIG_IPV6=y
# CONFIG_IPV6_PRIVACY is not set CONFIG_IPV6_ROUTER_PREF=y CONFIG_IPV6_ROUTE_INFO=y
# CONFIG_INET6_AH is not set
# CONFIG_INET6_ESP is not set
# CONFIG_INET6_IPCOMP is not set
# CONFIG_IPV6_MIP6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set CONFIG_INET6_TUNNEL=y CONFIG_INET6_XFRM_MODE_TRANSPORT=y CONFIG_INET6_XFRM_MODE_TUNNEL=y CONFIG_INET6_XFRM_MODE_BEET=y
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set CONFIG_IPV6_SIT=y
CONFIG_IPV6_TUNNEL=y
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_NETWORK_SECMARK is not set CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
# CONFIG_NF_CONNTRACK_ENABLED is not set
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
# CONFIG_NETFILTER_XT_TARGET_DSCP is not set
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
# CONFIG_NETFILTER_XT_MATCH_DCCP is not set
# CONFIG_NETFILTER_XT_MATCH_DSCP is not set
# CONFIG_NETFILTER_XT_MATCH_ESP is not set
# CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
# CONFIG_NETFILTER_XT_MATCH_POLICY is not set
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
# CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
CONFIG_NETFILTER_XT_MATCH_REALM=y
# CONFIG_NETFILTER_XT_MATCH_SCTP is not set
# CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
# CONFIG_NETFILTER_XT_MATCH_STRING is not set
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
# CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set

#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_TOS=y
# CONFIG_IP_NF_MATCH_RECENT is not set
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_AH=y
# CONFIG_IP_NF_MATCH_TTL is not set
CONFIG_IP_NF_MATCH_OWNER=y
CONFIG_IP_NF_MATCH_ADDRTYPE=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
# CONFIG_IP_NF_TARGET_ULOG is not set
CONFIG_IP_NF_TARGET_TCPMSS=y
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=y
# CONFIG_IP_NF_TARGET_TTL is not set
# CONFIG_IP_NF_RAW is not set
# CONFIG_IP_NF_ARPTABLES is not set

#
# IPv6: Netfilter Configuration (EXPERIMENTAL)
#
CONFIG_IP6_NF_QUEUE=y
# CONFIG_IP6_NF_IPTABLES is not set

#
# Bridge: Netfilter Configuration
#
# CONFIG_BRIDGE_NF_EBTABLES is not set

#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set

#
# TIPC Configuration (EXPERIMENTAL)
#
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
CONFIG_BRIDGE=y
CONFIG_VLAN_8021Q=y
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_FIFO=y
# CONFIG_NET_SCH_CLK_JIFFIES is not set
# CONFIG_NET_SCH_CLK_GETTIMEOFDAY is not set
CONFIG_NET_SCH_CLK_CPU=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=y
CONFIG_NET_SCH_HTB=y
# CONFIG_NET_SCH_HFSC is not set
CONFIG_NET_SCH_PRIO=y
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_SFQ=y
# CONFIG_NET_SCH_TEQL is not set
CONFIG_NET_SCH_TBF=y
CONFIG_NET_SCH_GRED=y
CONFIG_NET_SCH_DSMARK=y
CONFIG_NET_SCH_NETEM=y
CONFIG_NET_SCH_INGRESS=y

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
# CONFIG_NET_CLS_FW is not set
CONFIG_NET_CLS_U32=y
# CONFIG_CLS_U32_PERF is not set
# CONFIG_CLS_U32_MARK is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_EMATCH is not set
# CONFIG_NET_CLS_ACT is not set
# CONFIG_NET_CLS_POLICE is not set
# CONFIG_NET_CLS_IND is not set
# CONFIG_NET_ESTIMATOR is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
CONFIG_BT=y
CONFIG_BT_L2CAP=y
CONFIG_BT_SCO=y
CONFIG_BT_RFCOMM=y
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=y
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=y

#
# Bluetooth device drivers
#
CONFIG_BT_HCIUSB=m
CONFIG_BT_HCIUSB_SCO=y
# CONFIG_BT_HCIUART is not set
# CONFIG_BT_HCIBCM203X is not set
# CONFIG_BT_HCIBPA10X is not set
# CONFIG_BT_HCIBFUSB is not set
# CONFIG_BT_HCIDTL1 is not set
# CONFIG_BT_HCIBT3C is not set
# CONFIG_BT_HCIBLUECARD is not set
# CONFIG_BT_HCIBTUART is not set
# CONFIG_BT_HCIVHCI is not set
CONFIG_IEEE80211=y
# CONFIG_IEEE80211_DEBUG is not set
CONFIG_IEEE80211_CRYPT_WEP=y
CONFIG_IEEE80211_CRYPT_CCMP=y
CONFIG_IEEE80211_CRYPT_TKIP=y
CONFIG_IEEE80211_SOFTMAC=y
# CONFIG_IEEE80211_SOFTMAC_DEBUG is not set
CONFIG_WIRELESS_EXT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_SYS_HYPERVISOR is not set

#
# Connector - unified userspace <-> kernelspace linker #
CONFIG_CONNECTOR=y
# CONFIG_PROC_EVENTS is not set

#
# Memory Technology Devices (MTD)
#
CONFIG_MTD=m
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_CONCAT is not set
CONFIG_MTD_PARTITIONS=y
# CONFIG_MTD_REDBOOT_PARTS is not set

#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=m
CONFIG_MTD_BLOCK=m
# CONFIG_MTD_BLOCK_RO is not set
CONFIG_FTL=m
CONFIG_NFTL=m
# CONFIG_NFTL_RW is not set
CONFIG_INFTL=m
CONFIG_RFD_FTL=m
# CONFIG_SSFDC is not set

#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=m
CONFIG_MTD_JEDECPROBE=m
CONFIG_MTD_GEN_PROBE=m
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=m
CONFIG_MTD_CFI_AMDSTD=m
CONFIG_MTD_CFI_STAA=m
CONFIG_MTD_CFI_UTIL=m
CONFIG_MTD_RAM=m
CONFIG_MTD_ROM=m
# CONFIG_MTD_ABSENT is not set
# CONFIG_MTD_OBSOLETE_CHIPS is not set

#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
# CONFIG_MTD_PHYSMAP is not set
# CONFIG_MTD_PNC2000 is not set
# CONFIG_MTD_NETSC520 is not set
# CONFIG_MTD_TS5500 is not set
# CONFIG_MTD_SBC_GXX is not set
# CONFIG_MTD_AMD76XROM is not set
# CONFIG_MTD_ICHXROM is not set
# CONFIG_MTD_SCB2_FLASH is not set
# CONFIG_MTD_NETtel is not set
# CONFIG_MTD_L440GX is not set
# CONFIG_MTD_PCI is not set
# CONFIG_MTD_PLATRAM is not set

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
CONFIG_MTD_BLOCK2MTD=m

#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set

#
# NAND Flash Device Drivers
#
CONFIG_MTD_NAND=m
# CONFIG_MTD_NAND_VERIFY_WRITE is not set
# CONFIG_MTD_NAND_ECC_SMC is not set
CONFIG_MTD_NAND_IDS=m
# CONFIG_MTD_NAND_DISKONCHIP is not set
# CONFIG_MTD_NAND_CS553X is not set
# CONFIG_MTD_NAND_NANDSIM is not set

#
# OneNAND Flash Device Drivers
#
# CONFIG_MTD_ONENAND is not set

#
# Parallel port support
#
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
# CONFIG_PARPORT_SERIAL is not set
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_PC_PCMCIA is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
# CONFIG_PARPORT_1284 is not set

#
# Plug and Play support
#
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
# CONFIG_ISAPNP is not set
# CONFIG_PNPBIOS is not set
CONFIG_PNPACPI=y

#
# Block devices
#
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_NBD=y
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_MSI_LAPTOP is not set

#
# ATA/ATAPI/MFM/RLL support
#
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set

#
# SCSI low-level drivers
#
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set

#
# PCMCIA SCSI adapter support
#
# CONFIG_PCMCIA_AHA152X is not set
# CONFIG_PCMCIA_FDOMAIN is not set
# CONFIG_PCMCIA_NINJA_SCSI is not set
# CONFIG_PCMCIA_QLOGIC is not set
# CONFIG_PCMCIA_SYM53C500 is not set

#
# Serial ATA (prod) and Parallel ATA (experimental) drivers #
CONFIG_ATA=y
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIL24 is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5535 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_LEGACY is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PCMCIA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_QDI is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_WINBOND_VLB is not set

#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
# CONFIG_BLK_DEV_MD is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=y
CONFIG_DM_SNAPSHOT=y
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_IEEE1394=y

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_IEEE1394_OUI_DB is not set
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
CONFIG_IEEE1394_CONFIG_ROM_IP1394=y
# CONFIG_IEEE1394_EXPORT_FULL_API is not set

#
# Device Drivers
#
# CONFIG_IEEE1394_PCILYNX is not set
CONFIG_IEEE1394_OHCI1394=m

#
# Protocol Drivers
#
# CONFIG_IEEE1394_VIDEO1394 is not set
CONFIG_IEEE1394_SBP2=y
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394=y
# CONFIG_IEEE1394_DV1394 is not set
CONFIG_IEEE1394_RAWIO=y

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Network device support
#
CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
CONFIG_BONDING=y
# CONFIG_EQUALIZER is not set
CONFIG_TUN=y
# CONFIG_NET_SB1000 is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# PHY device support
#
# CONFIG_PHYLIB is not set

#
# Ethernet (10 or 100Mbit)
#
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set

#
# Tulip family network device support
#
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=y
# CONFIG_PCNET32_NAPI is not set
CONFIG_AMD8111_ETH=y
CONFIG_AMD8111E_NAPI=y
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_CS89x0 is not set
# CONFIG_DGRS is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=y
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_NET_POCKET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_TIGON3=y
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not set
CONFIG_S2IO=m
# CONFIG_S2IO_NAPI is not set
# CONFIG_MYRI10GE is not set
# CONFIG_NETXEN_NIC is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
CONFIG_NET_RADIO=y
CONFIG_NET_WIRELESS_RTNETLINK=y

#
# Obsolete Wireless cards support (pre-802.11)
#
# CONFIG_STRIP is not set
# CONFIG_ARLAN is not set
# CONFIG_WAVELAN is not set
# CONFIG_PCMCIA_WAVELAN is not set
# CONFIG_PCMCIA_NETWAVE is not set

#
# Wireless 802.11 Frequency Hopping cards support
#
# CONFIG_PCMCIA_RAYCS is not set

#
# Wireless 802.11b ISA/PCI cards support
#
# CONFIG_IPW2100 is not set
CONFIG_IPW2200=m
CONFIG_IPW2200_MONITOR=y
CONFIG_IPW2200_RADIOTAP=y
CONFIG_IPW2200_PROMISCUOUS=y
CONFIG_IPW2200_QOS=y
# CONFIG_IPW2200_DEBUG is not set
# CONFIG_AIRO is not set
# CONFIG_HERMES is not set
# CONFIG_ATMEL is not set

#
# Wireless 802.11b Pcmcia/Cardbus cards support
#
# CONFIG_AIRO_CS is not set
# CONFIG_PCMCIA_WL3501 is not set

#
# Prism GT/Duette 802.11(a/b/g) PCI/Cardbus support #
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set CONFIG_HOSTAP=m
CONFIG_HOSTAP_FIRMWARE=y CONFIG_HOSTAP_FIRMWARE_NVRAM=y
# CONFIG_HOSTAP_PLX is not set
# CONFIG_HOSTAP_PCI is not set CONFIG_HOSTAP_CS=m
# CONFIG_BCM43XX is not set CONFIG_ZD1211RW=m
# CONFIG_ZD1211RW_DEBUG is not set CONFIG_NET_WIRELESS=y

#
# PCMCIA network device support
#
# CONFIG_NET_PCMCIA is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=y
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
# CONFIG_PPP_MPPE is not set
CONFIG_PPPOE=y
# CONFIG_SLIP is not set
CONFIG_SLHC=y
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
CONFIG_NETCONSOLE=y
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_RX is not set
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_PCSPKR is not set
# CONFIG_INPUT_WISTRON_BTNS is not set
CONFIG_INPUT_UINPUT=m

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
# CONFIG_SERIAL_8250_CS is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
# CONFIG_TIPAR is not set

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_INTEL=y
CONFIG_HW_RANDOM_AMD=y
CONFIG_HW_RANDOM_GEODE=y
CONFIG_HW_RANDOM_VIA=y
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
CONFIG_AGP_INTEL=y
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=m
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set

#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
# CONFIG_CARDMAN_4000 is not set
# CONFIG_CARDMAN_4040 is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_HPET_MMAP=y
# CONFIG_HANGCHECK_TIMER is not set

#
# TPM devices
#
CONFIG_TCG_TPM=y
CONFIG_TCG_TIS=y
CONFIG_TCG_NSC=y
CONFIG_TCG_ATMEL=y
CONFIG_TCG_INFINEON=y
# CONFIG_TELCLOCK is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=y
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_ELEKTOR is not set
CONFIG_I2C_I801=y
CONFIG_I2C_I810=y
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_DS1374 is not set
CONFIG_SENSORS_EEPROM=m
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# SPI support
#
# CONFIG_SPI is not set
# CONFIG_SPI_MASTER is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Hardware Monitoring support
#
CONFIG_HWMON=y
# CONFIG_HWMON_VID is not set
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_K8TEMP is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
CONFIG_SENSORS_HDAPS=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set
# CONFIG_USB_DABUSB is not set

#
# Graphics support
#
CONFIG_FIRMWARE_EDID=y
CONFIG_FB=m
CONFIG_FB_DDC=m
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
CONFIG_FB_RADEON_DEBUG=y
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
CONFIG_FONTS=y
# CONFIG_FONT_8x8 is not set
# CONFIG_FONT_8x16 is not set
# CONFIG_FONT_6x11 is not set
# CONFIG_FONT_7x14 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
# CONFIG_FONT_MINI_4x6 is not set
# CONFIG_FONT_SUN8x16 is not set
CONFIG_FONT_SUN12x22=y
# CONFIG_FONT_10x18 is not set

#
# Logo configuration
#
# CONFIG_LOGO is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_DEVICE=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_DEVICE=y

#
# Sound
#
CONFIG_SOUND=y

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
# CONFIG_SND_SEQUENCER is not set
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
# CONFIG_SND_PCM_OSS_PLUGINS is not set
CONFIG_SND_RTCTIMER=y
# CONFIG_SND_DYNAMIC_MINORS is not set
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_AC97_CODEC=y
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set

#
# ISA devices
#
# CONFIG_SND_ADLIB is not set
# CONFIG_SND_AD1816A is not set
# CONFIG_SND_AD1848 is not set
# CONFIG_SND_ALS100 is not set
# CONFIG_SND_AZT2320 is not set
# CONFIG_SND_CMI8330 is not set
# CONFIG_SND_CS4231 is not set
# CONFIG_SND_CS4232 is not set
# CONFIG_SND_CS4236 is not set
# CONFIG_SND_DT019X is not set
# CONFIG_SND_ES968 is not set
# CONFIG_SND_ES1688 is not set
# CONFIG_SND_ES18XX is not set
# CONFIG_SND_GUSCLASSIC is not set
# CONFIG_SND_GUSEXTREME is not set
# CONFIG_SND_GUSMAX is not set
# CONFIG_SND_INTERWAVE is not set
# CONFIG_SND_INTERWAVE_STB is not set
# CONFIG_SND_OPL3SA2 is not set
# CONFIG_SND_OPTI92X_AD1848 is not set
# CONFIG_SND_OPTI92X_CS4231 is not set
# CONFIG_SND_OPTI93X is not set
# CONFIG_SND_MIRO is not set
# CONFIG_SND_SB8 is not set
# CONFIG_SND_SB16 is not set
# CONFIG_SND_SBAWE is not set
# CONFIG_SND_SGALAXY is not set
# CONFIG_SND_SSCAPE is not set
# CONFIG_SND_WAVEFRONT is not set

#
# PCI devices
#
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=y
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=y
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_AC97_POWER_SAVE=y

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# PCMCIA devices
#
# CONFIG_SND_VXPOCKET is not set
# CONFIG_SND_PDAUDIOCF is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y

#
# HID Devices
#
CONFIG_HID=y

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
CONFIG_USB_MULTITHREAD_PROBE=y
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' #

#

# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_KARMA is not set
CONFIG_USB_LIBUSUAL=y

#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_USB_HID_POWERBOOK is not set
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_ACECAD is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_TOUCHSCREEN is not set
# CONFIG_USB_YEALINK is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set
# CONFIG_USB_ATI_REMOTE2 is not set
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
CONFIG_USB_USBNET_MII=y
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=y
CONFIG_USB_NET_CDCETHER=m
# CONFIG_USB_NET_GL620A is not set
CONFIG_USB_NET_NET1080=m
# CONFIG_USB_NET_PLUSB is not set
# CONFIG_USB_NET_MCS7830 is not set
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
# CONFIG_USB_ALI_M5632 is not set
# CONFIG_USB_AN2720 is not set
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
# CONFIG_USB_EPSON2888 is not set
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_MON=y

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=y
# CONFIG_USB_SERIAL_CONSOLE is not set
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_AIRPRIME is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP2101 is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
CONFIG_USB_SERIAL_PL2303=y
CONFIG_USB_SERIAL_HP4X=y
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_DEBUG is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_TEST is not set

#
# USB DSL modem support
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# LED devices
#
# CONFIG_NEW_LEDS is not set

#
# LED drivers
#

#
# LED Triggers
#

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# EDAC - error detection and reporting (RAS) (EXPERIMENTAL) #
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
# CONFIG_EDAC_AMD76X is not set
# CONFIG_EDAC_E7XXX is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82875P is not set
# CONFIG_EDAC_I82860 is not set
# CONFIG_EDAC_R82600 is not set
CONFIG_EDAC_POLL=y

#
# Real Time Clock
#
# CONFIG_RTC_CLASS is not set

#
# DMA Engine support
#
# CONFIG_DMA_ENGINE is not set

#
# DMA Clients
#

#
# DMA Devices
#

#
# Virtualization
#
# CONFIG_KVM is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
# CONFIG_REISERFS_FS_XATTR is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_MINIX_FS=y
# CONFIG_ROMFS_FS is not set
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=y
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
CONFIG_CONFIGFS_FS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_JFFS2_FS=m
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_SUMMARY is not set
# CONFIG_JFFS2_FS_XATTR is not set
CONFIG_JFFS2_COMPRESSION_OPTIONS=y
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
# CONFIG_JFFS2_CMODE_NONE is not set
CONFIG_JFFS2_CMODE_PRIORITY=y
# CONFIG_JFFS2_CMODE_SIZE is not set
CONFIG_CRAMFS=m
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
# CONFIG_NFS_V4 is not set
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
# CONFIG_NFSD_V4 is not set
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
CONFIG_SMB_FS=y
# CONFIG_SMB_NLS_DEFAULT is not set
CONFIG_CIFS=y
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
CONFIG_NLS_CODEPAGE_932=y
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y

#
# Distributed Lock Manager
#
# CONFIG_DLM is not set

#
# Instrumentation Support
#
# CONFIG_PROFILING is not set
# CONFIG_KPROBES is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_RWSEMS is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_FRAME_POINTER is not set
CONFIG_FORCED_INLINING=y
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set

#
# Page alloc debug is incompatible with Software Suspend on i386 #
# CONFIG_DEBUG_RODATA is not set
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_MANAGER=y
# CONFIG_CRYPTO_HMAC is not set
# CONFIG_CRYPTO_XCBC is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_MD4 is not set
# CONFIG_CRYPTO_MD5 is not set
# CONFIG_CRYPTO_SHA1 is not set
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_GF128MUL is not set
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_586 is not set
# CONFIG_CRYPTO_SERPENT is not set
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_586 is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_TEA is not set
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_DEFLATE is not set
CONFIG_CRYPTO_MICHAEL_MIC=y
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_TEST is not set

#
# Hardware crypto devices
#
# CONFIG_CRYPTO_DEV_PADLOCK is not set
CONFIG_CRYPTO_DEV_GEODE=m

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_CRC_CCITT=y
# CONFIG_CRC16 is not set
CONFIG_CRC32=y
# CONFIG_LIBCRC32C is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_PLIST=y
CONFIG_IOMAP_COPY=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_KTIME_SCALAR=y

dmesg:

[    0.000000] Linux version 2.6.20-rc2 (ranma@navi) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #26 Mon Dec 25 14:00:08 CET 2006
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] sanitize start
[    0.000000] sanitize end
[    0.000000] copy_e820_map() start: 0000000000000000 size: 000000000009f000 end: 000000000009f000 type: 1
[    0.000000] copy_e820_map() type is E820_RAM
[    0.000000] copy_e820_map() start: 000000000009f000 size: 0000000000001000 end: 00000000000a0000 type: 2
[    0.000000] copy_e820_map() start: 00000000000dc000 size: 0000000000024000 end: 0000000000100000 type: 2
[    0.000000] copy_e820_map() start: 0000000000100000 size: 000000001fde0000 end: 000000001fee0000 type: 1
[    0.000000] copy_e820_map() type is E820_RAM
[    0.000000] copy_e820_map() start: 000000001fee0000 size: 0000000000015000 end: 000000001fef5000 type: 3
[    0.000000] copy_e820_map() start: 000000001fef5000 size: 000000000000b000 end: 000000001ff00000 type: 4
[    0.000000] copy_e820_map() start: 000000001ff00000 size: 0000000000100000 end: 0000000020000000 type: 2
[    0.000000] copy_e820_map() start: 00000000e0000000 size: 0000000010000000 end: 00000000f0000000 type: 2
[    0.000000] copy_e820_map() start: 00000000f0008000 size: 0000000000004000 end: 00000000f000c000 type: 2
[    0.000000] copy_e820_map() start: 00000000fec00000 size: 0000000000010000 end: 00000000fec10000 type: 2
[    0.000000] copy_e820_map() start: 00000000fed14000 size: 0000000000006000 end: 00000000fed1a000 type: 2
[    0.000000] copy_e820_map() start: 00000000fed20000 size: 0000000000070000 end: 00000000fed90000 type: 2
[    0.000000] copy_e820_map() start: 00000000fee00000 size: 0000000000001000 end: 00000000fee01000 type: 2
[    0.000000] copy_e820_map() start: 00000000ff000000 size: 0000000001000000 end: 0000000100000000 type: 2
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
[    0.000000]  BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 000000001fee0000 (usable)
[    0.000000]  BIOS-e820: 000000001fee0000 - 000000001fef5000 (ACPI data)
[    0.000000]  BIOS-e820: 000000001fef5000 - 000000001ff00000 (ACPI NVS)
[    0.000000]  BIOS-e820: 000000001ff00000 - 0000000020000000 (reserved)
[    0.000000]  BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
[    0.000000]  BIOS-e820: 00000000f0008000 - 00000000f000c000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
[    0.000000]  BIOS-e820: 00000000fed14000 - 00000000fed1a000 (reserved)
[    0.000000]  BIOS-e820: 00000000fed20000 - 00000000fed90000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
[    0.000000] 510MB LOWMEM available.
[    0.000000] Entering add_active_range(0, 0, 130784) 0 entries of 256 used
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA             0 ->     4096
[    0.000000]   Normal       4096 ->   130784
[    0.000000] early_node_map[1] active PFN ranges
[    0.000000]     0:        0 ->   130784
[    0.000000] On node 0 totalpages: 130784
[    0.000000]   DMA zone: 32 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 4064 pages, LIFO batch:0
[    0.000000]   Normal zone: 989 pages used for memmap
[    0.000000]   Normal zone: 125699 pages, LIFO batch:31
[    0.000000] DMI present.
[    0.000000] ACPI: RSDP (v002 IBM                                   ) @ 0x000f6bf0
[    0.000000] ACPI: XSDT (v001 IBM    TP-76    0x00001270  LTP 0x00000000) @ 0x1fee6f9b
[    0.000000] ACPI: FADT (v003 IBM    TP-76    0x00001270 IBM  0x00000001) @ 0x1fee7000
[    0.000000] ACPI: SSDT (v001 IBM    TP-76    0x00001270 MSFT 0x0100000e) @ 0x1fee71b4
[    0.000000] ACPI: ECDT (v001 IBM    TP-76    0x00001270 IBM  0x00000001) @ 0x1fef4d46
[    0.000000] ACPI: TCPA (v001 IBM    TP-76    0x00001270 PTL  0x00000001) @ 0x1fef4d98
[    0.000000] ACPI: MADT (v001 IBM    TP-76    0x00001270 IBM  0x00000001) @ 0x1fef4dca
[    0.000000] ACPI: MCFG (v001 IBM    TP-76    0x00001270 IBM  0x00000001) @ 0x1fef4e24
[    0.000000] ACPI: BOOT (v001 IBM    TP-76    0x00001270  LTP 0x00000001) @ 0x1fef4fd8
[    0.000000] ACPI: DSDT (v001 IBM    TP-76    0x00001270 MSFT 0x0100000e) @ 0x00000000
[    0.000000] ACPI: PM-Timer IO Port: 0x1008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] Processor #0 6:13 APIC version 20
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Enabling APIC mode:  Flat.  Using 1 I/O APICs
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] Allocating PCI resources starting at 30000000 (gap: 20000000:c0000000)
[    0.000000] Detected 1995.186 MHz processor.
[    2.815181] Built 1 zonelists.  Total pages: 129763
[    2.815183] Kernel command line: root=/dev/sda6 resume=/dev/sda5 vga=ext parport=auto ide0=noprobe ide1=noprobe libata.atapi_enabled=1 ro
[    2.815401] mapped APIC to ffff9000 (fee00000)
[    2.815404] mapped IOAPIC to ffff8000 (fec00000)
[    2.815406] Enabling fast FPU save and restore... done.
[    2.815408] Enabling unmasked SIMD FPU exception support... done.
[    2.815416] Initializing CPU#0
[    2.815473] CPU 0 irqstacks, hard=c05f3000 soft=c05f2000
[    2.815476] PID hash table entries: 2048 (order: 11, 8192 bytes)
[    2.815491] is_hpet_capable()
[    2.815493] trying to force-enable HPET
[    2.815498] RCBA already mapped at f0008000
[    2.815501] HPTC: RCBA Base is 0xf0008000, mapped at 0xffffc000 to 0xfffff000
[    2.815505] HPTC: RCBA 0x3404 is 0x00000080n<3>Intel HPET force-enabled at 0xfed00000
[    2.817499] Console: colour VGA+ 80x50
[    2.821573] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    2.821816] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    2.831460] Memory: 512836k/523136k available (3392k kernel code, 9880k reserved, 1444k data, 200k init, 0k highmem)
[    2.831572] virtual kernel memory layout:
[    2.831573]     fixmap  : 0xfffb3000 - 0xfffff000   ( 304 kB)
[    2.831574]     vmalloc : 0xe0800000 - 0xfffb1000   ( 503 MB)
[    2.831575]     lowmem  : 0xc0000000 - 0xdfee0000   ( 510 MB)
[    2.831576]       .init : 0xc05bb000 - 0xc05ed000   ( 200 kB)
[    2.831577]       .data : 0xc0450065 - 0xc05b90b8   (1444 kB)
[    2.831579]       .text : 0xc0100000 - 0xc0450065   (3392 kB)
[    2.832061] Checking if this processor honours the WP bit even in supervisor mode... Ok.
[    2.832297] hpet_enable
[    2.832382] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    2.832602] hpet0: 3 64-bit timers, 14318180 Hz
[    2.833675] Using HPET for base-timer
[    2.915669] Calibrating delay using timer specific routine.. 3994.20 BogoMIPS (lpj=6654729)
[    2.915836] Mount-cache hash table entries: 512
[    2.915981] CPU: After generic identify, caps: afe9fbff 00100000 00000000 00000000 00000180 00000000 00000000
[    2.915990] CPU: L1 I cache: 32K, L1 D cache: 32K
[    2.916101] CPU: L2 cache: 2048K
[    2.916171] CPU: After all inits, caps: afe9fbff 00100000 00000000 00002040 00000180 00000000 00000000
[    2.916176] Intel machine check architecture supported.
[    2.916248] Intel machine check reporting enabled on CPU#0.
[    2.916320] Compat vDSO mapped to ffffa000.
[    2.916396] CPU: Intel(R) Pentium(R) M processor 2.00GHz stepping 08
[    2.916543] Checking 'hlt' instruction... OK.
[    2.929131] ACPI: Core revision 20060707
[    2.945497] ENABLING IO-APIC IRQs
[    2.945753] ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
[    3.082402] NET: Registered protocol family 16
[    3.082649] ACPI: ACPI Dock Station Driver 
[    3.082749] ACPI: bus type pci registered
[    3.082824] PCI: Using MMCONFIG
[    3.083561] Setting up standard PCI resources
[    3.093993] ACPI: Interpreter enabled
[    3.094065] ACPI: Using IOAPIC for interrupt routing
[    3.094750] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11)
[    3.095684] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11)
[    3.096606] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11)
[    3.097521] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11)
[    3.098438] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11)
[    3.099369] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11)
[    3.100284] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 *5 6 7 9 10 11)
[    3.101201] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11)
[    3.101926] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    3.102000] PCI: Probing PCI hardware (bus 00)
[    3.103425] HPTC: RCBA Base is 0xf0008000
[    3.103498] HPTC: RCBA 0x3404 is 0x80
[    3.103566] HPTC: HPTC enabled
[    3.103635] HPTC: HPET located at 0xfed00000
[    3.103707] PCI quirk: region 1000-107f claimed by ICH6 ACPI/GPIO/TCO
[    3.103779] PCI quirk: region 1180-11bf claimed by ICH6 GPIO
[    3.103989] Boot video device is 0000:01:00.0
[    3.104433] PCI: Transparent bridge - 0000:00:1e.0
[    3.104583] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    3.109110] ACPI: Power Resource [PUBS] (on)
[    3.110023] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT]
[    3.110275] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EXP0._PRT]
[    3.110438] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.EXP2._PRT]
[    3.110626] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
[    3.112300] Linux Plug and Play Support v0.97 (c) Adam Belay
[    3.112376] pnp: PnP ACPI init
[    3.115896] pnp: PnP ACPI: found 13 devices
[    3.115984] intel_rng: FWH not detected
[    3.116140] SCSI subsystem initialized
[    3.116225] libata version 2.00 loaded.
[    3.116258] usbcore: registered new interface driver usbfs
[    3.116349] usbcore: registered new interface driver hub
[    3.116441] usbcore: registered new device driver usb
[    3.116549] PCI: Using ACPI for IRQ routing
[    3.116621] PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
[    3.215491] Bluetooth: Core ver 2.11
[    3.215587] NET: Registered protocol family 31
[    3.215657] Bluetooth: HCI device and connection manager initialized
[    3.215728] Bluetooth: HCI socket layer initialized
[    3.216289] ieee1394: Initialized config rom entry `ip1394'
[    3.216345] PCI: Bridge: 0000:00:01.0
[    3.216417]   IO window: 3000-3fff
[    3.216487]   MEM window: b0100000-b01fffff
[    3.216557]   PREFETCH window: c0000000-c7ffffff
[    3.216626] PCI: Bridge: 0000:00:1c.0
[    3.216694]   IO window: disabled.
[    3.216766]   MEM window: b0200000-b02fffff
[    3.216835]   PREFETCH window: disabled.
[    3.216906] PCI: Bridge: 0000:00:1c.2
[    3.216976]   IO window: 4000-4fff
[    3.217047]   MEM window: b2000000-b3ffffff
[    3.217117]   PREFETCH window: c8000000-c80fffff
[    3.217190] PCI: Bus 12, cardbus bridge: 0000:0b:00.0
[    3.217260]   IO window: 00005000-000050ff
[    3.217331]   IO window: 00005400-000054ff
[    3.217403]   PREFETCH window: d0000000-d3ffffff
[    3.217474]   MEM window: b8000000-bbffffff
[    3.217545] PCI: Bridge: 0000:00:1e.0
[    3.217615]   IO window: 5000-8fff
[    3.217686]   MEM window: b4000000-bfffffff
[    3.217757]   PREFETCH window: d0000000-d7ffffff
[    3.217834] ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
[    3.217973] PCI: Setting latency timer of device 0000:00:01.0 to 64
[    3.217986] ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 20 (level, low) -> IRQ 17
[    3.218125] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[    3.218140] ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 22 (level, low) -> IRQ 18
[    3.218278] PCI: Setting latency timer of device 0000:00:1c.2 to 64
[    3.218287] PCI: Setting latency timer of device 0000:00:1e.0 to 64
[    3.218298] ACPI: PCI Interrupt 0000:0b:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[    3.218448] NET: Registered protocol family 2
[    3.248830] IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
[    3.248952] TCP established hash table entries: 16384 (order: 4, 65536 bytes)
[    3.249071] TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
[    3.249167] TCP: Hash tables configured (established 16384 bind 8192)
[    3.249238] TCP reno registered
[    3.258925] Simple Boot Flag at 0x35 set to 0x1
[    3.259018] Machine check exception polling timer started.
[    3.259287] Installing knfsd (copyright (C) 1996 [email protected]).
[    3.259478] NTFS driver 2.1.27 [Flags: R/W].
[    3.259606] io scheduler noop registered
[    3.259714] io scheduler anticipatory registered (default)
[    3.259858] io scheduler deadline registered
[    3.259969] io scheduler cfq registered
[    3.261652] PCI: Setting latency timer of device 0000:00:01.0 to 64
[    3.261667] assign_interrupt_mode Found MSI capability
[    3.261754] Allocate Port Service[0000:00:01.0:pcie00]
[    3.261774] Allocate Port Service[0000:00:01.0:pcie03]
[    3.261816] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[    3.261852] assign_interrupt_mode Found MSI capability
[    3.261949] Allocate Port Service[0000:00:1c.0:pcie00]
[    3.261967] Allocate Port Service[0000:00:1c.0:pcie02]
[    3.261987] Allocate Port Service[0000:00:1c.0:pcie03]
[    3.262059] PCI: Setting latency timer of device 0000:00:1c.2 to 64
[    3.262095] assign_interrupt_mode Found MSI capability
[    3.262197] Allocate Port Service[0000:00:1c.2:pcie00]
[    3.262215] Allocate Port Service[0000:00:1c.2:pcie02]
[    3.262233] Allocate Port Service[0000:00:1c.2:pcie03]
[    3.262314] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    3.262387] ibmphpd: IBM Hot Plug PCI Controller Driver version: 0.6
[    3.262462] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    3.265962] decode_hpp: Could not get hotplug parameters. Use defaults
[    3.266059] acpiphp: Slot [1] registered
[    3.267122] acpiphp_ibm: ibm_find_acpi_device:  Failed to get device information<3>acpiphp_ibm: ibm_find_acpi_device:  Failed to get device information<3>acpiphp_ibm: ibm_find_acpi_device:  Failed to get device information<3>acpiphp_ibm: ibm_acpiphp_init: acpi_walk_namespace failed
[    3.269969] ACPI: AC Adapter [AC] (on-line)
[    3.278158] ACPI: Battery Slot [BAT0] (battery present)
[    3.278284] input: Power Button (FF) as /class/input/input0
[    3.278358] ACPI: Power Button (FF) [PWRF]
[    3.278459] input: Lid Switch as /class/input/input1
[    3.278532] ACPI: Lid Switch [LID]
[    3.278632] input: Sleep Button (CM) as /class/input/input2
[    3.278706] ACPI: Sleep Button (CM) [SLPB]
[    3.278995] ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
[    3.280635] ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
[    3.280857] ACPI: Processor [CPU] (supports 8 throttling states)
[    3.281966] ACPI: Thermal Zone [THM0] (63 C)
[    3.283336] Real Time Clock Driver v1.12ac
[    3.283432] Linux agpgart interface v0.101 (c) Dave Jones
[    3.283522] agpgart: Detected an Intel 915GM Chipset.
[    3.300594] agpgart: AGP aperture is 256M @ 0x0
[    3.300695] [drm] Initialized drm 1.1.0 20060810
[    3.300877] tpm_nsc tpm_nscl0: NSC TPM revision 2
[    3.301002] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
[    3.301249] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a NS16550A
[    3.302001] pnp: Device 00:09 activated.
[    3.302186] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A
[    3.302352] ACPI: PCI Interrupt 0000:00:1e.3[B] -> GSI 23 (level, low) -> IRQ 19
[    3.302495] ACPI: PCI interrupt for device 0000:00:1e.3 disabled
[    3.302599] parport: PnPBIOS parport detected.
[    3.302704] parport0: PC-style at 0x3bc (0x7bc), irq 7 [PCSPP(,...)]
[    3.303238] loop: loaded (max 8 devices)
[    3.303352] nbd: registered device at major 43
[    3.303761] Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006)
[    3.303837] bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details.
[    3.304025] pcnet32.c:v1.33 27.Jun.2006 [email protected]
[    3.304115] e100: Intel(R) PRO/100 Network Driver, 3.5.17-k2-NAPI
[    3.304185] e100: Copyright(c) 1999-2006 Intel Corporation
[    3.304292] tg3.c:v3.71 (December 15, 2006)
[    3.304377] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[    3.304520] PCI: Setting latency timer of device 0000:02:00.0 to 64
[    0.399999] eth0: Tigon3 [partno(BCM95751M) rev 4101 PHY(5750)] (PCI Express) 10/100/1000Base-T Ethernet 00:0a:e4:c1:27:01
[    0.399999] eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[    0.399999] eth0: dma_rwctrl[76180000] dma_mask[64-bit]
[    0.399999] PPP generic driver version 2.4.2
[    0.399999] PPP Deflate Compression module registered
[    0.399999] PPP BSD Compression module registered
[    0.403333] NET: Registered protocol family 24
[    0.403333] tun: Universal TUN/TAP device driver, 1.6
[    0.403333] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    0.403333] netconsole: not configured, aborting
[    0.403333] ahci 0000:00:1f.2: version 2.0
[    0.403333] ahci: probe of 0000:00:1f.2 failed with error -12
[    0.403333] ata_piix 0000:00:1f.2: version 2.00ac7
[    0.403333] ata_piix 0000:00:1f.2: MAP [ P0 P2 IDE IDE ]
[    0.403333] PCI: Setting latency timer of device 0000:00:1f.2 to 64
[    0.403333] ata1: SATA max UDMA/133 cmd 0x1F0 ctl 0x3F6 bmdma 0x18C0 irq 14
[    0.403333] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x18C8 irq 15
[    0.403333] scsi0 : ata_piix
[    0.563333] ata1.00: ATA-6, max UDMA/100, 195371568 sectors: LBA 
[    0.563333] ata1.00: ata1: dev 0 multi count 16
[    0.563333] ata1.00: applying bridge limits
[    0.573333] ata1.00: configured for UDMA/100
[    0.573333] scsi1 : ata_piix
[    0.886666] ata2.00: ATAPI, max UDMA/33
[    1.046666] ata2.00: configured for UDMA/33
[    1.046666] scsi 0:0:0:0: Direct-Access     ATA      FUJITSU MHV2100A 0084 PQ: 0 ANSI: 5
[    1.046666] SCSI device sda: 195371568 512-byte hdwr sectors (100030 MB)
[    1.046666] sda: Write Protect is off
[    1.046666] sda: Mode Sense: 00 3a 00 00
[    1.046666] SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.046666] SCSI device sda: 195371568 512-byte hdwr sectors (100030 MB)
[    1.046666] sda: Write Protect is off
[    1.046666] sda: Mode Sense: 00 3a 00 00
[    1.046666] SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.046666]  sda: sda1 sda2 sda3 < sda5 sda6 sda7 > sda4
[    1.113333] sd 0:0:0:0: Attached scsi disk sda
[    1.113333] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    1.116666] scsi 1:0:0:0: CD-ROM            MATSHITA DVD-RAM UJ-830S  1.02 PQ: 0 ANSI: 5
[    1.123333] sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
[    1.123333] Uniform CD-ROM driver Revision: 3.20
[    1.123333] sr 1:0:0:0: Attached scsi CD-ROM sr0
[    1.123333] sr 1:0:0:0: Attached scsi generic sg1 type 5
[    1.123333] ieee1394: raw1394: /dev/raw1394 device initialized
[    1.123333] Yenta: CardBus bridge found at 0000:0b:00.0 [1014:0532]
[    1.249999] Yenta: ISA IRQ mask 0x0438, PCI irq 16
[    1.249999] Socket status: 30000006
[    1.249999] pcmcia: parent PCI bridge I/O window: 0x5000 - 0x8fff
[    1.249999] cs: IO port probe 0x5000-0x8fff: clean.
[    1.249999] pcmcia: parent PCI bridge Memory window: 0xb4000000 - 0xbfffffff
[    1.249999] pcmcia: parent PCI bridge Memory window: 0xd0000000 - 0xd7ffffff
[    1.503333] usbcore: registered new interface driver usblp
[    1.503333] drivers/usb/class/usblp.c: v0.13: USB Printer Device Class driver
[    1.503333] usbcore: registered new interface driver libusual
[    1.503333] usbcore: registered new interface driver usbhid
[    1.503333] drivers/usb/input/hid-core.c: v2.6:USB HID core driver
[    1.503333] usbcore: registered new interface driver asix
[    1.503333] usbcore: registered new interface driver usbserial
[    1.503333] drivers/usb/serial/usb-serial.c: USB Serial support registered for generic
[    1.503333] usbcore: registered new interface driver usbserial_generic
[    1.503333] drivers/usb/serial/usb-serial.c: USB Serial Driver core
[    1.503333] drivers/usb/serial/usb-serial.c: USB Serial support registered for hp4X
[    1.503333] usbcore: registered new interface driver hp4X
[    1.503333] drivers/usb/serial/hp4x.c: HP4x (48/49) Generic Serial driver v1.00
[    1.503333] drivers/usb/serial/usb-serial.c: USB Serial support registered for pl2303
[    1.503333] usbcore: registered new interface driver pl2303
[    1.503333] drivers/usb/serial/pl2303.c: Prolific PL2303 USB to serial adaptor driver
[    1.503333] PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    1.509999] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.509999] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.509999] mice: PS/2 mouse device common for all mice
[    1.513333] input: AT Translated Set 2 keyboard as /class/input/input3
[    1.519999] i2c /dev entries driver
[    1.519999] ACPI: PCI Interrupt 0000:00:1f.3[A] -> GSI 23 (level, low) -> IRQ 19
[    1.519999] device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: [email protected]
[    1.519999] EDAC MC: Ver: 2.0.1 Dec 25 2006
[    1.549999] Advanced Linux Sound Architecture Driver Version 1.0.14rc1 (Wed Dec 20 08:11:48 2006 UTC).
[    1.549999] ACPI: PCI Interrupt 0000:00:1e.2[A] -> GSI 22 (level, low) -> IRQ 18
[    1.549999] PCI: Setting latency timer of device 0000:00:1e.2 to 64
[    1.816666] ACPI: EC: evaluating _Q75
[    2.133333] Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0
[    2.133333] serio: Synaptics pass-through port at isa0060/serio1/input0
[    2.176666] input: SynPS/2 Synaptics TouchPad as /class/input/input4
[    2.473333] intel8x0_measure_ac97_clock: measured 53330 usecs
[    2.473333] intel8x0: clocking to 48000
[    2.473333] ALSA device list:
[    2.473333]   #0: Intel ICH6 with AD1981B at 0xb0000800, irq 18
[    2.473333] netem: version 1.2
[    2.473333] u32 classifier
[    2.473333] Netfilter messages via NETLINK v0.30.
[    2.473333] ip_tables: (C) 2000-2006 Netfilter Core Team
[    2.553333] TCP bic registered
[    2.553333] TCP cubic registered
[    2.553333] TCP westwood registered
[    2.553333] TCP highspeed registered
[    2.553333] TCP vegas registered
[    2.553333] NET: Registered protocol family 1
[    2.553333] NET: Registered protocol family 10
[    2.553333] IPv6 over IPv4 tunneling driver
[    2.553333] NET: Registered protocol family 17
[    2.633333] Bridge firewalling registered
[    2.633333] Bluetooth: L2CAP ver 2.8
[    2.633333] Bluetooth: L2CAP socket layer initialized
[    2.633333] Bluetooth: SCO (Voice Link) ver 0.5
[    2.633333] Bluetooth: SCO socket layer initialized
[    2.633333] Bluetooth: RFCOMM socket layer initialized
[    2.633333] Bluetooth: RFCOMM TTY layer initialized
[    2.633333] Bluetooth: RFCOMM ver 1.8
[    2.633333] Bluetooth: BNEP (Ethernet Emulation) ver 1.2
[    2.633333] Bluetooth: BNEP filters: protocol multicast
[    2.633333] Bluetooth: HIDP (Human Interface Emulation) ver 1.1
[    2.633333] 802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
[    2.633333] All bugs added by David S. Miller <[email protected]>
[    2.633333] ieee80211: 802.11 data/management/control stack, git-1.1.13
[    2.633333] ieee80211: Copyright (C) 2004-2005 Intel Corporation <[email protected]>
[    2.633333] ieee80211_crypt: registered algorithm 'NULL'
[    2.633333] ieee80211_crypt: registered algorithm 'WEP'
[    2.633333] ieee80211_crypt: registered algorithm 'CCMP'
[    2.633333] ieee80211_crypt: registered algorithm 'TKIP'
[    2.633333] speedstep-centrino with X86_SPEEDSTEP_CENTRINO_ACPIconfig is deprecated.
[    2.633333]  Use X86_ACPI_CPUFREQ (acpi-cpufreq instead.
[    2.633333] Using IPI Shortcut mode
[    2.633333] ACPI: (supports S0 S3 S4 S5)
[    2.636666] Time: tsc clocksource has been installed.
[    2.643333] Time: hpet clocksource has been installed.
[    7.439999] IBM TrackPoint firmware: 0x0e, buttons: 3/3
[    7.696665] input: TPPS/2 IBM TrackPoint as /class/input/input5
[    7.703332] ACPI: EC: evaluating _Q75
[    7.879999] kjournald starting.  Commit interval 5 seconds
[    7.879999] EXT3-fs: mounted filesystem with ordered data mode.
[    7.879999] VFS: Mounted root (ext3 filesystem) readonly.
[    7.879999] Freeing unused kernel memory: 200k freed
[   11.453332] ACPI: PCI Interrupt 0000:0b:00.1[B] -> GSI 17 (level, low) -> IRQ 20
[   11.506665] ohci1394: fw-host0: OHCI-1394 1.0 (PCI): IRQ=[20]  MMIO=[b1000000-b10007ff]  Max Packet=[2048]  IR/IT contexts=[4/4]
[   11.516665] eth1394: eth0: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
[   11.679998] cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7
[   11.683332] cs: IO port probe 0x800-0x8ff: clean.
[   11.683332] cs: IO port probe 0xc00-0xcff: clean.
[   11.683332] cs: IO port probe 0xa00-0xaff: clean.
[   12.166665] Adding 1958000k swap on /dev/sda5.  Priority:10 extents:1 across:1958000k
[   12.319998] EXT3 FS on sda6, internal journal
[   12.693332] ibm_acpi: ThinkPad EC firmware 76HT16WW-1.06
[   12.693332] ibm_acpi: IBM ThinkPad ACPI Extras v0.13
[   12.693332] ibm_acpi: http://ibm-acpi.sf.net/
[   12.699998] ibm_acpi: fan_init: initial fan status is unknown, assuming it is in auto mode
[   12.783332] ieee1394: Host added: ID:BUS[0-00:1023]  GUID[000ae405314003e1]
[   13.063332] kjournald starting.  Commit interval 5 seconds
[   13.063332] EXT3-fs: mounted filesystem with ordered data mode.
[   13.066665] kjournald starting.  Commit interval 5 seconds
[   13.066665] EXT3 FS on sda7, internal journal
[   13.066665] EXT3-fs: mounted filesystem with ordered data mode.
[   13.493331] pcmcia: Detected deprecated PCMCIA ioctl usage from process: discover.
[   13.493331] pcmcia: This interface will soon be removed from the kernel; please expect breakage unless you upgrade to new tools.
[   13.493331] pcmcia: see http://www.kernel.org/pub/linux/utils/kernel/pcmcia/pcmcia.html for details.
[   15.979998] ieee1394: Node removed: ID:BUS[0-00:1023]  GUID[000ae405314003e1]
Signature · 1 line
Tobias						PGP: http://9ac7e0bc.uguu.de
maintainer
#188 lore raw

Re: [PATCH] mm: fix page_mkclean_one

From: Tobias Diedrich <[email protected]>
Date: Tue, 26 Dec 2006 17:17:00 +0100

Quoted 9 lines
Quoted 4 lines · Linus Torvalds

I don't think it's a page table issue any more, it just doesn't look likely with the ARM UP corruption. It's also not apparently even on a cacheline boundary, so it probably is really a dirty bit that got cleared wrogn due to some race with IO.

So, until now it's only been reported for SMP on i386? I'm seeing the issue on my Pentium-M Notebook (Thinkpad R52) over here, UP kernel, no preempt.

I've seen it on sparc64, UP kernel, no preempt.

#189 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 2 lines · David Miller

I've seen it on sparc64, UP kernel, no preempt.

Btw, having tried to debug the writeback code, there's one very special case that just makes me go "hmm".

If we have a buffer that is "busy" when we try to write back a page, we have this magic "wbc->sync_mode == WB_SYNC_NONE && wbc->nonblocking" mode, where we won't wait for it, but instead we'll redirty the page and redo the whole thing.

Looking at the code, that should all work, but at the same time, it triggers some of my debug messages about having a dirty page during writeback, and one way to trigger that debug message is to try to run rtorrent on the machine..

I dunno. Witht he writeback being suspicious, and the normal "block_write_full_page()" path being implicated in at least ext2, I just wonder. This is one of those "let's see if behaviour changes" patches, that I'm just throwing out there..

Linus

---

fs/buffer.c +1 -13
index 263f88e..4652ef1 100644
@@ -1653,19 +1653,7 @@ static int __block_write_full_page(struct inode *inode, struct page *page,
 	do {
 		if (!buffer_mapped(bh))
 			continue;
-		/*
-		 * If it's a fully non-blocking write attempt and we cannot
-		 * lock the buffer then redirty the page.  Note that this can
-		 * potentially cause a busy-wait loop from pdflush and kswapd
-		 * activity, but those code paths have their own higher-level
-		 * throttling.
-		 */
-		if (wbc->sync_mode != WB_SYNC_NONE || !wbc->nonblocking) {
-			lock_buffer(bh);
-		} else if (test_set_buffer_locked(bh)) {
-			redirty_page_for_writepage(wbc, page);
-			continue;
-		}
+		lock_buffer(bh);
 		if (test_clear_buffer_dirty(bh)) {
 			mark_buffer_async_write(bh);
 		} else {
#190 lore raw

Re: [PATCH] mm: fix page_mkclean_one

I have corrupted files...

Quoted 26 lines

---

fs/buffer.c +1 -13
index 263f88e..4652ef1 100644
@@ -1653,19 +1653,7 @@ static int __block_write_full_page(struct inode *inode, struct page *page,
 	do {
 		if (!buffer_mapped(bh))
 			continue;
-		/*
-		 * If it's a fully non-blocking write attempt and we cannot
-		 * lock the buffer then redirty the page.  Note that this can
-		 * potentially cause a busy-wait loop from pdflush and kswapd
-		 * activity, but those code paths have their own higher-level
-		 * throttling.
-		 */
-		if (wbc->sync_mode != WB_SYNC_NONE || !wbc->nonblocking) {
-			lock_buffer(bh);
-		} else if (test_set_buffer_locked(bh)) {
-			redirty_page_for_writepage(wbc, page);
-			continue;
-		}
+		lock_buffer(bh);
 		if (test_clear_buffer_dirty(bh)) {
 			mark_buffer_async_write(bh);
 		} else {
#191 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to David Miller
Quoted 2 lines · David Miller

I've seen it on sparc64, UP kernel, no preempt.

Ok, I still don't have a clue, but I think I at least have a new test-case.

It can probably be improved upon, but this would _seem_ to trigger the problem. Can people check?

You'd want to make sure you get page-put activity, by making TARGETSIZE be big enough to cause memory pressure (and rather than making it bigger, you might want to make your memory smaller instead, to make it run more quickly. Either using "mem=128M" or big compiles or something...).

If it finds corruption, you'll see something like

	Writing chunk 183858/183859 (99%)
	Chunk ..
	Chunk 120887 corrupted
	Chunk 122372 corrupted
	Chunk ...
	Checking chunk 183858/183859 (99%)

otherwise it will just say

	Writing chunk 183858/183859 (99%)
	Checking chunk 183858/183859 (99%)

and exit.

I didn't spend a lot of time verifying this, but I _was_ able to cause those "Chunk xxx corrupted" messages with this. There's probably a more efficient better way to do it, but this is better than trying to use rtorrent, and also makes any worries about what rtorrent does go away.

Of course, maybe it's this test-program that is buggy now, although it looks trivial enough that I don't think it is.

I think my earlier stress-tester may not have triggered this, because it just did all its writing in a linear order, so any LRU logic will happen to write back old pages that we are no longer touching. The randomization (and using a chunksize that isn't a multiple of a page-size) makes sure that we're actually going to have lots of rewriting going on.

I think the test-case could probably be improved by having a munmap() and page-cache flush in between the writing and the checking, to see whether that shows the corruption easier (and possibly without having to start paging in order to throw the pages out, which would simplify testing a lot). But I haven't tested. I decided to post this asap, now that I've recreated the corruption with something else, and something that is possibly easier to analyze..

Linus

----
#include <sys/mman.h>
#include <sys/fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <time.h>

#define TARGETSIZE (256 << 20)
#define CHUNKSIZE (1460)
#define NRCHUNKS (TARGETSIZE / CHUNKSIZE)
#define SIZE (NRCHUNKS * CHUNKSIZE)

static void fillmem(void *start, int nr)

{
	memset(start, nr, CHUNKSIZE);
}

static void checkmem(void *start, int nr)

{
	unsigned char c = nr, *p = start;
	int i;
	for (i = 0; i < CHUNKSIZE; i++) {
		if (*p++ != c) {
			printf("Chunk %d corrupted               \n", nr);
			return;
		}
	}
}

int main(int argc, char **argv)

{
	char *mapping;
	int fd, i;
	static int chunkorder[NRCHUNKS];

	/*
	 * Make some random ordering of writing the chunks to the
	 * memory map..
	 *
	 * Start with fully ordered..
	 */
	for (i = 0; i < NRCHUNKS; i++)
		chunkorder[i] = i;

	/* ..and then mix it up randomly */
	srandom(time(NULL));
	for (i = 0; i < NRCHUNKS; i++) {
		int index = (unsigned int) random() % NRCHUNKS;
		int nr = chunkorder[index];
		chunkorder[index] = chunkorder[i];
		chunkorder[i] = nr;
	}

	fd = open("mapfile", O_RDWR | O_TRUNC | O_CREAT, 0666);
	if (fd < 0)
		return -1;
	if (ftruncate(fd, SIZE) < 0)
		return -1;
	mapping = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
	if (-1 == (int)(long)mapping)
		return -1;

	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = chunkorder[i];
		printf("Writing chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		fillmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");
	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = i;
		printf("Checking chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		checkmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");
	
	return 0;
}
#192 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 6 lines · Linus Torvalds

I think the test-case could probably be improved by having a munmap() and page-cache flush in between the writing and the checking, to see whether that shows the corruption easier (and possibly without having to start paging in order to throw the pages out, which would simplify testing a lot).

I think the page-writeout is implicated, because I do seem to need it, but the page-cache flush does seem to make corruption _easier_ to see. I now seem about to trigger it with a 100MB file on a 256MB machine in a minute or so, with this slight modification.

I still don't see _why_, though. But maybe smarter people than me can see it..

Linus

---
#include <sys/mman.h>
#include <sys/fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <time.h>

#define TARGETSIZE (100 << 20)
#define CHUNKSIZE (1460)
#define NRCHUNKS (TARGETSIZE / CHUNKSIZE)
#define SIZE (NRCHUNKS * CHUNKSIZE)

static void fillmem(void *start, int nr)

{
	memset(start, nr, CHUNKSIZE);
}

static void checkmem(void *start, int nr)

{
	unsigned char c = nr, *p = start;
	int i;
	for (i = 0; i < CHUNKSIZE; i++) {
		if (*p++ != c) {
			printf("Chunk %d corrupted           \n", nr);
			return;
		}
	}
}

static char *remap(int fd, char *mapping)

{
	if (mapping) {
		munmap(mapping, SIZE);
		posix_fadvise(fd, 0, SIZE, POSIX_FADV_DONTNEED);
	}
	return mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
}

int main(int argc, char **argv)

{
	char *mapping;
	int fd, i;
	static int chunkorder[NRCHUNKS];

	/*
	 * Make some random ordering of writing the chunks to the
	 * memory map..
	 *
	 * Start with fully ordered..
	 */
	for (i = 0; i < NRCHUNKS; i++)
		chunkorder[i] = i;

	/* ..and then mix it up randomly */
	srandom(time(NULL));
	for (i = 0; i < NRCHUNKS; i++) {
		int index = (unsigned int) random() % NRCHUNKS;
		int nr = chunkorder[index];
		chunkorder[index] = chunkorder[i];
		chunkorder[i] = nr;
	}

	fd = open("mapfile", O_RDWR | O_TRUNC | O_CREAT, 0666);
	if (fd < 0)
		return -1;
	if (ftruncate(fd, SIZE) < 0)
		return -1;
	mapping = remap(fd, NULL);
	if (-1 == (int)(long)mapping)
		return -1;

	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = chunkorder[i];
		printf("Writing chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		fillmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");

	/* Unmap, drop, and remap.. */
	mapping = remap(fd, mapping);

	/* .. and check */
	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = i;
		printf("Checking chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		checkmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");
	
	return 0;
}
maintainer
#193 lore raw

Re: [PATCH] mm: fix page_mkclean_one

From: Linus Torvalds <[email protected]>
Date: Wed, 27 Dec 2006 16:39:43 -0800 (PST)

Quoted 10 lines · Linus Torvalds
Quoted 6 lines

I think the test-case could probably be improved by having a munmap() and page-cache flush in between the writing and the checking, to see whether that shows the corruption easier (and possibly without having to start paging in order to throw the pages out, which would simplify testing a lot).

Quoted 7 lines

I think the page-writeout is implicated, because I do seem to need it, but the page-cache flush does seem to make corruption _easier_ to see. I now seem about to trigger it with a 100MB file on a 256MB machine in a minute or so, with this slight modification.

I still don't see _why_, though. But maybe smarter people than me can see it..

FWIW this program definitely triggers the bug for me.

#194 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 5 lines · David Miller
Quoted 3 lines

I still don't see _why_, though. But maybe smarter people than me can see it..

FWIW this program definitely triggers the bug for me.

Ok, now that I have something simple to do repeatable stuff with, I can say what the pattern is.. It's not all that surprising, but it's still worth just stating for the record.

What happens is that when I do the "packetized writes" in random order, the _last_ write to a page occasionally just goes missing. It's not always at the end of a page, as shown by for example:

- A whole chunk got dropped:

	Chunk 2094 corrupted (0-1459)  (1624-3083)
	Expected 46, got 0
	Written as (30912)55414(10000)

That "Written as (x)y(z)" line means that the corrupted chunk was written as chunk #y, and the preceding and following chunks (that were _not_ corrupt) on the page was written as #x and #z respectively.

In other words, the missing chunk (which is still zero) was written much later than the ones that were ok, and never hit the disk. It's a contiguous chunk in the middle of the page (chunks are 1460 bytes in size)

The first line means that all bytes of the chunk (0-1459) were corrupted, and the values in parenthesis are the offsets within a page. In other words, this was a chunk in the _middle_ of a page.

- The missing data can also be at the beginning or ends of pages:

Beginning of the chunk missing, it was at the end of a page (page offsets 3288-4095) and the _next_ page got written out fine:

	Chunk 2126 corrupted (0-807)  (3288-4095)
	Expected 78, got 0
	Written as (32713)55573(14301)

End of a chunk missing, it was the beginning of a page (and the _previous_ page that contained the beginning of the chunk was written out fine)

	Chunk 2179 corrupted (1252-1459)  (0-207)
	Expected 131, got 0
	Written as (45189)55489(15515)

Now, the reason I say this isn't surprising is that this is entirely consistent with the dirty bit being dropped on the floor somewhere, and likely through some interaction with the previous changes being in the process of being written out.

Something (incorrectly) ends up deciding that it doesn't need to write the page, since it's already written, or alternatively clears the dirty bit too late (clears it because an _earlier_ write finished, never mind that the new dirty data didn't make it).

I also figured out that it's not the low-memory situation that does it, it really must be the "page_mkclean()" triggering. Becuase I can do

echo 5 > /proc/sys/vm/dirty_ratio echo 3 > /proc/sys/vm/dirty_background_ratio

to make it clean the pages much more aggressively than the default, and I can see corruption on my 256MB machine with just a 40MB shared file, and 70MB of memory consistently free.

So this thing is definitely giving some answers. It's NOT about low memory, and it very much seems to be about the whole "balance_dirty_ratio" thing. I don't think I triggered the actual low-memory stuff once in that situation..

So I have some more data on the behaviour, but I _still_ don't see the reason behind it. It's probably something really obvious once it's pointed out..

[ Modified test-program that tells you where the corruption happens (and
when the missing parts were supposed to be written out) appended, in case people care. ]

Linus

---
#include <sys/mman.h>
#include <sys/fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <time.h>

#define TARGETSIZE (100 << 20)
#define CHUNKSIZE (1460)
#define NRCHUNKS (TARGETSIZE / CHUNKSIZE)
#define SIZE (NRCHUNKS * CHUNKSIZE)

static void fillmem(void *start, int nr)

{
	memset(start, nr, CHUNKSIZE);
}

#define page_offset(buf, off) (0xfff & ((unsigned)(unsigned long)(buf)+(off)))

static int chunkorder[NRCHUNKS];

static int order(int nr)

{
	int i;
	if (nr < 0 || nr >= NRCHUNKS)
		return -1;
	for (i = 0; i < NRCHUNKS; i++)
		if (chunkorder[i] == nr)
			return i;
	return -2;
}

static void checkmem(void *buf, int nr)

{
	unsigned int start = ~0u, end = 0;
	unsigned char c = nr, *p = buf, differs = 0;
	int i;
	for (i = 0; i < CHUNKSIZE; i++) {
		unsigned char got = *p++;
		if (got != c) {
			if (i < start)
				start = i;
			if (i > end)
				end = i;
			differs = got;
		}
	}
	if (start < end) {
		printf("Chunk %d corrupted (%u-%u)  (%u-%u)            \n", nr, start, end,
			page_offset(buf, start), page_offset(buf, end));
		printf("Expected %u, got %u\n", c, differs);
		printf("Written as (%d)%d(%d)\n", order(nr-1), order(nr), order(nr+1));
	}
}

static char *remap(int fd, char *mapping)

{
	if (mapping) {
		munmap(mapping, SIZE);
		posix_fadvise(fd, 0, SIZE, POSIX_FADV_DONTNEED);
	}
	return mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
}

int main(int argc, char **argv)

{
	char *mapping;
	int fd, i;

	/*
	 * Make some random ordering of writing the chunks to the
	 * memory map..
	 *
	 * Start with fully ordered..
	 */
	for (i = 0; i < NRCHUNKS; i++)
		chunkorder[i] = i;

	/* ..and then mix it up randomly */
	srandom(time(NULL));
	for (i = 0; i < NRCHUNKS; i++) {
		int index = (unsigned int) random() % NRCHUNKS;
		int nr = chunkorder[index];
		chunkorder[index] = chunkorder[i];
		chunkorder[i] = nr;
	}

	fd = open("mapfile", O_RDWR | O_TRUNC | O_CREAT, 0666);
	if (fd < 0)
		return -1;
	if (ftruncate(fd, SIZE) < 0)
		return -1;
	mapping = remap(fd, NULL);
	if (-1 == (int)(long)mapping)
		return -1;

	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = chunkorder[i];
		printf("Writing chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		fillmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");

	/* Unmap, drop, and remap.. */
	mapping = remap(fd, mapping);

	/* .. and check */
	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = i;
		printf("Checking chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		checkmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");
	
	return 0;
}
#195 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 3 lines · Linus Torvalds

[ Modified test-program that tells you where the corruption happens (and
when the missing parts were supposed to be written out) appended, in case people care. ]

For the record, this is the output from a run on our ARM machine (32 MB RAM) with 2.6.18 + the following patches:

mm: tracking shared dirty pages
mm: balance dirty pages
mm: optimize the new mprotect() code a bit
mm: small cleanup of install_page()
mm: fixup do_wp_page()
mm: msync() cleanup

It is at all suprising that the second offset within a page can be less than the first offset within a page ? e.g.

Chunk 260 corrupted (1-1455) (2769-127)

$ ./linus-test
Writing chunk 279/280 (99%)
Chunk 256 corrupted (1-1455)  (1025-2479)
Expected 0, got 1
Written as (82)175(56)
Chunk 258 corrupted (1-1455)  (3945-1303)
Expected 2, got 3
Written as (56)51(20)
Chunk 260 corrupted (1-1455)  (2769-127)
Expected 4, got 5
Written as (20)30(18)
Chunk 262 corrupted (1-1455)  (1593-3047)
Expected 6, got 7
Written as (18)196(158)
Chunk 264 corrupted (1-1455)  (417-1871)
Expected 8, got 9
Written as (158)133(146)
Chunk 266 corrupted (1-1455)  (3337-695)
Expected 10, got 11
Written as (146)43(77)
Chunk 268 corrupted (1-1455)  (2161-3615)
Expected 12, got 13
Written as (77)251(211)
Chunk 270 corrupted (1-1455)  (985-2439)
Expected 14, got 15
Written as (211)257(231)
Chunk 272 corrupted (1-1455)  (3905-1263)
Expected 16, got 17
Written as (231)254(154)
Chunk 274 corrupted (1-1455)  (2729-87)
Expected 18, got 19
Written as (154)11(85)
Chunk 276 corrupted (1-1455)  (1553-3007)
Expected 20, got 21
Written as (85)230(134)
Chunk 278 corrupted (1-1455)  (377-1831)
Expected 22, got 23
Written as (134)233(103)
Checking chunk 279/280 (99%)

Gordon

Signature · 1 line
Gordon Farquharson
#196 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 5 lines · Gordon Farquharson

It is at all suprising that the second offset within a page can be less than the first offset within a page ? e.g.

Chunk 260 corrupted (1-1455) (2769-127)

No, that just means that it went over to the next page (so you actually had two consecutive pages that weren't written out).

That said, your output is very different from mine in another way. You don't have zeroes in your pages, rather the thing seems to have data from the next block (ie the chunk that should have 20 is reported as having 21 etc). You also have your offsets shifted up by one (ie offset 0 looks ok for you, and then you have a strange pattern of corruption at bytes 1...1455 instead of 0..1459.

You also seem to have an example of the _earlier_ writes being corrupted, rather than the later ones. For example (but it's also a page-crosser, so maybe that's part of it):

	Chunk 274 corrupted (1-1455)  (2729-87)
	Expected 18, got 19
	Written as (154)11(85)

says that block chunk 274 is the corrupt one, but it was written fairly early as #11, and the blocks around it (chunks 273 and 275) were actually written later.

For all I know, my test-program is buggy wrt the ordering printouts, though. Did you perhaps change the logic in any way?

Linus

#197 lore raw

Re: [PATCH] mm: fix page_mkclean_one

[Oops - forgot to hit "Reply to All" first time round.]

Hi Linus

Quoted 2 lines · Linus Torvalds

For all I know, my test-program is buggy wrt the ordering printouts, though. Did you perhaps change the logic in any way?

I don't think so. I did reduce the target size

#define TARGETSIZE (100 << 12)

to make the program finish a little quicker, and for some reason I get

linus-test.c: In function 'remap':
linus-test.c:61: error: 'POSIX_FADV_DONTNEED' undeclared (first use in this function)

when I compile the program, so I replaced POSIX_FADV_DONTNEED with 4 as defined in /usr/include/bits/fcntl.h.

Other than these two changes, the program is identical to the version you posted.

I have run the program a few times, and the output is pretty consistent. However, when I increase the target size, the difference between the expected and actual values is larger.

Written as (749)935(738)
Chunk 1113 corrupted (1-1455) (2965-323)
Expected 89, got 93
Written as (935)738(538)
Chunk 1114 corrupted (1-1455) (329-1783)
Expected 90, got 94
Written as (738)538(678)
Chunk 1115 corrupted (1-1455) (1789-3243)
Expected 91, got 95
Written as (538)678(989)
Chunk 1120 corrupted (1-1455) (897-2351)
Expected 96, got 100
Written as (537)265(1005)
Chunk 1121 corrupted (1-1455) (2357-3811)
Expected 97, got 101
Written as (265)1005(-1)

linus-test.c 2006-12-28 06:18:24.000000000 +0100 renamed from linus-test.c.orig 2006-12-28 06:17:24.000000000 +0100 +0 -0
@@ -6,7 +6,7 @@

#include <stdio.h>
#include <time.h>

-#define TARGETSIZE (100 << 20)
+#define TARGETSIZE (100 << 14)
#define CHUNKSIZE (1460)
#define NRCHUNKS (TARGETSIZE / CHUNKSIZE)
#define SIZE (NRCHUNKS * CHUNKSIZE)
@@ -61,7 +61,7 @@

{
       if (mapping) {
               munmap(mapping, SIZE);
-                posix_fadvise(fd, 0, SIZE, POSIX_FADV_DONTNEED);
+                posix_fadvise(fd, 0, SIZE, 4);
       }
       return mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

Gordon

Signature · 1 line
Gordon Farquharson
maintainer
#198 lore raw

Re: [PATCH] mm: fix page_mkclean_one

From: "Gordon Farquharson" <[email protected]>
Date: Wed, 27 Dec 2006 22:20:20 -0700

Quoted 8 lines

and for some reason I get

linus-test.c: In function 'remap':
linus-test.c:61: error: 'POSIX_FADV_DONTNEED' undeclared (first use in this function)

when I compile the program, so I replaced POSIX_FADV_DONTNEED with 4 as defined in /usr/include/bits/fcntl.h.

Me too, I added "-D_POSIX_C_SOURCE=200112" to "fix" this.

Perhaps Linus's GCC sets that by default and our's doesn't.

#199 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Hi David

Quoted 1 line · David Miller

Me too, I added "-D_POSIX_C_SOURCE=200112" to "fix" this.

That works for me. Thanks for the tip.

Gordon

Signature · 1 line
Gordon Farquharson
#200 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to Gordon Farquharson
Quoted 7 lines · Gordon Farquharson

I have run the program a few times, and the output is pretty consistent. However, when I increase the target size, the difference between the expected and actual values is larger.

Written as (749)935(738)
Chunk 1113 corrupted (1-1455) (2965-323)
Expected 89, got 93

This is not the corruption Linus is after. Note that the corruption starts at offset '1'. Also note that:

89 = 1113 & 255
93 = 1113 & 255 | (1113 >> 8)

and if you look at glibc's memset() function, you'll notice that's exactly what you expect if you pass a non-8bit value to it. Ergo, what you're seeing is utterly expected given glibc's memset() implementation on ARM.

Fixing Linus' test program to pass nr & 255 to memset results in clean passes on 2.6.9 on TheCus N2100 (IOP8032x) and 2.6.16.9 StrongARM machines (as would be expected.)

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#201 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 3 lines · Russell King

Fixing Linus' test program to pass nr & 255 to memset results in clean passes on 2.6.9 on TheCus N2100 (IOP8032x) and 2.6.16.9 StrongARM machines (as would be expected.)

Thanks for the fix, Russell.

I can now trigger the (real) problem by using a 25 MB file (100 << 18) and the Linksys NSLU2 (ARM, IXP420 processor, 32 MB RAM).

$ ./linus-test
Writing chunk 17954/17955 (99%)
Chunk 514 corrupted (0-1459)  (872-2331)
Expected 2, got 0
Written as (8479)11160(10312)
Chunk 516 corrupted (0-303)  (3792-4095)
Expected 4, got 0
Written as (10312)10569(4426)
Chunk 959 corrupted (0-691)  (3404-4095)
Expected 191, got 0
Written as (687)4881(1522)
Chunk 1895 corrupted (0-1459)  (1900-3359)
Expected 103, got 0
Written as (7746)8389(6231)
Chunk 2702 corrupted (0-1459)  (472-1931)
Expected 142, got 0
Written as (4866)7103(2409)
Chunk 3314 corrupted (0-1459)  (1064-2523)
Expected 242, got 0
Written as (4287)7064(1730)
Chunk 4043 corrupted (0-1459)  (444-1903)
Expected 203, got 0
Written as (6495)8509(4464)
Chunk 5180 corrupted (0-1459)  (1584-3043)
Expected 60, got 0
Written as (11056)12826(10797)
Chunk 5672 corrupted (0-991)  (3104-4095)
Expected 40, got 0
Written as (9944)4872(41)
Chunk 5793 corrupted (460-1459)  (0-999)
Expected 161, got 0
Written as (7059)5038(4377)
Chunk 6089 corrupted (0-1459)  (1620-3079)
Expected 201, got 0
Written as (4672)5230(4403)
Chunk 6545 corrupted (268-1459)  (0-1191)
Expected 145, got 0
Written as (3701)5969(4668)
Chunk 7578 corrupted (0-1459)  (584-2043)
Expected 154, got 0
Written as (10015)5082(1648)
Chunk 7880 corrupted (864-1459)  (0-595)
Expected 200, got 0
Written as (17869)5064(4745)
Chunk 8086 corrupted (0-1459)  (888-2347)
Expected 150, got 0
Written as (10206)11050(10374)
Chunk 8749 corrupted (0-1459)  (2212-3671)
Expected 45, got 0
Written as (15263)7132(4825)
Chunk 9068 corrupted (0-1459)  (1008-2467)
Expected 108, got 0
Written as (5557)7571(6771)
Chunk 9193 corrupted (812-1459)  (0-647)
Expected 233, got 0
Written as (9238)7277(4757)
Chunk 10032 corrupted (576-1459)  (0-883)
Expected 48, got 0
Written as (15741)10012(1753)
Chunk 10056 corrupted (0-1459)  (1696-3155)
Expected 72, got 0
Written as (5379)7431(262)
Chunk 10395 corrupted (0-1459)  (1020-2479)
Expected 155, got 0
Written as (21)7442(5902)
Chunk 10791 corrupted (0-1459)  (1644-3103)
Expected 39, got 0
Written as (4753)5925(5926)
Chunk 10792 corrupted (0-991)  (3104-4095)
Expected 40, got 0
Written as (5925)5926(8555)
Chunk 11036 corrupted (0-1103)  (2992-4095)
Expected 28, got 0
Written as (13755)14449(7458)
Chunk 11387 corrupted (644-1459)  (0-815)
Expected 123, got 0
Written as (10853)11459(9445)
Chunk 11586 corrupted (920-1459)  (0-539)
Expected 66, got 0
Written as (3769)11691(11123)
Chunk 11882 corrupted (0-1459)  (1160-2619)
Expected 106, got 0
Written as (10736)11696(2788)
Chunk 12397 corrupted (0-603)  (3492-4095)
Expected 109, got 0
Written as (2352)7515(2437)
Chunk 12669 corrupted (0-795)  (3300-4095)
Expected 125, got 0
Written as (1191)7661(5266)
Chunk 13162 corrupted (0-1459)  (2184-3643)
Expected 106, got 0
Written as (9383)13662(11544)
Chunk 14653 corrupted (0-27)  (4068-4095)
Expected 61, got 0
Written as (8100)9456(1275)
Chunk 17332 corrupted (0-367)  (3728-4095)
Expected 180, got 0
Written as (760)12247(1244)
Chunk 17445 corrupted (0-1459)  (772-2231)
Expected 37, got 0
Written as (8007)16481(14439)
Chunk 17556 corrupted (0-1007)  (3088-4095)
Expected 148, got 0
Written as (10113)10657(10477)
Chunk 17859 corrupted (0-995)  (3100-4095)
Expected 195, got 0
Written as (14472)14767(11426)
Checking chunk 17954/17955 (99%)

Gordon

Signature · 1 line
Gordon Farquharson
#202 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 4 lines · Gordon Farquharson

Thanks for the fix, Russell.

I can now trigger the (real) problem by using a 25 MB file (100 << 18) and the Linksys NSLU2 (ARM, IXP420 processor, 32 MB RAM).

Me too (using 100 << 18). Interestingly, I don't seem to get any corruption on a different ARM board, an IOP32x based machine with 128 MB RAM.

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#203 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to Russell King
Quoted 4 lines · Russell King

and if you look at glibc's memset() function, you'll notice that's exactly what you expect if you pass a non-8bit value to it. Ergo, what you're seeing is utterly expected given glibc's memset() implementation on ARM.

Guys, you _really_ should fix memset(). What you describe is a _bug_.

"memset()" takes an "int" as its argument (always has), and has to convert it to a byte _itself_. It may not be common, but it's perfectly normal, to pass it values outside 0-255 (negative values that still fit in a "signed char" in particular are very normal, but my usage of "let the thing truncate it itself" is also quite fine).

Quoted 1 line

Fixing Linus' test program to pass nr & 255 to memset

No. I'm almost certain that that is not a "fix", it's a workaround for a serious bug in your glibc crap.

But it does explain all the unexpected strange behaviour (and the really small writeback size - now it doesn't need any /proc/sys/vm/dirty_ratio assumptions to be explicable.

Linus

#204 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Russell King

and if you look at glibc's memset() function, you'll notice that's exactly what you expect if you pass a non-8bit value to it. Ergo, what you're seeing is utterly expected given glibc's memset() implementation on ARM.

Guys, you _really_ should fix memset(). What you describe is a _bug_.

Yup, but I have nothing to do with glibc because I refuse to do that silly copyright assignment FSF thing. Hopefully someone else can resolve it, but...

Quoted 4 lines
Quoted 1 line

Fixing Linus' test program to pass nr & 255 to memset

No. I'm almost certain that that is not a "fix", it's a workaround for a serious bug in your glibc crap.

_is_ a fix whether _you_ like it or not to work around the issue so people can at least run your test program. I'm not saying it's a proper fix though.

Of course, if you prefer to be mislead by incorrect bug reports...

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#205 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 4 lines · Russell King

Yup, but I have nothing to do with glibc because I refuse to do that silly copyright assignment FSF thing. Hopefully someone else can resolve it, but...

Yeah, me too.

Quoted 3 lines

_is_ a fix whether _you_ like it or not to work around the issue so people can at least run your test program. I'm not saying it's a proper fix though.

My point was that it wasn't a "fix", it's a "workaround". The _fix_ would be in glibc.

Nothing more..

Linus

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#208 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Gordon Farquharson

I don't think so. I did reduce the target size

#define TARGETSIZE (100 << 12)

Quoted 8 lines

That's just 400kB!

There's no way you should see corruption with that kind of value. It should all stay solidly in the cache.

Is this perhaps with ARM nommu or something else strange? It may be that the program just doesn't work at all if mmap() is faked out with a malloc or similar.

Definitely a question for the ARM gurus. I'm out of my depth.

Gordon

Signature · 1 line
Gordon Farquharson
#209 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 10 lines · Gordon Farquharson
Quoted 8 lines

That's just 400kB!

There's no way you should see corruption with that kind of value. It should all stay solidly in the cache.

Is this perhaps with ARM nommu or something else strange? It may be that the program just doesn't work at all if mmap() is faked out with a malloc or similar.

Definitely a question for the ARM gurus. I'm out of my depth.

The CPU has a MMU. For reference, it's a IXP4xx based device with 32 MB of memory.

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#210 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to Gordon Farquharson
Quoted 4 lines · Gordon Farquharson
Quoted 4 lines
Quoted 1 line

#define TARGETSIZE (100 << 12)

That's just 400kB!

Quoted 8 lines
Quoted 6 lines

There's no way you should see corruption with that kind of value. It should all stay solidly in the cache.

Is this perhaps with ARM nommu or something else strange? It may be that the program just doesn't work at all if mmap() is faked out with a malloc or similar.

Definitely a question for the ARM gurus. I'm out of my depth.

By the way, I just tried it with TARGETSIZE (100 << 12) on a different ARM machine (a Thecus N2100 based on an IOP32x chip with 128 MB of memory) and I see similar results to that from Gordon:

Writing chunk 279/280 (99%)
Chunk 256 corrupted (1-1455) (1025-2479) Expected 0, got 1
Written as (199)43(184)
Chunk 258 corrupted (1-1455) (3945-1303) Expected 2, got 3
Written as (184)74(145)
Chunk 260 corrupted (1-1455) (2769-127) Expected 4, got 5
Written as (145)89(237)
Chunk 262 corrupted (1-1455) (1593-3047) Expected 6, got 7
Written as (237)168(174)
Chunk 264 corrupted (1-1455) (417-1871) Expected 8, got 9
Written as (174)135(161)
Chunk 266 corrupted (1-1455) (3337-695) Expected 10, got 11
Written as (161)123(180)
Chunk 268 corrupted (1-1455) (2161-3615) Expected 12, got 13
Written as (180)13(19)
Chunk 270 corrupted (1-1455) (985-2439) Expected 14, got 15
Written as (19)172(106)
Chunk 272 corrupted (1-1455) (3905-1263) Expected 16, got 17
Written as (106)212(140)
Chunk 274 corrupted (1-1455) (2729-87) Expected 18, got 19
Written as (140)124(73)
Chunk 276 corrupted (1-1455) (1553-3007) Expected 20, got 21
Written as (73)151(52)
Chunk 278 corrupted (1-1455) (377-1831) Expected 22, got 23
Written as (52)215(99)
Checking chunk 279/280 (99%)

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#211 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 12 lines · Martin Michlmayr
Quoted 11 lines · Gordon Farquharson
Quoted 10 lines
Quoted 1 line

#define TARGETSIZE (100 << 12)

That's just 400kB!

There's no way you should see corruption with that kind of value. It should all stay solidly in the cache.

Is this perhaps with ARM nommu or something else strange? It may be that the program just doesn't work at all if mmap() is faked out with a malloc or similar.

Quoted 5 lines
Quoted 1 line

Definitely a question for the ARM gurus. I'm out of my depth.

By the way, I just tried it with TARGETSIZE (100 << 12) on a different ARM machine (a Thecus N2100 based on an IOP32x chip with 128 MB of memory) and I see similar results to that from Gordon:

Work around the glibc memset() problem by passing nr & 255, and re-run the test. You're getting false positives.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#212 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 6 lines · Russell King
Quoted 3 lines

By the way, I just tried it with TARGETSIZE (100 << 12) on a different ARM machine (a Thecus N2100 based on an IOP32x chip with 128 MB of memory) and I see similar results to that from Gordon:

Work around the glibc memset() problem by passing nr & 255, and re-run the test. You're getting false positives.

Yeah, I saw your message about this problem after I sent mine.

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#213 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to a message not in the archive
Quoted 4 lines · Linus Torvalds

That's just 400kB!

There's no way you should see corruption with that kind of value. It should all stay solidly in the cache.

100kB and 200kB files always succeed on the ARM system. 400kB and larger always seem to fail.

Does the following help interpret the results on ARM at all ?

$ free
            total       used       free     shared    buffers     cached
Mem:         30000      23620       6380          0        808      15676
-/+ buffers/cache:       7136      22864
Swap:        88316       3664      84652

Gordon

Signature · 1 line
Gordon Farquharson
#214 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 3 lines · Gordon Farquharson

100kB and 200kB files always succeed on the ARM system. 400kB and larger always seem to fail.

Oh, wow. Yeah, I've just repressed how tiny 32MB is. And especially if you lowered the /proc/sys/vm/dirty_ratio to a smaller percentage, I guess 400kB should be enough to cause writeback.

Ugh. I tested a 128MB machine a few weeks ago, and found it painful.

Linus

#215 lore raw

RE: [PATCH] mm: fix page_mkclean_one

In reply to Linus Torvalds

Linus Torvalds wrote on Wednesday, December 27, 2006 7:05 PM

Quoted 10 lines · David Miller
Quoted 5 lines
Quoted 3 lines

I still don't see _why_, though. But maybe smarter people than me can see it..

FWIW this program definitely triggers the bug for me.

Ok, now that I have something simple to do repeatable stuff with, I can say what the pattern is.. It's not all that surprising, but it's still worth just stating for the record.

Running the test code, git bisect points its finger at this commit. Reverting this commit on top of 2.6.20-rc2 doesn't trigger the bug from the test code.

edc79b2a46ed854595e40edcf3f8b37f9f14aa3f is first bad commit commit edc79b2a46ed854595e40edcf3f8b37f9f14aa3f

Author: Peter Zijlstra <[email protected]>
Date:   Mon Sep 25 23:30:58 2006 -0700

[PATCH] mm: balance dirty pages

Now that we can detect writers of shared mappings, throttle them. Avoids OOM by surprise.

#216 lore raw

RE: [PATCH] mm: fix page_mkclean_one

Chen, Kenneth wrote on Wednesday, December 27, 2006 9:55 PM

Quoted 18 lines

Linus Torvalds wrote on Wednesday, December 27, 2006 7:05 PM

Quoted 10 lines · David Miller
Quoted 5 lines
Quoted 3 lines

I still don't see _why_, though. But maybe smarter people than me can see it..

FWIW this program definitely triggers the bug for me.

Ok, now that I have something simple to do repeatable stuff with, I can say what the pattern is.. It's not all that surprising, but it's still worth just stating for the record.

Running the test code, git bisect points its finger at this commit. Reverting this commit on top of 2.6.20-rc2 doesn't trigger the bug from the test code.

edc79b2a46ed854595e40edcf3f8b37f9f14aa3f is first bad commit commit edc79b2a46ed854595e40edcf3f8b37f9f14aa3f

Quoted 7 lines
Author: Peter Zijlstra <[email protected]>
Date:   Mon Sep 25 23:30:58 2006 -0700

[PATCH] mm: balance dirty pages

Now that we can detect writers of shared mappings, throttle them. Avoids OOM by surprise.

Oh, never mind :-( I just didn't create enough write out pressure when test this. I just saw bug got triggered on a kernel I previously thought was OK.

maintainer
#217 lore raw

Re: [PATCH] mm: fix page_mkclean_one

From: "Chen, Kenneth W" <[email protected]>
Date: Wed, 27 Dec 2006 22:10:52 -0800

Quoted 24 lines

Chen, Kenneth wrote on Wednesday, December 27, 2006 9:55 PM

Quoted 23 lines

Linus Torvalds wrote on Wednesday, December 27, 2006 7:05 PM

Quoted 10 lines · David Miller
Quoted 5 lines
Quoted 3 lines

I still don't see _why_, though. But maybe smarter people than me can see it..

FWIW this program definitely triggers the bug for me.

Ok, now that I have something simple to do repeatable stuff with, I can say what the pattern is.. It's not all that surprising, but it's still worth just stating for the record.

Running the test code, git bisect points its finger at this commit. Reverting this commit on top of 2.6.20-rc2 doesn't trigger the bug from the test code.

edc79b2a46ed854595e40edcf3f8b37f9f14aa3f is first bad commit commit edc79b2a46ed854595e40edcf3f8b37f9f14aa3f

Author: Peter Zijlstra <[email protected]>
Date:   Mon Sep 25 23:30:58 2006 -0700

[PATCH] mm: balance dirty pages

Quoted 7 lines
Quoted 2 lines

Now that we can detect writers of shared mappings, throttle them. Avoids OOM by surprise.

Oh, never mind :-( I just didn't create enough write out pressure when test this. I just saw bug got triggered on a kernel I previously thought was OK.

Besides, I'm pretty sure that from the Debian bug entry it's been established that the dirty-page tracking changes from a few releases ago introduced this problem.

#218 lore raw

RE: [PATCH] mm: fix page_mkclean_one

In reply to Chen, Kenneth W
Quoted 4 lines · Kenneth W
Quoted 4 lines

Running the test code, git bisect points its finger at this commit. Reverting this commit on top of 2.6.20-rc2 doesn't trigger the bug from the test code.

Quoted 8 lines
Quoted 4 lines

[PATCH] mm: balance dirty pages

Now that we can detect writers of shared mappings, throttle them. Avoids OOM by surprise.

Oh, never mind :-( I just didn't create enough write out pressure when test this. I just saw bug got triggered on a kernel I previously thought was OK.

Btw, this is an important point - people have long felt that the new page balancing in 2.6.19 was to blame, but you've just confirmed the long-held suspicion (at least by me) that it's not actually a new bug at all, it's just that the dirty page balancing causes writeback to happen _earlier_, and thus is better able to _show_ a bug that we've likely had for a long long time.

Linus

#219 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to Linus Torvalds
Quoted 57 lines · Linus Torvalds
Quoted 5 lines · David Miller
Quoted 3 lines

I still don't see _why_, though. But maybe smarter people than me can see it..

FWIW this program definitely triggers the bug for me.

Ok, now that I have something simple to do repeatable stuff with, I can say what the pattern is.. It's not all that surprising, but it's still worth just stating for the record.

What happens is that when I do the "packetized writes" in random order, the _last_ write to a page occasionally just goes missing. It's not always at the end of a page, as shown by for example:

- A whole chunk got dropped:

	Chunk 2094 corrupted (0-1459)  (1624-3083)
	Expected 46, got 0
	Written as (30912)55414(10000)

That "Written as (x)y(z)" line means that the corrupted chunk was written as chunk #y, and the preceding and following chunks (that were _not_ corrupt) on the page was written as #x and #z respectively.

In other words, the missing chunk (which is still zero) was written much later than the ones that were ok, and never hit the disk. It's a contiguous chunk in the middle of the page (chunks are 1460 bytes in size)

The first line means that all bytes of the chunk (0-1459) were corrupted, and the values in parenthesis are the offsets within a page. In other words, this was a chunk in the _middle_ of a page.

- The missing data can also be at the beginning or ends of pages:

Beginning of the chunk missing, it was at the end of a page (page offsets 3288-4095) and the _next_ page got written out fine:

	Chunk 2126 corrupted (0-807)  (3288-4095)
	Expected 78, got 0
	Written as (32713)55573(14301)
   End of a chunk missing, it was the beginning of a page (and the 
   _previous_ page that contained the beginning of the chunk was written 
   out fine)

	Chunk 2179 corrupted (1252-1459)  (0-207)
	Expected 131, got 0
	Written as (45189)55489(15515)

Now, the reason I say this isn't surprising is that this is entirely consistent with the dirty bit being dropped on the floor somewhere, and likely through some interaction with the previous changes being in the process of being written out.

Quoted 4 lines

Something (incorrectly) ends up deciding that it doesn't need to write the page, since it's already written, or alternatively clears the dirty bit too late (clears it because an _earlier_ write finished, never mind that the new dirty data didn't make it).

There might be a narrow race between set_page_dirty and clear_page_dirty.

The test program is a process to write/read data. pdflush might write data to disk asynchronously. After pdflush writes a page to disk, it will call (either by softirq) clear_page_dirty to clear the dirty bit after getting the interrupt notification. But just after the page is written to disk and before the interrupt reports the result, the test process might change the data and unmap the area. When the area is unmapped, the page is marked as dirty again, but just after that, the interrupt arrives and the dirty bit is cleared, so the late data will not be written to disk.

Function zap_pte_range checks pte to set page dirty if needed, but it doesn't hold page lock. If the page lock is held before set page dirty, the race might be avoided.

Yanmin

#220 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 5 lines · Yanmin

The test program is a process to write/read data. pdflush might write data to disk asynchronously. After pdflush writes a page to disk, it will call (either by softirq) clear_page_dirty to clear the dirty bit after getting the interrupt notification.

That would indeed be a horrible bug. However, we don't do "clear_page_dirty()" _after_ the IO has completed, we do it _before_ the IO starts.

If you can actually find a place that does clear_page_dirty _after_ IO, then yes, you've just found the bug. But I haven't found it.

So the rule is _always_:

- call "clear_page_dirty_for_io()" with the page lock held, and _before_ the IO starts.
- do "set_page_writeback()" before unlocking the page again
- do a "end_page_writeback()" when the IO actually finishes.

and any code sequence that doesn't honor those rules would be buggy. A beer for anybody that finds it..

Linus

#221 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

[ Modified test-program that tells you where the corruption happens (and
when the missing parts were supposed to be written out) appended, in case people care. ]

Hi

2.6.18 (and 2.6.18.6) is ok, 2.6.19-rc1 is broken. I tried some snapshots between them but they hung before shell (2.6.18-git11, 2.6.18-git16, 2.6.18-git20, 2.6.18-git21). 2.6.18-git22 booted and was broken.

(UP, no preempt)

-Petri

#222 lore raw

Re: Re: [PATCH] mm: fix page_mkclean_one

In reply to Linus Torvalds

I set a qemu environment to test kernels: http://guichaz.free.fr/linux-bug/
I have corruption with every Fedora release kernel except the first, that is 2.4.22 works, but 2.6.5, 2.6.9, 2.6.11, 2.6.15 and 2.6.18-1.2798 exhibit some corruption.

Command line to test:

qemu root_fs -snapshot -kernel FC-kernels/FC2-vmlinuz-2.6.5-1.358 -append 'rw root=/dev/hda'

I get this kind of corruption: http://guichaz.free.fr/linux-bug/corruption.png

Signature · 1 line
Guillaume
#223 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 1 line · Guillaume Chazarain

I get this kind of corruption: http://guichaz.free.fr/linux-bug/corruption.png

Actually in qemu, I get three different behaviours:
- no corruption at all : with linux-2.4
- corruption only on the first chunks: before [PATCH] mm: balance dirty pages as identified by Kenneth
- corruption of all chunks: after the balance dirty pages patch

Bisecting in linux-2.5 land I found http://kernel.org/pub/linux/kernel/people/akpm/patches/2.5/2.5.66/2.5.66-mm3/broken-out/fadvise-flush-data.patch to cause the corruption for me.

The attached patch fixes the corruption for me.

Signature · 1 line
Guillaume
mm/fadvise.c +0 -3
diff -r 3859b1144d3a mm/fadvise.c
@@ -96,9 +96,6 @@ asmlinkage long sys_fadvise64_64(int fd,
 	case POSIX_FADV_NOREUSE:
 		break;
 	case POSIX_FADV_DONTNEED:
-		if (!bdi_write_congested(mapping->backing_dev_info))
-			filemap_flush(mapping);
-
 		/* First and last FULL page! */
 		start_index = (offset+(PAGE_CACHE_SIZE-1)) >> PAGE_CACHE_SHIFT;
 		end_index = (endbyte >> PAGE_CACHE_SHIFT);
#224 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 2 lines · Guillaume Chazarain

The attached patch fixes the corruption for me.

Well, that's a good hint, but it's really just a symptom. You effectively just made the test-program not even try to flush the data to disk, so the page cache would stay in memory, and you'd not see the corruption as well.

So you basically disabled the code that tried to trigger the bug more easily.

But the reason I say it's interesting is that "WB_SYNC_NONE" is very much implicated in mm/page-writeback.c, and if there is a bug triggered by WB_SYNC_NONE writebacks, then that would explain why page-writeback.c also fails..

Linus

#225 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 10 lines · On Thu, 28 Dec 2006 11:28:52 -0800 (PST) Linus Torvalds <[email protected]>…
Quoted 2 lines · Guillaume Chazarain

The attached patch fixes the corruption for me.

Well, that's a good hint, but it's really just a symptom. You effectively just made the test-program not even try to flush the data to disk, so the page cache would stay in memory, and you'd not see the corruption as well.

Quoted 8 lines

So you basically disabled the code that tried to trigger the bug more easily.

But the reason I say it's interesting is that "WB_SYNC_NONE" is very much implicated in mm/page-writeback.c, and if there is a bug triggered by WB_SYNC_NONE writebacks, then that would explain why page-writeback.c also fails..

It would be interesting to convert your app to do fsync() before FADV_DONTNEED. That would take WB_SYNC_NONE out of the picture as well (apart from pdflush activity).

#226 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 4 lines · Andrew Morton

It would be interesting to convert your app to do fsync() before FADV_DONTNEED. That would take WB_SYNC_NONE out of the picture as well (apart from pdflush activity).

I get corruption - but the whole point is that it's very much pdflush that should be writing these pages out.

Andrew - give my test-program a try. It can run in about 1 minute if you have a 256MB machine (I didn't, but "mem=256M" is my friend), and it seems to very consistently cause corruption.

What I do is:

# Make sure we write back aggressively
echo 5 > /proc/sys/vm/dirty_ratio

as root, and then just run the thing. Tons of corruption. But the corruption goes away if I just leave the default dirty ratio alone (but then I can increse the file size to trigger it, of course - but that also makes the test run a lot slower).

Now, with a pre-2.6.19 kernel, I bet you won't get the corruption as easily (at least with the "fsync()"), but that's less to do with anything new, and probably just because then you simply won't have any pdflushing going on - since the kernel won't even notice that you have tons of dirty pages ;)

It might also depend on the speed of your disk drive - the machine I test this on has a slow 4200 rpm laptop drive in it, and that probably makes things go south more easily. That's _especially_ true if this is related to any "bdi_write_congested()" logic.

Now, it could also be related to various code snippets like

	...
	if (wbc->sync_mode != WB_SYNC_NONE)
		wait_on_page_writeback(page);

	if (PageWriteback(page) ||
			!clear_page_dirty_for_io(page)) {
		unlock_page(page);
		continue;
	}
	...

where the WB_SYNC_NONE case will hit the "PageWriteback()" and just not do the writeback at all (but it also won't clear the dirty bit, so it's certainly not an *OBVIOUS* bug).

We also have code like this ("pageout()"):

	if (clear_page_dirty_for_io(page)) {
		int res;
		struct writeback_control wbc = {
			.sync_mode = WB_SYNC_NONE,
			..
		}
		...
		res = mapping->a_ops->writepage(page, &wbc);

and in this case, if the "WB_SYNC_NONE" means that the "writepage()" call won't do anything at all because of congestion, then that would be a _bad_ thing, and would certainly explain how something didn't get written out.

But that particular path should only trigger for the "shrink_page_list()" case, and it's not the case I seem to be testing with my "low dirty_ratio" testing.

Linus

#include <sys/mman.h>
#include <sys/fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <time.h>

#define TARGETSIZE (22 << 20)
#define CHUNKSIZE (1460)
#define NRCHUNKS (TARGETSIZE / CHUNKSIZE)
#define SIZE (NRCHUNKS * CHUNKSIZE)

static void fillmem(void *start, int nr)

{
	memset(start, nr, CHUNKSIZE);
}

#define page_offset(buf, off) (unsigned)((unsigned long)(buf)+(off)-(unsigned long)(mapping))

static int chunkorder[NRCHUNKS];
static char *mapping;

static int order(int nr)

{
	int i;
	if (nr < 0 || nr >= NRCHUNKS)
		return -1;
	for (i = 0; i < NRCHUNKS; i++)
		if (chunkorder[i] == nr)
			return i;
	return -2;
}

static void checkmem(void *buf, int nr)

{
	unsigned int start = ~0u, end = 0;
	unsigned char c = nr, *p = buf, differs = 0;
	int i;
	for (i = 0; i < CHUNKSIZE; i++) {
		unsigned char got = *p++;
		if (got != c) {
			if (i < start)
				start = i;
			if (i > end)
				end = i;
			differs = got;
		}
	}
	if (start < end) {
		printf("Chunk %d corrupted (%u-%u)  (%x-%x)            \n", nr, start, end,
			page_offset(buf, start), page_offset(buf, end));
		printf("Expected %u, got %u\n", c, differs);
		printf("Written as (%d)%d(%d)\n", order(nr-1), order(nr), order(nr+1));
	}
}

static char *remap(int fd, char *mapping)

{
	if (mapping) {
		munmap(mapping, SIZE);
//		fsync(fd);
		posix_fadvise(fd, 0, SIZE, POSIX_FADV_DONTNEED);
	}
	return mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
}

int main(int argc, char **argv)

{
	int fd, i;

	/*
	 * Make some random ordering of writing the chunks to the
	 * memory map..
	 *
	 * Start with fully ordered..
	 */
	for (i = 0; i < NRCHUNKS; i++)
		chunkorder[i] = i;

	/* ..and then mix it up randomly */
	srandom(time(NULL));
	for (i = 0; i < NRCHUNKS; i++) {
		int index = (unsigned int) random() % NRCHUNKS;
		int nr = chunkorder[index];
		chunkorder[index] = chunkorder[i];
		chunkorder[i] = nr;
	}

	fd = open("mapfile", O_RDWR | O_TRUNC | O_CREAT, 0666);
	if (fd < 0)
		return -1;
	if (ftruncate(fd, SIZE) < 0)
		return -1;
	mapping = remap(fd, NULL);
	if (-1 == (int)(long)mapping)
		return -1;

	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = chunkorder[i];
		printf("Writing chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		fillmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");

	/* Unmap, drop, and remap.. */
	mapping = remap(fd, mapping);

	/* .. and check */
	for (i = 0; i < NRCHUNKS; i++) {
		int chunk = i;
		printf("Checking chunk %d/%d (%d%%)     \r", i, NRCHUNKS, 100*i/NRCHUNKS);
		checkmem(mapping + chunk * CHUNKSIZE, chunk);
	}
	printf("\n");

	/* Clean up for next time */
	sleep(5);
	sync();
	sleep(5);
	munmap(mapping, SIZE);
	close(fd);
	unlink("mapfile");
	
	return 0;
}
maintainer
#227 lore raw

Re: [PATCH] mm: fix page_mkclean_one

From: Linus Torvalds <[email protected]>
Date: Thu, 28 Dec 2006 12:14:31 -0800 (PST)

Quoted 2 lines

I get corruption - but the whole point is that it's very much pdflush that should be writing these pages out.

I think what might be happening is that pdflush writes them out fine, however we don't trap writes by the application _during_ that writeout.

These corruptions look exactly as if:

1) pdflush begins writeback of page X
2) page goes to disk
3) application writes a chunk to the page
4) pdflush et al. think the page is clean, so it gets tossed, losing the writes done in #3

So there's a missing PTE change in there, so that we never get proper re-dirtying of the page if the application tries to write to the page during the writeback.

It's something that will only occur with writeback and MAP_SHARED writable access to the file pages. That's why we never see this with normal filesystem writes, since those explicitly manage the page dirty state.

I think the dirty balancing logic etc. isn't where the problems are, to me it's a PTE state update issue for sure.

#228 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 2 lines

I think what might be happening is that pdflush writes them out fine, however we don't trap writes by the application _during_ that writeout.

Yeah. I believe that more exactly it happens if the very last write to the page causes a writeback (due to dirty balancing) while another writeback for the page is already happening.

As usual in these cases, I have zero proof.

Quoted 2 lines

It's something that will only occur with writeback and MAP_SHARED writable access to the file pages.

It's the do_wp_page -> balance_dirty_pages -> generic_writepages path for sure. Maybe it's enough to change

                        if (wbc->sync_mode != WB_SYNC_NONE)
                                wait_on_page_writeback(page);

                        if (PageWriteback(page) ||
                            !clear_page_dirty_for_io(page)) {
                                unlock_page(page);
                                continue;
                        }

to

                        if (wbc->sync_mode != WB_SYNC_NONE)
                                wait_on_page_writeback(page);

                        if (PageWriteback(page)) {
                        	    redirty_page_for_writepage(wbc, page);
                                unlock_page(page);
                                continue;
                        }

                        if (!clear_page_dirty_for_io(page)) {
                                unlock_page(page);
                                continue;
                        }

or something along those lines. Completely untested of course :-)

Segher

#229 lore raw

Re: [PATCH] mm: fix page_mkclean_one

Quoted 9 lines · Segher Boessenkool
Quoted 3 lines

I think what might be happening is that pdflush writes them out fine, however we don't trap writes by the application _during_ that writeout.

Yeah. I believe that more exactly it happens if the very last write to the page causes a writeback (due to dirty balancing) while another writeback for the page is already happening.

As usual in these cases, I have zero proof.

I actually have proof to the contrary, ie I have traces that say "the write was started" after the last write.

And the VM layer in this area is actually fairly sane and civilized. It has a bit that says "writeback in progress", and if that bit is set, it simply _will_not_ start a new write. It even has various BUG_ON()'s to that effect.

So everything I have ever seen says that the VM layer is actually doing everything right.

Quoted 11 lines

It's the do_wp_page -> balance_dirty_pages -> generic_writepages path for sure. Maybe it's enough to change

                        if (wbc->sync_mode != WB_SYNC_NONE)
                                wait_on_page_writeback(page);

                        if (PageWriteback(page) ||
                            !clear_page_dirty_for_io(page)) {
                                unlock_page(page);
                                continue;
                        }

Notive how this one basically says:

- if it's under writeback, don't even clear the page dirty flag.

Your suggested change:

Quoted 5 lines
                        if (wbc->sync_mode != WB_SYNC_NONE)
                                wait_on_page_writeback(page);

                        if (PageWriteback(page)) {
                        	    redirty_page_for_writepage(wbc, page);

makes no sense, because we simply never _did_ the "clear_page_dirty()" if the thing was under writeback in the first place. That's how C conditionals work. So there's no reason to "redirty" it, because it wasn't cleaned in the first place.

I've double- and triple-checked the dirty bits, including having traces that actually say that the IO was started (from a VM perspective) _after_ the last write was done. The IO just didn't hit the disk.

I'm personally fairly convinced that it's not a VM issue, but a "IO issue". Either in a low-level filesystem or in some of the fs/buffer.c helper routines.

But I'd love to be proven wrong.

I do have a few interesting details from the trace I haven't really analyzed yet. Here's the trace for events on one of the pages that was corrupted. Note how the events are numbered (there were 171640 events total), so the thing you see is just a small set of events from the whole big trace, but it's the ones that talk about _that_ particular page.

I've grouped them so hat "consecutive" events group together. That just means that no events on any other pages happened in between those events, and it is usually a sign that it's really one single call-chain that causes all the events.

For example, for the first group of three events (44366-44368), it's the page fault that brings in the page, and since it's a write-fault, it will not only map the page, it will mark the page itself dirty and then also set the TAG_DIRTY on the mapping. So the "group" is just really a result of one single event happening, which causes several things to happen to that page. That's exactly what you'd expect.

Anyway, here is the list of events that page went through:

   44366  PG 00000f6d: mm/memory.c:2254 mapping at b789fc54 (write)
   44367  PG 00000f6d: mm/page-writeback.c:817 setting dirty
   44368  PG 00000f6d: fs/buffer.c:738 setting TAG_DIRTY
   64231  PG 00000f6d: mm/page-writeback.c:872 clean_for_io
   64232  PG 00000f6d: mm/rmap.c:451 cleaning PTE b789f000
   64233  PG 00000f6d: mm/page-writeback.c:914 set writeback
   64234  PG 00000f6d: mm/page-writeback.c:916 setting TAG_WRITEBACK
   64235  PG 00000f6d: mm/page-writeback.c:922 clearing TAG_DIRTY

67570 PG 00000f6d: mm/page-writeback.c:891 end writeback 67571 PG 00000f6d: mm/page-writeback.c:893 clearing TAG_WRITEBACK

   76705  PG 00000f6d: mm/page-writeback.c:817 setting dirty
   76706  PG 00000f6d: fs/buffer.c:725 dirtied buffers
   76707  PG 00000f6d: fs/buffer.c:738 setting TAG_DIRTY
  105267  PG 00000f6d: mm/page-writeback.c:872 clean_for_io
  105268  PG 00000f6d: mm/rmap.c:451 cleaning PTE b789f000
  105269  PG 00000f6d: mm/page-writeback.c:914 set writeback
  105270  PG 00000f6d: mm/page-writeback.c:916 setting TAG_WRITEBACK
  105271  PG 00000f6d: mm/page-writeback.c:922 clearing TAG_DIRTY
  105272  PG 00000f6d: mm/page-writeback.c:891 end writeback
  105273  PG 00000f6d: mm/page-writeback.c:893 clearing TAG_WRITEBACK

128032 PG 00000f6d: mm/memory.c:670 unmapped at b789f000

132662 PG 00000f6d: mm/filemap.c:119 Removing page cache

148278 PG 00000f6d: mm/memory.c:2254 mapping at b789f000 (read)

166326 PG 00000f6d: mm/memory.c:670 unmapped at b789f000

171958 PG 00000f6d: mm/filemap.c:119 Removing page cache

And notice that big grouping of seven events (105267-105273). The five first events really _do_ make sense together: it's our page cleaning that happens. But notice how the "end writeback" happens _immediately_.

Here's another page cleaning event for the page that preceded that page, and did _not_ get corrupted:

  105262  PG 00000f6c: mm/page-writeback.c:872 clean_for_io
  105263  PG 00000f6c: mm/rmap.c:451 cleaning PTE b789e000
  105264  PG 00000f6c: mm/page-writeback.c:914 set writeback
  105265  PG 00000f6c: mm/page-writeback.c:916 setting TAG_WRITEBACK
  105266  PG 00000f6c: mm/page-writeback.c:922 clearing TAG_DIRTY

108437 PG 00000f6c: mm/page-writeback.c:891 end writeback 108438 PG 00000f6c: mm/page-writeback.c:893 clearing TAG_WRITEBACK

and this looks a lot more like what you'd expect: other thngs happened in between the "clear dirty, set writeback" stage and the "end writeback" stage. That's what you'd expect to see if there was actually overlapping IO and/or work.

(And notice that that _was_ what you saw even for the corrupted page for the _first_ writeback: you saw the group-of-five that indicated a page cleaning event had started, and then a group-of-two to indicate that the writeback finished).

So I find this kind of pattern really suspicious. We have a missing writeout, and my traces show (I didn't analyze this _particular_ one closely, but I did the previous trace for another page that I posted) that the writeback was actually started after the write that went missing was done. AND I have this trace that seems to show that the writeback basically completed immediately, with no other work in between.

That to me says: "somebody didn't actually write out out". The VM layer asked the filesystem to do the write, but the filesystem just didn't do it. I personally think it's because some buffer-head BH_dirty bit got scrogged, but it could be some event that makes the filesystem simply not do the IO because it thinks the "disk queues are too full", so it just says "IO completed", without actually doing anything at all.

Now, the fact that it apparently happens for all of ext2, ext3 and reiserfs (but NOT apparently with "data=writeback"), makes me suspect that there is some common interaction, and that it's somehow BH-related (they all share much of the buffer head infrastructure). So it doesn't look like it's just a bug in one random filesystem, I think it's a bug in some buffer-head infrastructure/helper function.

So I don't think it's "core VM". I don't think it's the "page cache". I think we handle the dirty state correctly at that level.

It looks more like "buffer cache" or "filesystem" to me by now.

(Btw, don't get me wrong - the above sequence numbers are in no way *proof* of anything. You could get big groups for one page just because something ended up being synchronous. I'll add some timestamps to my traces to make it easier to see where there was real IO going on and where there wasn't).

Linus

#230 lore raw

Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 3 lines · Linus Torvalds

So everything I have ever seen says that the VM layer is actually doing everything right.

That was true, but at the same time, it's not. Let me explain.

Quoted 4 lines

That to me says: "somebody didn't actually write out out". The VM layer asked the filesystem to do the write, but the filesystem just didn't do it. I personally think it's because some buffer-head BH_dirty bit got scrogged

Ok, I have proof of this now.

Here's a trace (with cycle counts), and with a new trace event added: this is for another corrupted page. I have:

 49105  PG 000015d8 (14800): mm/page-writeback.c:872 clean_for_io
 49106  PG 000015d8 (6900): mm/rmap.c:451 cleaning PTE b7fa6000
 49107  PG 000015d8 (9900): mm/page-writeback.c:914 set writeback
 49108  PG 000015d8 (6480): mm/page-writeback.c:916 setting TAG_WRITEBACK
 49109  PG 000015d8 (7110): mm/page-writeback.c:922 clearing TAG_DIRTY
 49110  PG 000015d8 (7190): fs/buffer.c:1713 no IO underway
 49111  PG 000015d8 (6180): mm/page-writeback.c:891 end writeback
 49112  PG 000015d8 (6460): mm/page-writeback.c:893 clearing TAG_WRITEBACK

where that first column is the trace event number again, and the "PG 000015d8" is that corrupted page. The thing in the parenthesis is "CPU cycles since last event), and the important part to note is that this is indeed all one single thing with no actual IO anywhere (~7000 CPU cycles may sound like a lot, but (a) it's not that many cache misses and (b) a lot of it is the logging overhead - back-to-back log events will take about 3500 cycles) just because it does the actual ASCII printk() etc.

Also, the new event is:

fs/buffer.c:1713 no IO underway

which is just the

	if (nr_underway == 0)

case in fs/buffer.c

And I now finally really believe that I fully understand the corruption, but I don't have a simple solution, much less a patch.

What the problem basically boils down to is that "set_page_dirty()" is supposed to be a mark for dirtying THE WHOLE PAGE, but it really is not "the whole page when the 'set_page_dirty()' itself happens", but more of a "the next writepage() needs to write back the whole page" thing.

And that's not what "__set_page_dirty_buffers()" really does.

Because what "__set_page_dirty_buffers()" does is that AT THE TIME THE "set_page_dirty()" IS CALLED, it will mark all the buffers on that page as dirty. That may _sound_ like what we want, but it really isn't. Because by the time "writepage()" is actually called (which can be MUCH MUCH later), some internal filesystem activity may actually have cleaned one or more of those buffers in the meantime, and now we call "writepage()" (which really wants to write them _all_), and it will write only part of them, or none at all.

So the VM thought that since it did a "writepage()", all the dirty state at that point got written back. But it didn't - the filesystem could have written back part or all of the page much earlier, and the writepage() actually does nothing at all.

Both filesystem and VM actually _think_ they do the right thing, because they simply have totally different expectations. The filesystem thinks that it should care about dirty buffers (that got marked dirty _after_ they were dirtied), while the filesystem thinks that it cares about dirty _pages_ (that got dirted at any time _before_ "writepage()" was called).

Neither is really "wrong", per se, it's just that the two parts have different expectations, and the _combination_ just doesn't work. "set_page_dirty()" at some point meant "the writes have been done", but these days it really means something else.

Now, the reason there is no trivial patch is not that this is conceptually really hard to fix. I can see several different approaches to fixing it, but they all really boil down to two alternatives:

(a) splitting the one "PG_dirty" bit up into two bits: the "PG_writescheduled" bit and the "PG_alldirty" bit.

The "PG_write_scheduled" bit would be the bit that the filesystem would set when it has pending dirty data that it wrote itself (and that may not cover the whole page), and is the part of PG_dirty that sets the PAGECACHE_TAG_DIRTY. It's also what forces "writepage()" to be called.

The "PG_alldirty" bit is just an additional "somebody else dirtied random parts of this page, and we don't know what" flag, which is set by "set_page_dirty()" in addition to doing the PG_write_scheduled stuff. We would test-and-clear it at "writepage()" time, and pass it in to "writepages()" to tell the writepage() function that it can't just write out its own small limited notion of what is dirty.

(There are various variations on this whole theme: instead of having a flag to "writepage()", we could split the "whole page" case out as a separate callback or similar)

(b) making sure that all "set_page_dirty()" calls are _after_ the page has been marked dirty (which in the case of memory mapped pages would mean that we would _not_ call it when we mark the page writable at all, we would call it when we _remove_ the dirty bit and mark it unwritable). That would have the nice fearture that it wouldn't require any FS-level changes, which would be a nice thing - it would basically make the VM dirty accounting work the way the FS layer now already expects it to.

I think (b) is conceptually simpler, and I think I'll try it tomorrow after I've slept on it. The reason I mention (a) at all is that I like the conceptual notion of telling he filesystem ahead of time that "you're going to get a full dirty page", because what (b) will inevitably lead to is that the filesystem will maintain its own partial state, and then effectively just before it gets the writepage() notification, it will be told it was all pointless, because we just dirtied the whole thing.

IOW, the advantage of (a) is also it's disadvantage: it tells the filesystem more. The disadvantage is that it would require VFS interface changes exactly to do that (ie the "mapping->set_page_dirty()" semantics would also be split up into two, and it would now be a "prepare to write the whole page during the next 'writepage()'" thing).

So to recap: the VM layer really expected "writepage()" to act as if it wrote out the whole page. It doesn't. Not in the presense of the buffer layer and the filesystem having written out some buffers independently of the VM layer earlier.

I think this also explains why "data=ordered" broke, and "data=writeback" didn't. When ext3 does "ordered" writebacks, it will do file data writebacks on its own, in _its_ order. In contrast, when it does "data=writeback", it will do the writebacks exactly as the VM presents them, and won't write any buffers on its own - which makes the bug go away, because now VM and FS end up agreeing about the semantics of "writepage()".

Andrew, do you see anything wrong in my thinking?

Peter - on a VM level, the fix would be:

- remove the "set_page_dirty()" from the page fault path, and just set the PAGECACHE_TAG_DIRTY instead.

- clear_page_dirty_for_io() would now need to check the mappings of the page even if it wasn't marked PG_dirty (or we'd have another page flag for the "page is dirty in page tables"), which is kind of a mixture of
(a) and (b) cases above, except we don't expose it to the FS.

- if it was dirty in the page tables, we do a "set_page_dirty()" after cleaning the page tables, and then the rest of "clear_page_dirty_for_io()" really boils down to a simple "TestAndClearDirty(page)"

Hmm? I'd love it if somebody else wrote the patch and tested it, because I'm getting sick and tired of this bug ;)

Linus

#231 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 3 lines · Linus Torvalds

Hmm? I'd love it if somebody else wrote the patch and tested it, because I'm getting sick and tired of this bug ;)

Who the hell am I kidding? I haven't been able to sleep right for the last few days over this bug. It was really getting to me.

And putting on the thinking cap, there's actually a fairly simple an nonintrusive patch. It still has a tiny tiny race (see the comment), but I bet nobody can really hit it in real life anyway, and I know several ways to fix it, so I'm not really _that_ worried about it.

The patch is mostly a comment. The "real" meat of it is actually just a few lines.

Can anybody get corruption with this thing applied? It goes on top of plain v2.6.20-rc2.

Linus

----

mm/page-writeback.c +37 -8
index b3a198c..ec01da1 100644
@@ -862,17 +862,46 @@ int clear_page_dirty_for_io(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);
 
-	if (!mapping)
-		return TestClearPageDirty(page);
-
-	if (TestClearPageDirty(page)) {
-		if (mapping_cap_account_dirty(mapping)) {
-			page_mkclean(page);
+	if (mapping && mapping_cap_account_dirty(mapping)) {
+		/*
+		 * Yes, Virginia, this is indeed insane.
+		 *
+		 * We use this sequence to make sure that
+		 *  (a) we account for dirty stats properly
+		 *  (b) we tell the low-level filesystem to
+		 *      mark the whole page dirty if it was
+		 *      dirty in a pagetable. Only to then
+		 *  (c) clean the page again and return 1 to
+		 *      cause the writeback.
+		 *
+		 * This way we avoid all nasty races with the
+		 * dirty bit in multiple places and clearing
+		 * them concurrently from different threads.
+		 *
+		 * Note! Normally the "set_page_dirty(page)"
+		 * has no effect on the actual dirty bit - since
+		 * that will already usually be set. But we
+		 * need the side effects, and it can help us
+		 * avoid races.
+		 *
+		 * We basically use the page "master dirty bit"
+		 * as a serialization point for all the different
+		 * threds doing their things.
+		 *
+		 * FIXME! We still have a race here: if somebody
+		 * adds the page back to the page tables in
+		 * between the "page_mkclean()" and the "TestClearPageDirty()",
+		 * we might have it mapped without the dirty bit set.
+		 */
+		if (page_mkclean(page))
+			set_page_dirty(page);
+		if (TestClearPageDirty(page)) {
 			dec_zone_page_state(page, NR_FILE_DIRTY);
+			return 1;
 		}
-		return 1;
+		return 0;
 	}
-	return 0;
+	return TestClearPageDirty(page);
 }
 EXPORT_SYMBOL(clear_page_dirty_for_io);
 
#232 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 14 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

Hmm? I'd love it if somebody else wrote the patch and tested it, because I'm getting sick and tired of this bug ;)

Who the hell am I kidding? I haven't been able to sleep right for the last few days over this bug. It was really getting to me.

And putting on the thinking cap, there's actually a fairly simple an nonintrusive patch. It still has a tiny tiny race (see the comment), but I bet nobody can really hit it in real life anyway, and I know several ways to fix it, so I'm not really _that_ worried about it.

Quoted 5 lines

The patch is mostly a comment. The "real" meat of it is actually just a few lines.

Can anybody get corruption with this thing applied? It goes on top of plain v2.6.20-rc2.

Tested with rtorrent and there is no corruption.

Quoted 63 lines

Linus

----

mm/page-writeback.c +37 -8
index b3a198c..ec01da1 100644
@@ -862,17 +862,46 @@ int clear_page_dirty_for_io(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);
 
-	if (!mapping)
-		return TestClearPageDirty(page);
-
-	if (TestClearPageDirty(page)) {
-		if (mapping_cap_account_dirty(mapping)) {
-			page_mkclean(page);
+	if (mapping && mapping_cap_account_dirty(mapping)) {
+		/*
+		 * Yes, Virginia, this is indeed insane.
+		 *
+		 * We use this sequence to make sure that
+		 *  (a) we account for dirty stats properly
+		 *  (b) we tell the low-level filesystem to
+		 *      mark the whole page dirty if it was
+		 *      dirty in a pagetable. Only to then
+		 *  (c) clean the page again and return 1 to
+		 *      cause the writeback.
+		 *
+		 * This way we avoid all nasty races with the
+		 * dirty bit in multiple places and clearing
+		 * them concurrently from different threads.
+		 *
+		 * Note! Normally the "set_page_dirty(page)"
+		 * has no effect on the actual dirty bit - since
+		 * that will already usually be set. But we
+		 * need the side effects, and it can help us
+		 * avoid races.
+		 *
+		 * We basically use the page "master dirty bit"
+		 * as a serialization point for all the different
+		 * threds doing their things.
+		 *
+		 * FIXME! We still have a race here: if somebody
+		 * adds the page back to the page tables in
+		 * between the "page_mkclean()" and the "TestClearPageDirty()",
+		 * we might have it mapped without the dirty bit set.
+		 */
+		if (page_mkclean(page))
+			set_page_dirty(page);
+		if (TestClearPageDirty(page)) {
 			dec_zone_page_state(page, NR_FILE_DIRTY);
+			return 1;
 		}
-		return 1;
+		return 0;
 	}
-	return 0;
+	return TestClearPageDirty(page);
 }
 EXPORT_SYMBOL(clear_page_dirty_for_io);
 
#233 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds

Hey nice work Linus!

Quoted 9 lines · Linus Torvalds
Quoted 1 line · Linus Torvalds

Hmm? I'd love it if somebody else wrote the patch and tested it, because I'm getting sick and tired of this bug ;)

Who the hell am I kidding? I haven't been able to sleep right for the last few days over this bug. It was really getting to me.

And putting on the thinking cap, there's actually a fairly simple an nonintrusive patch.

Yeah *this* makes more sense. And in retrospect it was simple, we can't just throw out pte dirtiness information if the page doesn't have all buffers dirtied.

Quoted 1 line

It still has a tiny tiny race (see the comment), but I bet nobody can really hit it in real life anyway, and I know several ways to fix it, so I'm not really _that_ worried about it.

Well the race isn't a data loss one, is it? Just a case where the pte may be dirty but the page dirty state not accounted for.

Can we fix it by just putting the page_mkclean back inside the TestClearPageDirty check, and re-clearing PG_dirty after redoing the set_page_dirty?

Quoted 68 lines

The patch is mostly a comment. The "real" meat of it is actually just a few lines.

Can anybody get corruption with this thing applied? It goes on top of plain v2.6.20-rc2.

Linus

----

mm/page-writeback.c +37 -8
index b3a198c..ec01da1 100644
@@ -862,17 +862,46 @@ int clear_page_dirty_for_io(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);
 
-	if (!mapping)
-		return TestClearPageDirty(page);
-
-	if (TestClearPageDirty(page)) {
-		if (mapping_cap_account_dirty(mapping)) {
-			page_mkclean(page);
+	if (mapping && mapping_cap_account_dirty(mapping)) {
+		/*
+		 * Yes, Virginia, this is indeed insane.
+		 *
+		 * We use this sequence to make sure that
+		 *  (a) we account for dirty stats properly
+		 *  (b) we tell the low-level filesystem to
+		 *      mark the whole page dirty if it was
+		 *      dirty in a pagetable. Only to then
+		 *  (c) clean the page again and return 1 to
+		 *      cause the writeback.
+		 *
+		 * This way we avoid all nasty races with the
+		 * dirty bit in multiple places and clearing
+		 * them concurrently from different threads.
+		 *
+		 * Note! Normally the "set_page_dirty(page)"
+		 * has no effect on the actual dirty bit - since
+		 * that will already usually be set. But we
+		 * need the side effects, and it can help us
+		 * avoid races.
+		 *
+		 * We basically use the page "master dirty bit"
+		 * as a serialization point for all the different
+		 * threds doing their things.
+		 *
+		 * FIXME! We still have a race here: if somebody
+		 * adds the page back to the page tables in
+		 * between the "page_mkclean()" and the "TestClearPageDirty()",
+		 * we might have it mapped without the dirty bit set.
+		 */
+		if (page_mkclean(page))
+			set_page_dirty(page);
+		if (TestClearPageDirty(page)) {
 			dec_zone_page_state(page, NR_FILE_DIRTY);
+			return 1;
 		}
-		return 1;
+		return 0;
 	}
-	return 0;
+	return TestClearPageDirty(page);
 }
 EXPORT_SYMBOL(clear_page_dirty_for_io);
 
Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#234 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 7 lines · Nick Piggin
Quoted 4 lines

It still has a tiny tiny race (see the comment), but I bet nobody can really hit it in real life anyway, and I know several ways to fix it, so I'm not really _that_ worried about it.

Well the race isn't a data loss one, is it? Just a case where the pte may be dirty but the page dirty state not accounted for.

Right. We should be picking it up eventually, since it's still in the page tables, but if we've lost sight of the page dirtyness we won't react correctly to msync() and/or fdatasync(). So we don't _lose_ the data, we just might not write it out in a timely manner if we ever hit the race.

Quoted 3 lines

Can we fix it by just putting the page_mkclean back inside the TestClearPageDirty check, and re-clearing PG_dirty after redoing the set_page_dirty?

I considered it, but quite frankly, if we did it that way, I'd really like to just fix the whole insane "set_page_dirty()" instead.

I think set_page_dirty() should be split up. One thing that confused me mentally was that almost all of the dirty handling was actualyl done only if PG_dirty wasn't already set, so the _bulk_ of set_page_dirty() really ends up being

	if (!TestSetPageDirty(page)) {
		.. we just marked the page dirty, it was clean before,
		   so we need to add it to the queues etc ..
	}

and that's the part that I (and probably others) always really thought about.

But then we have the _one_ thing that runs outside of that "do only once per dirty bit" logic, and it's the buffer dirtying. If we had had two separate operations for this all: "set_dirty_every_time()" and the regular "set_dirty()", I don't think this would have been nearly as confusing.

(And then the difference between "__set_page_dirty_nobuffers()" and "__set_page_dirty_buffers()" really boils down to one doing the "everytime" _and_ the "once per dirty" checks and the other one doing just the "once per dirty bit" act - and we could rename the damn things to something saner too).

If we split it up that way, then the whole clear_page_dirty_for_io() logic would boil down to

	if (TestClearPageDirty(page)) {
		if (page_mkclean(page))
			set_dirty_every_time();
		return 1;
	}
	return 0;

and we wouldn't even need to do any of the "clear dirty again" kind of idiocy, because the "set_dirty_every_time()" stuff is the one that doesn't even care about the state of the PG_dirty bit - it's done regardless, and doesn't really touch it.

That's what I wanted to do, but with the current "set_page_dirty()" setup, I think my patch makes reasonable sense.

Linus

#235 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 2 lines

Hmm? I'd love it if somebody else wrote the patch and tested it, because I'm getting sick and tired of this bug ;)

Who the hell am I kidding? I haven't been able to sleep right for the last few days over this bug. It was really getting to me.

And putting on the thinking cap, there's actually a fairly simple an nonintrusive patch. [...]

ok, your patch seems to fix the testcase here too on -rc2-rt.

[ Damn, i should have slept a bit more, that would have saved me a ~4
hour debug and tracing session today to analyze your testcase, just to find your patch and your explanation on lkml, right after i sent my analysis and workaround patch ;-) At least now we know it from two independent tracing results that the suspect code is the same. ]

Ingo

#236 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

[...] The patch is mostly a comment. The "real" meat of it is actually just a few lines.

Can anybody get corruption with this thing applied? It goes on top of plain v2.6.20-rc2.

No corruption with the testcase here. Will check with rtorrent too later today but I suppose it will work just fine. Nice work! It has been interesting (and educating) to follow this bug-hunt :)

/Martin

#237 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Can anybody get corruption with this thing applied? It goes on top of plain v2.6.20-rc2.

It works for me now, both your testcase as well as an installation of Debian on this ARM device. I manually applied the patch to 2.6.19.

Thanks.

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#238 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 8 lines · Martin Michlmayr
Quoted 4 lines · Linus Torvalds

Can anybody get corruption with this thing applied? It goes on top of plain v2.6.20-rc2.

Quoted 6 lines

It works for me now, both your testcase as well as an installation of Debian on this ARM device. I manually applied the patch to 2.6.19.

Thanks.

Hi Martin,

Can you post a diff against 2.6.19?

Thanks,
Steve

Signature · 3 lines
"They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty decreases."  (Thomas Jefferson)
#239 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 4 lines · Stephen Clark
Quoted 2 lines

It works for me now, both your testcase as well as an installation of Debian on this ARM device. I manually applied the patch to 2.6.19.

Can you post a diff against 2.6.19?

mm/page-writeback.c +35 -6
@@ -893,16 +893,45 @@
 {
 	struct address_space *mapping = page_mapping(page);
 
-	if (mapping) {
+	if (mapping && mapping_cap_account_dirty(mapping)) {
+		/*
+		 * Yes, Virginia, this is indeed insane.
+		 *
+		 * We use this sequence to make sure that
+		 *  (a) we account for dirty stats properly
+		 *  (b) we tell the low-level filesystem to
+		 *      mark the whole page dirty if it was
+		 *      dirty in a pagetable. Only to then
+		 *  (c) clean the page again and return 1 to
+		 *      cause the writeback.
+		 *
+		 * This way we avoid all nasty races with the
+		 * dirty bit in multiple places and clearing
+		 * them concurrently from different threads.
+		 *
+		 * Note! Normally the "set_page_dirty(page)"
+		 * has no effect on the actual dirty bit - since
+		 * that will already usually be set. But we
+		 * need the side effects, and it can help us
+		 * avoid races.
+		 *
+		 * We basically use the page "master dirty bit"
+		 * as a serialization point for all the different
+		 * threds doing their things.
+		 *
+		 * FIXME! We still have a race here: if somebody
+		 * adds the page back to the page tables in
+		 * between the "page_mkclean()" and the "TestClearPageDirty()",
+		 * we might have it mapped without the dirty bit set.
+		 */
+		if (page_mkclean(page))
+			set_page_dirty(page);
 		if (TestClearPageDirty(page)) {
-			if (mapping_cap_account_dirty(mapping)) {
-				page_mkclean(page);
-				dec_zone_page_state(page, NR_FILE_DIRTY);
-			}
+			dec_zone_page_state(page, NR_FILE_DIRTY);
 			return 1;
 		}
 		return 0;
-	}
+ 	}
 	return TestClearPageDirty(page);
 }
 EXPORT_SYMBOL(clear_page_dirty_for_io);
Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#240 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds
Quoted 27 lines · On Fri, 29 Dec 2006 02:48:35 -0800 (PST) Linus Torvalds <[email protected]>…
mm/page-writeback.c +27 -0
+	if (mapping && mapping_cap_account_dirty(mapping)) {
+		/*
+		 * Yes, Virginia, this is indeed insane.
+		 *
+		 * We use this sequence to make sure that
+		 *  (a) we account for dirty stats properly
+		 *  (b) we tell the low-level filesystem to
+		 *      mark the whole page dirty if it was
+		 *      dirty in a pagetable. Only to then
+		 *  (c) clean the page again and return 1 to
+		 *      cause the writeback.
+		 *
+		 * This way we avoid all nasty races with the
+		 * dirty bit in multiple places and clearing
+		 * them concurrently from different threads.
+		 *
+		 * Note! Normally the "set_page_dirty(page)"
+		 * has no effect on the actual dirty bit - since
+		 * that will already usually be set. But we
+		 * need the side effects, and it can help us
+		 * avoid races.
+		 *
+		 * We basically use the page "master dirty bit"
+		 * as a serialization point for all the different
+		 * threds doing their things.
+		 *
+		 * FIXME! We still have a race here: if somebody

mm/page-writeback.c int clear_page_dirty_for_io(struct page *page) · line 901

Quoted 10 lines
mm/page-writeback.c +8 -0
+		 * adds the page back to the page tables in
+		 * between the "page_mkclean()" and the "TestClearPageDirty()",
+		 * we might have it mapped without the dirty bit set.
+		 */
+		if (page_mkclean(page))
+			set_page_dirty(page);
+		if (TestClearPageDirty(page)) {
 			dec_zone_page_state(page, NR_FILE_DIRTY);
+			return 1;
 		}

- Presumably reiser3's ordered-data mode has the same problem. And ext4, of course. Dunno about other filesytems.

- The above change means that we do extra writeout. If a page is dirtied once, kjournald will write it and then pdflush will come along and needlessly write it again.

But otoh, if a mapping is being repeatedly dirtied, kjournald will write the page once per 30 seconds (dirty_expire_centisecs) and pdflush will write the page once per 30 seconds as well. But we _should_ be writing it once per five seconds (kjournald commit interval). So we're still ahead ;)

- Poor old IO accounting broke again.

- People were saying that ext2 and ext3,data=writeback were also showing corruption. What's up with that?

- For a long time I've wanted to nuke the current ext3/jbd ordered-data implementation altogether, and just make kjournald call into the standard writeback code to do a standard suberblock->inodes->pages walk.

I think it'd be fairly straightforward to do. We'd need to teach the writeback code to be able to skip dirty pages which don't have a disk mapping, so that kjournald doesn't end up waiting for kjournald to free up journal space..

Would need to avoid possible deadlocks where someone calls ext3_force_commit() or otherwise does a synchronous commit while holding VFS locks.

reiser3 and ext4 could be converted too.

Not a short-term project, but this would avoid the problem.

- It's pretty obnoxious that the VM now sets a clean page "dirty" and then proceeds to modify its contents. It would be nice to stop doing that.

We could stop marking the page dirty in do_wp_page() and create a new VM counter "NR_PTE_DIRTY", which means

"number of mapping_cap_account_dirty() pages which have a dirty pte pointing at them".

Or, perhaps

"number of dirty ptes which point at mapping_cap_account_dirty() pages".

Which can be larger, but the writeout code will probably cope.

Then we take NR_PTE_DIRTY into account in the dirty-page balancing act. So

- do_wp_page() will still run balance_dirty_pages()

- but it would no longer run set_page_dirty().

- But it needs to run mark_inode_dirty() so the fs-writeback code notices the file.

- And mapping_tagged(mapping, PAGECACHE_TAG_DIRTY) becomes insufficient.

The tricky part here is "how do we do the writeback"? The pte-dirty,!PageDirty pages aren't tagged as dirty in the radix-tree and writeback needs to find them so that it can effectively do an msync() on them. Walking all the mm's and vma's would be insane. Visiting all the pages in the file would also probably be insane.

Perhaps this can be solved by adding a new radix-tree tag which means "this page might have dirty ptes pointing at it". For each file writeback would do a radix-tree walk of these pages, cleaning-and-write-protecting ptes, marking the corresponding pages dirty and clearing their PAGECACHE_TAG_PTE_DIRTY tags.

Then we can fix the mapping_tagged(mapping, PAGECACHE_TAG_DIRTY) problem by doing

    mapping_tagged(mapping, PAGECACHE_TAG_DIRTY) ||
	mapping_tagged(mapping, PAGECACHE_TAG_PTE_DIRTY)

or, better,

	mapping_tagged(mapping,
		(1<<PAGECACHE_TAG_DIRTY)|(1<<PAGECACHE_TAG_PTE_DIRTY))

perhaps.

The msync() code would need to be taught to call the PAGECACHE_TAG_PTE_DIRTY walker for the appropriate page range.

This is also not a quick-fix.

#241 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 1 line · On Fri, 29 Dec 2006 14:16:32 -0800 Andrew Morton <[email protected]> wrote:

- Poor old IO accounting broke again.

No it didn't - we're relying upon the behaviour of __set_page_dirty_buffers() against an already-dirty page.

#242 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

- The above change means that we do extra writeout. If a page is dirtied once, kjournald will write it and then pdflush will come along and needlessly write it again.

There's zero extra writeout for any flushing that flushes BY PAGES.

Only broken flushers that flush by buffer heads (which really really really shouldn't be done any more: welcome to the 21st century) will cause extra writeouts. And those extra writeouts are obviously required for all the dirty state to actually hit the disk - which is the point of the patch.

So they're not "extra" - they are "required for correct working".

But I can't stress the fact enough that people SHOULD NOT do writeback by buffer heads. The buffer head has been purely an "IO entity" for the last several years now, and it's not a cache entity. Anybody who does writeback by buffer heads is basically bypassing the real cache (the page cache), and that's why all the problems happen.

I think ext3 is terminally crap by now. It still uses buffer heads in places where it really really shouldn't, and as a result, things like directory accesses are simply slower than they should be. Sadly, I don't think ext4 is going to fix any of this, either.

It's all just too inherently wrongly designed around the buffer head (which was correct in 1995, but hasn't been correct for a long time in the kernel any more).

Quoted 1 line

- Poor old IO accounting broke again.

No. That's why I used "set_page_dirty()" and did it that strange ugly way ("set page dirty, even though it's already dirty, and even though the very next thing we will do is TestClearPageDirty???").

That code looks strange as a result, which is why it now has more comments on it than actual code ;)

Quoted 2 lines

- People were saying that ext2 and ext3,data=writeback were also showing corruption. What's up with that?

I thought the "ext3,data=writeback" case was reported to be fine by several people?

I'm not sure about ext2. I didn't look at what it did based on buffer heads. I would have expected it to be ok.

That said, at least one report was later shown to be bogus (errors due to out of disk, not due to actual errors ;).

Quoted 3 lines

- For a long time I've wanted to nuke the current ext3/jbd ordered-data implementation altogether, and just make kjournald call into the standard writeback code to do a standard suberblock->inodes->pages walk.

I really would like to see less of the buffer-head-based stuff, and yes, more of the normal inode page walking. I don't think you can "order" accesses within a page anyway, exactly because of memory mapping issues, so any page ordering is not about buffer heads on the page itself, it should be purely about metadata.

Quoted 3 lines

- It's pretty obnoxious that the VM now sets a clean page "dirty" and then proceeds to modify its contents. It would be nice to stop doing that.

No. I think this really the fundamental confusion people had. People thought that setting the page dirty meant that it was no longer being modified. It hasn't meant that in a LONG time - ever since the whole DIRTY_TAG thing, the most important part of the PG_dirty thing has really been that it's now efficiently findable by the writeout logic.

And that is very much what the whole page accounting _depends_ on. When we mmap a page, we need to mark it "findable" as dirty _before_ people actually start writing to it, because it's too late afterwards.

Quoted 5 lines

We could stop marking the page dirty in do_wp_page() and create a new VM counter "NR_PTE_DIRTY", which means

"number of mapping_cap_account_dirty() pages which have a dirty pte pointing at them".

Well, then you need to change what PAGE_MAPPING_TAG_DIRTY means too.

That's very fundamental. That DIRTY _tag_ is now even more important than the PG_dirty bit itself, since that's what we actually use to _access_ those things.

Linus

#243 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 4 lines · Linus Torvalds

I think ext3 is terminally crap by now. It still uses buffer heads in places where it really really shouldn't, and as a result, things like directory accesses are simply slower than they should be. Sadly, I don't think ext4 is going to fix any of this, either.

Not just ext3; ocfs2 is using the jbd layer as well. I think we're going to have to put this (a rework of jbd2 to use the page cache) on the ext4 todo list, and work with the ocfs2 folks to try to come up with something that suits their needs as well. Fortunately we have this filesystem/storage summit thing coming up in the next few months, and we can try to get some discussion going on the linux-ext4 mailing list in the meantime. Unfortunately, I don't think this is going to be trivial.

If we do get this fixed for ext4, one interesting question is whether people would accept a patch to backport the fixes to ext3, given the the grief this is causing the page I/O and VM routines. OTOH, reiser3 probably has the same problems, and I suspect the changes to ext3 to cause it to avoid buffer heads, especially in order to support for filesystem blocksizes < pagesize, are going to be sufficiently risky in terms of introducing regressions to ext3 that they would probably be rejected on those grounds. So unfortunately, we probably are going to have to support flushes via buffer heads for the foreseeable future.

- Ted

#244 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 4 lines · Theodore Tso

If we do get this fixed for ext4, one interesting question is whether people would accept a patch to backport the fixes to ext3, given the the grief this is causing the page I/O and VM routines.

I don't think backporting is the smartest option (unless it's done _way_ later), but the real problem with it isn't actually the VM behaviour, but simply the fact that cached performance absoluely _sucks_ with the buffer cache.

With the physically indexed buffer cache thing, you end up always having to do these complicated translations into block numbers for every single access, and at some point when I benchmarked it, it was a huge overhead for doing simple things like readdir.

It's also a major pain for read-ahead, exactly partly due to the high cost of translation - because you can't cheaply check whether the next block is there, the cost of even asking the question "should I try to read ahead?" is much much higher. As a result, read-ahead is seriously limited, because it's so expensive for the cached case (which is still hopefully the _common_ case).

So because read-ahead is limited, the non-cached case then _really_ sucks.

It was somewhat fixed in a really god-awful fashion by having ext3_readdir() actually do _readahead_ though the page cache, even though it does everything else through the buffer cache. And that just happens to work because we hopefully have physically contiguous blocks, but when that isn't true, the readahead doesn't do squat.

It's really quite fundamentally broken. But none of that causes any problems for the VM, since directories cannot be mmap'ed anyway. But it's really pitiful, and it really doesn't work very well. Of course, other filesystems _also_ suck at this, and other operating systems haev even MORE problems, so people don't always seem to realize how horribly horribly broken this all is.

I really wish somebody would write a filesystem that did large cold-cache directories well. Open some horrible file manager on /usr/bin with cold caches, and weep. The biggest problem is the inode indirection, but at some point when I looked at why it sucked, it was doing basically synchronous single-buffer reads on the directory too, because readahead didn't work properly.

I was hoping that something like SpadFS would actually take off, because it seemed to do a lot of good design choices (having inodes in-line in the directory for when there are no hardlinks is probably a requirement for a good filesystem these days. The separate inode table had its uses, but indirection in a filesystem really does suck, and stat information is too important to be indirect unless it absolutely has to).

But I suspect it needs more than somebody who just wants to get his thesis written ;)

Linus

#245 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Theodore Tso
Quoted 6 lines · On Fri, 29 Dec 2006 18:32:07 -0500 Theodore Tso <[email protected]> wrote:
Quoted 4 lines · Linus Torvalds

I think ext3 is terminally crap by now. It still uses buffer heads in places where it really really shouldn't, and as a result, things like directory accesses are simply slower than they should be. Sadly, I don't think ext4 is going to fix any of this, either.

Quoted 8 lines

Not just ext3; ocfs2 is using the jbd layer as well. I think we're going to have to put this (a rework of jbd2 to use the page cache) on the ext4 todo list, and work with the ocfs2 folks to try to come up with something that suits their needs as well. Fortunately we have this filesystem/storage summit thing coming up in the next few months, and we can try to get some discussion going on the linux-ext4 mailing list in the meantime. Unfortunately, I don't think this is going to be trivial.

I suspect it would be insane to move any part of JBD (apart from the ordered-data flush) to use pagecache. The whole thing is fundamentally block-based. But only for metadata - there's no strong reason why ext3/4 needs to manipulate file data via buffer_heads if data=journal and chattr +j aren't in use.

We could possibly move ext3/4 directories out of the blockdev pagecache and into per-directory pagecache, but that wouldn't change anything - the journalling would still be block-based.

Adam Richter spent considerable time a few years ago trying to make the mpage code go direct-to-BIO in all cases and we eventually gave up. The conceptual layering of page<->blocks<->bio is pretty clean, and it is hard and ugly to fully optimise away the "block" bit in the middle.

buffer_heads become more important with large PAGE_CACHE_SIZE. I'd expect nobh mode to be quite inefficient with some workloads on 64k pages. We need that representation of the state (and location) of the block-sized hunks which make up the page.

Quoted 10 lines

If we do get this fixed for ext4, one interesting question is whether people would accept a patch to backport the fixes to ext3, given the the grief this is causing the page I/O and VM routines. OTOH, reiser3 probably has the same problems, and I suspect the changes to ext3 to cause it to avoid buffer heads, especially in order to support for filesystem blocksizes < pagesize, are going to be sufficiently risky in terms of introducing regressions to ext3 that they would probably be rejected on those grounds. So unfortunately, we probably are going to have to support flushes via buffer heads for the foreseeable future.

We'll see.

#246 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 5 lines · Andrew Morton

Adam Richter spent considerable time a few years ago trying to make the mpage code go direct-to-BIO in all cases and we eventually gave up. The conceptual layering of page<->blocks<->bio is pretty clean, and it is hard and ugly to fully optimise away the "block" bit in the middle.

Using the buffer cache as a translation layer to the physical address is fine. That's what _any_ block device will do.

I'm not at all sayign that "buffer heads must go away". They work fine.

What I'm saying is that

- if you index by buffer heads, you're screwed.
- if you do IO by starting at buffer heads, you're screwed.

Both indexing and writeback decisions should be done at the page cache layer. Then, when you actually need to do IO, you look at the buffers. But you start from the "page". YOU SHOULD NEVER LOOK UP a buffer on its own merits, and YOU SHOULD NEVER DO IO on a buffer head on its own cognizance.

So by all means keep the buffer heads as a way to keep the "virtual->physical" translation. It's what they were designed for. But they were _originally_ also designed for "lookup" and "driving the start of IO", and that is wrong, and has been wrong for a long time now, because

- lookup based on physical address is fundamentally slow and inefficient. You have to look up the virtual->physical translation somewhere else, so it's by design an unnecessary indirection _and_ that "somewere else" is also by definition filesystem-specific, so you can't do any of these things at the VFS layer.

Ergo: anything that needs to look up the physical address in order to find the buffer head is BROKEN in this day and age. We look up the _virtual_ page cache page, and then we can trivially find the buffer heads within that page thanks to page->buffers.

Example: ext2 vs ext3 readdir. One of them sucks, the other doesn't.

- starting IO based on the physical entity is insane. It's insane exactly _because_ the VM doesn't actually think in physical addresses, or in buffer-sized blocks. The VM only really knows about whole pages, and all the VM decisions fundamentally have to be page-based. We don't ever "free a buffer". We free a whole page, and as such, doing writeback based on buffers is pointless, because it doesn't actually say anything about the "page state" which is what the VM tracks.

But neither of these means that "buffer_head" itself has to go away. They both really boil down to the same thing: you should never KEY things by the buffer head. All actions should be based on virtual indexes as far as at all humanly possible.

Once you do lookup and locking and writeback _starting_ from the page, it's then easy to look up the actual buffer head within the page, and use that as a way to do the actual _IO_ on the physical address. So the buffer heads still exist in ext2, for example, but they don't drive the show quite as much.

(They still do in some areas: the allocation bitmaps, the xattr code etc. But as long as none of those have big VM footprints, and as long as no _common_ operations really care deeply, and as long as those data structures never need to be touched by the VM or VFS layer, nobody will ever really care).

The directory case comes up just because "readdir()" actually is very common, and sometimes very slow. And it can have a big VM working set footprint ("find"), so trying to be page-based actually really helps, because it all drives things like writeback on the _right_ issues, and we can do things like LRU's and writeback decisions on the level that really matters.

I actually suspect that the inode tables could benefit from being in the page cache too (although I think that the inode buffer address is actually "physical", so there's no indirection for inode tables, which means that the virtual vs physical addressing doesn't matter). For directories, there definitely is a big cost to continually doing the virtual->physical translation all the time.

Linus

#247 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds
Quoted 10 lines · On Fri, 29 Dec 2006 14:42:51 -0800 (PST) Linus Torvalds <[email protected]>…
Quoted 4 lines · Andrew Morton

- The above change means that we do extra writeout. If a page is dirtied once, kjournald will write it and then pdflush will come along and needlessly write it again.

There's zero extra writeout for any flushing that flushes BY PAGES.

Quoted 7 lines

Only broken flushers that flush by buffer heads (which really really really shouldn't be done any more: welcome to the 21st century) will cause extra writeouts. And those extra writeouts are obviously required for all the dirty state to actually hit the disk - which is the point of the patch.

So they're not "extra" - they are "required for correct working".

They're extra. As in "can be optimised away".

Quoted 3 lines

But I can't stress the fact enough that people SHOULD NOT do writeback by buffer heads. The buffer head has been purely an "IO entity" for the last several years now, and it's not a cache entity.

The buffer_head is not an IO container. It is the kernel's core representation of a disk block. Usually (but not always) it is backed by some memory which is in pagecache. We can feed buffer_heads into IO containers via submit_bh(), but that's far from the only thing we use buffer_heads for. We should have done s/buffer_head/block/g years ago.

JBD implements physical block-based journalling, so it is 100% appropriate that JBD deal with these disk blocks using their buffer_head representation.

That being said, ordered-data mode isn't really part of the JBD journalling system at all (the data doesn't get journalled!) - ordered-mode is an add-on to the JBD journal to make the metadata which we're about to journal point at more-likely-to-be-correct data.

JBD's ordered-mode writeback is just a sync and I see no conceptual problems with killing its old buffer_head based sync and moving it into the 21st century.

Quoted 6 lines

Anybody who does writeback
by buffer heads is basically bypassing the real cache (the page cache), and that's why all the problems happen.

I think ext3 is terminally crap by now. It still uses buffer heads in places where it really really shouldn't,

The ordered-data mode flush: sure. The rest of JBD's use of buffer_heads is quite appropriate.

Quoted 3 lines

and as a result, things like
directory accesses are simply slower than they should be. Sadly, I don't think ext4 is going to fix any of this, either.

I thought I fixed the performance problem?

Somewhat nastily, but as ext3 directories are metadata it is appropriate that modifications to them be done in terms of buffer_heads (ie: blocks).

Quoted 9 lines

It's all just too inherently wrongly designed around the buffer head (which was correct in 1995, but hasn't been correct for a long time in the kernel any more).

Quoted 1 line

- Poor old IO accounting broke again.

No. That's why I used "set_page_dirty()" and did it that strange ugly way ("set page dirty, even though it's already dirty, and even though the very next thing we will do is TestClearPageDirty???").

nfs_set_page_dirty() and reiserfs_set_page_dirty() should now bail if PageDirty() to avoid needless work.

Quoted 9 lines
Quoted 3 lines

- For a long time I've wanted to nuke the current ext3/jbd ordered-data implementation altogether, and just make kjournald call into the standard writeback code to do a standard suberblock->inodes->pages walk.

I really would like to see less of the buffer-head-based stuff, and yes, more of the normal inode page walking. I don't think you can "order" accesses within a page anyway, exactly because of memory mapping issues, so any page ordering is not about buffer heads on the page itself, it should be purely about metadata.

In this context ext3's "ordered" mode means "sync the file contents before journalling the metadata which points at it".

Quoted 7 lines
Quoted 3 lines

- It's pretty obnoxious that the VM now sets a clean page "dirty" and then proceeds to modify its contents. It would be nice to stop doing that.

No. I think this really the fundamental confusion people had. People thought that setting the page dirty meant that it was no longer being modified.

No. Setting a page (or bh, or inode) dirty means "this is known to have been modified". ie: this cached entity is now out of sync with backing store.

Ho hum. I don't care much, really. But then, I understand how all this stuff works. Try explaining to someone the relationship between pte-dirtiness, page-dirtiness, radix-tree-dirtiness and buffer_head-dirtiness.

#248 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 2 lines · Andrew Morton

They're extra. As in "can be optimised away".

Sure. Don't use buffer heads.

Quoted 2 lines

The buffer_head is not an IO container. It is the kernel's core representation of a disk block.

Please come back from the 90's.

The buffer heads are nothing but a mapping of where the hardware block is. If you use it for anything else, you're basically screwed.

Quoted 3 lines

JBD implements physical block-based journalling, so it is 100% appropriate that JBD deal with these disk blocks using their buffer_head representation.

And as long as it does that, you just have to face the fact that it's going to perform like crap, including what you call "extra" writes, and what I call "deal with it".

Btw, you can make pages be physically indexed too, but they obviously
(a) won't be coherent with any virtual mapping laid on top of it
(b) will be _physical_, so any readahead etc will be based on physical addresses too.

Quoted 1 line

I thought I fixed the performance problem?

No, you papered over it, for the reasonably common case where things were physically contiguous - exactly by using a physical page cache, so now it can do read-ahead based on that. Then, because the pages contain buffer heads, the directory accesses can look up buffers, and if it was all physically contiguous, it all works fine.

But if you actually want virtualluy indexed caching (and all _users_ want it), it really doesn't work.

Quoted 2 lines

Somewhat nastily, but as ext3 directories are metadata it is appropriate that modifications to them be done in terms of buffer_heads (ie: blocks).

No. There is nothing "appropriate" about using buffer_heads for metadata.

It's quite proper - and a hell of a lot more efficient - to use virtual page-caching for metadata too.

Look at the ext2 readdir() implementation, and compare it to the crapola horror that is ext3. Guess what? ext2 uses virtually indexed metadata, and as a result it is both simpler, smaller and a LOT faster than ext3 in accessing that metadata.

Face it, Andrew, you're wrong on this one. Really. Just take a look at ext2_readdir().

[ I'm not saying that ext2_readdir() is _beautiful_. If it had been
written with the page cache in mind, it would probably have been done very differently. And it doesn't do any readahead, probably because nobody cared enough, but it should be trivial to add, and it would automatically "do the right thing" just because it's much easier at the page cache level.

But I _am_ saying that compared to ext3, the ext2 readdir is a work of art. ]

"metadata" has _zero_ to do with "physically indexed". There is no correlation what-so-ever. If you think there is a correlation, it's all in your mind.

Linus

#249 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 9 lines · On Fri, 29 Dec 2006 16:11:44 -0800 (PST) Linus Torvalds <[email protected]>…
Quoted 5 lines

JBD implements physical block-based journalling, so it is 100% appropriate that JBD deal with these disk blocks using their buffer_head representation.

And as long as it does that, you just have to face the fact that it's going to perform like crap, including what you call "extra" writes, and what I call "deal with it".

It is quite tiresome to delete things which your interlocutor said and to then restate them as if it were some sort of relevation.

Quoted 4 lines
Quoted 2 lines

Somewhat nastily, but as ext3 directories are metadata it is appropriate that modifications to them be done in terms of buffer_heads (ie: blocks).

No. There is nothing "appropriate" about using buffer_heads for metadata.

I said "modification".

Quoted 1 line

[stuff about directory reads elided]

#250 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 7 lines · Andrew Morton
Quoted 5 lines
Quoted 3 lines

Somewhat nastily, but as ext3 directories are metadata it is appropriate that modifications to them be done in terms of buffer_heads (ie: blocks).

No. There is nothing "appropriate" about using buffer_heads for metadata.

I said "modification".

You said "metadata".

Why do you think directories are any different from files? Yes, they are metadata. So what? What does that have to do with anything?

They should still use virtual indexes, the way files do. That doesn't preclude them from using buffer-heads to mark their (partial-page) modifications and for keeping the virtual->physical translations cached.

I mean, really. Look at ext2. It does exactly that. It keeps the directories in the page cache - virtually indexed. And it even modifies them there. Exactly the same way it modifies regular file data.

It all works exactly the same way it works for regular files. It uses

	page->mapping->a_ops->prepare_write(NULL, page, from, to);
	... do modification ...
	ext2_commit_chunk(page, from, to);

exactly the way regular file data works.

That's why I'm saying there is absolutely _zero_ thing about "metadata" here, or even about "modifications". It all works better in a virtual cache, because you get all the support that we give to page caches.

So I really don't understand why you make excuses for ext3 and talk about "modifications" and "metadata". It was a fine design ten years ago. It's not really very good any longer.

I suspect we're stuck with the design, but that doesn't make it any _better_.

Linus

#251 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 7 lines · On Fri, 29 Dec 2006 16:58:41 -0800 (PST) Linus Torvalds <[email protected]>…
Quoted 4 lines · Andrew Morton
Quoted 4 lines
Quoted 3 lines

Somewhat nastily, but as ext3 directories are metadata it is appropriate that modifications to them be done in terms of buffer_heads (ie: blocks).

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

No. There is nothing "appropriate" about using buffer_heads for metadata.

I said "modification".

You said "metadata".

Why do you think directories are any different from files? Yes, they are metadata. So what? What does that have to do with anything?

We journal the contents of directories. Fully. So we handle their dirty data at the block (ie: buffer_head) level. When someone tries to dirty part of a directory we need to cheat and not mark that part of the page as dirty and we need to then write the block to the journal and then mark the block as really dirty for checkpointing (but still attached to the journal) and all that goop.

The regular page-based writeback doesn't apply until the block has been written to the journal. At that stage the block is considered dirty against its real position on disk. It will then be written back by pdflush via the blockdev inode -> blkdev_writepage(). Unless kjournald needs to do an early flush to reclaim the journal space, in which case kjournald will write the block itself.

Quoted 5 lines

So I really don't understand why you make excuses for ext3 and talk about "modifications" and "metadata". It was a fine design ten years ago. It's not really very good any longer.

As I said in another apparently-neglected email:

: We could possibly move ext3/4 directories out of the blockdev pagecache and : into per-directory pagecache, but that wouldn't change anything - the : journalling would still be block-based.

We already have all the code in place to journal blocks which are cached in an address_space other than the blockdev inode's: ext3_journalled_aops.

#252 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

Because what "__set_page_dirty_buffers()" does is that AT THE TIME THE "set_page_dirty()" IS CALLED, it will mark all the buffers on that page as dirty. That may _sound_ like what we want, but it really isn't. Because by the time "writepage()" is actually called (which can be MUCH MUCH later), some internal filesystem activity may actually have cleaned one or more of those buffers in the meantime, and now we call "writepage()" (which really wants to write them _all_), and it will write only part of them, or none at all.

I'm confused. Does this mean that if "fs blocksize"=="VM pagesize" this bug can't trigger? But I thought at least one of people reporting corruption was using a filesystem with a 4k block size on an i386?

- Ted

#253 lore raw

Re: Ok, explained.. (was Re: [PATCH] mm: fix page_mkclean_one)

Quoted 3 lines · Theodore Tso

I'm confused. Does this mean that if "fs blocksize"=="VM pagesize" this bug can't trigger?

No. Even if there is just a single buffer-head, if the filesystem ever writes out that _single_ buffer-head out of turn (ie before the VM actually asks it to, with "->writepage()"), then the same issue will happen.

In fact, a bigger fs blocksize will likely just make this easier to trigger (although I doubt it makes a big difference), since any out-of-order buffer flushback will happen for the whole page, rather than just a part of the page.

So the "problem" really ends up being that the filesystem does flushing that the VM isn't aware of, so when the VM did "set_page_dirty()" at an earlier time, the VM _expected_ the "->writepages()" call that happened much later to write the whole page - but because the FS had flushed things behind it backs even _before_ the "->writepage" happens, by the time the VM actually asks for the page to be written out, the FS layer won't actually write it all out any more.

Blocksize doesn't matter, the only thing that matters is whether something writes out data on a buffer-cache level, not on a "page cache" level. Ext3 apparently does this in "ordered" data more at least (and hey, I suspect that the code that tries to release buffer head data might try to do it on its own too).

Linus

#254 lore raw

PATCH 1/1 fix data corruption bug in __block_write_full_page()

In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

I do have a few interesting details from the trace I haven't really analyzed yet. Here's the trace for events on one of the pages that was corrupted. Note how the events are numbered (there were 171640 events total), so the thing you see is just a small set of events from the whole big trace, but it's the ones that talk about _that_ particular page.

i've extended the tracer in -rt to trace all relevant pagetable, pagecache, buffer-cache and IO events and coupled the tracer to your test.c code. The corruption happens here:

    test-2126  0.... 3756170us+: trace_page (cf20ebd8 b6a2c000 0)
 pdflush-2006  0.... 6432909us+: trace_page (cf20ebd8 b6a2c000 4200420)
    test-2126  0.... 8135596us+: trace_page (cf20ebd8 b6a2c000 4200420)
    test-2126  0D... 9012933us+: do_page_fault (8048900 4 b6a2c000)
    test-2126  0.... 9023278us+: trace_page (cf262f24 b6a2c000 0)
    test-2126  0.... 9023305us > sys_prctl (000000d8 b6a2c000 000000ac)

address 0xb6a2c000 is the one that shows the corruption. Now, this address is mapped to page cf262f24 when the bug happened, but it had page 0xcf20ebd8 mapped to it 3 seconds ago, which has this history:

    test-2126  0.... 3756413us+: trace_page (cf20ebd8 0 0)
    test-2126  0.... 3756469us+: trace_page (cf20ebd8 0 0)
    test-2126  0.... 3757341us+: trace_page (cf20ebd8 10 0)
  IRQ-14-402   0.... 3759332us+: trace_page (cf20ebd8 ffffffff 0)
  IRQ-14-402   0.... 3759376us+: trace_page (cf20ebd8 ffffffff 0)
    test-2126  0.... 5104662us+: trace_page (cf20ebd8 b6a2c400 0)
    test-2126  0.... 5104687us+: trace_page (cf20ebd8 1 0)
 pdflush-2006  0.... 6432909us+: trace_page (cf20ebd8 b6a2c000 4200420)
 pdflush-2006  0.... 6432952us+: trace_page (cf20ebd8 ffffffff 4200420)
 pdflush-2006  0.... 6432986us+: trace_page (cf20ebd8 1 4200420)
 pdflush-2006  0.... 6433022us+: trace_page (cf20ebd8 4096 4200420)
 pdflush-2006  0.... 6433061us+: trace_page (cf20ebd8 0 4200420)
 pdflush-2006  0.... 6433112us+: trace_page (cf20ebd8 0 4200420)
 pdflush-2006  0.... 6433154us+: trace_page (cf20ebd8 0 4200420)
 pdflush-2006  0.... 6433303us+: trace_page (cf20ebd8 11 4200420)
 pdflush-2006  0.... 6433343us+: trace_page (cf20ebd8 13 4200420)
 pdflush-2006  0.... 6433382us+: trace_page (cf20ebd8 14 4200420)
 pdflush-2006  0.... 6433421us+: trace_page (cf20ebd8 15 4200420)
 pdflush-2006  0.... 6433460us+: trace_page (cf20ebd8 ffffffff 4200420)
 pdflush-2006  0.... 6433504us+: trace_page (cf20ebd8 ffffffff 4200420)
    test-2126  0.... 8135596us+: trace_page (cf20ebd8 b6a2c000 4200420)

in particular timestamp 6433421us is interesting:

 pdflush-2006  0.... 6433504us+: trace_page (cf20ebd8 ffffffff 4200420)
 pdflush-2006  0.... 6433526us : trace_page()<-test_clear_page_writeback()<-end_page_writeback()<-__block_write_full_page()
 pdflush-2006  0.... 6433526us+: block_write_full_page()<-ext3_ordered_writepage()<-generic_writepages()<-(-1)()

i.e. the page got its pending writeback cancelled in block_write_full_page(), without any IRQ#14 activity whatsoever! That looks quite suspect. It is this piece of code in __block_write_full_page():

                /*
                 * The page was marked dirty, but the buffers were
                 * clean.  Someone wrote them back by hand with
                 * ll_rw_block/submit_bh.  A rare case.
                 */
                ....
                if (uptodate)
                        SetPageUptodate(page);
                end_page_writeback(page);

A 'rare case' ... hm. So i tried a quick workaround below, just to keep us from marking the page clean, to see whether the corruption goes away
- and i was unable to trigger the corruption after half an hour of testing, while before it triggered within 10 seconds!

now this patch is only an ugly hack, but the bug definitely seems to be related to buffer management, as you suspected.

Ingo

---
 fs/buffer.c |    1 +
 1 file changed, 1 insertion(+)
fs/buffer.c +1 -0
Index: linux/fs/buffer.c
===================================================================
@@ -1702,6 +1702,7 @@ done:
 		} while (bh != head);
 		if (uptodate)
 			SetPageUptodate(page);
+		set_page_dirty(page);
 		end_page_writeback(page);
 		/*
 		 * The page and buffer_heads can be released at any time from
#255 lore raw

Re: [patch] fix data corruption bug in __block_write_full_page()

Quoted 10 lines · Ingo Molnar

i've extended the tracer in -rt to trace all relevant pagetable, pagecache, buffer-cache and IO events and coupled the tracer to your test.c code. The corruption happens here:

    test-2126  0.... 3756170us+: trace_page (cf20ebd8 b6a2c000 0)
 pdflush-2006  0.... 6432909us+: trace_page (cf20ebd8 b6a2c000 4200420)
    test-2126  0.... 8135596us+: trace_page (cf20ebd8 b6a2c000 4200420)
    test-2126  0D... 9012933us+: do_page_fault (8048900 4 b6a2c000)
    test-2126  0.... 9023278us+: trace_page (cf262f24 b6a2c000 0)
    test-2126  0.... 9023305us > sys_prctl (000000d8 b6a2c000 000000ac)

This tracer definitly looks interesting. Could you send a splitout patch with it to lkml for review?

#256 lore raw

Re: [patch] fix data corruption bug in __block_write_full_page()

[Cc:-ed lkml]

Quoted 12 lines · Christoph Hellwig
Quoted 10 lines · Ingo Molnar

i've extended the tracer in -rt to trace all relevant pagetable, pagecache, buffer-cache and IO events and coupled the tracer to your test.c code. The corruption happens here:

    test-2126  0.... 3756170us+: trace_page (cf20ebd8 b6a2c000 0)
 pdflush-2006  0.... 6432909us+: trace_page (cf20ebd8 b6a2c000 4200420)
    test-2126  0.... 8135596us+: trace_page (cf20ebd8 b6a2c000 4200420)
    test-2126  0D... 9012933us+: do_page_fault (8048900 4 b6a2c000)
    test-2126  0.... 9023278us+: trace_page (cf262f24 b6a2c000 0)
    test-2126  0.... 9023305us > sys_prctl (000000d8 b6a2c000 000000ac)
Quoted 2 lines

This tracer definitly looks interesting. Could you send a splitout patch with it to lkml for review?

Find it below - it's ontop of the tracer included in 2.6.20-rc2-rt3. it's very ad-hoc, based on Linus' test utility. I can write such a tracer in 30 minutes so i usually throw them away. I literally wrote dozens of tracer variants for specific bugs in the past few years.

Note: this particular one tracks page contents as well from kernel-space, that's how i was able to see where the corruption happened. That assumes that there's no highmem on the box. Also, the pte value tracking portion is only for i386 - etc. etc. Note: for the bug to be visible i didnt need the per-page tracking portion of the tracer - the key was to track page contents, and to track how virtual addresses map to physical pages, and how their IO happens.

This patch is /not/ for merging: this patch too undescores my years long experience that static tracepoints included in the generic kernel are just pointless in general - i dont want to see such cruft in the kernel, and they amass with time. The union of all ad-hoc tracing hacks i had in the past would be thousands of static tracepoints - and that's just /me/. If we pick only a handful they wont help us find the most difficult bugs and they'll only create additional 'demand' for 'more' - leading to an endless fight.

The best method i think is to use the source code itself (Linus used printks) - or if any infrastructure is to be used then ad-hoc "scriptlets" via SystemTap can find the really difficult bugs - and in the long run systemtap suits that purpose best. If systemtap were ubiquous we could have sent scriptlets to users who experienced the bugs, for them to install them dynamically. Systemtap makes it plain obvious that tracepoints are 1) detached from the source code and are 2) are temporary and ad-hoc in nature. It doesnt create undue pressure to include more and more static tracepoints.

Ingo

----------->

 fs/buffer.c                       |    1 
 include/asm-i386/pgtable-2level.h |    4 +
 include/linux/mm_types.h          |   22 +++++++++
 kernel/sys.c                      |   15 ++++++
 mm/Makefile                       |    2 
 mm/memory.c                       |    2 
 mm/page-writeback.c               |   33 ++++++++++++--
 mm/page_alloc.c                   |    3 +
 mm/page_trace.c                   |   84 ++++++++++++++++++++++++++++++++++++++
 mm/rmap.c                         |    2 
 10 files changed, 159 insertions(+), 9 deletions(-)
fs/buffer.c +1 -0
Index: linux/fs/buffer.c
===================================================================
@@ -1590,6 +1590,7 @@ static int __block_write_full_page(struc
 	int nr_underway = 0;
 
 	BUG_ON(!PageLocked(page));
+	trace_page(page, blocksize);
 
 	last_block = (i_size_read(inode) - 1) >> inode->i_blkbits;
 
include/asm-i386/pgtable-2level.h +3 -1
Index: linux/include/asm-i386/pgtable-2level.h
===================================================================
@@ -13,7 +13,9 @@
  */
 #ifndef CONFIG_PARAVIRT
 #define set_pte(pteptr, pteval) (*(pteptr) = pteval)
-#define set_pte_at(mm,addr,ptep,pteval) set_pte(ptep,pteval)
+struct mm_struct;
+extern void trace_set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte_val);
+#define set_pte_at(mm,addr,ptep,pteval) trace_set_pte_at(mm,addr,ptep,pteval)
 #define set_pmd(pmdptr, pmdval) (*(pmdptr) = (pmdval))
 #endif
 
include/linux/mm_types.h +22 -0
Index: linux/include/linux/mm_types.h
===================================================================
@@ -5,9 +5,29 @@
 #include <linux/threads.h>
 #include <linux/list.h>
 #include <linux/spinlock.h>
+#include <linux/stacktrace.h>
 
 struct address_space;
 
+struct page;
+struct seq_file;
+
+#define PAGE_TRACE_DEPTH	16
+#define PAGE_TRACE_NR		20
+
+struct page_trace_entry {
+	unsigned long	timestamp;
+	char		comm[17];
+	int		pid;
+	int		nr_entries;
+	unsigned long	info;
+	unsigned long	content;
+	unsigned long	entries[PAGE_TRACE_DEPTH];
+};
+
+extern void trace_page(struct page *page, unsigned long info);
+extern void print_page_trace(struct seq_file *m, struct page *page);
+
 /*
  * Each physical page in the system has a struct page associated with
  * it to keep track of whatever it is we are using the page for at the
@@ -62,6 +82,8 @@ struct page {
 	void *virtual;			/* Kernel virtual address (NULL if
 					   not kmapped, ie. highmem) */
 #endif /* WANT_PAGE_VIRTUAL */
+	int trace_idx;
+	struct page_trace_entry trace[PAGE_TRACE_NR];
 };
 
 #endif /* _LINUX_MM_TYPES_H */
kernel/sys.c +15 -0
Index: linux/kernel/sys.c
===================================================================
@@ -2067,6 +2067,21 @@ asmlinkage long sys_prctl(int option, un
 {
 	long error;
 
+	if (option == 999) {
+		unsigned long addr = arg2;
+		struct vm_area_struct *vma = find_vma(current->mm, addr);
+		struct page *page = NULL;
+
+		printk("page trace, got addr %08lx, vma %p\n", addr, vma);
+		if (vma) {
+			page = follow_page(vma, addr, FOLL_GET);
+			if (page) {
+				print_page_trace(NULL, page);
+				put_page(page);
+			}
+		}
+		return 0;
+	}
 #ifdef CONFIG_EVENT_TRACE
 	if (option == PR_SET_TRACING) {
 		if (arg2)
mm/Makefile +1 -1
Index: linux/mm/Makefile
===================================================================
@@ -9,7 +9,7 @@ mmu-$(CONFIG_MMU)	:= fremap.o highmem.o 
 
 obj-y			:= bootmem.o filemap.o mempool.o oom_kill.o fadvise.o \
 			   page_alloc.o page-writeback.o pdflush.o \
-			   readahead.o swap.o truncate.o vmscan.o \
+			   readahead.o swap.o truncate.o vmscan.o page_trace.o \
 			   prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
 			   $(mmu-y)
 
mm/memory.c +2 -0
Index: linux/mm/memory.c
===================================================================
@@ -451,6 +451,8 @@ struct page *vm_normal_page(struct vm_ar
 	 * The PAGE_ZERO() pages and various VDSO mappings can
 	 * cause them to exist.
 	 */
+
+	trace_page(pfn_to_page(pfn), addr);
 	return pfn_to_page(pfn);
 }
 
mm/page-writeback.c +27 -6
Index: linux/mm/page-writeback.c
===================================================================
@@ -762,8 +762,10 @@ int __set_page_dirty_nobuffers(struct pa
 		struct address_space *mapping = page_mapping(page);
 		struct address_space *mapping2;
 
-		if (!mapping)
+		if (!mapping) {
+			trace_page(page, 1);
 			return 1;
+		}
 
 		write_lock_irq(&mapping->tree_lock);
 		mapping2 = page_mapping(page);
@@ -781,8 +783,10 @@ int __set_page_dirty_nobuffers(struct pa
 			/* !PageAnon && !swapper_space */
 			__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
 		}
+		trace_page(page, 1);
 		return 1;
 	}
+	trace_page(page, 0);
 	return 0;
 }
 EXPORT_SYMBOL(__set_page_dirty_nobuffers);
@@ -806,6 +810,7 @@ EXPORT_SYMBOL(redirty_page_for_writepage
 int fastcall set_page_dirty(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);
+	int ret;
 
 	if (likely(mapping)) {
 		int (*spd)(struct page *) = mapping->a_ops->set_page_dirty;
@@ -813,12 +818,17 @@ int fastcall set_page_dirty(struct page 
 		if (!spd)
 			spd = __set_page_dirty_buffers;
 #endif
-		return (*spd)(page);
+		ret = (*spd)(page);
+		trace_page(page, ret);
+		return ret;
 	}
 	if (!PageDirty(page)) {
-		if (!TestSetPageDirty(page))
+		if (!TestSetPageDirty(page)) {
+			trace_page(page, 1);
 			return 1;
+		}
 	}
+	trace_page(page, 0);
 	return 0;
 }
 EXPORT_SYMBOL(set_page_dirty);
@@ -840,6 +850,7 @@ int set_page_dirty_lock(struct page *pag
 	lock_page_nosync(page);
 	ret = set_page_dirty(page);
 	unlock_page(page);
+	trace_page(page, ret);
 	return ret;
 }
 EXPORT_SYMBOL(set_page_dirty_lock);
@@ -915,13 +926,17 @@ int test_clear_page_writeback(struct pag
 
 		write_lock_irqsave(&mapping->tree_lock, flags);
 		ret = TestClearPageWriteback(page);
-		if (ret)
+		trace_page(page, ret);
+		if (ret) {
 			radix_tree_tag_clear(&mapping->page_tree,
 						page_index(page),
 						PAGECACHE_TAG_WRITEBACK);
+			trace_page(page, ret);
+		}
 		write_unlock_irqrestore(&mapping->tree_lock, flags);
 	} else {
 		ret = TestClearPageWriteback(page);
+		trace_page(page, ret);
 	}
 	return ret;
 }
@@ -936,17 +951,23 @@ int test_set_page_writeback(struct page 
 
 		write_lock_irqsave(&mapping->tree_lock, flags);
 		ret = TestSetPageWriteback(page);
-		if (!ret)
+		trace_page(page, ret);
+		if (!ret) {
 			radix_tree_tag_set(&mapping->page_tree,
 						page_index(page),
 						PAGECACHE_TAG_WRITEBACK);
-		if (!PageDirty(page))
+			trace_page(page, ret);
+		}
+		if (!PageDirty(page)) {
 			radix_tree_tag_clear(&mapping->page_tree,
 						page_index(page),
 						PAGECACHE_TAG_DIRTY);
+			trace_page(page, ret);
+		}
 		write_unlock_irqrestore(&mapping->tree_lock, flags);
 	} else {
 		ret = TestSetPageWriteback(page);
+		trace_page(page, ret);
 	}
 	return ret;
 
mm/page_alloc.c +3 -0
Index: linux/mm/page_alloc.c
===================================================================
@@ -1420,6 +1420,8 @@ nopage:
 		show_mem();
 	}
 got_pg:
+	if (page)
+		trace_page(page, order);
 	return page;
 }
 
@@ -1468,6 +1470,7 @@ void __pagevec_free(struct pagevec *pvec
 fastcall void __free_pages(struct page *page, unsigned int order)
 {
 	if (put_page_testzero(page)) {
+		trace_page(page, order);
 		if (order == 0)
 			free_hot_page(page);
 		else
mm/page_trace.c new file +84 -0

84 lines not on this page · raw message

mm/rmap.c +1 -1
Index: linux/mm/rmap.c
===================================================================
@@ -452,7 +452,7 @@ static int page_mkclean_one(struct page 
 		entry = ptep_clear_flush(vma, address, pte);
 		entry = pte_wrprotect(entry);
 		entry = pte_mkclean(entry);
-		set_pte_at(vma, address, pte, entry);
+		set_pte_at(mm, address, pte, entry);
 		lazy_mmu_prot_update(entry);
 		ret = 1;
 	}
#257 lore raw

Re: [patch] fix data corruption bug in __block_write_full_page()

Quoted 4 lines · Ingo Molnar

Find it below - it's ontop of the tracer included in 2.6.20-rc2-rt3. it's very ad-hoc, based on Linus' test utility. I can write such a tracer in 30 minutes so i usually throw them away. I literally wrote dozens of tracer variants for specific bugs in the past few years.

Ah, I though this was a general purpose tracer. Question solved, thanks :)

I was just tired of writing my own special purpose tracers all the time aswell.

#258 lore raw

Re: [PATCH] mm: fix page_mkclean_one

In reply to Andrew Morton
Quoted 22 lines · Andrew Morton
Quoted 10 lines · On Thu, 28 Dec 2006 11:28:52 -0800 (PST) Linus Torvalds <[email protected]>…
Quoted 2 lines · Guillaume Chazarain

The attached patch fixes the corruption for me.

Well, that's a good hint, but it's really just a symptom. You effectively just made the test-program not even try to flush the data to disk, so the page cache would stay in memory, and you'd not see the corruption as well.

Quoted 8 lines

So you basically disabled the code that tried to trigger the bug more easily.

But the reason I say it's interesting is that "WB_SYNC_NONE" is very much implicated in mm/page-writeback.c, and if there is a bug triggered by WB_SYNC_NONE writebacks, then that would explain why page-writeback.c also fails..

Quoted 3 lines

It would be interesting to convert your app to do fsync() before FADV_DONTNEED. That would take WB_SYNC_NONE out of the picture as well (apart from pdflush activity).

I did fdatasync(), tried remapping before unmapping... nogo here.

#259 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Martin Michlmayr
Quoted 8 lines · Martin Michlmayr
Quoted 1 line · Andrei Popa

With all three patches I have corruption....

I've completed one installation with Linus' patch plus the two from Andrew successfully, but I'm currently trying again... but I really need a better testcase since an installation takes about an hour. Andrei, which torrent do you download as a testcase? It would be good if someone could suggest a torrent which is legal and not too large.

Hi everyone,

I have been reading this thread for the last few days, but have been silent. I have 3 torrents here for testing, if you want.

You can easily reproduce with "rtorrent", if you:

- Have a completly downloaded one, no matter what size
- Corrupt the download with
dd if=/dev/zero of=download.file bs=16k count=1
- Restart 'rtorrent', hash-check fails
- It will download 1 piece that was corrupted.

The important part here is that rtorrent transfers one piece, using its own code sequence to write to the file.

Let me offer to test until Saturday afternoon CET,
I have a cloned git repository, downloaded torrent files and "apt".

My systems that are affected are:

Linux oscar 2.6.18 SMP (2x450Mhz Intel P3)
(rolled back to 2.6.18 but can boot latest git)

Linux tony 2.6.20-git UP
(can be tested using all kinds of "apt" operations)

Both machines are using:
IDE -> MD-RAID1 -> LVM -> EXT3 (data=ordered)
SCSI -> MD-RAID5 -> .....

I don't want to disturb your technical discussion, just offering some help in testing.

Regards,
Patrick

#260 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Martin Michlmayr
Quoted 8 lines · Martin Michlmayr
Quoted 1 line · Andrei Popa

With all three patches I have corruption....

I've completed one installation with Linus' patch plus the two from Andrew successfully, but I'm currently trying again... but I really need a better testcase since an installation takes about an hour. Andrei, which torrent do you download as a testcase? It would be good if someone could suggest a torrent which is legal and not too large.

It's a 1.4GB file torrent split in 84 rar files and there are many seeders. I download with ~ 5MB/sec. The torrent is private.

#261 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

Andrew located at least one bug: we run cancel_dirty_page() too late in "truncate_complete_page()", which means that do_invalidatepage() ends up not clearing the page cache.

His patch is appended.

Thanks. I'll try this out later today.

Quoted 3 lines

But it sounds like I probably misunderstood something, because I thought that Martin had acknowledged that this patch actually worked for him. Which sounded very similar to your setup (he has a 32M ARM box too, no?)

Yup, we have the same machines (Linksys NSLU2) and are running the same test case (installing Debian). However, I'm not sure what kernel version he had used for his latest test. I presumed 2.6.20-git, whereas I had used 2.6.19.

Quoted 2 lines

Maybe it's mount option issue? I've got data=ordered on my machine, are you perhaps runnign with something else?

We are also using ordered.

/dev/scsi/host0/bus0/target0/lun0/part1 /target ext3 rw,data=ordered 0 0

Gordon

Signature · 1 line
Gordon Farquharson
#262 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Gordon Farquharson
Quoted 4 lines · Gordon Farquharson

generating these files, pkgcache.bin grows to 12582912 bytes, and when apt-get finishes, pkgcache.bin is 6425533 bytes and srcpkgcache.bin is 64254483 bytes. This time, when apt-get exited, it had only created pkgcache.bin which was still 12582912 bytes.

Yes, same here:

sh-3.1# ls -l /var/cache/apt/
total 5252
drwxr-xr-x 3 root root 12288 Dec 22 04:41 archives
-rw-r--r-- 1 root root 12582912 Dec 22 04:45 pkgcache.bin
-rw-r--r-- 1 root root 8554 Dec 22 04:45 srcpkgcache.bin

Gordon, does it fail for you where it normally does (installing initramfs-tools) or much later? For me, the installer was able to install initramfs-tools and the kernel, but apt now hangs at "Select and install software".

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#263 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Review

Quoted 5 lines · Martin Michlmayr

sh-3.1# ls -l /var/cache/apt/
total 5252
drwxr-xr-x 3 root root 12288 Dec 22 04:41 archives

(quoted diff) +0 -2
-rw-r--r-- 1 root root 12582912 Dec 22 04:45 pkgcache.bin
-rw-r--r-- 1 root root     8554 Dec 22 04:45 srcpkgcache.bin

This listing is a little different to what I got. For me, srcpkgcache.bin did not exist when apt eventually finished. Did you notice whether the install took a lot longer than usual ?

Quoted 4 lines

Gordon, does it fail for you where it normally does (installing initramfs-tools) or much later? For me, the installer was able to install initramfs-tools and the kernel, but apt now hangs at "Select and install software".

apt didn't hang for me, it just took 20 to 30 minutes to complete building the package database. Usually, it takes less than a minute. The installer stopped because it could not find a kernel to install. I have seen this failure mde before, and as you have previously pointed out, is probably the same problem (corrupted apt cache files), just a different manifestation.

Gordon

Signature · 1 line
Gordon Farquharson
#264 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

That's obviously a bug worth fixing on its own. Do you know when it started?

My last merge, just before 2.6.19-rc1.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#265 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 5 lines · Russell King
Quoted 2 lines · Linus Torvalds

That's obviously a bug worth fixing on its own. Do you know when it started?

My last merge, just before 2.6.19-rc1.

Obviously 2.6.20-rc1.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#266 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Martin, Andrei, does this make any difference for your corruption cases?

Works for me.

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#267 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Martin Michlmayr
Quoted 5 lines · Martin Michlmayr
Quoted 1 line · Peter Zijlstra

page_mkclean_one() fix

This patch doesn't fix my problem (apt segfaults on ARM because its database is corrupted).

Are you using IDE in PIO mode? If so, the bug probably lies there.

As I've said repeatedly when asked by IDE folk to test their PIO-based cache coherency fixes, I am unable to reproduce the bug, ergo I am unable to test the fix.

(Some people, such as Jeff Garzik to name names, took that as me being entirely unreasonable and un-cooperative. But consider carefully - how can _anyone_ test something that they can't produce. I consider Jeff's comments extremely very childish in that respect.)

Hence, as far as I'm aware, Linux on PIO-based IDE ARM hardware remains utterly *unsafe*.

Sorry.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#268 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 4 lines · Russell King
Quoted 2 lines

This patch doesn't fix my problem (apt segfaults on ARM because its database is corrupted).

Are you using IDE in PIO mode? If so, the bug probably lies there.

I'm using usb-storage. It's used to access an external IDE drive in an USB enclosure but I don't think it matters that it's IDE since we're using the SCSI layer to talk to it, right?

Signature · 2 lines
Martin Michlmayr
http://www.cyrius.com/
#269 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

Quoted 9 lines · Martin Michlmayr
Quoted 4 lines · Russell King
Quoted 2 lines

This patch doesn't fix my problem (apt segfaults on ARM because its database is corrupted).

Are you using IDE in PIO mode? If so, the bug probably lies there.

I'm using usb-storage. It's used to access an external IDE drive in an USB enclosure but I don't think it matters that it's IDE since we're using the SCSI layer to talk to it, right?

USB generally uses DMA so you're probably safe.

Signature · 3 lines
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:
#270 lore raw

Re: [PATCH] mm: fix page_mkclean_one (was: 2.6.19 file content corruption on ext3)

In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

fix page_mkclean_one()

it had several issues:
- it failed to flush the cache
- it failed to flush the tlb
- it failed to do s390 (s390 guys, please verify this is now correct)

Sorry, page_mkclean is broken for s390. But it has already been broken before your change. It is only more broken now.

Quoted 27 lines
mm/rmap.c +10 -11
@@ -440,22 +440,23 @@ static int page_mkclean_one(struct page
 	if (address == -EFAULT)
 		goto out;
 
-	pte = page_check_address(page, mm, address, &ptl);
-	if (!pte)
+	ptep = page_check_address(page, mm, address, &ptl);
+	if (!ptep)
 		goto out;
 
-	if (!pte_dirty(*pte) && !pte_write(*pte))
-		goto unlock;
-
-	entry = ptep_get_and_clear(mm, address, pte);
-	entry = pte_mkclean(entry);
-	entry = pte_wrprotect(entry);
-	ptep_establish(vma, address, pte, entry);
-	lazy_mmu_prot_update(entry);
-	ret = 1;
+	while (pte_dirty(*ptep) || pte_write(*ptep)) {
+		pte_t entry = ptep_get_and_clear(mm, address, ptep);
+		flush_cache_page(vma, address, pte_pfn(entry));
+		flush_tlb_page(vma, address);
+		(void)page_test_and_clear_dirty(page); /* do the s390 thing */
+		entry = pte_wrprotect(entry);
+		entry = pte_mkclean(entry);
+		set_pte_at(vma, address, ptep, entry);

mm/rmap.c static int page_mkclean_one(struct page · line 462

Quoted 10 lines
mm/rmap.c +4 -2
+		lazy_mmu_prot_update(entry);
+		ret = 1;
+	}
 
-unlock:
-	pte_unmap_unlock(pte, ptl);
+	pte_unmap_unlock(ptep, ptl);
 out:
 	return ret;
 }

1) pte_dirty() is always false. The reason is that s390 keeps the dirty bit information in the storage key and not the pte. If pte_write is false as well nothing is done. There really should be a

	if (page_test_and_clear_dirty(page))
		ret = 1;

at the end of page_mkclean.

2) Please use ptep_clear_flush instead of ptep_get_and_clear + flush_tlb_page. The former uses an optimization on s390 that flushes just one TLB, the later flushes every TLB of the current mm.

My try to fix this up is attached. It moves the flush_cache_page after the flush_tlb_page (see asm-generic/pgtable.h for the generic definition of ptep_clear_flush that is used for i386). I hope this doesn't break anything else.

Signature · 37 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.

---
 mm/rmap.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff -urpN linux-2.6/mm/rmap.c linux-2.6-mkclean/mm/rmap.c
--- linux-2.6/mm/rmap.c	2006-12-20 15:49:01.000000000 +0100
+++ linux-2.6-mkclean/mm/rmap.c	2006-12-20 15:51:14.000000000 +0100
@@ -445,10 +445,8 @@ static int page_mkclean_one(struct page 
 		goto out;
 
 	while (pte_dirty(*ptep) || pte_write(*ptep)) {
-		pte_t entry = ptep_get_and_clear(mm, address, ptep);
+		pte_t entry = ptep_clear_flush(vma, address, ptep);
 		flush_cache_page(vma, address, pte_pfn(entry));
-		flush_tlb_page(vma, address);
-		(void)page_test_and_clear_dirty(page); /* do the s390 thing */
 		entry = pte_wrprotect(entry);
 		entry = pte_mkclean(entry);
 		set_pte_at(vma, address, ptep, entry);
@@ -490,6 +488,8 @@ int page_mkclean(struct page *page)
 		if (mapping)
 			ret = page_mkclean_file(mapping, page);
 	}
+	if (page_test_and_clear_dirty(page))
+		ret = 1;
 
 	return ret;
 }
#271 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra

Also, what is this page_test_and_clear_dirty() business, that seems to be exclusively s390 btw. However they do seem to need this.

Quoted 2 lines

But the "ptep_get_and_clear() + flush_tlb_page()" sequence should hopefully also work.

Yeah, probably, not optimally so on some archs that don't actually need the flush though. And as above, I wonder about s390.

Simple, the s390 architecture does not keep the dirty bit in the pte but in something called the storage key. For each physical page there is one associated storage key. It is accessed with special instructions like "iske", "sske" or "rrbe". To clear the dirty bit the storage key of a page is read with iske, the bit is cleared and the storage key is stored back with sske. That means that clearing the dirty bit is not an atomic operation. rrbe is used to test and clear the referenced bit (young/old infomation) and is atomic in regard to other storage key operations. If you think about it, the storage keys are quite nice for the operating system, page_referenced() can be implemented with a single test "page_test_and_clear_young()". No need to read all the ptes pointing to the page. The downside is that the storage keys have a cost on the hardware side.

Signature · 8 lines
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.
#272 lore raw
In reply to Linus Torvalds
Quoted 1 line · Linus Torvalds

Pls test.

Is good. Only s390 remains a question.

Another point, change_protection() also does a cache flush, should we too?

mm/rmap.c static int page_mkclean_one(struct page *page, struct vm_area_struct *vma) · line 450

Quoted 9 lines

----

mm/rmap.c +0 -0
index d8a842a..eec8706 100644
@@ -448,9 +448,10 @@ static int page_mkclean_one(struct page *page, struct vm_area_struct *vma)
 		goto unlock;
 
 	entry = ptep_get_and_clear(mm, address, pte);
          flush_cache_page(vma, address, pte_pfn(entry));
Quoted 9 lines
mm/rmap.c +2 -1
+	flush_tlb_page(vma, address);
 	entry = pte_mkclean(entry);
 	entry = pte_wrprotect(entry);
-	ptep_establish(vma, address, pte, entry);
+	set_pte_at(mm, address, pte, entry);
 	lazy_mmu_prot_update(entry);
 	ret = 1;
 
#273 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds

OR:

- page_mkclean_one() is simply buggy.

GOLD!

Quoted 8 lines

it seems to work with all this (full diff against current git).

/me rebuilds full kernel to make sure...
reboot...
test... pff the tension...
yay, still good!

Andrei; would you please verify.

I have corrupted files.

Quoted 158 lines

The magic seems to be in the extra tlb flush after clearing the dirty bit. Just too bad ptep_clear_flush_dirty() needs ptep not entry.

drivers/connector/connector.c +1 -2
index 5e7cd45..2b8893b 100644
@@ -135,8 +135,7 @@ static int cn_call_callback(struct cn_msg *msg, void (*destruct_data)(void *), v
 	spin_lock_bh(&dev->cbdev->queue_lock);
 	list_for_each_entry(__cbq, &dev->cbdev->queue_list, callback_entry) {
 		if (cn_cb_equal(&__cbq->id.id, &msg->id)) {
-			if (likely(!test_bit(WORK_STRUCT_PENDING,
-					     &__cbq->work.work.management) &&
+			if (likely(!delayed_work_pending(&__cbq->work) &&
 					__cbq->data.ddata == NULL)) {
 				__cbq->data.callback_priv = msg;
 
fs/buffer.c +1 -17
index d1f1b54..263f88e 100644
@@ -2834,7 +2834,7 @@ int try_to_free_buffers(struct page *page)
 	int ret = 0;
 
 	BUG_ON(!PageLocked(page));
-	if (PageWriteback(page))
+	if (PageDirty(page) || PageWriteback(page))
 		return 0;
 
 	if (mapping == NULL) {		/* can this still happen? */
@@ -2845,22 +2845,6 @@ int try_to_free_buffers(struct page *page)
 	spin_lock(&mapping->private_lock);
 	ret = drop_buffers(page, &buffers_to_free);
 	spin_unlock(&mapping->private_lock);
-	if (ret) {
-		/*
-		 * If the filesystem writes its buffers by hand (eg ext3)
-		 * then we can have clean buffers against a dirty page.  We
-		 * clean the page here; otherwise later reattachment of buffers
-		 * could encounter a non-uptodate page, which is unresolvable.
-		 * This only applies in the rare case where try_to_free_buffers
-		 * succeeds but the page is not freed.
-		 *
-		 * Also, during truncate, discard_buffer will have marked all
-		 * the page's buffers clean.  We discover that here and clean
-		 * the page also.
-		 */
-		if (test_clear_page_dirty(page))
-			task_io_account_cancelled_write(PAGE_CACHE_SIZE);
-	}
 out:
 	if (buffers_to_free) {
 		struct buffer_head *bh = buffers_to_free;
mm/memory.c +28 -0
index c00bac6..60e0945 100644
@@ -1842,6 +1842,33 @@ void unmap_mapping_range(struct address_space *mapping,
 }
 EXPORT_SYMBOL(unmap_mapping_range);
 
+static void check_last_page(struct address_space *mapping, loff_t size)
+{
+	pgoff_t index;
+	unsigned int offset;
+	struct page *page;
+
+	if (!mapping)
+		return;
+	offset = size & ~PAGE_MASK;
+	if (!offset)
+		return;
+	index = size >> PAGE_SHIFT;
+	page = find_lock_page(mapping, index);
+	if (page) {
+		unsigned int check = 0;
+		unsigned char *kaddr = kmap_atomic(page, KM_USER0);
+		do {
+			check += kaddr[offset++];
+		} while (offset < PAGE_SIZE);
+		kunmap_atomic(kaddr, KM_USER0);
+		unlock_page(page);
+		page_cache_release(page);
+		if (check)
+			printk(KERN_ERR "%s: BADNESS: truncate check %u\n", current->comm, check);
+	}
+}
+
 /**
  * vmtruncate - unmap mappings "freed" by truncate() syscall
  * @inode: inode of the file used
@@ -1875,6 +1902,7 @@ do_expand:
 		goto out_sig;
 	if (offset > inode->i_sb->s_maxbytes)
 		goto out_big;
+	check_last_page(mapping, inode->i_size);
 	i_size_write(inode, offset);
 
 out_truncate:
mm/page-writeback.c +1 -1
index 237107c..f561e72 100644
@@ -957,7 +957,7 @@ int test_set_page_writeback(struct page *page)
 EXPORT_SYMBOL(test_set_page_writeback);
 
 /*
- * Return true if any of the pages in the mapping are marged with the
+ * Return true if any of the pages in the mapping are marked with the
  * passed tag.
  */
 int mapping_tagged(struct address_space *mapping, int tag)
mm/rmap.c +9 -8
index d8a842a..900229a 100644
@@ -432,7 +432,7 @@ static int page_mkclean_one(struct page *page, struct vm_area_struct *vma)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long address;
-	pte_t *pte, entry;
+	pte_t *ptep, entry;
 	spinlock_t *ptl;
 	int ret = 0;
 
@@ -440,22 +440,23 @@ static int page_mkclean_one(struct page *page, struct vm_area_struct *vma)
 	if (address == -EFAULT)
 		goto out;
 
-	pte = page_check_address(page, mm, address, &ptl);
-	if (!pte)
+	ptep = page_check_address(page, mm, address, &ptl);
+	if (!ptep)
 		goto out;
 
-	if (!pte_dirty(*pte) && !pte_write(*pte))
+	if (!pte_dirty(*ptep) && !pte_write(*ptep))
 		goto unlock;
 
-	entry = ptep_get_and_clear(mm, address, pte);
-	entry = pte_mkclean(entry);
+	entry = ptep_get_and_clear(mm, address, ptep);
 	entry = pte_wrprotect(entry);
-	ptep_establish(vma, address, pte, entry);
+	ptep_establish(vma, address, ptep, entry);
+	ret = ptep_clear_flush_dirty(vma, address, ptep) ||
+		page_test_and_clear_dirty(page);
 	lazy_mmu_prot_update(entry);
 	ret = 1;
 
 unlock:
-	pte_unmap_unlock(pte, ptl);
+	pte_unmap_unlock(ptep, ptl);
 out:
 	return ret;
 }
#274 lore raw
Quoted 11 lines · Andrei Popa
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds

OR:

- page_mkclean_one() is simply buggy.

GOLD!

it seems to work with all this (full diff against current git).

Quoted 8 lines
Quoted 6 lines

/me rebuilds full kernel to make sure...
reboot...
test... pff the tension...
yay, still good!

Andrei; would you please verify.

I have corrupted files.

drad; and with this patch:
http://lkml.org/lkml/2006/12/20/112

/me goes rebuild his kernel and try more than 3 times

#275 lore raw
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Andrei Popa
Quoted 15 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds

OR:

- page_mkclean_one() is simply buggy.

GOLD!

it seems to work with all this (full diff against current git).

/me rebuilds full kernel to make sure...
reboot...
test... pff the tension...
yay, still good!

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Andrei; would you please verify.

I have corrupted files.

drad; and with this patch:
http://lkml.org/lkml/2006/12/20/112

Hash check on download completion found bad chunks, consider using "safe_sync".

Quoted 3 lines

/me goes rebuild his kernel and try more than 3 times

#276 lore raw
Quoted 20 lines · Andrei Popa
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Andrei Popa
Quoted 16 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds

OR:

- page_mkclean_one() is simply buggy.

GOLD!

it seems to work with all this (full diff against current git).

/me rebuilds full kernel to make sure...
reboot...
test... pff the tension...
yay, still good!

Andrei; would you please verify.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I have corrupted files.

drad; and with this patch:
http://lkml.org/lkml/2006/12/20/112

Hash check on download completion found bad chunks, consider using "safe_sync".

*sigh* back to square 1.

and I need to look at my reproduction case ;-(

Thanks for testing.

#277 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 7 lines
mm/rmap.c +0 -0
index d8a842a..3f9061e 100644
@@ -448,7 +448,7 @@ static int page_mkclean_one(struct page 
 		goto unlock;
 
Quoted 9 lines

mm/rmap.c static int page_mkclean_one(struct page · line 454

Quoted 6 lines
mm/rmap.c +1 -1
 	entry = ptep_get_and_clear(mm, address, pte);
-	entry = pte_mkclean(entry);
+	/*entry = pte_mkclean(entry);*/
 	entry = pte_wrprotect(entry);
 	ptep_establish(vma, address, pte, entry);
 	lazy_mmu_prot_update(entry);

The above patch is bad. It's always going to hide the bug, but it hides it by just not doing anything at all.

Not quite, it does wrprotect still, so further updates will trigger the do_wp_page() path and call set_page_dirty().

So we could make 'something' that would keep the tracking working and not create corruption, say something like this:

However I'll try and figure out how we get so terribly confused on the PG_dirty state that we have to clean it and fall back to pte_dirty. That is the real issue we have.

---
 include/linux/rmap.h |    6 ++++++
 mm/page-writeback.c  |    3 ++-
 mm/rmap.c            |   23 ++++++++++++++++++-----
 3 files changed, 26 insertions(+), 6 deletions(-)
mm/rmap.c +18 -5
Index: linux-2.6-git/mm/rmap.c
===================================================================
@@ -428,7 +428,8 @@ int page_referenced(struct page *page, i
 	return referenced;
 }
 
-static int page_mkclean_one(struct page *page, struct vm_area_struct *vma)
+static int page_mkcw_one(struct page *page,
+			 struct vm_area_struct *vma, int make_clean)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long address;
@@ -448,7 +449,8 @@ static int page_mkclean_one(struct page 
 		goto unlock;
 
 	entry = ptep_get_and_clear(mm, address, pte);
-	entry = pte_mkclean(entry);
+	if (make_clean)
+		entry = pte_mkclean(entry);
 	entry = pte_wrprotect(entry);
 	ptep_establish(vma, address, pte, entry);
 	lazy_mmu_prot_update(entry);
@@ -460,7 +462,8 @@ out:
 	return ret;
 }
 
-static int page_mkclean_file(struct address_space *mapping, struct page *page)
+static int page_mkcw_file(struct address_space *mapping,
+			  struct page *page, int make_clean)
 {
 	pgoff_t pgoff = page->index << (PAGE_CACHE_SHIFT - PAGE_SHIFT);
 	struct vm_area_struct *vma;
@@ -478,7 +481,7 @@ static int page_mkclean_file(struct addr
 	return ret;
 }
 
-int page_mkclean(struct page *page)
+static int page_mkcw(struct page *page, int make_clean)
 {
 	int ret = 0;
 
@@ -487,12 +490,22 @@ int page_mkclean(struct page *page)
 	if (page_mapped(page)) {
 		struct address_space *mapping = page_mapping(page);
 		if (mapping)
-			ret = page_mkclean_file(mapping, page);
+			ret = page_mkcw_file(mapping, page, make_clean);
 	}
 
 	return ret;
 }
 
+int page_mkclean(struct page *page)
+{
+	return page_mkcw(page, 1);
+}
+
+int page_wrprotect(struct page *page)
+{
+	return page_mkcw(page, 0);
+}
+
 /**
  * page_set_anon_rmap - setup new anonymous rmap
  * @page:	the page to add the mapping to
include/linux/rmap.h +6 -0
Index: linux-2.6-git/include/linux/rmap.h
===================================================================
@@ -110,6 +110,7 @@ unsigned long page_address_in_vma(struct
  * returns the number of cleaned PTEs.
  */
 int page_mkclean(struct page *);
+int page_wrprotect(struct page *);
 
 #else	/* !CONFIG_MMU */
 
@@ -125,6 +126,11 @@ static inline int page_mkclean(struct pa
 	return 0;
 }
 
+static inline int page_wrprotect(struct page *page)
+{
+	return 0;
+}
+
 
 #endif	/* CONFIG_MMU */
 
mm/page-writeback.c +2 -1
Index: linux-2.6-git/mm/page-writeback.c
===================================================================
@@ -872,7 +872,8 @@ int test_clear_page_dirty(struct page *p
 		 * page is locked, which pins the address_space
 		 */
 		if (mapping_cap_account_dirty(mapping)) {
-			page_mkclean(page);
+			if (page_wrprotect(page))
+				set_page_dirty();
 			dec_zone_page_state(page, NR_FILE_DIRTY);
 		}
 		return 1;
#278 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Peter Zijlstra

This should be safe; page_mkclean walks the rmap and flips the pte's under the pte lock and records the dirty state while iterating. Concurrent faults will either do set_page_dirty() before we get around to doing it or vice versa, but dirty state is not lost.

Ok, I really liked this patch, but the more I thought about it, the more I started to doubt the reasons for liking it.

Well this implements my suggestion to redirty the page if there were dirty ptes. I think it is a good fix (whether or not it fixes Andrei's bug, it does fix a bug), though maybe _slightly_ suboptimal.

Quoted 10 lines

I think we have some core fundamental problem here that this patch is needed at all.

So let's think about this: we apparently have two cases of "clear_page_dirty()":

- the one that really wants to clear the bit unconditionally (Andrew calls this the "must_clean_ptes" case, which I personally find to be a really confusing name, but whatever)

- the other case. The case that doesn't want to really clear the pte dirty bits.

I don't think this characterises it correctly. Think about how it worked before the page_mkclean went in there.

We really _never_ want to just clear pte dirty bits, because that would be a data loss situation[*]. The only reason we clear PG_dirty is because some filesystem may have cleaned each buffer without realising it has cleaned the whole page. But if you have a dirty pte, then all bets are off: a buffer with a clear dirty bit can not be considered clean.

Before the dirty page tracking, it was fine to clear PG_dirty here, because we would pick up the pte dirty info later on. After the page dirty tracking, clearing pte dirty is a bug here, and re-accounting the dirty page is arguably the minimal fix.

[*] except in the truncate case where we are happy to throw out dirty data,
but in that case there would be no ptes anyway.

The only thing I would suggest is not applying Andrew's patch at all, and do the special casing in try_to_free_buffers(). I've attached a patch for comments.

Quoted 3 lines

and I thought your patch made sense, because it saved away the pte state in the page dirty state, and that matches my mental model, but the more I think about it, the less sense that whole "the other case" situation makes AT ALL.

Why does "the other case" exist at all? If you want to clear the dirty page flag, what is _ever_ the reason for not wanting to drop PTE dirty information? In other words, what possible reason can there ever be for saying "I want this page to be clean", while at the same time saying "but if it was dirty in the page tables, don't forget about that state".

We never want to drop dirty data! (ignoring the truncate case, which is handled privately by truncate anyway)

This whole exercise is not about cleaning or dirtying or fogetting the actual *data* in the page. It is about bringing the pagecache's notion of whether the page is dirty or clean in line with the (more uptodate) filesystem's notion.

After dirty write accounting, we also threw in "the virtual memory manager's notion", but got that case slightly wrong.

As unlikely as this race is for SMP systems, I think it is easily possible for PREEMPT kernels. And they have featured in all bug reports, AFAIKS.

Signature · 1 line
SUSE Labs, Novell Inc.
fs/buffer.c +11 -1
Index: linux-2.6/fs/buffer.c
===================================================================
@@ -2852,7 +2852,17 @@ int try_to_free_buffers(struct page *pag
 		 * This only applies in the rare case where try_to_free_buffers
 		 * succeeds but the page is not freed.
 		 */
-		clear_page_dirty(page);
+
+		/*
+		 * If the page has been dirtied via the user mappings, then
+		 * clean buffers does not indicate the page data is actually
+		 * clean! Only clear the page dirty bit if there are no dirty
+		 * ptes either.
+		 *
+		 * If there are dirty ptes, then the page must be uptodate, so
+		 * the above concern does not apply.
+		 */
+		clear_page_dirty_sync_ptes(page);
 	}
 out:
 	if (buffers_to_free) {
include/linux/page-flags.h +6 -0
Index: linux-2.6/include/linux/page-flags.h
===================================================================
@@ -254,6 +254,7 @@ static inline void SetPageUptodate(struc
 struct page;	/* forward declaration */
 
 int test_clear_page_dirty(struct page *page);
+int test_clear_page_dirty_sync_ptes(struct page *page);
 int test_clear_page_writeback(struct page *page);
 int test_set_page_writeback(struct page *page);
 
@@ -262,6 +263,11 @@ static inline void clear_page_dirty(stru
 	test_clear_page_dirty(page);
 }
 
+static inline void clear_page_dirty_sync_ptes(struct page *page)
+{
+	test_clear_page_dirty_sync_ptes(page);
+}
+
 static inline void set_page_writeback(struct page *page)
 {
 	test_set_page_writeback(page);
mm/page-writeback.c +37 -4

65 lines not on this page · raw message

#279 lore raw
Quoted 3 lines · Nick Piggin

We never want to drop dirty data! (ignoring the truncate case, which is handled privately by truncate anyway)

Bzzt.

SURE we do.

We absolutely do want to drop dirty data in the writeout path.

How do you think dirty data ever _becomes_ clean data?

In other words, yes, we _do_ want to test-and-clear all the pgtable bits _and_ the PG_dirty bit. We want to do it for:
- writeout
- truncate
- possibly a "drop" event (which could be a case for a journal entry that becomes stale due to being replaced or something - kind of "truncate" on metadata)

because both of those events _literally_ turn dirty state into clean state.

In no other circumstance do we ever want to clear a dirty bit, as far as I can tell.

Linus

#280 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Nick Piggin

We never want to drop dirty data! (ignoring the truncate case, which is handled privately by truncate anyway)

Quoted 7 lines

Bzzt.

SURE we do.

We absolutely do want to drop dirty data in the writeout path.

How do you think dirty data ever _becomes_ clean data?

I wouldn't have thought it becomes clean by dropping it ;) Is this a trick question? My answer is that we clean a page by by taking some action such that the underlying data matches the data in RAM...

We don't "drop" any data until it has been cleaned (again, ignoring things like truncate for a minute). That's a bug! And try_to_free_buffers() is called from places outside the writeout path. This is our bug (or at least, one of our bugs that appears to have the same triggers and symptoms as people are reporting).

[...]

Quoted 1 line

In no other circumstance do we ever want to clear a dirty bit, as far as I can tell.

Exactly. And that is exactly what try_to_free_buffers is doing now.

I still think you should have a look at the patch.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#281 lore raw
Quoted 4 lines · Nick Piggin

I wouldn't have thought it becomes clean by dropping it ;) Is this a trick question? My answer is that we clean a page by by taking some action such that the underlying data matches the data in RAM...

Sure.

Quoted 2 lines

We don't "drop" any data until it has been cleaned (again, ignoring things like truncate for a minute). That's a bug!

Actually, it's the other way around. We have to drop the dirty bits BEFORE cleaning. If we clean first, and _then_ drop the dirty bits, THAT is a bug, because the dirty bits can now refer to _new_ dirty data that didn't get written out.

So the proper sequence is _literally_ to mark the page clean FIRST. Drop all the dirty bits, but not the _data_ obviously (ie you have a reference to the page). And _then_ you do the writeout to actually clean the data itself.

So you actually state it exactly the wrogn way around.

We MUST clear the dirty bits before we do the IO that actually cleans the data. Exactly because if new writes keep on happening, if we do it in the other order, we'll drop dirty data on the floor.

Quoted 6 lines
Quoted 2 lines

In no other circumstance do we ever want to clear a dirty bit, as far as I can tell.

Exactly. And that is exactly what try_to_free_buffers is doing now.

I still think you should have a look at the patch.

I claim that dropping dirty bits AFTER the IO is always wrong. Try_to_free_buffers() must never touch the dirty bits at all, because by definition that thing happens after the IO has actually been done.

Anbd yes, I looked at your patch. And it looks a million times cleaner than Andrew's patch. However, it's already been tested multiple times, and totally REMOVING the "clear_page_dirty()" from try_to_free_buffers() still resulted in the corruption.

That said, I think your patch is worth it just as a cleanup. Much nicer than Andrews code, also from a naming standpoint. So I'm not actually disagreeing about the patch itself, but I _am_ saying that I don't actually see the point of ever moving the dirty bits around.

So I repeat: we have the case where we really want to _remove_ the dirty bits (because we're going to write the current state of the page to disk, and we need to clear the dirty bits BEFORE we do that). That's the one that makes sense, and that's the code we want to run before doing IO. It's the "clear_dirty_bits_for_io()" case.

The code that doesn't make sense is the "shuffle the dirty bits around" In other words: when does it actually make sense to call your (well-implemented, don't get me wrong) "test_clear_page_dirty_sync_ptes()" function? It doesn't _fix_ anything. It just shuffles dirty bits from one place to another. What was the point again?

If the point is "try_to_free_buffers()", then my argument was that I had a much simpler solution: "Just don't do that then". My simple patch sadly didn't fix the data corruption, so the data corruption comes from something ELSE than try_to_free_buffers().

Linus

#282 lore raw
Quoted 6 lines · Linus Torvalds

The code that doesn't make sense is the "shuffle the dirty bits around" In other words: when does it actually make sense to call your (well-implemented, don't get me wrong) "test_clear_page_dirty_sync_ptes()" function? It doesn't _fix_ anything. It just shuffles dirty bits from one place to another. What was the point again?

Let me try to phrase that another way, in terms that you defined.

In other words, look at your test_clear_page_dirty_sync_ptes() function.

First, start out from the _inner_ part, the:

	if (mapping_cap_account_dirty(mapping)) {
		if (page_mkclean(page))
			set_page_dirty(page);
	}

part.

This the one that both you and I agree is a "working" situation: we are moving the dirty bits from the pte into the "struct page", and we both agree that this is fine. No dirty bits get lost. You even make a BIG DEAL about the fact that no dirty bits get lost.

So begin by just explaining:
- why do it?

Why shuffle the dirty bits around? Why not just _leave_ the PG_dirty bit on the "struct page", and simply leave it all at that? I agree that the above doesn't lose any dirty bits, but what I'm asking for is WHAT IS THE POINT?

So that is the code that we both agree "works", but I personally don't see the _point_ of. However, that's actually not even important, because I don't even care about the point. I wanted to bring that up just in order to then ignore it, and look at the stuff _around: it, namely the other part in "test_clear_page_dirty_sync_ptes()":

	int test_clear_page_dirty_sync_ptes(struct page *page)
	{
		if (test_clear_page_dirty_leave_ptes(page)) {
			.. do the inner part ..
			return 1;
		}
		return 0;
	}

Now, the above is the OUTER part. Please realize that this DOES actually drop the PG-dirty bit. So ignore the inner part entirely (which is a no-op for the case where the page isn't mapped), and explain to me why it's ok to DROP the dirty bit in the outer part, when you tried to say that it was NOT ok to drop it in the inner part?

NOTICE? First you make a BIG DEAL about how dirty bits should never get lost, but THE VERY SAME FUNCTION actually very much on purpose DOES drop the dirty bit for when it's not in the page tables.

In fact, if you just call that function twice, the first time it will MOVE the dirty bits from the PTE to the "struct page *", and the _second_ time it will just clear the dirty bit from the "struct page *". You end up with a clean page. It returned the same return value BOTH TIMES, even though it did two very different things (once just moving dirty bits around, and the second time actually _removing_ the dirty bit entirely).

Again, I have a very simple claim: I claim that NONE of the "test_clear_page_dirty()" functions make any sense what-so-ever. They're all wrong.

The "funny" part is, that the only thing that Andrei reports actually fixed his corruption (apart from the patch tjhat just stops removign the dirty bits from the PTE's _entirely_) is actually the part where he had an "#if 0 .. #endif" around basically _all_ of the "test_clear_page_dirty()" function (ie he had mis-understood what I asked for, and put it outside the _outer_ if(), rather than putting it around the inner one).

So I claim:
- there is ONE and only ONE place where you can really drop the dirty
bits: it's when you're going to immediately afterwards do a writeout.

This is the "clear_page_dirty_for_io()"

- all the other "[test_and_]clear_dirty*()" functions seem to be outright buggy and bogus. Shuffling dirty bits around from the page tables to the "struct page *" (after having _cleared_ that "very important" PG_dirty bit just before - apparently it wasn't that important after all, was it?) is insane.

Nobody has actually ever explained why "test_clear_page_dirty()" is good at all.

- Why is it ever used instead of "clear_page_dirty_for_io()"?

- What is the difference?

- Why would you EVER want to clear bits just in the "struct page *" or just in the PTE's?

- Why is it EVER correct to clear dirty bits except JUST BEFORE THE IO?

In other words, I have a theory:

"A lot of this is actually historical cruft. Some of it may even be code
that was never supposed to work, but because we maintained _other_ dirty bits in the PTE's, and never touched them before, we never even realized that the code that played with PG_dirty was totally insane"

Now, that's just a theory. And yeah, it may be stated a bit provocatively. It may not be entirely correct. I'm just saying.. maybe it is?

And yes, we actually really _do_ have a data-point from Andrei that says that if you just make "test_clear_page_dirty()" a no-op, the corruption goes away. It was unintentional, bit hey, it's a real datapoint.

See the email from Andrei:

	Subject: Re: 2.6.19 file content corruption on ext3
	From: Andrei Popa <[email protected]>
	Date: Tue, 19 Dec 2006 01:48:11 +0200
	Message-Id: <1166485691.6977.6.camel@localhost>

and look at what remains of his "test_clear_page_dirty()".

Scary, isn't it? And a big hint that "test_clear_page_dirty()" is just totally BOGUS.

And the thing is, I think it's bogus just because I don't understand why it would EVER be ok to drop those dirty bits _except_ very much just before doing the IO that makes it non-dirty (where "truncate()" is really a special case where the IO ends up being not done, but it's the same kind of situation).

Linus

#283 lore raw
Quoted 21 lines · Linus Torvalds

Nobody has actually ever explained why "test_clear_page_dirty()" is good at all.

- Why is it ever used instead of "clear_page_dirty_for_io()"?

- What is the difference?

- Why would you EVER want to clear bits just in the "struct page *" or just in the PTE's?

- Why is it EVER correct to clear dirty bits except JUST BEFORE THE IO?

In other words, I have a theory:

"A lot of this is actually historical cruft. Some of it may even be code
that was never supposed to work, but because we maintained _other_ dirty bits in the PTE's, and never touched them before, we never even realized that the code that played with PG_dirty was totally insane"

Now, that's just a theory. And yeah, it may be stated a bit provocatively. It may not be entirely correct. I'm just saying.. maybe it is?

Quoted 12 lines · Andrew Morton

try_to_free_buffers() clears the page's dirty state if it successfully removed the page's buffers.

Background for this:

- a process does a one-byte-write to a file on a 64k pagesize, 4k blocksize ext3 filesystem. The page is now PageDirty, !PgeUptodate and has one dirty buffer and 15 not uptodate buffers.

- kjournald writes the dirty buffer. The page is now PageDirty, !PageUptodate and has a mix of clean and not uptodate buffers.

Quoted 8 lines

- try_to_free_buffers() removes the page's buffers. It MUST now clear PageDirty. If we were to leave the page dirty then we'd have a dirty, not uptodate page with no buffer_heads.

We're screwed: we cannot write the page because we don't know which sections of it contain garbage. We cannot read the page because we don't know which sections of it contain modified data. We cannot free the page because it is dirty.

However!! this is not true for mapped pages because mapped pages must have the whole (16k in akpm's example) page loaded. Hence I suspect that what Andrei did by accident - remove the if (mapping) case in test_clean_dirty_pages() - is actually totally correct.

#284 lore raw
Quoted 47 lines · Peter Zijlstra
Quoted 21 lines · Linus Torvalds

Nobody has actually ever explained why "test_clear_page_dirty()" is good at all.

- Why is it ever used instead of "clear_page_dirty_for_io()"?

- What is the difference?

- Why would you EVER want to clear bits just in the "struct page *" or just in the PTE's?

- Why is it EVER correct to clear dirty bits except JUST BEFORE THE IO?

In other words, I have a theory:

"A lot of this is actually historical cruft. Some of it may even be code
that was never supposed to work, but because we maintained _other_ dirty bits in the PTE's, and never touched them before, we never even realized that the code that played with PG_dirty was totally insane"

Now, that's just a theory. And yeah, it may be stated a bit provocatively. It may not be entirely correct. I'm just saying.. maybe it is?

Quoted 12 lines · Andrew Morton

try_to_free_buffers() clears the page's dirty state if it successfully removed the page's buffers.

Background for this:

- a process does a one-byte-write to a file on a 64k pagesize, 4k blocksize ext3 filesystem. The page is now PageDirty, !PgeUptodate and has one dirty buffer and 15 not uptodate buffers.

- kjournald writes the dirty buffer. The page is now PageDirty, !PageUptodate and has a mix of clean and not uptodate buffers.

Quoted 8 lines

- try_to_free_buffers() removes the page's buffers. It MUST now clear PageDirty. If we were to leave the page dirty then we'd have a dirty, not uptodate page with no buffer_heads.

We're screwed: we cannot write the page because we don't know which sections of it contain garbage. We cannot read the page because we don't know which sections of it contain modified data. We cannot free the page because it is dirty.

Quoted 4 lines

However!! this is not true for mapped pages because mapped pages must have the whole (16k in akpm's example) page loaded. Hence I suspect that what Andrei did by accident - remove the if (mapping) case in test_clean_dirty_pages() - is actually totally correct.

Obviously I need my morning shot, 64k ofcourse.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#286 lore raw
Quoted 10 lines · On Tue, 19 Dec 2006 20:56:50 +1100 Nick Piggin <[email protected]> wrote:
Quoted 3 lines · Linus Torvalds

NOTICE? First you make a BIG DEAL about how dirty bits should never get lost, but THE VERY SAME FUNCTION actually very much on purpose DOES drop the dirty bit for when it's not in the page tables.

try_to_free_buffers is quite a special case, where we're transferring the page dirty metadata from the buffers to the page. I think Andrew would have a better grasp of it so he could correct me, but what it does is legitimate.

Well it used to be. After 2.6.19 it can do the wrong thing for mapped pages. But it turns out that we don't feed it mapped pages, apart from pagevec_strip() and possibly races against pagefaults.

Quoted 2 lines

I think it could be very likely that indeed the bug is a latent one in a clear_page_dirty caller, rather than dirty-tracking itself.

The only callers are try_to_free_buffers(), truncate and a few scruffy possibly-wrong-for-fsync filesytems which aren't being used here.

<spots a race in do_no_page()>

If a write-fault races with a read-fault and the write-fault loses, we forget to mark the page dirty.

Something like this, but it's probably wrong - I didn't try very hard (am feeling ill, and vaguely grumpy)

From: Andrew Morton <[email protected]>

---

 mm/memory.c |   12 ++++++++++++
 1 file changed, 12 insertions(+)
mm/memory.c renamed from mm/memory.c~a +12 -0
diff -puN mm/memory.c~a mm/memory.c
@@ -2264,10 +2264,22 @@ retry:
 		}
 	} else {
 		/* One of our sibling threads was faster, back out. */
+		if (write_access) {
+			/*
+			 * We might have raced against a read-fault.  We still
+			 * need to dirty the page.
+			 */
+			dirty_page = vm_normal_page(vma, address, *page_table);
+			if (dirty_page) {
+				get_page(dirty_page);
+				goto dirty_it;
+			}
+		}
 		page_cache_release(new_page);
 		goto unlock;
 	}
 
+dirty_it:
 	/* no need to invalidate: a not-present page shouldn't be cached */
 	update_mmu_cache(vma, address, entry);
 	lazy_mmu_prot_update(entry);

_

#287 lore raw
Quoted 9 lines · Andrew Morton
Quoted 5 lines · On Tue, 19 Dec 2006 20:56:50 +1100 Nick Piggin <[email protected]> wrote:
Quoted 1 line · Linus Torvalds

NOTICE? First you make a BIG DEAL about how dirty bits should never get lost, but THE VERY SAME FUNCTION actually very much on purpose DOES drop the dirty bit for when it's not in the page tables.

Quoted 8 lines
Quoted 4 lines

try_to_free_buffers is quite a special case, where we're transferring the page dirty metadata from the buffers to the page. I think Andrew would have a better grasp of it so he could correct me, but what it does is legitimate.

Well it used to be. After 2.6.19 it can do the wrong thing for mapped pages.

Yes, that is what I was trying to get at.

Quoted 2 lines

But it turns out that we don't feed it mapped pages, apart from
pagevec_strip() and possibly races against pagefaults.

True, and I think we have pretty well established that this isn't the cause of Andrei's problem, but I think we all agree it is *a* bug?

And surely Andrei's data corruption will be of the same flavour in that test_clear_page_dirty somewhere is now stripping pte dirty bits where it shouldn't? (because it went away after Peter nooped that behaviour)

Quoted 4 lines
Quoted 2 lines

I think it could be very likely that indeed the bug is a latent one in a clear_page_dirty caller, rather than dirty-tracking itself.

Quoted 8 lines

The only callers are try_to_free_buffers(), truncate and a few scruffy possibly-wrong-for-fsync filesytems which aren't being used here.

<spots a race in do_no_page()>

If a write-fault races with a read-fault and the write-fault loses, we forget to mark the page dirty.

Hmm.. in that case will the pte still be readonly, and thus the write faulter will have to try again I think?

Quoted 42 lines

Something like this, but it's probably wrong - I didn't try very hard (am feeling ill, and vaguely grumpy)

From: Andrew Morton <[email protected]>

---

 mm/memory.c |   12 ++++++++++++
 1 file changed, 12 insertions(+)

diff -puN mm/memory.c~a mm/memory.c

mm/memory.c renamed from mm/memory.c~a +12 -0
@@ -2264,10 +2264,22 @@ retry:
 		}
 	} else {
 		/* One of our sibling threads was faster, back out. */
+		if (write_access) {
+			/*
+			 * We might have raced against a read-fault.  We still
+			 * need to dirty the page.
+			 */
+			dirty_page = vm_normal_page(vma, address, *page_table);
+			if (dirty_page) {
+				get_page(dirty_page);
+				goto dirty_it;
+			}
+		}
 		page_cache_release(new_page);
 		goto unlock;
 	}
 
+dirty_it:
 	/* no need to invalidate: a not-present page shouldn't be cached */
 	update_mmu_cache(vma, address, entry);
 	lazy_mmu_prot_update(entry);

_

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#288 lore raw
In reply to Andrew Morton
Quoted 4 lines · On Tue, 19 Dec 2006 02:32:55 -0800 Andrew Morton <[email protected]> wrote:

<spots a race in do_no_page()>

If a write-fault races with a read-fault and the write-fault loses, we forget to mark the page dirty.

No that isn't right, is it. The writer just retakes the fault and all the right things happen. Ho hum.

#289 lore raw
In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 6 lines · On Tue, 19 Dec 2006 20:56:50 +1100 Nick Piggin <[email protected]> wrote:
Quoted 3 lines · Linus Torvalds

NOTICE? First you make a BIG DEAL about how dirty bits should never get lost, but THE VERY SAME FUNCTION actually very much on purpose DOES drop the dirty bit for when it's not in the page tables.

Quoted 8 lines
Quoted 4 lines

try_to_free_buffers is quite a special case, where we're transferring the page dirty metadata from the buffers to the page. I think Andrew would have a better grasp of it so he could correct me, but what it does is legitimate.

Well it used to be. After 2.6.19 it can do the wrong thing for mapped pages. But it turns out that we don't feed it mapped pages, apart from pagevec_strip() and possibly races against pagefaults.

So how about this:

mm/page-writeback.c +3 -0
Index: linux-2.6-git/mm/page-writeback.c
===================================================================
@@ -859,6 +859,9 @@ int test_clear_page_dirty(struct page *p
 	struct address_space *mapping = page_mapping(page);
 	unsigned long flags;
 
+	if (page_mapped(page))
+		return 0;
+
 	if (!mapping)
 		return TestClearPageDirty(page);
 
#290 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Andrew Morton

Well it used to be. After 2.6.19 it can do the wrong thing for mapped pages. But it turns out that we don't feed it mapped pages, apart from pagevec_strip() and possibly races against pagefaults.

So how about this:

Well that's still racy. Anyway several earlier patches (including the one I posted) closed this race. Some were still reported to trigger corruption IIRC.

Quoted 17 lines

Index: linux-2.6-git/mm/page-writeback.c
===================================================================

mm/page-writeback.c +3 -1
@@ -859,6 +859,9 @@ int test_clear_page_dirty(struct page *p
 	struct address_space *mapping = page_mapping(page);
 	unsigned long flags;
 
+	if (page_mapped(page))
+		return 0;
+
 	if (!mapping)
 		return TestClearPageDirty(page);
 
 
 
-
Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#291 lore raw
Quoted 8 lines · Nick Piggin
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Andrew Morton

Well it used to be. After 2.6.19 it can do the wrong thing for mapped pages. But it turns out that we don't feed it mapped pages, apart from pagevec_strip() and possibly races against pagefaults.

Quoted 5 lines
Quoted 1 line

So how about this:

Well that's still racy. Anyway several earlier patches (including the one I posted) closed this race. Some were still reported to trigger corruption IIRC.

I can't remember a patch that removes mapped pages from this code path, however I could have missed it. All out removing the mapping branch in ttfb() did also fix the problem - which is a superset of page_mapped().

I'm now building a kernel with this patch, and will submit that to rtorrent with mem=256M on a 1k ext3 filesystem on x86_64 smp preempt.

---
 fs/buffer.c |   32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)
fs/buffer.c +31 -1
Index: linux-2.6/fs/buffer.c
===================================================================
@@ -2798,11 +2798,38 @@ static inline int buffer_busy(struct buf
 		(bh->b_state & ((1 << BH_Dirty) | (1 << BH_Lock)));
 }
 
+/*
+ * AKPM sayeth:
+ *
+ * - a process does a one-byte-write to a file on a 64k pagesize, 4k
+ *   blocksize ext3 filesystem.  The page is now PageDirty, !PgeUptodate and
+ *   has one dirty buffer and 15 not uptodate buffers.
+ *
+ * - kjournald writes the dirty buffer.  The page is now PageDirty,
+ *   !PageUptodate and has a mix of clean and not uptodate buffers.
+ *
+ * - try_to_free_buffers() removes the page's buffers.  It MUST now clear
+ *   PageDirty.  If we were to leave the page dirty then we'd have a dirty, not
+ *   uptodate page with no buffer_heads.
+ *
+ *   We're screwed: we cannot write the page because we don't know which
+ *   sections of it contain garbage.  We cannot read the page because we don't
+ *   know which sections of it contain modified data.  We cannot free the page
+ *   because it is dirty.
+ *
+ * However for mapped pages this is not true; mapped pages will be fully
+ * loaded and thus cannot have not uptodate buffers.
+ *
+ * Hence allow the PG_dirty bit to stay for pages that had no not uptodate
+ * buffers (and assert that mapped pages never have those).
+ */
+
 static int
 drop_buffers(struct page *page, struct buffer_head **buffers_to_free)
 {
 	struct buffer_head *head = page_buffers(page);
 	struct buffer_head *bh;
+	int uptodate = 1;
 
 	bh = head;
 	do {
@@ -2818,11 +2845,14 @@ drop_buffers(struct page *page, struct b
 
 		if (!list_empty(&bh->b_assoc_buffers))
 			__remove_assoc_queue(bh);
+		if (!buffer_uptodate(bh))
+			uptodate = 0;
 		bh = next;
 	} while (bh != head);
 	*buffers_to_free = head;
 	__clear_page_buffers(page);
-	return 1;
+	VM_BUG_ON(page_mapped(page) && !uptodate);
+	return !uptodate;
 failed:
 	return 0;
 }
#292 lore raw
In reply to Andrew Morton
Quoted 9 lines · Andrew Morton
Quoted 2 lines · On Tue, 19 Dec 2006 20:56:50 +1100 Nick Piggin <[email protected]> wrote:

I think it could be very likely that indeed the bug is a latent one in a clear_page_dirty caller, rather than dirty-tracking itself.

The only callers are try_to_free_buffers(), truncate and a few scruffy possibly-wrong-for-fsync filesytems which aren't being used here.

Well truncate/invalidate will not operate on mapped pages (barring the very-unlikely truncate/invalidate vs fault races). We can ignore those filesystems as they don't include ext3. Which brings us back to try_to_free_buffers().

Maybe it is something else entirely, but did try_to_free_buffers ever get completely cleared? Or was some of Andrei's corruption possibly leftover on-disk corruption from a previous kernel?

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#293 lore raw
In reply to a message not in the archive
Quoted 4 lines · Nick Piggin

Counterexample? Well AFAIKS, the clearing of PG_dirty in ttfb() in response to finding all buffers clean is perfectly valid. What makes you think otherwise?

If the page really is clean, then why the heck cant' we just clean the page table bits too?

Either it's clean or it isn't. If all the buffers being clean means that the page is clean, then it's clean. WE SHOULD NOT THINK THAT PTE'S ARE ANY DIFFERENT.

I really don't see your point. Is it clean? If it is, then clear the damn dirty bits from the page tables too. Don't go pussyfooting around the issue and confuse yourself and everybody but me by saying "but if it's dirty in the page tables, it's magically dirty". NO.

It really is that simple. Is it clean or not?

If it's clean, you can remove ALL the dirty bits. Not just some.

Linus

#294 lore raw

Btw,
here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

There is one case where the kernel actually forcibly writes zeroes into a
file: when we're writing a page that straddles the "inode->i_size" boundary. See the various writepages in fs/buffer.c, they all contain variations on that theme (although most of them aren't as well commented as this snippet):

        /*
         * The page straddles i_size.  It must be zeroed out on each and every
         * writepage invocation because it may be mmapped.  "A file is mapped
         * in multiples of the page size.  For a file that is not a multiple of
         * the  page size, the remaining memory is zeroed when mapped, and
         * writes to that region are not written out to the file."
         */
        kaddr = kmap_atomic(page, KM_USER0);
        memset(kaddr + offset, 0, PAGE_CACHE_SIZE - offset);
        flush_dcache_page(page);
        kunmap_atomic(kaddr, KM_USER0);

Now, this should _matter_ only for user processes that are buggy, and that have written to the page _before_ extending it with ftruncate(). That's definitely a serious bug, but it's one that can do totally undetected depending on when the actual write-out happens.

So what I'm saying is that if we end up writing things earlier thanks to the more aggressive dirty-page-management thing in 2.6.19, we might actually just expose a long-time userspace bug that was just a LOT harder to trigger before..

I'm not saying this is the cause of all this, but we've been tearing our hair out, and it migth be worthwhile trying this really really really stupid patch that will notice when that happens at truncate() time, and tell the user that he's a total idiot. Or something to that effect.

Maybe the reason this is so easy to trigger with rtorrent is not because rtorrent does some magic pattern that triggers a kernel bug, but simply because rtorrent itself might have a bug.

Ok, so it's a long shot, but it's still worth testing, I suspect. The patch is very simple: whenever we do an _expanding_ truncate, we check the last page of the _old_ size, and if there were non-zero contents past the old size, we complain.

As an attachement is a test-program that _should_ trigger a kernel message like

a.out: BADNESS: truncate check 17000

for good measure, just so that you can verify that the patch works and actually catches this case.

(The 17000 number is just the one-hundred _invalid_ 0xaa bytes - out of the 200 we wrote - that were summed up: 100*0xaa == 17000. Anything non-zero is always a bug).

I doubt this is really it, but it's worth trying. If you fill out a page, and only do "ftruncate()" in response to SIGBUS messages (and don't truncate to whole pages), you could potentially see zeroes at the end of the page exactly because _writeout_ cleared the page for you! So it _could_ explain the symptoms, but only if user-space was horribly horribly broken.

Linus

----

mm/memory.c +28 -0
index c00bac6..79cecab 100644
@@ -1842,6 +1842,33 @@ void unmap_mapping_range(struct address_space *mapping,
 }
 EXPORT_SYMBOL(unmap_mapping_range);
 
+static void check_last_page(struct address_space *mapping, loff_t size)
+{
+	pgoff_t index;
+	unsigned int offset;
+	struct page *page;
+
+	if (!mapping)
+		return;
+	offset = size & ~PAGE_MASK;
+	if (!offset)
+		return;
+	index = size >> PAGE_SHIFT;
+	page = find_lock_page(mapping, index);
+	if (page) {
+		unsigned int check = 0;
+		unsigned char *kaddr = kmap_atomic(page, KM_USER0);
+		do {
+			check += kaddr[offset++];
+		} while (offset < PAGE_SIZE);
+		kunmap_atomic(kaddr,KM_USER0);
+		unlock_page(page);
+		page_cache_release(page);
+		if (check)
+			printk("%s: BADNESS: truncate check %u\n", current->comm, check);
+	}
+}
+
 /**
  * vmtruncate - unmap mappings "freed" by truncate() syscall
  * @inode: inode of the file used
@@ -1875,6 +1902,7 @@ do_expand:
 		goto out_sig;
 	if (offset > inode->i_sb->s_maxbytes)
 		goto out_big;
+	check_last_page(mapping, inode->i_size);
 	i_size_write(inode, offset);
 
 out_truncate:

#include <sys/mman.h>
#include <sys/fcntl.h>
#include <unistd.h>
#include <string.h>

int main(int argc, char **argv)

{
	char *mapping;
	int fd;

	fd = open("mapfile", O_RDWR | O_TRUNC | O_CREAT, 0666);
	if (fd < 0)
		return -1;
	if (ftruncate(fd, 10) < 0)
		return -1;
	mapping = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
	if (-1 == (int)(long)mapping)
		return -1;
	memset(mapping, 0x55, 10);
	if (ftruncate(fd, 100) < 0)
		return -1;
	memset(mapping, 0xaa, 200);
	if (ftruncate(fd, 200) < 0)
		return -1;
	return 0;
}
#295 lore raw
Quoted 3 lines · Linus Torvalds

here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

Btw, here's a simpler test-program that actually shows the difference between 2.6.18 and 2.6.19 in action, and why it could explain why a program like rtorrent might show corruption behavious that it didn't show before.

	#include <sys/mman.h>
	#include <sys/fcntl.h>
	#include <unistd.h>
	#include <string.h>
	
	int main(int argc, char **argv)
	{
		char *mapping;
		int fd;
	
		fd = open("mapfile", O_RDWR | O_TRUNC | O_CREAT, 0666);
		if (fd < 0)
			return -1;
		if (ftruncate(fd, 10) < 0)
			return -1;
		mapping = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
		if (-1 == (int)(long)mapping)
			return -1;
		memset(mapping, 0xaa, 20);
		sync();
		if (ftruncate(fd, 40) < 0)
			return -1;
		memset(mapping + 20, 0x55, 20);
		write(1, mapping, 40);
		return 0;
	}

Notice the "sync()" in between the "memset()" and the "ftruncate()". In 2.6.18, that would normally do absolutely _nothing_ to the shared memory mapping, becuase we simply couldn't track pages that were dirty in the page tables.

So in 2.6.18, if you try this, with

	./a.out | od -x

you should see something like

	0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa
	0000020 aaaa aaaa 5555 5555 5555 5555 5555 5555
	0000040 5555 5555 5555 5555
	0000050

which matches your memset() patterns: 20 bytes of 0xaa, and 20 bytes of 0x55.

HOWEVER.

In 2.6.19, because we actually track dirty data so much better, "sync()" will actually be smart enough to write out the dirty mmap'ed data too. But since the user program has only allocated ten bytes for it in the file, when it is written out, the rest of the page is cleared. When you then write the last 20 bytes (after _properly_ allocating memory for them), you should now see a pattern like

	0000000 aaaa aaaa aaaa aaaa aaaa 0000 0000 0000
	0000020 0000 0000 5555 5555 5555 5555 5555 5555
	0000040 5555 5555 5555 5555
	0000050

instead: with ten bytes of zero in between, because the data that couldn't be written out was cleared.

So 2.6.19 is strictly _better_, but exactly because it's tracking dirty status much more precisely, you'll see certain user-level bugs much more easily.

NOTE NOTE NOTE! The code really _was_ buggy in 2.6.18 too, and you _can_ get the zeroes in the middle of the file with an older kernel. But in older kernels, you need to be really really unlucky, and have the page cleaned by strong memory pressure. In 2.6.19, any "sync()" activity (includign from the outside) will clean the page, so a user program with this bug can just be made to trigger the bug much more easily.

Linus

#296 lore raw
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

I'm sad to say this doesn't trigger :-(

#297 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

I'm sad to say this doesn't trigger :-(

Oh, well. It was a theory.

Linus

#298 lore raw
	On Tue, 19 Dec 2006 14:51:55 -0800 (PST)
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Peter Zijlstra
Quoted 1 line · Linus Torvalds
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines · Linus Torvalds
Quoted 3 lines

here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

I'm sad to say this doesn't trigger :-(

Oh, well. It was a theory.

Well... we'd need to see (corruption && this-not-triggering) to be sure.

Peter, have you been able to trigger the corruption?

#299 lore raw
Quoted 3 lines · Andrew Morton

Well... we'd need to see (corruption && this-not-triggering) to be sure.

Peter, have you been able to trigger the corruption?

Yes; however the mail I send describing that seems to be lost in space.

/me quotes from the send folder:

Quoted 17 lines

The bad new is, that doesn't help either. The good news is I can reproduce it.

What I did to achieve that:

- get a sizable torrent from legaltorrents.com / or create a torrent
yourself that is around ~600M and has multiple files.

- start a tracker, and multiple seeds (I used three machines here)

- pull the torrent on a fourth machine

the seeding machines don't much matter of course.

the fourth machine was a dual core x86-64 with an SMP kernel and PREEMPT, mem=256M (so that the torrent is quite a bit larger and does require writeout) and I used an ext3 partition with 1k blocks.

#300 lore raw
Quoted 27 lines · Peter Zijlstra
Quoted 3 lines · Andrew Morton

Well... we'd need to see (corruption && this-not-triggering) to be sure.

Peter, have you been able to trigger the corruption?

Yes; however the mail I send describing that seems to be lost in space.

/me quotes from the send folder:

Quoted 17 lines

The bad new is, that doesn't help either. The good news is I can reproduce it.

What I did to achieve that:

- get a sizable torrent from legaltorrents.com / or create a torrent
yourself that is around ~600M and has multiple files.

- start a tracker, and multiple seeds (I used three machines here)

- pull the torrent on a fourth machine

the seeding machines don't much matter of course.

the fourth machine was a dual core x86-64 with an SMP kernel and PREEMPT, mem=256M (so that the torrent is quite a bit larger and does require writeout) and I used an ext3 partition with 1k blocks.

PS. this was a reply to:
http://lkml.org/lkml/2006/12/19/121

#301 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Andrew Morton

Well... we'd need to see (corruption && this-not-triggering) to be sure.

Peter, have you been able to trigger the corruption?

Yes; however the mail I send describing that seems to be lost in space.

Btw, can somebody actually explain the mess that is ext3 "dirtying".

Ext3 does NOT use __set_page_dirty_buffers. It does

	static int ext3_journalled_set_page_dirty(struct page *page)
	{
	        SetPageChecked(page);
	        return __set_page_dirty_nobuffers(page);
	}

and uses that "Checked" bit as a "whole page is dirty" bit (which it tests in "writepage()".

You realize what this all means? It means that ANYTHING that actually clears the _real_ dirty bit won't actually be doing anything at all for ext3, because the Checked bit will still stay set, and any IO down the line on that page would totally ignore the dirty bits on the buffer heads and just write out everything.

That is "The Mess(tm)".

It also basically means that anything that clears the dirty bit without just calling "writepage()" had _better_ call "invalidatepage()" for the whole page, because otherwise the PageChecked bit will never be cleared as far as I can see. Happily, at least ext3 seems to _test_ for that case in the release_page() function, so it appears that we do do this.

But this seems to just strengthen my argument: you can NEVER clean a page, unless you (a) do IO on it immediately afterwards (writeback) or (b) invalidate it entirely (truncate).

I'd really like to see just those two functions exist. Preferably in a form where you can see easily that we actually follow those rules. Rather than having a confusing set of "clear_page_dirty()" and "test_and_clear_page_dirty()" functions that are called from random places.

IOW, I think the "clear_page_dirty_for_io()" is fine (it's case (a)) above, and then we should probably have a "cancel_dirty_page()" function that does all the current clear_page_dirty() but also makes sure that we actually call the invalidate_page() function itself.

Hmm?

Linus

#302 lore raw
Quoted 16 lines · On Tue, 19 Dec 2006 16:03:49 -0800 (PST) Linus Torvalds <[email protected]>…
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Andrew Morton

Well... we'd need to see (corruption && this-not-triggering) to be sure.

Peter, have you been able to trigger the corruption?

Yes; however the mail I send describing that seems to be lost in space.

Btw, can somebody actually explain the mess that is ext3 "dirtying".

Ext3 does NOT use __set_page_dirty_buffers. It does

Quoted 8 lines
	static int ext3_journalled_set_page_dirty(struct page *page)
	{
	        SetPageChecked(page);
	        return __set_page_dirty_nobuffers(page);
	}

and uses that "Checked" bit as a "whole page is dirty" bit (which it tests in "writepage()".

This is purely for data=journal, which is rarely used.

In journalled-data mode, write(), write-fault, etc are not allowed to dirty the pages and buffers, because the data has to be written to the journal first. After the data has been written to the journal we only then mark buffers (and hence pages) dirty as far as the VFS is concerned. For checkpointing the data back to its real place on the disk.

For MAP_SHARED pages ext3 cheats madly and doesn't journal the data at all. In all journalling modes, MAP_SHARED data follows the regular ext2-style handling. Which is a bit of a wart.

#303 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

here's a totally new tangent on this: it's possible that user code is simply BUGGY.

Quoted 11 lines

I'm sad to say this doesn't trigger :-(

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi all,

I ran it a number of times on 2.6.16-1.2115_FC4 and always got ./a.out | od -x
0000000 aaaa aaaa aaaa aaaa aaaa aaaa aaaa aaaa 0000020 aaaa aaaa 5555 5555 5555 5555 5555 5555 0000040 5555 5555 5555 5555

but running it on 2.6.19-rc5 I always get zeros in the middle.

Steve

Signature · 3 lines
"They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty decreases."  (Thomas Jefferson)
#304 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Linus Torvalds

here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

Btw, here's a simpler test-program that actually shows the difference between 2.6.18 and 2.6.19 in action, and why it could explain why a program like rtorrent might show corruption behavious that it didn't show before.

Kinda late to the discussion, but I guess I could summarize what rtorrent actually does, or should be doing.

When downloading a new torrent, it will create the files and truncate them to the final size. It will never call truncate after this and the files will remain sparse until data is downloaded. A 'piece' is mapped to memory using MAP_SHARED, which will be page aligned on single file torrents but unlikely to be so on multi-file torrents.

So on multi-file torrents it'll often end up with two mappings overlapping with one page, each of which only write to their own part the page. These will then be sync'ed with MS_ASYNC, or MS_SYNC if low on disk space. After that it might be unmapped, then mapped as read-only.

I haven't thought of asking if single file torrents are ok.

Rakshasa

#305 lore raw
In reply to Linus Torvalds

* Linus Torvalds:

Quoted 3 lines

Now, this should _matter_ only for user processes that are buggy, and that have written to the page _before_ extending it with ftruncate().

APT seems to properly extend the file before mapping it, by writing a zero byte at the desired position (creating a hole).

24986 open("/var/cache/apt/pkgcache.bin", O_RDWR|O_CREAT|O_TRUNC, 0666) = 6

24986 lseek(6, 12582911, SEEK_SET) = 12582911
24986 write(6, "\0", 1) = 1

24986 mmap(NULL, 12582912, PROT_READ|PROT_WRITE, MAP_SHARED, 6, 0) = 0x2b6578636000

24986 msync(0x2b6578636000, 7464112, MS_SYNC) = 0
24986 msync(0x2b6578636000, 8656, MS_SYNC) = 0
24986 munmap(0x2b6578636000, 12582912) = 0
24986 ftruncate(6, 7464112) = 0
24986 fstat(6, {st_mode=S_IFREG|0644, st_size=7464112, ...}) = 0 24986 mmap(NULL, 7464112, PROT_READ, MAP_SHARED, 6, 0) = 0x2b6578636000

APT's code is pretty convoluted, though, and there might be some code path in it that gets it wrong. 8-P

#306 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

Btw,
here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

depmod: BADNESS: written outside isize 22183

---

fs/buffer.c +13 -0
index d1f1b54..5db9fd9 100644
@@ -2393,6 +2393,17 @@ int nobh_commit_write(struct file *file, struct page *page,
 }
 EXPORT_SYMBOL(nobh_commit_write);
 
+static void __check_tail_zero(char *kaddr, unsigned int offset)
+{
+	unsigned int check = 0;
+	do {
+		check += kaddr[offset++];
+	} while (offset < PAGE_CACHE_SIZE);
+	if (check)
+		printk(KERN_ERR "%s: BADNESS: written outside isize %u\n",
+				current->comm, check);
+}
+
 /*
  * nobh_writepage() - based on block_full_write_page() except
  * that it tries to operate without attaching bufferheads to
@@ -2437,6 +2448,7 @@ int nobh_writepage(struct page *page, get_block_t *get_block,
 	 * writes to that region are not written out to the file."
 	 */
 	kaddr = kmap_atomic(page, KM_USER0);
+	__check_tail_zero(kaddr, offset);
 	memset(kaddr + offset, 0, PAGE_CACHE_SIZE - offset);
 	flush_dcache_page(page);
 	kunmap_atomic(kaddr, KM_USER0);
@@ -2604,6 +2616,7 @@ int block_write_full_page(struct page *page, get_block_t *get_block,
 	 * writes to that region are not written out to the file."
 	 */
 	kaddr = kmap_atomic(page, KM_USER0);
+	__check_tail_zero(kaddr, offset);
 	memset(kaddr + offset, 0, PAGE_CACHE_SIZE - offset);
 	flush_dcache_page(page);
 	kunmap_atomic(kaddr, KM_USER0);
#307 lore raw
Quoted 7 lines · On Thu, 21 Dec 2006 14:03:20 +0100 Peter Zijlstra <[email protected]>…
Quoted 4 lines · Linus Torvalds

Btw,
here's a totally new tangent on this: it's possible that user code is
simply BUGGY.

depmod: BADNESS: written outside isize 22183

akpm:/usr/src/module-init-tools-3.3-pre1> grep -r mmap .
./zlibsupport.c: map = mmap(0, *size, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);

So presumably it's in a library.

akpm:/usr/src/25> ldd /sbin/depmod

        linux-gate.so.1 =>  (0xffffe000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x46afa000)
        /lib/ld-linux.so.2 (0x4631d000)

worrisome.

#308 lore raw
In reply to Linus Torvalds
Quoted 21 lines · Linus Torvalds
Quoted 3 lines · Nick Piggin

We never want to drop dirty data! (ignoring the truncate case, which is handled privately by truncate anyway)

Bzzt.

SURE we do.

We absolutely do want to drop dirty data in the writeout path.

How do you think dirty data ever _becomes_ clean data?

In other words, yes, we _do_ want to test-and-clear all the pgtable bits _and_ the PG_dirty bit. We want to do it for:
- writeout
- truncate
- possibly a "drop" event (which could be a case for a journal entry that becomes stale due to being replaced or something - kind of "truncate" on metadata)

Quoted 5 lines

because both of those events _literally_ turn dirty state into clean state.

In no other circumstance do we ever want to clear a dirty bit, as far as I can tell.

i admit this may not be entirely relevant, but it seems like a good place to bring up an old problem: when a disk dies with lots of queued writes it can totally bring a system to its knees... even after the disk is removed. i wrote up something about this a while ago:

http://lkml.org/lkml/2005/8/18/243

so there's another reason to "clear a dirty bit"... well, in fact -- drop the pages entirely.

-dean

#309 lore raw
In reply to Nick Piggin
Quoted 110 lines · Nick Piggin

plain text document attachment (fs-fix.patch)
Index: linux-2.6/fs/buffer.c
===================================================================

fs/buffer.c +11 -1
@@ -2852,7 +2852,17 @@ int try_to_free_buffers(struct page *pag
 		 * This only applies in the rare case where try_to_free_buffers
 		 * succeeds but the page is not freed.
 		 */
-		clear_page_dirty(page);
+
+		/*
+		 * If the page has been dirtied via the user mappings, then
+		 * clean buffers does not indicate the page data is actually
+		 * clean! Only clear the page dirty bit if there are no dirty
+		 * ptes either.
+		 *
+		 * If there are dirty ptes, then the page must be uptodate, so
+		 * the above concern does not apply.
+		 */
+		clear_page_dirty_sync_ptes(page);
 	}
 out:
 	if (buffers_to_free) {

Index: linux-2.6/include/linux/page-flags.h
===================================================================

include/linux/page-flags.h +6 -0
@@ -254,6 +254,7 @@ static inline void SetPageUptodate(struc
 struct page;	/* forward declaration */
 
 int test_clear_page_dirty(struct page *page);
+int test_clear_page_dirty_sync_ptes(struct page *page);
 int test_clear_page_writeback(struct page *page);
 int test_set_page_writeback(struct page *page);
 
@@ -262,6 +263,11 @@ static inline void clear_page_dirty(stru
 	test_clear_page_dirty(page);
 }
 
+static inline void clear_page_dirty_sync_ptes(struct page *page)
+{
+	test_clear_page_dirty_sync_ptes(page);
+}
+
 static inline void set_page_writeback(struct page *page)
 {
 	test_set_page_writeback(page);

Index: linux-2.6/mm/page-writeback.c
===================================================================

mm/page-writeback.c +30 -4
@@ -844,9 +844,10 @@ EXPORT_SYMBOL(set_page_dirty_lock);
 
 /*
  * Clear a page's dirty flag, while caring for dirty memory accounting. 
+ * Does not clear pte dirty bits.
  * Returns true if the page was previously dirty.
  */
-int test_clear_page_dirty(struct page *page)
+static int test_clear_page_dirty_leave_ptes(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);
 	unsigned long flags;
@@ -862,10 +863,8 @@ int test_clear_page_dirty(struct page *p
 			 * We can continue to use `mapping' here because the
 			 * page is locked, which pins the address_space
 			 */
-			if (mapping_cap_account_dirty(mapping)) {
-				page_mkclean(page);
+			if (mapping_cap_account_dirty(mapping))
 				dec_zone_page_state(page, NR_FILE_DIRTY);
-			}
 			return 1;
 		}
 		write_unlock_irqrestore(&mapping->tree_lock, flags);
@@ -873,9 +872,43 @@ int test_clear_page_dirty(struct page *p
 	}
 	return TestClearPageDirty(page);
 }
+
+/*
+ * As above, but does clear dirty bits from ptes
+ */
+int test_clear_page_dirty(struct page *page)
+{
+	struct address_space *mapping = page_mapping(page);
+
+	if (test_clear_page_dirty_leave_ptes(page)) {
+		if (mapping_cap_account_dirty(mapping))
+			page_mkclean(page);
+		return 1;
+	}
+	return 0;
+}
 EXPORT_SYMBOL(test_clear_page_dirty);
 
 /*
+ * As above, but redirties page if any dirty ptes are found (and then only
+ * if the mapping accounts dirty pages, otherwise dirty ptes are left dirty
+ * but the page is cleaned).
+ */
+int test_clear_page_dirty_sync_ptes(struct page *page)
+{
+	struct address_space *mapping = page_mapping(page);
+
+	if (test_clear_page_dirty_leave_ptes(page)) {
+		if (mapping_cap_account_dirty(mapping)) {
+			if (page_mkclean(page))
+				set_page_dirty(page);

mm/page-writeback.c int test_clear_page_dirty(struct page *p · line 914

Quoted 10 lines
mm/page-writeback.c +7 -0
+		}
+		return 1;
+	}
+	return 0;
+}
+
+/*
  * Clear a page's dirty flag, while caring for dirty memory accounting.
  * Returns true if the page was previously dirty.
  *

Hmm, not quite; It certainly look better than the extra ,[01] tagged to test_clear_page_dirty() though. Although I would have expected it the other way around - test_clear_pages_dirty_sync_ptes to be the default case and test_clear_pages_dirty_clean_ptes to be used in clear_page_dirty_for_io().

Anyway it has the same issues as the others. See what happens when you run two test_clear_page_dirty_sync_ptes() consecutively, you still loose PG_dirty even though the page might actually be dirty.

#310 lore raw
Quoted 131 lines · Peter Zijlstra
Quoted 110 lines · Nick Piggin

plain text document attachment (fs-fix.patch)
Index: linux-2.6/fs/buffer.c
===================================================================

fs/buffer.c +11 -1
@@ -2852,7 +2852,17 @@ int try_to_free_buffers(struct page *pag
 		 * This only applies in the rare case where try_to_free_buffers
 		 * succeeds but the page is not freed.
 		 */
-		clear_page_dirty(page);
+
+		/*
+		 * If the page has been dirtied via the user mappings, then
+		 * clean buffers does not indicate the page data is actually
+		 * clean! Only clear the page dirty bit if there are no dirty
+		 * ptes either.
+		 *
+		 * If there are dirty ptes, then the page must be uptodate, so
+		 * the above concern does not apply.
+		 */
+		clear_page_dirty_sync_ptes(page);
 	}

out:

	if (buffers_to_free) {

Index: linux-2.6/include/linux/page-flags.h
===================================================================

include/linux/page-flags.h +0 -0
@@ -254,6 +254,7 @@ static inline void SetPageUptodate(struc

struct page; /* forward declaration */

int test_clear_page_dirty(struct page *page); +int test_clear_page_dirty_sync_ptes(struct page *page); int test_clear_page_writeback(struct page *page); int test_set_page_writeback(struct page *page);

include/linux/page-flags.h +5 -0
@@ -262,6 +263,11 @@ static inline void clear_page_dirty(stru
 	test_clear_page_dirty(page);
 }
 
+static inline void clear_page_dirty_sync_ptes(struct page *page)
+{
+	test_clear_page_dirty_sync_ptes(page);
+}
+
 static inline void set_page_writeback(struct page *page)
 {
 	test_set_page_writeback(page);

Index: linux-2.6/mm/page-writeback.c
===================================================================

mm/page-writeback.c +30 -4
@@ -844,9 +844,10 @@ EXPORT_SYMBOL(set_page_dirty_lock);
 
 /*
 * Clear a page's dirty flag, while caring for dirty memory accounting. 
+ * Does not clear pte dirty bits.
 * Returns true if the page was previously dirty.
 */
-int test_clear_page_dirty(struct page *page)
+static int test_clear_page_dirty_leave_ptes(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);
 	unsigned long flags;
@@ -862,10 +863,8 @@ int test_clear_page_dirty(struct page *p
 			 * We can continue to use `mapping' here because the
 			 * page is locked, which pins the address_space
 			 */
-			if (mapping_cap_account_dirty(mapping)) {
-				page_mkclean(page);
+			if (mapping_cap_account_dirty(mapping))
 				dec_zone_page_state(page, NR_FILE_DIRTY);
-			}
 			return 1;
 		}
 		write_unlock_irqrestore(&mapping->tree_lock, flags);
@@ -873,9 +872,43 @@ int test_clear_page_dirty(struct page *p
 	}
 	return TestClearPageDirty(page);
 }
+
+/*
+ * As above, but does clear dirty bits from ptes
+ */
+int test_clear_page_dirty(struct page *page)
+{
+	struct address_space *mapping = page_mapping(page);
+
+	if (test_clear_page_dirty_leave_ptes(page)) {
+		if (mapping_cap_account_dirty(mapping))
+			page_mkclean(page);
+		return 1;
+	}
+	return 0;
+}
 EXPORT_SYMBOL(test_clear_page_dirty);
 
 /*
+ * As above, but redirties page if any dirty ptes are found (and then only
+ * if the mapping accounts dirty pages, otherwise dirty ptes are left dirty
+ * but the page is cleaned).
+ */
+int test_clear_page_dirty_sync_ptes(struct page *page)
+{
+	struct address_space *mapping = page_mapping(page);
+
+	if (test_clear_page_dirty_leave_ptes(page)) {
+		if (mapping_cap_account_dirty(mapping)) {
+			if (page_mkclean(page))
+				set_page_dirty(page);

mm/page-writeback.c int test_clear_page_dirty(struct page *p · line 914

Quoted 10 lines
mm/page-writeback.c +7 -0
+		}
+		return 1;
+	}
+	return 0;
+}
+
+/*
 * Clear a page's dirty flag, while caring for dirty memory accounting.
 * Returns true if the page was previously dirty.
 *

Hmm, not quite; It certainly look better than the extra ,[01] tagged to test_clear_page_dirty() though. Although I would have expected it the other way around - test_clear_pages_dirty_sync_ptes to be the default case and test_clear_pages_dirty_clean_ptes to be used in clear_page_dirty_for_io().

Quoted 3 lines

Anyway it has the same issues as the others. See what happens when you run two test_clear_page_dirty_sync_ptes() consecutively, you still loose PG_dirty even though the page might actually be dirty.

How can this happen? We'll only test_clear_page_dirty_sync_ptes again after buffers have been reattached, and subsequently cleaned. And in that case if the ptes are still clean at this point then the page really is clean.

Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#311 lore raw
Quoted 9 lines · Nick Piggin
Quoted 4 lines

Anyway it has the same issues as the others. See what happens when you run two test_clear_page_dirty_sync_ptes() consecutively, you still loose PG_dirty even though the page might actually be dirty.

How can this happen? We'll only test_clear_page_dirty_sync_ptes again after buffers have been reattached, and subsequently cleaned. And in that case if the ptes are still clean at this point then the page really is clean.

Why do you talk about buffers being reattached? Are you still in some world where "try_to_free_buffers()" matters? Have you not followed the discussion? Why do you ignore my MUCH SIMPLER patch that just removed all this crap ENTIRELY from "try_to_free_buffers()", and the exact same corruption happened?

Forget about "try_to_free_buffers()". Please apply this patch to your tree first. That gets rid of _one_ copy of totally insane code that did all the wrong things.

Only after you have applied this patch should you look at the code again. Realizing that the corruption still happens.

So forget about buffers already. That piece of code was crap.

Linus

---

fs/buffer.c +1 -17
index d1f1b54..263f88e 100644
@@ -2834,7 +2834,7 @@ int try_to_free_buffers(struct page *page)
 	int ret = 0;
 
 	BUG_ON(!PageLocked(page));
-	if (PageWriteback(page))
+	if (PageDirty(page) || PageWriteback(page))
 		return 0;
 
 	if (mapping == NULL) {		/* can this still happen? */
@@ -2845,22 +2845,6 @@ int try_to_free_buffers(struct page *page)
 	spin_lock(&mapping->private_lock);
 	ret = drop_buffers(page, &buffers_to_free);
 	spin_unlock(&mapping->private_lock);
-	if (ret) {
-		/*
-		 * If the filesystem writes its buffers by hand (eg ext3)
-		 * then we can have clean buffers against a dirty page.  We
-		 * clean the page here; otherwise later reattachment of buffers
-		 * could encounter a non-uptodate page, which is unresolvable.
-		 * This only applies in the rare case where try_to_free_buffers
-		 * succeeds but the page is not freed.
-		 *
-		 * Also, during truncate, discard_buffer will have marked all
-		 * the page's buffers clean.  We discover that here and clean
-		 * the page also.
-		 */
-		if (test_clear_page_dirty(page))
-			task_io_account_cancelled_write(PAGE_CACHE_SIZE);
-	}
 out:
 	if (buffers_to_free) {
 		struct buffer_head *bh = buffers_to_free;
#312 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Nick Piggin
Quoted 3 lines

Anyway it has the same issues as the others. See what happens when you run two test_clear_page_dirty_sync_ptes() consecutively, you still loose PG_dirty even though the page might actually be dirty.

Quoted 7 lines
Quoted 4 lines

How can this happen? We'll only test_clear_page_dirty_sync_ptes again after buffers have been reattached, and subsequently cleaned. And in that case if the ptes are still clean at this point then the page really is clean.

Why do you talk about buffers being reattached? Are you still in some world where "try_to_free_buffers()" matters? Have you not followed the

I'm talking about fixing just the race Andrew noticed via inspection. No it doesn't appear to fix Andrei's problem, unfortunately. But it needs to be fixed all the same, doesn't it?

Quoted 7 lines

discussion? Why do you ignore my MUCH SIMPLER patch that just removed all this crap ENTIRELY from "try_to_free_buffers()", and the exact same corruption happened?

Forget about "try_to_free_buffers()". Please apply this patch to your tree first. That gets rid of _one_ copy of totally insane code that did all the wrong things.

Only after you have applied this patch should you look at the code again. Realizing that the corruption still happens.

So forget about buffers already. That piece of code was crap.

Now I'm not exactly sure how ext3 (or any other) filesystems make use of this particular feature of try_to_free_buffers(), but it is clear from the comments what it is for. So your patch isn't really a minimal fix (ie. it would require an OK from all filesystems, wouldn't it?)

Or did I miss a mail where you reasoned that it is safe to make this change (/me goes to reread the thread)...

Quoted 41 lines

Linus

---

fs/buffer.c +1 -17
index d1f1b54..263f88e 100644
@@ -2834,7 +2834,7 @@ int try_to_free_buffers(struct page *page)
 	int ret = 0;
 
 	BUG_ON(!PageLocked(page));
-	if (PageWriteback(page))
+	if (PageDirty(page) || PageWriteback(page))
 		return 0;
 
 	if (mapping == NULL) {		/* can this still happen? */
@@ -2845,22 +2845,6 @@ int try_to_free_buffers(struct page *page)
 	spin_lock(&mapping->private_lock);
 	ret = drop_buffers(page, &buffers_to_free);
 	spin_unlock(&mapping->private_lock);
-	if (ret) {
-		/*
-		 * If the filesystem writes its buffers by hand (eg ext3)
-		 * then we can have clean buffers against a dirty page.  We
-		 * clean the page here; otherwise later reattachment of buffers
-		 * could encounter a non-uptodate page, which is unresolvable.
-		 * This only applies in the rare case where try_to_free_buffers
-		 * succeeds but the page is not freed.
-		 *
-		 * Also, during truncate, discard_buffer will have marked all
-		 * the page's buffers clean.  We discover that here and clean
-		 * the page also.
-		 */
-		if (test_clear_page_dirty(page))
-			task_io_account_cancelled_write(PAGE_CACHE_SIZE);
-	}
 out:
 	if (buffers_to_free) {
 		struct buffer_head *bh = buffers_to_free;
Signature · 2 lines
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 
#313 lore raw
Quoted 8 lines · Nick Piggin

Now I'm not exactly sure how ext3 (or any other) filesystems make use of this particular feature of try_to_free_buffers(), but it is clear from the comments what it is for. So your patch isn't really a minimal fix (ie. it would require an OK from all filesystems, wouldn't it?)

Or did I miss a mail where you reasoned that it is safe to make this change (/me goes to reread the thread)...

I'm saying it had _better_ be safe, and no, low-level filesystems don't actually matter.

The page has to be cleanable _some_ way. So if we test for "page_dirty()" at the top, and just refuse to do it in try_to_free_pages(), we still know that the _proper_ page cleaning had better clean it. Because ttfp() is never going to clean the page in the general case _anyway_.

So I'm really saying:

- the page WILL be cleaned by the real page cleaning action (ie memory pressure or sync or something else causing us to go through the bog-standard page-based writeout.

Does anybody dispute this?

- the "ttfp()" hack was a HACK. It was an ugly and nasty hack even when it was first introduced. It gets doubly worse now that we know we have something wrong with page cleaning, and it has distracted from the real problem.

- I removed tha ugly and disgusting hack entirely at first, but Andrew points out that he really wants to keep the buffers there, because the buffers being clean actually say something. That, together with the fact that as long as the page is dirty, the buffers really do end up have a job to do, made me add a much smaller hack to replace the big ugly one ("don't even try, if the page is marked dirty").

- so with that thing in place, there isn't even any change in behaviour wrt the buffers and low-level filesystems. It's just that we make them a bit harder to get rid of. But arguably that shouldn't actually ever really _happen_ anyway (because I think it's a BUG if the page is marked dirty but none of the buffers are), so I think that part is a non-issue.

In other words, ttfp() _never_ had anything to do with "page cleaning". Not originally, not with the horrible hack, and not with my patch.

Trying to mix it in just caused a bug that _everybody_ agrees is a bug. It's not the bug we're chasing, but we've got three different patches to fix it (Andrew's, mine and yours), and mine is the simplest one by far especially in the long run, because it just REMOVES the ugly dependency.

And yes, I probably care more about "in the long run" than most. To me, a bug is a bug even if it's _just_ a maintenance headache. Andrews patch made things _worse_ ("magic insane flag"), and while yours didn't make the code worse, it still introduced the notion of a totally insane "clean the page but if the PTE's are dirty, do something else" notion.

IF THE PAGE TRULY IS CLEAN (and both you and Andrew claim it is, if all buffers are clean - since you mark it clean in the non-mapped case) THEN YOU SHOULD BE ABLE TO CLEAN THE PAGE TABLE BITS TOO.

And by claiming that the page table bits are different from PG_dirty, you're just making the issues worse. They shouldn't be. That's what the whole point of Peter's patch was: PG_dirty fundmentally _means_ that the page tables might be dirty too. That was the whole _point_ of doing all this in 2.6.19 in the first place.

So if you cannot accept that page table bits should be on "equal footing" with PG_dirty, then you should just say "Let's remove Peter's patch entirely".

Linus