[Fwd: Page allocator doubt]

Fwd: Page allocator doubt Luciano A. Stertz 3 participants 8 messages Nov 11, 2004 – Nov 12, 2004 Memory management

lore

Share & tools

lore

Luciano A. Stertz

author

2004-11-11 14:33 UTC #0 lore raw

May someone help me? I tried kernelnewbies, but got no answers.

TIA,
Luciano Stertz

-------- Original Message --------
Subject: Page allocator doubt
Date: Wed, 10 Nov 2004 15:31:41 -0200
From: Luciano A. Stertz <[email protected]>
To: KERNEL <[email protected]>

I have a doubt about the page allocation. I need to allocate a number
of contiguous pages to later initialize and add them to the page cache. I'm doing something like that:

	struct address_space *x = &area->vm_file->f_dentry->d_inode->i_data;
	struct page *page = alloc_pages(mapping_gfp_mask(x)|__GFP_COLD, order);

	for (i = 0; i< 1<<order; i++) {
		struct page *pg = page + i;
		printk("Page count of page %i is %i\n", i, page_count(pg));
		printk("Page address: 0x%lx\n", page_address(pg));
	}

	For my surprise, for order = 4 I got the following output:

	Page count of page 0 is 1
	Page address: 0xe00000003c1e0000
	Page count of page 1 is 0
	Page address: 0xe00000003c1e4000
	Page count of page 2 is 0
	Page address: 0xe00000003c1e8000
	Page count of page 3 is 0
	Page address: 0xe00000003c1ec000

	Only the first page got it page counter incremented. Is this expected?

As far as I understand, if page_count is 0 the page is free.
Looking at page_alloc, I found set_page_refs (below), and it really
sets the page count only for the first page for machines with MMU.
I'm confused... are these pages really allocated to me?

I'm running kernel 2.6.8-rc3 on a IPF machine.

Thanks in advance for any help!

Luciano Stertz

static inline void set_page_refs(struct page *page, int order) {

#ifdef CONFIG_MMU
    set_page_count(page, 1);
#else
    int i;

    /*
     * We need to reference all the pages for this order, otherwise if
     * anyone accesses one of the pages with (get/put) it will be freed.
     */
    for (i = 0; i < (1 << order); i++)
        set_page_count(page+i, 1);

#endif /* CONFIG_MMU */
}

Signature · 16 lines

Luciano A. Stertz
[email protected]
T&T Engenheiros Associados Ltda
http://www.tteng.com.br
Fone/Fax (51) 3224 8425

--
Kernelnewbies: Help each other learn about the Linux kernel.
Archive:       http://mail.nl.linux.org/kernelnewbies/
FAQ:           http://kernelnewbies.org/faq/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Hansen

2004-11-11 19:10 UTC #1 lore raw

Quoted 1 line · Luciano A. Stertz

Only the first page got it page counter incremented. Is this expected?

Yes.

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Luciano A. Stertz

author

2004-11-11 19:27 UTC #2 lore raw

Quoted 6 lines · Dave Hansen

Quoted 1 line · Luciano A. Stertz
Only the first page got it page counter incremented. Is this expected?
Yes.

But... are they allocated to me, even with page_count zeroed? Do I need
to do get_page on the them? Sorry if it's a too lame question, but I still didn't understand and found no place to read about this.

Luciano Stertz
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Hansen

2004-11-11 19:36 UTC #3 lore raw

Quoted 3 lines · Luciano A. Stertz

But... are they allocated to me, even with page_count zeroed? Do I need
to do get_page on the them? Sorry if it's a too lame question, but I still didn't understand and found no place to read about this.

Do you see anywhere in the page allocator where it does a loop like yours?

        for (i = 1; i< 1<<order; i++)
		get_page(page + i);

When you do a multi-order allocation, the first page represents the whole group and they're treated as a whole. As you've noticed, breaking them up requires a little work.

Why don't you post all of the code that you're using so that we can tell what you're doing? There might be a better way. Drivers probably shouldn't be putting stuff in the page cache all by themselves.

-- Dave

Luciano A. Stertz

author

2004-11-11 20:22 UTC #4 lore raw

Quoted 11 lines · Dave Hansen

Quoted 2 lines · Luciano A. Stertz
But... are they allocated to me, even with page_count zeroed? Do I need
to do get_page on the them? Sorry if it's a too lame question, but I still didn't understand and found no place to read about this.
Do you see anywhere in the page allocator where it does a loop like yours?
        for (i = 1; i< 1<<order; i++)
		get_page(page + i);

Actually this loop isn't mine. It's part of the page allocator, but
it's only executed on systems without a MMU.

Quoted 7 lines

When you do a multi-order allocation, the first page represents the whole group and they're treated as a whole. As you've noticed, breaking them up requires a little work.
Why don't you post all of the code that you're using so that we can tell what you're doing? There might be a better way. Drivers probably shouldn't be putting stuff in the page cache all by themselves.

Unhappily I can't post any code yet, but I'll try to give an insight of
what we're trying to do.
It's not a driver. We're doing an implementation to allow the kernel to
execute compressed files, decompressing pages on demand.
These files will usually be compressed in small blocks, typically 4kb.
But if they got compressed in blocks bigger then a page (say 8kb blocks on a 4kb page system), the kernel will have more than one decompressed page each time a block have to be decompressed; and I'd like to add them both to the page cache.
So, seems I would have to break multi-order allocated pages. Is this
possible / viable? If not, maybe I'll have to work only with small blocks, but I wouldn't like to...

Quoted 9 lines

-- Dave
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Hansen

2004-11-11 20:34 UTC #5 lore raw

Quoted 8 lines · Luciano A. Stertz

Quoted 7 lines · Dave Hansen
Quoted 3 lines · Luciano A. Stertz
But... are they allocated to me, even with page_count zeroed? Do I need
to do get_page on the them? Sorry if it's a too lame question, but I still didn't understand and found no place to read about this.

Quoted 7 lines

Quoted 5 lines
Do you see anywhere in the page allocator where it does a loop like yours?
        for (i = 1; i< 1<<order; i++)
		get_page(page + i);
Actually this loop isn't mine. It's part of the page allocator, but
it's only executed on systems without a MMU.

Well, what does that tell you? How can page_count(page[i]) be non-zero unless someone goes and sets it like that for pages other than the first (0th) one?

Quoted 9 lines

Unhappily I can't post any code yet, but I'll try to give an insight of
what we're trying to do.
It's not a driver. We're doing an implementation to allow the kernel to
execute compressed files, decompressing pages on demand.
These files will usually be compressed in small blocks, typically 4kb.
But if they got compressed in blocks bigger then a page (say 8kb blocks on a 4kb page system), the kernel will have more than one decompressed page each time a block have to be decompressed; and I'd like to add them both to the page cache.

Why do 2 *uncompressed* blocks have to be physically contiguous? If
you're decompressing and you need more than one page, just allocate another one. I understand that your algorithms may not be optimized for this right now, but that's what you get for doing it in the kernel. :)

Quoted 3 lines

So, seems I would have to break multi-order allocated pages. Is this
possible / viable? If not, maybe I'll have to work only with small blocks, but I wouldn't like to...

It's possible, but you shouldn't do it. Multi-order pages are a very valuable commodity and should be reserved for things that *ABSOLUTELY* need them, like DMA buffers. If you ever get a system up for a while and under a lot of memory pressure, those non-order-zero allocations are going to start failing all over the place.

Make your code handle all order-0 pages now. You'll need to do it eventually.

-- Dave

Marcelo Tosatti

2004-11-11 21:21 UTC #6 lore raw

In reply to Luciano A. Stertz

Quoted 23 lines · Luciano A. Stertz

Quoted 7 lines · Dave Hansen
Quoted 3 lines · Luciano A. Stertz
But... are they allocated to me, even with page_count zeroed? Do I need to do get_page on the them? Sorry if it's a too lame question, but I
still didn't understand and found no place to read about this.
Quoted 5 lines
Do you see anywhere in the page allocator where it does a loop like yours?
       for (i = 1; i< 1<<order; i++)
		get_page(page + i);
Actually this loop isn't mine. It's part of the page allocator, but
it's only executed on systems without a MMU.
Quoted 7 lines
When you do a multi-order allocation, the first page represents the whole group and they're treated as a whole. As you've noticed, breaking them up requires a little work.
Why don't you post all of the code that you're using so that we can tell what you're doing? There might be a better way. Drivers probably shouldn't be putting stuff in the page cache all by themselves.

Quoted 12 lines

Unhappily I can't post any code yet, but I'll try to give an insight of what we're trying to do.
It's not a driver. We're doing an implementation to allow the kernel to execute compressed files, decompressing pages on demand. These files will usually be compressed in small blocks, typically 4kb. But if they got compressed in blocks bigger then a page (say 8kb
blocks on a 4kb page system), the kernel will have more than one decompressed page each time a block have to be decompressed; and I'd like to add them both to the page cache.
So, seems I would have to break multi-order allocated pages. Is this
possible / viable? If not, maybe I'll have to work only with small blocks, but I wouldn't like to...

Why do you need the pages to be physically contiguous?

I dont see any reason for that requirement - you can use discontiguous physical pages which are virtually contiguous (so your decompression code wont need to care about non adjacent pieces of memory).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Luciano A. Stertz

author

2004-11-12 11:37 UTC #7 lore raw

Quoted 41 lines · Marcelo Tosatti

Quoted 28 lines · Luciano A. Stertz
Quoted 8 lines · Dave Hansen
Quoted 3 lines · Luciano A. Stertz
But... are they allocated to me, even with page_count zeroed? Do I need to do get_page on the them? Sorry if it's a too lame question, but I
still didn't understand and found no place to read about this.
Quoted 5 lines
Do you see anywhere in the page allocator where it does a loop like yours?
      for (i = 1; i< 1<<order; i++)
		get_page(page + i);
Actually this loop isn't mine. It's part of the page allocator, but
it's only executed on systems without a MMU.
Quoted 7 lines
When you do a multi-order allocation, the first page represents the whole group and they're treated as a whole. As you've noticed, breaking them up requires a little work.
Why don't you post all of the code that you're using so that we can tell what you're doing? There might be a better way. Drivers probably shouldn't be putting stuff in the page cache all by themselves.
Quoted 9 lines
Unhappily I can't post any code yet, but I'll try to give an insight
of what we're trying to do.
It's not a driver. We're doing an implementation to allow the kernel
to execute compressed files, decompressing pages on demand.
These files will usually be compressed in small blocks, typically
4kb. But if they got compressed in blocks bigger then a page (say 8kb
blocks on a 4kb page system), the kernel will have more than one decompressed page each time a block have to be decompressed; and I'd like to add them both to the page cache.
So, seems I would have to break multi-order allocated pages. Is this
possible / viable? If not, maybe I'll have to work only with small blocks, but I wouldn't like to...

Quoted 5 lines

Why do you need the pages to be physically contiguous?
I dont see any reason for that requirement - you can use discontiguous physical
pages which are virtually contiguous (so your decompression code wont need to care about non adjacent pieces of memory).

Thanks Dave and Marcelo. You're obviously right. I'll do it.

Luciano

Signature · 10 lines

Luciano A. Stertz
[email protected]
T&T Engenheiros Associados Ltda
http://www.tteng.com.br
Fone/Fax (51) 3224 8425
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>